Master Project: I spy with my little eye

Contents

Winter semester 2018/19 & Summer semester 2019

Supervisors: Felix Putze, Celeste Mason, Dennis Küster, Cognitive  Systems Lab 

Short description: An intelligent system should be able to play "I spy with my little eye" with a human player. By observing the human being and the scene, the system should recognize and describe increasingly difficult objects as reliably as possible. For this, numerous existing sensors can be combined with each other.

In the classic child game "I spy with my little eye", a player secretly selects and describes an object in a scene, e.g. based on color, shape or function. Another player tries to guess the object. This project aims to develop an intelligent system for the role of the rater.

The system will use a number of sensors and data processing components to extract and aggregate relevant information. Using AI's artificial intelligence and machine learning methods, the system first analyzes the description given by the player and his / her eye movements with respect to visible objects within the scene. These informatons are combined to identify probable targets. Through a user interface, the system then communicates its hypotheses to the human player.

In the basic configuration of the system, the following functions should be implemented:

• Processing of textually entered statements about the selected object and their linking to a semantic database
• Utilization of existent neural network architectures for the identification of objects from camera images of the scene
• Interpretation of the player's gaze movements and fixatons, recorded using a mobile eye tracker
• Linking the components via a flexible middleware and implementing the gameplay in a graphical interface

For further details, please refer to the below documents. Unfortunately, these are not presently available in English.