HoloAI

Supervisor: Daniel Reich, Moritz Meier, Dennis Küster, Felix Putze, Tanja Schultz

Project period: winter semester 2019/2020 and summer semester 2020

Brief description

HoloAI stands for "holographic Artificial Intelligence". Holographic projections are a popular motif in science fiction pop culture. In Star Wars there is mobile holographic telephony, in the game Half-Life you are guided through the tutorial with holographic assistants, and in Star Trek even a holographic ship's doctor is used.

The holoAI project has as its goal the development of a holographic agent that reacts interactively to the requests of its users. Part of the project is the design of a three-dimensional, humanoid avatar as well as the development of a powerful backend using modern AI technology. Equipped with multimodal sensor technology, the agent is able to react to speech, facial expressions and gestures of humans and to recognize returning users. Located in the Cartesium building, visitors are welcomed and informed about site-specific facts. The setup is part of the scientific focus "Mind, Media, Machines" (MMM).

 
Details

Part of the development is a framework for easy implementation of content and functions of an agent for the holographic platform, as well as the agent and its behaviour itself. The participants will deal with questions in the areas of machine learning, automatic analysis of voice and video, and real-time processing of large data streams. In order to make such a project a success, we provide you with numerous state-of-the-art tools that support the development of complex interactive systems: For programming we will mainly use Python. It is planned that the individual modules will be connected via the ROS middleware (Robot Operating System). The design of the 3D avatar will be realized with Unity and Blender. Furthermore, the development of the agent's cognitive abilities is important, such as the recognition of body poses or the acquisition of attention based on facial expressions. Modern tools from the field of machine learning and image processing are also used for this purpose. For speech processing, our speech recognition components and a modern pipeline for speech comprehension based on neural networks are used.

To carry out the project, we provide the hardware for the projection of the avatar, the necessary sensor technology, workstations for teamwork and a test environment. Furthermore, numerous software tools are available for a quick start. We offer intensive support and are very interested in the results, because we want to use them for our own research. During design and implementation, principles of agile project management as well as tools of modern software development will be used.


Organizational details

The offer is aimed at students in the bachelor's degree program in computer science. Good programming skills or experience with 3D modelling or animation are required. If interested, we would like to continue the project as a subsequent master project (until September 2021). The project prepares students for the master profiles KIKR and DMI, depending on the focus.
 

Daniel Reich
Universität Bremen

Recommended events for preparation

•  Basics of machine learning (SS 2019, Bachelor)
•  Biosignals and user interfaces (SS 2019, Bachelor)
•  Automatic speech recognition (WS 2019/20, Master)