Multimodal Interaction

In this course we will set the basics for an understanding of multimodal communication between humans and multimodal interaction between humans and machines. We will start with clarifying the basic principles of human-human communication and human-machine interaction. We will then describe the processes taking place in humans when perceiving auditory, visual and tactile signals, as well as how these perceptions are integrated in order to form a multimodal perception. The signals can be generated and received by machines which are able to interact with humans in limited domains. The set-up of such machines will be discussed, and limitations as well as potential solutions to overcome these limitations will be explained.

Week 1  :  Multimedia and Multimodality; Hearing and Speech; Vision
Week 2  :   Other Senses, Integration and Cognition; Multimodal Perception; Human Multimodal Interaction
Week 3  : Multimodal Input and Output Systems 
Week 4  :  Multimodal Interactive Systems, Virtual Environments