Context based multimodal fusion

11 years 6 months ago
Context based multimodal fusion
We present a generic approach to multimodal fusion which we call context based multimodal integration. Key to this approach is that every multimodal input event is interpreted and enriched with respect to its local turn context. This local turn context comprises all previously recognized input events and the dialogue state that both belong to the same user turn. We show that a production rule system is an elegant way to handle this context based multimodal integration and we describe a first implementation of the so-called PATE system. Finally, we present results from a first evaluation of this approach as part of a human-factors experiment with the COMIC system. Categories and Subject Descriptors H.5 [User Interfaces]: input devices and strategies, natural language, prototyping, theory and methods, user-centered design. General Terms Algorithms, Design, Human Factors Keywords Multimodal integration, fusion, speech and pen input, multimodal dialogue systems
Norbert Pfleger
Added 01 Jul 2010
Updated 01 Jul 2010
Type Conference
Year 2004
Where ICMI
Authors Norbert Pfleger
Comments (0)