Speech and sketching for multimodal design

9 years 3 months ago
Speech and sketching for multimodal design
While sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. We studied the behavior of people sketching and speaking, and from this derived a set of rules for segmenting and aligning the signals from both modalities. Once the inputs are aligned, we use both modalities in interpretation. The result is a more natural interface to our system. Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation (e.g., HCI)]: User Interfaces—Natural language, Graphical user interfaces (GUI), Evaluation/methodology, Input devices and strategies (e.g., mouse, touchscreen), Interaction styles (e.g., commands, menus, forms, direct manipulation), User-centered design, Voice I/O General Terms Performance, Design, Experimentation, Human Factors Keywords speech, sketch, multimodal interaction
Aaron Adler, Randall Davis
Added 30 Jun 2010
Updated 30 Jun 2010
Type Conference
Year 2004
Where IUI
Authors Aaron Adler, Randall Davis
Comments (0)