A Data-Driven Paradigm to Understand Multimodal Communication in Human-Human and Human-Robot Interaction

Data-driven knowledge discovery is becoming a new trend in various scientific fields. In light of this, the goal of the present paper is to introduce a novel framework to study one interesting topic in cognitive and behavioral studies -- multimodal communication between human-human and human-robot interaction. We present an overall solution from data capture, through data coding and validation, to data analysis and visualization. In data collection, we have developed a multimodal sensing system to gather fine-grained video, audio and human body movement data. In data analysis, we propose a hybrid solution based on visual data mining and information-theoretic measures. We suggest that this data-driven paradigm will lead not only to breakthroughs in understanding multimodal communication, but will also serve as a successful case study to demonstrate the promise of data-intensive discovery which can be applied in various research topics in cognitive and behavioral studies.

• Type: Journal
• Year: 2010
• Where: IDA
• Authors: Chen Yu, Thomas G. Smith, Shohei Hidaka, Matthias Scheutz, Linda B. Smith
• Keywords: Behavioral Studies, Data Analysis, IDA 2010, Information Technology, Multimodal Communication