Gesture Salience as a Hidden Variable for Coreference Resolution and Keyframe Extraction

Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and non-communicative hand motions may confuse natural language processing (NLP) and impede learning. People have little difficulty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by at...

• Type: Journal
• Year: 2008
• Where: JAIR
• Authors: Jacob Eisenstein, Regina Barzilay, Randall Davis
• Keywords: Gesture Salience, Gestures, JAIR 2008, Natural Language