Sciweavers

Share
NAACL
2004

A Salience-Based Approach to Gesture-Speech Alignment

9 years 2 months ago
A Salience-Based Approach to Gesture-Speech Alignment
One of the first steps towards understanding natural multimodal language is aligning gesture and speech, so that the appropriate gestures ground referential pronouns in the speech. This paper presents a novel technique for gesture-speech alignment, inspired by saliencebased approaches to anaphoric pronoun resolution. We use a hybrid between data-driven and knowledge-based mtehods: the basic structure is derived from a set of rules about gesture salience, but the salience weights themselves are learned from a corpus. Our system achieves 95% recall and precision on a corpus of transcriptions of unconstrained multimodal monologues, significantly outperforming a competitive baseline.
Jacob Eisenstein, Chris Mario Christoudias
Added 31 Oct 2010
Updated 31 Oct 2010
Type Conference
Year 2004
Where NAACL
Authors Jacob Eisenstein, Chris Mario Christoudias
Comments (0)
books