Sciweavers

Share
AAAI
2007

Turning Lectures into Comic Books Using Linguistically Salient Gestures

9 years 4 months ago
Turning Lectures into Comic Books Using Linguistically Salient Gestures
Creating video recordings of events such as lectures or meetings is increasingly inexpensive and easy. However, reviewing the content of such video may be time-consuming and difficult. Our goal is to produce a “comic book” summary, in which a transcript is augmented with keyframes that disambiguate and clarify accompanying text. Unlike most previous keyframe extraction systems which rely primarily on visual cues, we present a linguistically-motivated approach that selects keyframes that contain salient gestures. Rather than learning gesture salience directly, it is estimated by measuring the contribution of gesture to understanding other discourse phenomena. More specifically, we bootstrap from multimodal coreference resolution to identify gestures that improve performance. We then select keyframes that capture these gestures. Our model predicts gesture salience as a hidden variable in a conditional framework, with observable features from both the visual and textual modalities....
Jacob Eisenstein, Regina Barzilay, Randall Davis
Added 02 Oct 2010
Updated 02 Oct 2010
Type Conference
Year 2007
Where AAAI
Authors Jacob Eisenstein, Regina Barzilay, Randall Davis
Comments (0)
books