Attention driven reference resolution in multimodal contexts

13 years 5 months ago
Attention driven reference resolution in multimodal contexts
In recent years a a number of psycholinguistic experiments have pointed to the interaction between language and vision. In particular, the interaction between visual attention and linguistic reference. In parallel with this, several theories of discourse have attempted to provide an account of the relationship between types of referential expressions on the one hand and the degree of mental activation on the other. Building on both of these traditions, this paper describes an attention based approach to visually situated reference resolution. The framework uses the relationship between referential form and preferred mode of interpretation as a basis for a weighted integration of linguistic and visual attention scores for each entity in the multimodal context. The resulting integrated attention scores are then used to rank the candidate referents during the resolution process, with the candidate scoring the highest selected as the referent. One advantage of this approach is that the res...
John D. Kelleher
Added 10 Dec 2010
Updated 10 Dec 2010
Type Journal
Year 2006
Where AIR
Authors John D. Kelleher
Comments (0)