Sciweavers

Share
CSL
2002
Springer

Learning visually grounded words and syntax for a scene description task

8 years 10 months ago
Learning visually grounded words and syntax for a scene description task
A spoken language generation system has been developed that learns to describe objects in computer-generated visual scenes. The system is trained by a `show-and-tell' procedure in which visual scenes are paired with natural language descriptions. Learning algorithms acquire probabilistic structures which encode the visual semantics of phrase structure, word classes, and individual words. Using these structures, a planning algorithm integrates syntactic, semantic, and contextual constraints to generate natural and unambiguous descriptions of objects in novel scenes. The system generates syntactically well-formed compound adjective noun phrases, as well as relative spatial clauses. The acquired linguistic structures generalize from training data, enabling the production of novel word sequences which were never observed during training. The output of the generation system is synthesized using word-based concatenative synthesis drawing from the original training speech corpus. In eva...
Deb K. Roy
Added 18 Dec 2010
Updated 18 Dec 2010
Type Journal
Year 2002
Where CSL
Authors Deb K. Roy
Comments (0)
books