Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions

Abstract—We consider performing a sequence of mobile manipulation tasks with instructions given in natural language (NL). Given a new environment, even a simple task such as of boiling water would be performed quite differently depending on the presence, location and state of the objects. We start by collecting a dataset of task descriptions in free-form natural language and the corresponding grounded task-logs of the tasks performed in an online robot simulator. We then build a library of verbenvironment-instructions that represents the possible instructions for each verb in that environment—these may or may not be valid for a different environment and task context. We present a model that takes into account the variations in natural language, and ambiguities in grounding them to robotic instructions with appropriate environment context and task constraints. Our model also handles incomplete or noisy NL instructions. Our model is based on an energy function that encodes such prope...

• Type: Journal
• Year: 2016
• Where: IJRR
• Authors: Dipendra Kumar Misra, Jaeyong Sung, Kevin Lee, Ashutosh Saxena
• Keywords: IJRR 2016