SGL: Symbolic Goal Learning in a Hybrid, Modular Framework for Human Instruction Following - Semantic Textual Similarity Dataset ﻿

Xu, Ruinian; Chen, Hongyi; Lin, Yunzhi; Vela, Patricio A.

Title:

SGL: Symbolic Goal Learning in a Hybrid, Modular Framework for Human Instruction Following - Semantic Textual Similarity Dataset

dc.contributor.author	Xu, Ruinian
dc.contributor.author	Chen, Hongyi
dc.contributor.author	Lin, Yunzhi
dc.contributor.author	Vela, Patricio A.
dc.contributor.corporatename	Georgia Institute of Technology. School of Electrical and Computer Engineering	en_US
dc.date.accessioned	2022-03-04T20:59:17Z
dc.date.available	2022-03-04T20:59:17Z
dc.date.issued	2022-02
dc.description	This item contains one of three datasets which supplement the manuscript https://arxiv.org/abs/2202.12912	en_US
dc.description.abstract	The affiliated paper investigates robot manipulation based on human instruction with ambiguous requests. The intent is to compensate for imperfect natural language via visual observations. Early symbolic methods, based on manually defined symbols, built modular framework consist of semantic parsing and task planning for producing sequences of actions from natural language requests. Modern connectionist methods employ deep neural networks to automatically learn visual and linguistic features and map to a sequence of low-level actions, in an endto-end fashion. These two approaches are blended to create a hybrid, modular framework: it formulates instruction following as symbolic goal learning via deep neural networks followed by task planning via symbolic planners. Connectionist and symbolic modules are bridged with Planning Domain Definition Language. The vision-and-language learning network predicts its goal representation, which is sent to a planner for producing a task-completing action sequence. For improving the flexibility of natural language, we further incorporate implicit human intents with explicit human instructions. To learn generic features for vision and language, we propose to separately pretrain vision and language encoders on scene graph parsing and semantic textual similarity tasks. Benchmarking evaluates the impacts of different components of, or options for, the vision-and-language learning model and shows the effectiveness of pretraining strategies. Manipulation experiments conducted in the simulator AI2THOR show the robustness of the framework to novel scenarios.	en_US
dc.description.sponsorship	National Science Foundation Awared #2026611	en_US
dc.identifier.uri	http://hdl.handle.net/1853/66304
dc.publisher	Georgia Institute of Technology	en_US
dc.relation.issupplementto	https://arxiv.org/abs/2202.12912
dc.subject	Deep learning in grasping and manipulation	en_US
dc.subject	AI-enabled robotics	en_US
dc.subject	Representation learning	en_US
dc.title	SGL: Symbolic Goal Learning in a Hybrid, Modular Framework for Human Instruction Following - Semantic Textual Similarity Dataset	en_US
dc.title.alternative	SGL: Symbolic Goal Learning in a Hybrid, Modular Framework for Human Instruction Following	en_US
dc.type	Dataset	en_US
dspace.entity.type	Publication
local.contributor.corporatename	School of Electrical and Computer Engineering
local.contributor.corporatename	College of Engineering
relation.isOrgUnitOfPublication	5b7adef2-447c-4270-b9fc-846bd76f80f2
relation.isOrgUnitOfPublication	7c022d60-21d5-497c-b552-95e489a06569