(Georgia Institute of Technology, 2024-04-29)
Truong, Joanne
The goal of Artificial Intelligence is to “construct useful intelligent systems”, such as mobile robots, to assist in our day-to-day lives. For these mobile robotic assistants to be useful in the real-world, they must skillfully navigate complex environments (e.g., delivering packages from one building to another). However, training robots in the real-world can be slow, dangerous, expensive, and difficult to reproduce. Thus, one paradigm in robot learning is to leverage simulation for training robots (where gathering experience is scalable, safe, cheap, and reproducible) before being deployed in the real world. However, no simulator is perfect; AI systems learn to “cheat” by exploiting imperfections. Thus, how can we train robots in imperfect simulators while ensuring that the learned skills generalize to reality?
In this thesis, we will argue that simulators need not be perfect to be useful; they don’t need to model everything about the world, only what’s necessary for generalization. We present 1) Sim2Real Correlation Coefficient for measuring and optimizing performance correlation between simulation and reality, enabling confident evaluation. 2) Bi-directional Domain Adaptation (BDA), and Kinematic-to-Dynamic Transfer (Kin2Dyn), sample-efficient methods for reducing the sim2real gap. BDA and Kin2Dyn improve robot learning and generalization to the real-world by utilizing abstracted physics and simple adaptation models learned from small amounts of real-world data. 3) IndoorSim-to-OutdoorReal, an end-to-end learned approach that enables visual navigation in out-of-distribution environments zero-shot. We show that simulators can be used for real-world transfer without having to apriori design and model the deployment scenario. 4) Implicit Map Cross Modal Attention, a vision and language navigation model that utilizes structured implicit maps for navigating in an environment over time. Structured memory representation and training paradigms enable navigation for robots that occupy the same environment for long periods of time.