(Georgia Institute of Technology, 2020-05)
Srirangam, Bharat V.
We as humans take our ability to digest a scene and extract its context knowledge to be for granted. There are several senses involved including but not limited to sight, hearing, and touch. This also includes our ability to combine information from the different senses to enrich our understanding. In the healthcare robotics space, a lot of success has been met with emulating or attempting to emulate these abilities of humans in everyday processes. To be specific, one process, context inference, is a very subconscious but powerful process. When someone picks up a glass cup, they can feel that the cup is made of glass where their hand meets the cup but are still able to infer that the rest of the cup is also glass. This is a powerful example of how humans take local information and use it to derive global context. In this paper, we attempt to emulate this process through a PR2 robot by developing a way to segment a scene for the various objects in the scene. The PR2 can then estimate the material of the different objects using a pre-trained neural network that takes in spectroscopy measurements and pictures of a small patch of each object. This would thereby allow the PR2 to emulate the same ability to abstract its local information to a global context as the material of each object would be determined by a small measurement. We set up a table with an arrangement of objects and test different approaches to segmenting this scene to provide points of interest and measurement to the PR2. Some objects that were used include pots, glasses, mugs, sweaters, and scissors. After testing 3 different approaches to segmentation, a 3D analysis based one was able to sufficiently segment the scenes and provided the PR2 with enough information to make proper measurements and make reasonable estimates. Finally, we demonstrate how a PR2 robot can do all of this and leverage this system to estimate the materials of everyday objects so that it can infer interactions with these objects. From this work, we find that we are one step closer to providing robots with the same advantage that we have to mix partial contextual understandings to make better globally informed decisions.