(Georgia Institute of Technology, 2022-05)
Chaganti, Sidhartha
Multimodal learning enables networks to consider mulitple perspectives or modalities of a scene when performing activity recognition. We propose networks which use attention to better focus on select portions of data along both embedding-space and time. Additionally, we propose using a projection network to project from one modality to another in order to use a multimodal network even when the second modality is unavailable during inference time. We observe that adding attention leads to better performance and that using projected data retains most of the performance from the multimodal architectures.
(Georgia Institute of Technology, 2022-05)
Agrawal, Pranay
95% of deaf children are born to hearing parents and lack continuous exposure to language, which often inhibits learning. We are developing Adaptive CopyCat, an educational game where Deaf children communicate with the computer via American Sign Language (ASL) in order to improve their language skills and working memory. While previous versions of CopyCat relied on custom hardware such as colored gloves with accelerometers for sign verification, our current version of the game utilizes off-the-shelf 4K RGB depth cameras and pose estimators. Before re-creating the game for Deaf children, we evaluate the efficacy of our current CopyCat ASL recognition system with 12 adults. Average user-independent sentence and word accuracies were 85.1% and 95.4%, respectively. To improve the accuracy when new users are introduced, we developed a progressive training model that can adapt to a new user's signing as they play the game. This approach produced a 5% absolute increase in sentence accuracy. To test for generality, a 13th user was recruited six months after the initial experiment and achieved similarly high accuracies. These promising results suggest that our recognizer will be sufficiently accurate for verifying children's signing while playing Adaptive CopyCat.