Human Hand Joint and Mesh Reconstruction Based on RGB Images
Author(s)
Cheng, Juntao
Advisor(s)
Editor(s)
Collections
Supplementary to:
Permanent Link
Abstract
Hand pose and shape estimation, or hand mesh and joints reconstruction, has been popular since the extensive use of deep learning techniques for its utility in game design, AR/VR applications, and human-machines interactions. The task is estimating the real-world or camera space coordinates of a hand from an RGB image, or video, of the hand or a person, with possible assisting inputs such as depth images, heat maps, and silhouette. In this work, we will try to reconstruct hand pose and shape from RGB images as frames from videos to assist evaluating the condition of patients possibly experiencing stroke or seizure. Our model will contain a spiral encoder and decoders to reconstruct the mesh as well as projecting the mesh to get the joints prediction, and another transformer for taking in predictions from the previous model and proceed to predict about the existence of illness, with some Large Language Model(LLM) at the end to adjust for the small errors and smoothing the movements.
Sponsor
Date
Extent
Resource Type
Text
Resource Subtype
Undergraduate Research Option Thesis