Title:
ASL Fingerspelling Recognition Through Hidden Markov Models
ASL Fingerspelling Recognition Through Hidden Markov Models
Author(s)
So, Matthew
Advisor(s)
Starner, Thad E.
Editor(s)
Collections
Supplementary to
Permanent Link
Abstract
American Sign Language (ASL) is notable for its unique grammatical structures such as classifiers and its use in a more complex multi-dimensional (spatial) medium, rather than a uni-dimensional (auditory) medium. As a result, most current recognition or translation systems require expensive equipment such as accelerometers and depth cameras, restrictions on recognizable phrases, and/or low recognition accuracy. However, ASL fingerspelling representing signs corresponding to the letters of the alphabet, a subset of ASL, may be able to engage and encourage parents for initial learning. In this project, we have analyzed data sufficiency by collecting training data from three users on 36 coarticulated fingerspelled words of varying lengths using cameras recording hand at a size of approximately 256 by 256 pixels. We have also tested model recognition accuracy on training data using uniletter and triletter Gaussian Mixture-Hidden Markov Models (GMM-HMMs) and using various levels of data augmentation involving including finger features (locations) centered around the location of the hand and including wrist features. We have found that triletter GMM-HMMs produce up to 96% letter and word accuracy rates on user-dependent tests when centered hand features and wrist features are included in training data. The results indicate that fingerspelling recognition on mobile devices is viable and should be further investigated.
Sponsor
Date Issued
2022-05
Extent
Resource Type
Text
Resource Subtype
Undergraduate Thesis