(Georgia Institute of Technology, 2016-08)
Labean, Tyler J.
The purpose of this research was to create a wearable system that recognizes
gestures of the user, allowing interaction through hand gestures. The user wears a hat
mounted with a regular optical camera and a thermal camera. The combination of these
two heterogeneous video streams was used to recognize the user’s gestures in many
conditions and environments. First, corners were detected from contrast stretched images
using the Shi-Tomasi method. The movement of these corners was then tracked using
Lucas-Kanade optical flow analysis. Groups of corners that moved together were defined
using hierarchical cluster linkage analysis. To determine how these groups moved with
time, a connected components analysis was employed. The motion path was reduced into
its cardinal and semi cardinal vector components to encode the motion vector.
Subsequently, this data was used to train hidden Markov models for each gesture and
each camera. After the evaluation of gesture priority over all hidden Markov models,
principal components analysis was performed on this gesture prioritized set to train a one
vs one Multiclass recognizer. Finally, a confusion matrix was generated indicating a
recognition success rate of 87%. An analysis was performed on the robustness of the
algorithm under various luminance, heat and image variance conditions. The contribution
of combining optical and thermal video streams vs utilizing either as a single video
stream input and found to be a great advantage. Additionally, a video database of gestures
was created and will be released so that other researchers can compare algorithms and
benchmarks using the same data-set.