Title:
Multimodal assessment of neuropsychiatric disorders using audiovisual recordings
Multimodal assessment of neuropsychiatric disorders using audiovisual recordings
Author(s)
Jiang, Zifan
Advisor(s)
Clifford, Gari D.
Editor(s)
Collections
Supplementary to
Permanent Link
Abstract
Over one billion people worldwide live with a neuropsychiatric disorder, yet most do not have access to adequate diagnosis and care. Accurate, fast, and accessible detection of those disorders is critical to early and effective interventions. Over the last decade, digitally administered assessments have emerged as one of the most promising approaches. Moreover, the increasing use of telemedicine in psychiatry and neurology in recent years presented an unprecedented opportunity to use audiovisual data for accessible neuropsychiatric assessments without the limitation of geographical location and specialized hardware.
This dissertation describes the use of low-cost audiovisual data collected from in-lab and remote mobile devices to assess neuropsychiatric conditions by extracting and combining various behavioral and physiological indicators. First, we showed that facial and speech emotions can be effectively estimated from audiovisual data collected in interviews and used for major depressive disorder evaluation. Then, we presented the automated assessment of cognitive impairment using facial emotions and viewing behaviors recognized from videos passively collected from a mobile device. Lastly, we further improved the scalability and accessibility by extracting facial, vocal, linguistic, and cardiovascular features from audiovisual data collected remotely from heterogeneous mobile devices and validating them in both clinician-rated and self-rated mental health condition evaluation.
Sponsor
Date Issued
2023-11-28
Extent
Resource Type
Text
Resource Subtype
Dissertation