Organizational Unit:

School of Interactive Computing

Permanent Link

https://hdl.handle.net/1853/70783

Parent Organization

Organizational Unit

College of Computing

ArchiveSpace Name Record

https://finding-aids.library.gatech.edu/agents/corporate_entities/1113

Full item page

Publication Search Results

Now showing 1 - 10 of 13

Passive Sensing Frameworks for the Future of Information Workers

(Georgia Institute of Technology, 2023-07-30) Swain, Vedant Das Das

Work sustains our livelihoods and is key to leading a fulfilling life. Improving our effectiveness at work helps us progress toward our goals and reclaim our lives for other activities. Traditionally we have used surveys to understand what makes workers more effective. However, these approaches do not sufficiently reflect workers as a part of a complex ecology --- comprising their daily activities, social dynamics, and the larger community. My thesis posits an alternative and more holistic approach. We can gain a more naturalistic understanding of worker effectiveness by leveraging everyday digital technology dispersed in their ecology as passive sensors. I focus my studies on information workers, a significant portion of white-collar work. This dissertation demonstrates the potential of repurposing everyday digital technology as an ecological lens to explain their performance and wellbeing. I have studied various technology readily available in information work, such as wearables, mobiles, desktops, Bluetooth beacons, WiFi router networks, and social media. My research presents (i) the utility of passively explaining worker wellbeing with behavioral traces and (ii) the acceptability of deploying such technologies for information work. In my studies, I applied statistical modeling and machine learning to show new ways to clarify indicators of worker experiences at the individual, group, and organizational levels. Later, I took a worker-centric perspective to situate such algorithmic inferences in today's work paradigm and describe the methodological and socio-technical challenges. My dissertation contributes to the future of work across multiple dimensions. First, it adds to behavioral computing research by showing computationally efficient and versatile opportunities to model passively collected behavioral traces and provides insight into worker effectiveness. Next, it refines organizational science by providing new opportunities to explain worker experiences by accounting for previously unforeseen behavioral dynamics. Last, it highlights the limits of this approach and provides evidence to suggest how these technologies should (and should not) manifest in the workplace. Collectively, my research aims to help workers by underscoring passive sensing practices that are more holistic, accurate, and humane.
Understanding Social Requirements for Social Media Powered Artificial Intelligence (SOMPAI) for Mental Health Care

(Georgia Institute of Technology, 2023-07-25) Yoo, Dong Whi

In the field of medicine, mental health is unique as it entirely depends on the patient’s ability to express their cognitive and emotional states, symptom progression, and interpersonal relationships. This reliance creates extra challenges for patients and results in less effective evaluations and treatments. Recent advancements in artificial intelligence (AI) have been proposed as a means to develop more objective criteria and evidence for mental health treatments, with AI technologies being viewed as the potential solution to critical issues in mental health care, such as delayed, inaccurate, and ineffective care delivery. To support mental health practices, researchers from computer science, mental health, and other related fields have collaborated on developing AI models. However, despite decades of effort, these advancements have not been successfully integrated into real-world mental health contexts. This discrepancy between AI research and mental health practices can be framed as the socio-technical gap. It represents the intellectual challenge arising from differences between what technologies can provide and what users want to achieve in social contexts. Researchers in Computer-Supported Cooperative Work (CSCW) have pointed out that a socio-technical gap exists because users possess diverse roles, tasks, and procedures in social contexts and can fluidly navigate between them. However, technologies often lack understanding and flexibility in supporting these changes. AI technologies currently face a similar socio-technical gap as they lack social requirements such as nuance, flexibility, and ambiguity tolerance. Emerging research in Human-AI Interaction has revealed more detailed accounts of the social requirements for AI models. These studies propose working closely with AI technology end-users to understand the requirements from their perspectives. Building on these research efforts, this dissertation investigates the social requirements for mental health AI technologies from the standpoint of patients and clinicians, identifying them and envisioning design implications for future AI technologies. Mental health AI technologies utilize various data types, such as electronic health records, sensor-based data, and social media data. This dissertation focuses on social media-powered AI (SoMPAI) as many mental health patients use social media for treatment and recovery purposes, including self-disclosure, help-seeking, and peer support. Social media data can be a valuable source for understanding the mental health of people who are active on social media, particularly adolescents and young adults, who are the primary target of mental health treatment. Mental health patients’ social media data contain a rich amount of written text used in psychoanalysis for several mental disorders. Social media data can also reflect patients’ social activities, which can be valuable to mental health clinicians who must infer patients’ daily social lives from their self-reports during consultations. Therefore, this dissertation aims to understand the social requirements for social media-powered AI from both clinicians’ and patients’ perspectives by utilizing human-centered design approaches and working closely with clinicians and patients to understand their expectations and concerns. This dissertation contributes to several domains: it expands the concept of social requirements and the socio-technical gap in CSCW as they relate to mental health AI technologies; it provides empirical evidence, including the perspectives of mental health patients and clinicians, on expectations and concerns regarding AI models, contributing to recent Human-AI Interaction research; and the design implications of this dissertation will help develop implementable mental health AI technologies that can support current mental health practices.
Eating Behavior In-The-Wild and Its Relationship to Mental Well-Being

(Georgia Institute of Technology, 2022-12-13) Bin Morshed, Mehrab

The motivation for eating is beyond survival. Eating serves as means for socializing, exploring cultures, etc. Computing researchers have developed various eating detection technologies that can leverage passive sensors available on smart devices to automatically infer when and, to some extent, what an individual is eating. However, despite their significance in eating literature, crucial contextual information such as meal company, type of food, location of meals, the motivation of eating episodes, the timing of meals, etc., are difficult to detect through passive means. More importantly, the applications of currently developed automated eating detection systems are limited. My dissertation addresses several of these challenges by combining the strengths of passive sensing technologies and EMAs (Ecological Momentary Assessment). EMAs are a widely adopted tool used across a variety of disciplines that can gather in-situ information about individual experiences. In my dissertation, I demonstrate the relationship between various eating contexts and the mental well-being of college students and information workers through naturalistic studies. The contributions of my dissertation are four-fold. First, I develop a real-time meal detection system that can detect meal-level episodes and trigger EMAs to gather contextual data about one’s eating episode. Second, I deploy this system in a college student population to understand their eating behavior during day-to-day life and investigate the relationship of these eating behaviors with various mental well-being outcomes. Third, based on the limitations of passive sensing systems to detect short and sporadic chewing episodes present in snacking, I develop a snacking detection system and operationalize the definition of snacking in this thesis. Finally, I investigate the causal relationship between stress levels experienced by remote information workers during their workdays and its effect on lunchtime. This dissertation situates the findings in an interdisciplinary context, including ubiquitous computing, psychology, and nutrition.
Compute-proximal Energy Harvesting for Mobile Environments: Fundamentals, Applications, and Tools

(Georgia Institute of Technology, 2021-12-13) Park, Jung Wook

Over the past two decades, we have witnessed remarkable achievements in computing, sensing, actuating, and communications capabilities of ubiquitous computing applications. However, due to the limitations in stable energy supply, it is difficult to make the applications ubiquitous. Batteries have been considered a promising technology for this problem, but their low energy density and sluggish innovation have constrained the utility and expansion of ubiquitous computing. Two key techniques—energy harvesting and power management—have been studied as alternatives to overcome the battery limitations. Compared to static environments such as homes or buildings, there are more energy harvesting opportunities in mobile environments since ubiquitous systems can generate various forms of energy as they move. Most of the previous studies in this regard have been focused on human movements for wearable computing, while other mobile environments (e.g., cars, motorcycles, and bikes) have received limited attention. In this thesis, I present a class of energy harvesting approaches called compute-proximal energy harvesting, which allows us to develop energy harvesting technology where computing, sensing, and actuating are needed in vehicles. Computing includes sensing phenomena, executing instructions, actuating components, storing information, and communication. Proximal considers the harvesting of energy available around the specific location where computation is needed, reducing the need for excessive wiring. A primary goal of this new approach is to mitigate the effort associated with the installation and field deployment of self-sustained computing and lower the entry barriers to developing self-sustainable systems for vehicles. In this thesis, I first select an automobile as a promising case study and discuss the opportunities, challenges, and design guidelines of compute-proximal energy harvesting with practical yet advanced examples in the automotive domain. Second, I present research in the design of small-scale wind energy harvesters and the implementation and evaluation of two advanced safety sensing systems—a blind spot monitoring system and a lane detection system—with the harvested power from wind. Finally, I conduct a study to democratize the lessons learned from the automotive case studies for makers and people with no prior experience in energy harvesting technology. In this study, I seek to understand what problems they have encountered and what possible solutions they have considered while dealing with energy harvesting technology. Based on the findings, I develop a comprehensive energy harvesting toolkit and examine its utility, usability, and creativity through a series of workshops.
Deriving Sensor-based Complex Human Activity Recognition Models Using Videos

(Georgia Institute of Technology, 2021-10-14) Kwon, Hyeokhyen

With the ever-increasing number of ubiquitous and mobile devices, Human Activity Recognition (HAR) using wearables has become a central pillar in ubiquitous and mobile computing. HAR systems commonly adopt machine learning approaches, which use supervised training on labeled datasets. Recent success in HAR has come along with the advances in supervised training techniques, namely deep learning models, which also have made dramatic breakthroughs in various domains, such as computer vision, natural language processing, and speech recognition. Across domains, the keys to derive robust recognition models, which strongly generalize across application boundaries, were highly complex analysis models and large-scale labeled datasets to serve the data-hungry nature of deep learning models. Although the field of HAR has seen first, substantial success from using deep learning models, the complexity of HAR models is still constrained, mainly due to the typically only small-scale datasets. Conventionally, sensor datasets are collected in user studies in a laboratory environment. The process is very labor-intensive, recruiting participants is expensive, and annotations are time-consuming. As a consequence, the sensor data collection often results in only a limited size of a labeled dataset, where a model derived from such a small-scale dataset is not likely to generalize well. My research develops a framework, namely IMUTube, that can potentially alleviate the limitations of large-scale labeled data collection in sensor-based HAR, which is the most pressing issue to limit the model performance in HAR systems. I aim to harvest existing video data from large-scale repositories, such as YouTube. IMUTube is a system that bridges the modality gap between videos and wearable sensors by tracking human motions captured in videos. Once the motion information is extracted from the videos, the information is transformed to virtual Inertial Measurement Unit (IMU) sensor signals for various on-body locations. The collection of virtual IMU data from a large amount of videos is then used for deriving HAR systems that can be used in real-world settings. The overarching idea is appealing due to the sheer size of readily accessible video repositories and the availability of weak labels in the form of video titles and descriptions. The IMUTube framework automatically extracts motion information from arbitrary human activity videos and is thereby not limited to specific scenes or viewpoints by integrating techniques from the fields of computer vision, computer graphics, and signal processing. Tracking 3D motion information from unrestricted online video poses multiple challenges, such as fast camera motion, noise, lighting changes, occlusions, and so on. IMUTube automatically identifies artifacts in the video that challenges robust motion tracking to generate high-quality virtual IMU data only from those video segments that exhibit the least noise. Using IMUTube, I show that complex models, which could not have been derived using the typical, small-scale datasets of real IMU sensor readings, have become trainable with the weakly-labeled virtual IMU dataset collected from many videos. The availability of more complex HAR models represents the first step towards research opportunities to design sophisticated, deep learning models that shall capture sensor data more effectively than the state-of-the-art. Overall, my work opens up research opportunities for the human activity recognition community to generate large-scale labeled datasets in an automated, cost-effective manner. Having access to larger-scale datasets opens up possibilities for deriving more robust and more complex activity recognition models that can be employed in entirely new application scenarios.
Ubiquitous Self-Powered Ambient Light Sensing Surfaces

(Georgia Institute of Technology, 2021-05-01) Zhang, Dingtian

Many human activities interfere with ambient light in a predictable and detectable way in that our activities implicitly or explicitly block the paths of ambient light in our environment. This dissertation explores sensing of ambient light interference patterns as a general-purpose signal at the surface level of everyday objects for activity recognition as well as novel interaction techniques. Two sensing systems, OptoSense and the Computational Photodetector, are developed as self-powered, cost-effective, and privacy-preserving computational materials that sense ambient light interference patterns and detect a wide variety of implicit and explicit human activities on surfaces of everyday objects. This work shows a promising path of a ubiquitous computational material that weaves into the fabric of everyday surfaces, and discusses the challenges and opportunities from the power, cost, privacy, form factors, and application perspectives.
The internet of living things: Enabling increased information flow in dog—human interactions

(Georgia Institute of Technology, 2017-04-25) Alcaidinho, Joelle Marie

The human–canine relationship is one of the oldest relationships between a hu- man and an animal. Even with this longevity and unique living arrangement, there is still a great deal that we don’t know about our dogs. What do we want to know and how can computing help provide avenues for dogs to tell us more? To address the question of “what do people wish their dogs could tell them?” In an un- published survey of UK dog-owners, the most frequent request was to know about their dog’s emotional state and the most frequent response regarding what they wish their dogs would tell them was about what they love and what they are thinking. These responses dominated the survey, outnumbering even the responses regarding the dog’s physical needs like toileting. This hunger for more and better information from dogs has created a boom in the number of devices targeting these desires with unverified claims that have appeared on the market within the past 5 years. Clearly there is a need for more research, particularly in computing, in this space. While my dissertation unfortunately does not provide a love–detector or dog–thought–decoder, it does lay out the space for what wearables on dogs could provide today and in the near future. My focus is on addressing the information asymmetry between dogs and people, specifically by using wearable computing to provide more and richer in- formation from the dog to more people. To do this, I break down the space into three categories of interactions. Within each of these categories I present research that explores how these interactions can work in the field through prototype systems. This area of research, Animal–Human–Computer Interaction is new, and the area of Canine–Centered–Computing is younger still. With the state of these fields in mind, that my goal with this dissertation is to help frame this space as it pertains to dogs and wearable computing.
Enabling In situ & context-based motion gesture design

(Georgia Institute of Technology, 2017-04-05) Parnami, Aman

Motion gestures, detected through body-worn inertial sensors, are an expressive, fast to access input technique, which is ubiquitously supported by mobile and wearable devices. Recent work on gesture authoring tools has shown that interaction designers can create and evaluate gesture recognizers in stationary and controlled environments. However, we still lack a generalized understanding of their design process and how to enable in situ and context-based motion gesture design. This dissertation advances our understanding of these problems in two ways. First, by characterizing the factors impacting a gesture designer's process, as well as their gesture designs and tools. Second, by demonstrating rapid motion gesture design in a variety of new contexts. Specifically, this dissertation presents: (1) a novel triadic framework that enhances our understanding of the motion gestures, their designers, and the factors influencing design of authoring tools; (2) the first ever explorations of in situ and context-based prototyping of motion gestures through development of two generations of a smartphone-based tool, Mogeste, followed by Comoge; and (3) a description of the challenges and advantages of designing motion gestures in situ, based on the first user study with both professional as well as student interaction designers.
Understanding visual analysis processes from user interactions using visual analytics

(Georgia Institute of Technology, 2016-11-15) Han, Yi

Understanding the visual analysis process taken by people when using a visualization application can help its designers improve the application. This goal is typically achieved by observing usage sessions. Unfortunately, many visualization applications are now deployed online so their use is occurring remotely. These remote usages make it very difficult for designers to directly observe usage sessions in person. A solution to the problem is to analyze interaction logs. While interaction logs are easy to collect remotely and at scale, they can be difficult to analyze because they require an analyst to make many difficult decisions about event organization and pattern discovery. For example, which events are irrelevant to the analysis and should be removed? Which events should be grouped because they are related to the same feature? Which events lead to meaningful patterns that help to understand user behaviors? An analyst needs to be able to make these decisions to identify different types of patterns and insights based on an analysis goal. If the analysis goal changes during the process, these decisions need to be revisited in order to obtain the best analysis results. Because of the subjective nature of the analysis process and such decisions, flexibility is required so the process cannot be fully automated. Every decision requires additional effort from an analyst that could reduce the practicality of the analysis process. Therefore, an effective interaction analysis method needs to balance the tradeoffs of flexibility and practicality to best support analysts. Visual analytics provides a promising solution to this problem because it leverages human’s broadband visual analysis abilities with the support of computational methods. For flexibility, the interactive visualizations can ensure an analyst can dynamically adjust decisions in every step of the process to maximize the variety of patterns that could be identified. For practicality, visualizations can help speed up the data inspection and decision-making process while computational methods can reduce the labor in efficiently extracting potentially useful patterns. Therefore, in this thesis I employ visual analytics in a visual interaction analysis framework to achieve flexibility and practicality in the visual analysis process for identifying patterns in interaction logs. I evaluate the framework by applying it to multiple visualization applications to assess the effectiveness of the analysis process and the usefulness of the patterns discovered.
Automatic eating detection in real-world settings with commodity sensing

(Georgia Institute of Technology, 2016-01-07) Thomaz, Edison

Motivated by challenges and opportunities in nutritional epidemiology and food journaling, ubiquitous computing researchers have proposed numerous techniques for automated dietary monitoring (ADM) over the years. Although progress has been made, a truly practical system that can automatically recognize what people eat in real-world settings remains elusive. This dissertation addresses the problem of ADM by focusing on practical eating moment detection. Eating detection is a foundational element of ADM since automatically recognizing when a person is eating is required before identifying what and how much is being consumed. Additionally, eating detection can serve as the basis for new types of dietary self-monitoring practices such as semi-automated food journaling. In this thesis, I show that everyday eating moments such as breakfast, lunch, and dinner can be automatically detected in real-world settings by opportunistically leveraging sensors in practical, off-the-shelf wearable devices. I refer to this instrumentation approach as "commodity sensing". The work covered by this thesis encompasses a series of experiments I conducted with a total of 106 participants where I explored a variety of sensing modalities for automatic eating moment detection. The modalities studied include first-person images taken with wearable cameras, ambient sounds, and on-body inertial sensors. I discuss the extent to which first-person images reflecting everyday experiences can be used to identify eating moments using two approaches: human computation, and by employing a combination of state-of-the-art machine learning and computer vision techniques. Furthermore, I also describe privacy challenges that arise with first-person photographs. Next, I present results showing how certain sounds associated with eating can be recognized and used to infer eating activities. Finally, I elaborate on findings from three studies focused on the use of on-body inertial sensors (head and wrists) to recognize eating moments both in a semi-controlled laboratory setting and in real-world conditions. I conclude by relating findings and insights to practical applications, and highlighting opportunities for future work.