Deriving Bespoke Human Activity Recognition Systems for Smart Homes
Author(s)
Hiremath, Shruthi K.
Advisor(s)
Editor(s)
Collections
Supplementary to:
Permanent Link
Abstract
Smart Homes have come a long way: From research laboratories in the early days, through periods of (almost) neglect, to their recent revival in real-world environments enabled by the existence of commodity devices and robust, standardized software frameworks. With such availability, human activity recognition (HAR) in smart homes has become attractive for many real-world applications, especially in the domain of Ambient Assisted Living (AAL). Yet, building an activity recognition system for specific smart homes, which are specialized spaces with varying home layouts and inhabited by individuals with idiosyncratic behaviors and habits, is a non-trivial endeavor. For real-world deployments, privacy and logistical concerns essentially rule out the possibility of third parties being able to collect the much-needed annotated sensor data while the resident already lives in their smart home.
My dissertation addresses these challenges by first defining the Lifespan of a HAR system for smart homes, comprising three phases: i) bootstrapping; ii) update and extend; and iii) extend the capabilities of the recognition model for complex HAR tasks. The Lifespan comprises components that are used to derive functional HAR systems quickly with minimal yet targeted involvement of residents for the bootstrapping and the update and extend procedure. Integrating external context information requires no resident supervision, only the layouts of the individual smart homes. This is achieved through building
novel analysis procedures and data analysis pipelines corresponding to the aforementioned phases.
The contributions of my dissertation are three-fold. First, I develop an initial bootstrapping procedure aimed at addressing the beginning of the life span of HAR resulting in a system that is capable of recognizing relevant and prominent activities. Second, I build on the bootstrapped system and introduce an effective update and extension procedure for continuous improvements of HAR systems. The goal is to improve the segmentation accuracy of the HAR system corresponding to the prominent activities identified in the bootstrapping phase. Finally, I extend the capabilities of the recognition model, by utilizing large language models (LLMs) as contextual knowledge bases. Contextual information is encoded through the use of language-based descriptions and the use of LLMs. Specifically, the LLMs are used to identify the structural constructs that make up complex activity sequences. Combining identified structural constructs helps with improving the recognition procedure and identifying changes to the sequences of these structural constructs aids in observing changes to routine patterns.
In the bootstrapping process, I identify activity segments that correspond to prominent activities, which frequently occur over extended periods. However, this system only captures a subset of these activity instances. Therefore, in the update and extend phase, I increase the length of the identified activity segments. Smart homes encompass more than just data from sensor-triggered events. By incorporating external context, I demonstrate how HAR can leverage contextual data to improve recognition accuracy, particularly for short-duration and infrequent activity instances.
The work presented in this dissertation establishes a fully functional recognition process for ambient-sensing-based HAR. My objective is to develop a HAR system that can be deployed in any home from the ground up, without the need for prolonged data collection in each specific setting. Additionally, I evaluate the role of external context in these environments and explore how HAR can benefit from such information.
Sponsor
Date
2025-04-15
Extent
Resource Type
Text
Resource Subtype
Dissertation