[00:00:05]
>> I think everybody for showing up and they after the end of the semester far they didn't I have a true confession from a about the reason why this is not. Being told in early April but he says the hell now. We're both of the Early in the project you're remembering there is not a good myself to be a problem the week but I've also seen a few moonie to come visit and give a talk so to my chagrin I didn't discover this anywhere.

[00:00:38]
But more aimless every version of to just your schedule answered I was good for you personally shows you fully got a little bit different. But of course that made it more difficult for all of you and we really appreciate you being here so this is the book finishes off our off more aging Court Room Series for the year and we're having to walk away sure it's from from Washington State University and recover huge heat from the University of Memphis and then for this one you wrote yes he's a mean.

[00:01:13]
Breezy Hoover so he slipped a distinguished muscle in some. Orders very recently after her degree above the she moved to Washington State where she moved up the ranch uses this role for Professor and she said in the process of her a work areas including the so-called YOU socialization nurse like you were the clinical director for a while and then you somehow got out of that Russia last week a great way to not get that back you know you're going to cause these so clearly you saw the you you say there were no percent but if you're the interesting track record of research interest and funding she has one of the most diverse research funding for fully was ever seen in part because of the intersection of the word with technology engineering computer science but also.

[00:02:06]
Hold your own so all of you will be told us a lot about them sort of want to learn from your 1st began work work usually gruesome work in another car mission production memory term. Traditional regime reproved illusions which of them are still there but see below where she was doing a lot of very good.

[00:02:27]
Things you need an area of generally aging model guard. And the ways in which you were one or 2 to use technology for meaningful support for goals from memory problems ocean for me that's all that that's your heard give me more moments of her we're so sorry respect you so much for crowded room rooting for during your talk about using technology to support oracles from the White House thank you for that here Jackson and thank you everyone for being here today on what is a nice day outside and a celebration of the end of the semester to go so I'm excited to share the work that our team has been doing it C.E.O. and just want to point out that this really takes an army to do this work so it's really a team of investigators across multiple disciplines including computer science look see I went the wrong way.

[00:03:27]
OK So long term goal of my work is really to extend the functional independence of individuals with cognitive impairment so if this looks like the functional trajectory of an older adult I would like to keep that adult to a higher level for a longer period of time. You're all aware that America is eating so by 2031 in 5 Americans so that will be over the age of $65.00 so that's going to be 20 percent of our population.

[00:04:01]
Now this changing demographics is bringing with it a change in the number of individuals 2 are having difficulty living independently one of the big causes of that is Alzheimer's disease the statistics now say that 5800000 Americans are living with Alzheimer's disease. When they see this it always makes me my gosh every 65 seconds someone develops timers disease and 520-5013 point 5000000 people will have Alzheimer's disease.

[00:04:34]
So this is something that we need to be critically thinking about in terms of our health care right now virtually all our clinical clinical trials for Alzheimer's drugs have failed in fact they recently pulled a drug and amyloid drug so we need to be thinking creatively about how we are going to be working with our aging population so the sciences increase in the number of individuals who are having difficulty living independently we're seeing a rise in health care costs and because of the changing demographic right from the pyramid to no one for the pyramid We're also going to be seeing a reduction in the number of caregivers who are able to care for aging population.

[00:05:23]
So this is really presenting a health care challenge for us and how is it that we're going to keep our aging population functioning independently and with high quality of play now our lab believes one of those answers might come with technology a way of being innovative with our health care and one of the things that I think is important to keep in mind when you're looking into Vell thing these technologies that you want to consider all aspects of the aging population so if I'm working with someone with M.C.I. I don't want to create a technology that's going to take more work by the caregiver and put more stress on the caregiver so I need to be keeping all of this in mind when we're talking these technologies.

[00:06:10]
In terms of kind of a trajectory of change are all probably familiar with kind of the normal aging process we've see some changes in cognition and we now know that individuals who are on this trajectory to eventually develop some type of dementia actually are showing changes in their brain even 15 to 20 years before we 1st start to see those initial cognitive changes a lot of my work is big geared towards individuals in the earliest stages or mild cognitive impairment where they're starting to show some changes in cognition $1.00 to $2.00 standard deviations below what we would expect to see in terms of normal aging and they're also starting to have difficulty with more complex everyday activities so managing their finances cooking complex recipes but it's not to the point where they're unable to live independently.

[00:07:09]
All right my goal for today is to introduce you to some of our work on smart environments and then talk about how we're using that for real time prompting I'm going to introduce our digital memory notebook and then talk about how we're partnering up with the Smart home and some of our robotic activities support data and then talk about some of the challenges and opportunities involved in this research OK so smart environments so smart environments are in the environment that can possibly in continuously pick up information about what the human is doing in the environment and I'm very interested in smart environments for their potential to assist with health assessment and also with intervention.

[00:07:54]
One of the reasons that I got involved in this work in the 1st place is that a lot of our intervention work is trying to make a difference in terms of people's everyday functioning but then the question becomes if I'm not trying to make their memory better how am I going to assess their every day functioning and we all know there's problems with self an informant report questionnaires rate we can bring them into the laboratory we can do our performance based test in our cognitive tasks but we're missing a lot of things that are happening out there in the environment the way they're constantly eating or all the distractions in their environment so what if there is a way for me to get data in real time in the environment the other thing that I have been really interested in is the opportunities to do real time intervention so to be able to intervene in the environment when they need that intervention and this type of technology has been helping us get closer to realizing some of these things OK so how do we get information in the 1st place kind of what what forms the basis of the data that we use.

[00:08:58]
We have different types of sensors that we put in individuals homes and we also have a smart home test bed on campus the mainstay of our abuse motion sensor so these are infrared sensors that we put mostly on the ceilings and we can modify them so they can either capture on large area like a room or maybe a small area and maybe there is work desk that a person uses and if a person walks under into that space then the center is going to register on it and when they walk out of that space it's going to register and off of that and so we're going to see that it sees on off the bends from multiple sensors that will allow us to track where a person is and also service the activity recognition this is the sensor that can go on a door cabinet and this works when it separates it's going to measure an open event and then when it's together it's going to measure close to that so we can know whether a cabinet that someone stores their medication in for example has been opened.

[00:10:05]
These Or I didn't sensors that we can put on different objects and if the object is in motion then it's going to match then it's going to record that that object is in motion and this is one of the ways that we can track individual objects these are new sun service that we are just starting to work with one of the challenges with objects sensors in the past is that they only state charge for about 72 hours so you can't really deploy them in people's homes and ask them to corral you can change all their sensors I recently 2 hours these are so close to the role I have all for up to 6 months although that has not been our experience.

[00:10:48]
OK So this is a picture of our Smart home Tests found on the campus. When we bring people into the Smart home we test them in the body space so it looks like a typical apartment with a kitchen and a living room we have a lot more sensors in our Smart home than we use in people's homes we also have sensors that look at kind of the temperature that can measure the temperature of the water whether the stove is turned on and but mostly we're using a lot of the data from the motion sensors that I shared with you earlier we also before that this is kind of what the setup looks like for when we're doing experiments that we have to experimenters upstairs and they're communicating the video conference to the individual who's downstairs so only the participate and is in the environment and we ask them to do different types of task of daily living and mass are doing that we have an extensive coding system so we can code the quality of what they're doing as well as each step of what they're doing that gives the ground truth for the sun survey stand out but it also allows us to code behaviorally the quality of how they're performing every day activities and so we've brought probably now close to a $1000.00 golf in Maryland that have healthy have mild cognitive impairment have dementia and all the different sorts of neurological conditions.

[00:12:26]
This is our Smart home in a box we've actually installed smart phones for had people install their own smart home in over 140 houses to date on some of which we followed people for more than 5 years and basically. Over time we learned a lot of things so and this really when we were putting smart homes in people's homes they could see the switches and we'd have some people just go and turn the whole system off because it's a light right so you're just going to turn it off so that bore out our Smart home in a box to protect it and so people wouldn't turn it off and then we also worked on ways to have people the able to install their own smart phones then when you put things out in the wild you learn a lot of things that you don't in a very controlled environment so some of our 1st work were in.

[00:13:21]
Aging communities where most of the doors all look the same when we started having people put these in their own homes and their front doors were all different configurations sometimes our you know sensors stink connect with each other or the refrigerator was the big one that was used so much in the sensors would fall off and then we realized we could just put one sensor in the refrigerator and that simplified things so much more and actually gave us more data so so lots of things that you learned taking things from kind of the test that into the real world environment OK so those form the basis of kind of sensing the information to be able to work with that to kind of actually view assessment and intervention we need to identify our kind of tag what it is that people are doing and so our computer scientists have been working on algorithms that will recognize different activities that people are doing.

[00:14:20]
And let me start this. So basically what they're taking is all this information that's coming from the different sensors that's registering on and off advance and they are using that data identify and if I click something that made that go off they're using that information to identify an activity that that individual is involved in and to date we have a list of about 40 different activities that we can identify fairly well and most of them represent activities of daily living that most people would be engaged in so you know things like sleeping in a transition into the van for a minute my.

[00:15:07]
Working the eating grooming but we're also going to see that there's lots of different activities that are individual to a person and so you know this is kind of a high Here's our activities of daily living here is our sleep and then there's all these other activities that are specific to an individual so we've also been looking at algorithms that Kinko into the data and identify patterns that are specific to different individuals and this one I've used for a while because it really is my favorite we have students living in the Smart home when we're not running studies and we're like why insist pattern because we can kind of tell this is being this is you know they're in the bathroom and what is that us and we showed it to the student and she said I watch T.V. I get up to get something to eat and I go back and watch T.V. and so she did that so many times that we were able to identify the patterns in the data and we've identified things like scrapbooking patterns so things that you know are very specific for different individuals.

[00:16:14]
Now understanding like recognizing activities and being able to discover new activities is also important and being able to predict future activity so if we know for example that someone typically eats their breakfast and then takes their medication if we learn that pattern we can predict that after they use their practice they're going to take their medication and if they don't then engage in that activity of taking their medication and that becomes an opportunity to do some real time intervention with them.

[00:16:50]
OK so we've since we've finally identified some of these activities one of things that I want to point out is that even though this could be like a purely all Thomas system we see it as one that involves a clinician in the loop and so that's actually what we named one of our principal a clinician in the loop because the clinician plays a very important role in monitoring and working with the individual and identifying what sort of the interventions for example might be needed so in some of our early cross sectional work that we did in the test that we wanted to know whether we could take this information from the sensors and say anything about a person's everyday functioning so could the sensors actually tell us about the quality of their everyday functioning and we have people with invention of mild cognitive impairment and healthy older adults come in and complete a number of different tasks but this is our complex everyday task of the day out tasks and they have to multitask and do a number of different things to prepare for a day out and we coded this the accuracy behaviorally and then we also had the sensors drive a score that was to represent their accuracy and we got a pretty nice correlation of about 0.62 and so we were pretty excited in the sense that this gave some preliminary data that we could probably get some information about the person's functioning from the sensor data.

[00:18:23]
Now one of the things in terms of how do people live as far Combs they're going to become their own baseline data so you're going to be monitoring a person over time you can use some of the career data initially to give you some information about like what would be a cooking activity or grooming activity but ultimately you're going to be deriving a lot of information not specific to the individual so this represents 2 individuals this is the healthy individual and we used we have been running a lot of different kind of behavior change sort of algorithms and this one has to do with a sleep change score and these are this is over months of this it's 25 or 30 months that this person was living in a smart home and this was a healthy individual and you see very little change in their sleep and this is our change algorithm for someone who progressed to M.C.I. and you can see that we're picking up at about 110 some pretty significant changes have happening with their sleep schedule so one of the things that we're a very high interest in doing is being able to use this type of information for a proactive intervention it is a clinician many times you know a person will come in and I'll see them and they'll be complaining of memory problems and then Lo I'll see them again a year later well it could be the 1st time I see them you know I see them on a down day the next time I see them I see them on the up day they look the same but over time if I had more continuous data that trajectory of decline might be more obvious.

[00:20:00]
OK so how can we use that sort of information for real time prompting from. So we know that prompts can increase independence and activity at people with cognitive impairment we also know they work for all of us already many of us rely on Kronk's to do many of the things that we do in everyday life most of our comps are time straight you set your.

[00:20:28]
Calendar for a certain time that reminds you that you need to go somewhere some of our smartphones now can do location based promptings So you're driving by a gas station and you have it set to remind you when you drive by a gas station what we're really interested in and is doing more activity aware context face prompting because this is what caregivers do right they see what the person is doing and then they make decisions based on the context that I'll intervene with a person and so we're trying to drive prompting that's going to be more similar to what caregiver might do in the everyday volume and.

[00:21:08]
So this is the example I guess I kind of gave this one an earlier the idea that it will use our activity recognition and discovery out in the algorithms to recognize an activity has occurred like eating breakfast or eating lunch. OK Pick whatever you want lunch or breakfast.

[00:21:29]
Then the algorithm is going to predict that the person should take their medication and if they don't engage in that activity then it can prompt them to take their medication now because it's all full system it can also recognize if they don't see them that medication activity and it can wreak prompt if necessary so this is one of the ways that we can engage in this real time intervention now just to let you know in our earliest studies we've been working with prompting more proactive health related behavior such as walking or exercising or gauging in yoga I steer away from not a huge sense because I don't want to prompt someone to overdose on their medication so you know there is those safety issues are important and there's a whole host of questions that you can be asking in terms of problem then and so we started with things like you know 1st of all if someone with M.C.I. is like you've got to respond to a problem and so and some of our early studies we have people come in and do different activities of daily living and we've created program to how to time the different stuff so errors could occur and then preprogramed that interact prompt that just kind of oriented them back to the task and direct prompt which told them what to do and then a multi-modal prompt and we had people just naturalistically do these tasks and it was like a 3 hour battery and we told them when they 1st came in that these problems might happen but it was like 2 hours later that they might have received any problems so there was no training on what would happen here.

[00:23:12]
Here's what our indirect prompt would sound like. The still. So I didn't tell the person to turn the stove off it just kind of guided them back to the task the director prompt would tell them exactly what to do and then the multi-modal prompt. Would actually hew them.

[00:23:40]
Fish is going to cue her to go over to the computer screen. Later on the a tablet but the phone's. Very early on. And then it's going to show her. Way and if it all works well then the activity continues on. In the 1st study we were really excited to find that individuals with multi didn't mean M.C.I. although they needed more prompts they responded really well to just not interact props or prompt that just.

[00:24:25]
You know directed that kind of indirectly back to the task and we thought you know the specialist social psychology people don't want to be told what to do. But we said well you know we really need to test that to kind of better understand this the whole prompting So then in the next study we set up conditions where older adults came in and we had them all make an error in the task and then we gave them either bit interact the director multi-modal prompting and we asked them which prompt they like best and you know did some crisis wires and turns out that if you're reporting that you're having more cognitive complaints you actually like that multi-modal Q So you want more support if you're reporting your cognitive complaints you like that direct you know I don't really like the indirect some thinking well maybe people think if they're going to get a prompt in mind of what just tell me what it is they want me to do.

[00:25:18]
But these are you know some important questions and they also suggest that flexibility needs to be built into the system right as people are progressing with some of their impairment it seems like they're interested in having more and more support and the other thing that we've been working on is when is the best time to prompting fall so if I am wrong thing for someone with cognitive impairment in the middle of another cognitively engaging task am I just going to make the situation worse for them because now they've got 2 things that maybe aren't going to get done so we are working on trying to minimize those interruptions by creating algorithms that will be able to detect transitions between activities and be able to prompt people drain those transition points and we found through some of our early studies that it is true that people will actually engage in the 1st prompt drain transition activities more often and this was a recent study that we actually did in the real world so we have people living in smartphones and they had tablets and we knew when they were home and 4 times during the day we either prompted them randomly or re prompted during transitions between activities to complete side ecological momentary assessment information one of them was about the actual activity that they were doing so that allowed us to look at how well our activity recognition algorithms were doing another was about the timing of our prompts and we found that when we prompted people or tried to use our transition to tax and that actually increased their M. a response as it actually increased sunlight above 80 percent which is actually really good for it D.N.A. research too so I think that there could be even wider applications for for this type of work.

[00:27:09]
So in terms of my own research and interested in these prompting interventions on both for prevention so being able to help people that integrate new healthy brain aging behaviors into their everyday life but also to support every day activities and to support compensation and it's. So I'm now going to want to talk a little bit about our digital memory notebook and how we've been partnering that with a smart home.

[00:27:43]
OK So in terms of compensation as we get older older adults tend to use more external strategies I'm a big believer in external strategies having worked with a lot of individuals with cognitive impairment I think there is a role for internal strategies in new Monex but I think when you're trying to teach people to manage everything in their everyday life external strategies are really an important key in our own work we've also seen that the use of these compensatory strategies increases as one can enter a side M.C.I. stage but then as that load gets to be too much these compensatory strategy of start to fall off.

[00:28:22]
Just because I didn't know where it's been I wonder if you guys where are our website architect for aging. Website this is a website that kind is an act and if I have time I'll show you a video but it has technologies that can assist older adults with things like memory falls per vention hearing vision but it's just a really great resource so I just want to.

[00:28:52]
To put that up there so that you guys know about that but it kind of got me a little off topic so sorry about that OK. So benefits of compensation one of the things that we recently dead was go into people's homes and have them do everyday tasks of daily living but as they were doing these tasks we were not only cutting the quality in which they were doing but we were also coding compensatory strategies that they were using So were they using assistive technology is where they're making use of routine Did they have things out in the location that we could see them so we had about 10 different categories of compensation we coded.

[00:29:33]
This the mean age of this group was 7374 percent we're seeing now and basically the outcome injury here that I'm fighting is a prospective memory task so they had to remember to do something when we came the 2nd time 2 hours into the suction and we were interested in seeing whether their use of compensation would actually account for their ability to complete the task over and above our measures of cognition or performance based measures of everyday functioning their self reported their everyday functioning and we did find that compensation was a significant protect their Even after all of those other things one of the other things that we found was that cognition moderated that effect so that if you were at the average range of cognition compensation actually assisted you more so if you were high and you were pretty much getting doing the task no matter how much compensation you would use you you were fine but in the average range the more compensation you use the more chance you had of getting the best score on that task as we got down into the 4 line range the fact it wasn't so significant and then people who were having more significant.

[00:30:52]
Called these with cognition compensation was it making a difference but this was spontaneous compensation in people's homes so this has nothing to do with training people to use compensation and so that's something we're hoping is a place where we can intervene because we know that compensation can increase every day functioning.

[00:31:18]
So. It was very very large a grove possibles that was that it could be career disco solution which whatever they were doing ready to leave all the extra weight of some car was considered purpose rate so a lot of times for this house get money and get something like putting the medication Boggo like out in a visible place and some people were setting timers some people were saying multiple timers so you know there were different sorts of strategies that people were using but we were able to see them and code for them.

[00:31:53]
One of the things that we're doing in kind of a follow up to this right now is we're actually tagging different sort of compensatory strategies that people are using with our AM sensors and in between tasks we're also having them do a different perspective memory task and tagging aspects of the prospective memory task with the item sensor so we can actually pick up on that when there are going in using those things.

[00:32:21]
OK so our memory notebook I've done work in this area for many years. And really the memory notebook is a practice standard in stroke and T.V. I read dilatation we've demonstrated in others that these sorts of compensatory AIDS can be of assistance to individuals with C.I. can help them play and schedule and carry out deadly activities and the reason I really like this is because it can help both of retrospective memories so if they record things that they did in the past then they've got a record of that information but it can also help them plan for the future as well as initiate every day activities because they can plan for their day to write and they can goal set in stone longer term goals to go.

[00:33:11]
The digital memory know Paula application that we've developed we've developed it iteratively with individuals with mild cognitive impairment. And we took it through 5 iterations the 1st iteration was actually an Android app and it became clear right away that people didn't like that as much as kind of using IO assets just much easier for them to use so then we switched over to an Iowa here and we started working primarily with individuals with mild cognitive impairment as well as their caregivers.

[00:33:51]
So what we did is we have them come into the lab the show the hour and then we have them complete a number of different tasks so it might be something like imagine you have a doctor's appointment now we want you to add this to your to do this and we looked at how fast they could do these things and then after they were done everything we did some satisfaction in terms of usability and overall satisfaction of the system and then we asked them open ended question and and we continuously work to improve that design and what we wanted to do was get people to the point where on our question errors in terms of satisfaction and we were getting scores 8 or 9 are bought so people were very satisfied and we also wanted to get usability to like a 2 or below and by our 5th generation we were successful in reaching that goal and this just shows kind of some of the changes people got faster in their reaction times and their ratings that.

[00:34:51]
Kind of satisfaction and usability increased. We also found that. The app was pretty broadly user friendly so we had people rate their familiarity with technology their use the scheduling tools and we also did a moka just to get kind of a general level of cognition and we found that.

[00:35:16]
You know how fast and kind of their readings were not dependent and want to end their prior technology use or scheduling so we feel like we are making something that was pretty broad that individuals with cognitive impairment could use we didn't find and this is typical in the literature that if you can complete tasks faster you tend to rate them higher and so that and that's what most people found this is it shows you some of that are going to design and even then we try to apply a lot of the principles of contrast and you know things that you would.

[00:35:50]
Hughes and designing for older adults we still have to keep improving and improving on the I didn't know I'll show you are a video of our final but you can kind of see how we added a note section and then we eventually added the help section and our 1st individual that we had to take the digital memory notebook home to work with it for a week she got stuck in the help because it was like a video it looked exactly like the real book you know the REAL out did and she didn't know how to get out of it so then we had to like put you know a thing at the end of the video and how to get out of that so you know all these things that you learn from actually having people use things this is an example of our add new task to the age of one of the things that we it's not here that we did a lot of work on is just the curse for the.

[00:36:43]
Time being so you know a lot of the pickers have like the rolling thing which older adults do not like I don't know how many different pickers we went through to find one that worked really easy for them to be able to put in the time. I mean just OK Well let me show you the digital numbering up and then I'll talk a little bit more about that.

[00:37:04]
It's you know memory nearly application was developed a collaboration with older adults remembering him and to helps memory and everyday functioning it is easy to learn e e this and if it's design features like a lot of targets simple icons and text support the digital memory nobody has for me 10 minutes in the whole feature would be a tutorials.

[00:37:29]
The Today he is the home base it has a To Do list and I'm really time schedule that allows users to keep track of activities and events and information about activities completely throughout the day can help support American type characters that's such as get togethers with friends or me he's watched tech support encourages users to add detail information about activities these entries appear gray on the time schedule and to do list because they happen in the past.

[00:38:05]
The easy entry allows easier treadle of the details out of later time. The today page also facilitates scheduling abends the user wants to complete throughout the day supporting planning time management and IT TO BE initiation. To help users keep track of daily activities and foster feelings of accomplishment activities can be checked off when completed.

[00:38:32]
The digital memory notebook also supports memory for future that it's such as appointments travel or birthdays the calendar page is used to add events to future dates. The user can also label events as high priority set reminder alarms and or have the event repeat. To get back to current state simply press the go team today button.

[00:39:02]
The notes page can ease to record a variety of information such as shopping lists and addresses. Users can also keep a journal and daily experiences supporting memory recall and the silicium notional processing even management. Pictures can be added to note and you know entries to record visual information.

[00:39:27]
A search feature allows quick and easy access to entries. The 4th main component the profile page is used to store input into the from region about the user such as contacts and medication lists. Finally to support continued use the additional memory notebook presents motivation quotes and encourages recording accomplishments.

[00:39:54]
OK So looks OK so that's an interjection into the digital memory no. So I'm really there to train individuals in using the digital memory notebook and one of the things that has been pretty exciting about this is we're using 5 sessions of training where is it used to take maybe 10 to 20 sessions to teach someone how to use a paper and pencil memory you know so for example one of the 1st things in training a paper and pencil memory notebook is teaching them how to make sure they're on the correct day so they can be recording information for that day here if you're not on the correct day you press the go to Dave and takes you right to the crack day when you open it up you're on the correct day the other thing is helping them learn an automatic routine for looking at and using the digital memory notebook used to be we have to teach them how to wear a watch alarm to Paret to learn to pair it with a digital memory notebook now the alarm is just associated with the no buck they go right to the notebook so so many things that you can utilize to when you can use technology that meet Trini easier we use a systematic learning map to train the different parts of the digital memory no fuck so we work with them in their homes you've really got to understand the types of functional difficulties that they're having and how our system 10 S You know best benefit then we help to acquire.

[00:41:28]
You know using the digital memory notebook and that involves establishing that overlearned routine of looking at and using the digital money notebook and we've got. Each session has some problem solving goals that they need to utilize we teach them to apply it then in their everyday lives and then also to adapt it to different situations.

[00:41:54]
Underlying all that's the kind of you self-determination theories that we're trying to teach them to be more autonomous and to develop confidence in being able to use them as. The caregiver. Can be involved in the training but it's usually just to the extent that we're training the individual who is having the memory difficulties and then we kind of check in with them at their something that we might want them to assist sweat but for the most part we're trying to help the person be more autonomous and the able to use a system that was designed by people with M.C.I. to be easy to use.

[00:42:36]
This is some data from one of our pilot participants and I could have thrown a couple of different examples up there this is a 7 year old Latino female she had 14 years of education her Arbonne score was at the 3rd percentile so those this kind of the borderline range of cognition and she was reporting that she was having difficulty remembering to complete each words people point minutes difficulty recalling information about past events such as movies difficulties with time management difficulties with organization and task initiation she was worried that her husband was having to do too many things for her she was also.

[00:43:15]
Feeling very guilty because she was missing appointments with her friends and that was upsetting to her and to then so we talked her to use the system over the 5 sessions which we do in the period of one mind this is RELIABLE of change docs course I'm here is the normative data for these different measures the 1st humanitarianism measures of everyday functioning and every day memory lapses and you can see that we were able and we want to take these down we were able to take these down and this is 6 months after the intervention so we're done training her she's using this for 6 months and this is this.

[00:43:53]
Data itself is really exciting because a lot of times our interventions are short lived. So here we don't take this down to the normative range but we would expect if she's got memory problems she's got significant changes in her cognition but we were able to show a significant How liable to change these are emotions that look at things like life satisfaction in her coping Self Advocacy her quality of life and you can see that we were able to bring all these measures into the normative range after the intervention so we were pretty excited about that data.

[00:44:30]
We're currently running a clinical trial and this is one of our 1st case examples where we have how our sample is getting the digital memory notebook alone the other is getting the digital memory notebook that's partnered with the Smart home so the Smart home is able to serve as a queuing for them if they're not using it so a spouse wouldn't have to do that.

[00:44:53]
Also And also you can read information into the digital memory notebook but this is the 64 year old gentleman who was diagnosed with M.C.I.. At the end of that intervention which was 5 months later and we are getting some qualitative information from him and he said it's the 1st thing I look at when I get up in the morning because it's sitting in a prominent place and that's one thing that we work with down the line I've not missed a medication I put on my To Do List the To Do list is kind of a motivator to get things done it or against me to the day of the week and put the entries into the things that I accomplished in the day I guess that's what you want me to do with it and it's working so he had a lot of positive things to say now the other really cool thing about this is that we can see in real time how people are using the digital memory no this is post intervention so I don't have when he was learning the intervention I have for 2 months after he completed our intervention training this here is the use of the help and you can see that he's not having to use the help very much at all and this is the daily uses of the digital memory Novak and you can see one post intervention he was using it on about 10 different occasions during the day and he kind of settles out all into late around you know 6 to 7 uses of a digital memory notebook a day which is pretty good and that's what we want them to be doing to be checking it and to be guiding them through their everyday activities this piece looks at the different.

[00:46:30]
Aspects of the digital memory know so that today pages like that mean where they can you know schedule their daily of that it's then that's in the blue and we see that he's making good use of. The calendars and the purple and he's making good use of the calendar the journal is in the green and he told us initially he says I'm not interested in the journal and yet we are not interested in the journal you didn't use it at all the notes pages and the read and it looks like he's not making quite as good use of the notes over time it as he could you know maybe that's something we need to do a little bit more training on our booster could be he's getting what he needs out of you know these 2 sections and that's that's what's working for him.

[00:47:18]
We also coded it in terms of whether he's putting in things in the red which are important for prospect of memory for a planning in the future or. The Blue which is retrospective memories so he's recording things after they happen and you can see that he is using it for both prospect in retrospect it takes a task and the green is whether he is like checking off completely that sense so he's also making use of you know tech checking when he has completed events.

[00:47:50]
OK I said one of the things that. We're able to do with the Smart home partnership mass arises Now once you have the Smart home this must make information a stance an algorithm that recognizes these tires and that's a tease this information transfer D.N.A. and the power of that Iraqi mission.

[00:48:10]
That to me is displayed on the D.N.A. and the individual can choose to and he wants. To be incorrect so that's another thing that we can do we can feed information into the digital memory notebooks though they don't have to record all of that information. The other thing that we're using it for is for doing activity where transition base prompting so trying to prompt then at opportune times to use their digital memory notebook if they're not engaging in regular use of that.

[00:48:45]
The other thing that we're hoping to do eventually is also be able to prompt people after like a usual van So let's say they had visitors come over that might be a good time that they might want to record some information into their digital memory notebook so can we pick up on those sorts of events and then prompt them to use their digital memory notebook if they're not I all right last part.

[00:49:10]
One of the things that we were you know most of our work is done it's the person has to go to something so we wanted to see if we could design something that would actually come to the person and we had a robotics person who has what I need to work with us on a project so he said Sure why not you know what we'll try this if people become more significant repaired this could be there any more and this turns out to be a very difficult question to get everything working together to show the concept so you know we have to be useful to detect errors in real time as they're happening so we've got sensors that are on objects and we can only do this now for very learned activities so our system knows all the objects involved in the pattern of person does it but then we've also got to be able to have the robot stand in space know how to move through the state be able to approach the person you were against itself directly to the person and then the robot ask if they need any help and one of the hopes that they can get is actually leading them to an object so I'll show you the example you can see it's a very low tech world by.

[00:50:30]
Which just spent a week in someone's home my biggest concern was I the person was going to fall over because they were going to grab on to the robot and everything was but it didn't happen so I was very relieved. About some robot he's on it's. The most kindest slow or.

[00:50:54]
So our ratings of our 1st study was her because I don't know all aspects of our robot actually buy it. And then a robotic person has taken another position out of academia. So we had someone help draw like the different aspects of the room but you can see this this is a very contrived situation where she should have water that plant before the other plant but someone could you know one of the other plant before that plant so there's lots of things to to work on for prompting we've been most successful in terms of prompting initiation of activities but trying to prompt when an actual error is occurring when the person is doing an activity is very difficult and in fact in our 1st studies we did screw up dead but then I wanted to kind of get a feel for the types of errors that people would do so we gave them scenarios and then we asked them just to kind of you know make an error that might be like someone with Alzheimer's he's gotten confused and we have things where people would like take the sugar and just you know pour all the sugar into the old meal Well how am I going to pick that up unless I like have something tagged on the sugar I mean it's going to end up not being good to eat right but it's going to be something very hard for me to pick up so we're going to see that there is definitely limitations to this work which leads into kind of challenges and opportunities.

[00:52:29]
So some of the challenges are making intervention technologies that are going to be acceptable to the an user so you can see that one of the things that we do is work extensively with the user when we're developing these technologies because if they're not technology is that the end users are going to be comfortable with them they're not going to use them so the 1st time I walked into the Smart home pointing out that my colleagues said can you get people to bring this and put this into people's homes and there were wires everywhere there are wires that were coming from the middle of the ceiling down into the phone they were below so I was like.

[00:53:05]
Probably not alive we want to use video and people so as I said let's see if we get people to come stage here for 2 nights with the video that's. And even work so you know there is a lot of kind of warning in terms of you know the user has to be comfortable with that otherwise the technology is not going to be use.

[00:53:25]
The other thing is you know if we want to do these activities where real time intervention 1st of all can we detect situations where we can use them as and then can we deliver can we deliver these prompts interventions with false positives so you know a lot of the Falls software has a high rate of false positives and if you have too many false positives people aren't going to want to use the system so that's going to be an important thing.

[00:53:53]
We need to be thinking about technologies that don't require a lot of new learning or like our digital memory notebook and for working with the user to but design something that is going to be simple for them to use something that's low in cost that protects privacy. Something that probably is going to have to adjust to the users' needs especially in working with cognitive impairment that's going to progress how are we going to keep people motivated to use these interventions though another thing that works flooring some of our studies now is intervention and here and send you know we can get the state and real time when we're prompting the you know when is it that they actually going to initiate and follow through on the prompt and when Aren't they going to leave here.

[00:54:37]
Sensory battery life I was talking about the problems with sensors so that's an issue of training the future generations that we've been fortunate not to have some training programs in the area of German technology and to be able to be training students across different disciplines. But what are the opportunity as well there is a lot of opportunities for health innovations and for ways that we can extend that functional independence and keep people at this high level for a longer period of time so it talks about real time intervention the idea of being able to do functional substance for early detection you know could be something as simple as a person is like walking around the smartphone differently and maybe they don't realize it's taking them longer to do task and they're walking around differently but maybe that's something that's going to cue us and that something is changing or are you know a person and there is an acute event and a person doesn't get out of bed and it's 1 o'clock and they're up and they're usually active needed after the event or someone needs to go check on the person improve safety being able to do those proactive interventions reducing the burden of care givers improving quality of life and then also things like monitoring inherence to interventions or medication.

[00:55:54]
So I want to thank my contributor. Diane Koch is my collaborator in computer science we've been working together now for about 12 years she does. In the computer science aspects of things Larry Holder was very important for the development of digital memory notebook he actually after our we started our clinical trial and digital memory notebook kept crashing the rebuilt the whole thing for us and now hardly has any problems that we're so happy.

[00:56:27]
Crandall's also in computer science and physician and also a collaborator in psychology but I also want to thank all the students in the participants are funding agency and I guess I can take questions now we have like 5 minutes. Yes you. Look. Like. The one who was. All this for.

[00:57:10]
Yeah so I mean if it's going to be something that can actually be like a smart home in a box that can come off the shelf we have to have some algorithm that can pick up the kind of general activity so we've been working on that kind of generalizing from one home to another home and there are things that you know most people do you know they groom they eat in the kitchen you know so that there are certain things that you can generalize pretty well across homes and sometimes I can actually increase the algorithm in the home by having that information of how to time but there are also other things that you really have to get the learning from the person individual way to be able to develop so you are using both approaches.

[00:57:54]
Yes really. We're all in on the whole series. Of who says this person is real or. Really. Really good and the whole routine in the room goes from an earlier this in perspective you probably have your. Daughter A lot of the world over the moon. Is sort of.

[00:58:25]
Over those words but somebody who is an. Older brain and. You're with us. Really screwed. Yeah I mean we've done some studies like our our firsts our 1st study like looked at very scripted activities and then you could say like well that's not really how we do things in everyday life right so then we did multi-tasking activities where we had people come in and we told them you know these are the different tasks now after they did do them in any order you want just multitask be as efficient as you can and then also the question of well what happens if there's 2 people in the environment right like if there's one person you know you can do OK whether there's 2 that's a really hard question for the algorithm so definitely accuracy decreases as in those multitasking situations but we can still call part those different activities one of the things that we have learned is that when they are stupid was definitely a very hard question and the computer scientists are still working on it but people are creatures that have that So if you can kind of feel like where that person was and then see where they're going you can usually figure out like who that person is just because of way you know the how beds and in the environment but we've tried mostly to work in France where there's only one individual and even for a week the accuracy of our.

[00:59:53]
Activity recognition algorithms it becomes complicated of 32 people in the house because if the person is the one in the study then we're like well really to test its accuracy you have to say whether the other person might be engaging in that activity because we just recognize that someone's you know in the kitchen cooking maybe it's not need but you know maybe it's my husband who foods in the house.

[01:00:18]
So yeah definitely routines that makes it much easier one person in their home there's lots of things that help the accuracy better but there's also lots of things that make the actor seem much more. Gets because you have a little bit of her that she was a push toward good explosions or a few words it's you who says you know I mean we talked about that but of whole nother level of complexity one we have to ask the person to wear something and we've really because we're working with cognitive impairment trying not to do that although we've got a whole nother thing with wearable set that we've been doing with activity recognition.

[01:00:58]
But. Mountain of data that comes in then is. So we've been trying to keep the system as low in cost as we can and try to develop methods for parsing those out with the data that we were told. So you will sure it's a results of the old.

[01:01:25]
Yeah so we have definitely been working with wearables. And we've been like using P.M.A. with wearables where people can put in what they're doing and in fact. My colleague had one of the really annoying experiment where like every 10 seconds the person was spose from telling them what they were doing grow they know people that what they do best but really now we've been working on things where they put it and more often I think as the system learns it doesn't need to ask them as often what activity is that they're doing so we are trying to find ways to use wearables to tell us more about activities outside the home but then we haven't even started to touch like how decide to integrate with the system so we actually have an experiment with.

[01:02:12]
A way or goal the kind that integrate with the digital memory no fuck to tell. You. Why. Yeah people don't want cameras and proponents There is also like a lot of times and a camera there is a group and a tiger house the same thing. Well I think they are using cameras but what they're doing is they're changing that image that you see from a camera to a person that looks like a block or so from a so to the person they don't see the person but I don't really know like where that transition in this for like to you know like I'm not exactly sure how that's all locked down but our robot actually does happen here because it's the only way for it to see the environment.

[01:03:10]
So that all kind of different thing it's not like we're using it to see what the person seeing but we can't track the environment without it but we don't have to record that information and so if you know once it play tracks it it can just get rid.

[01:03:28]
Of. All right exactly. Now the whole. Our sensor is a aren't very good for false although then a recent study that we have or we have a nurse that talks with individuals on a weekly basis you know person with Parkinson's disease and follow he called the nurse and said I felt today I want you to go through the data and tell me like when I fell and she was able to come up with 2 different events that she thought were Falls and she did she did get a fall for one of the but it's a little bit but you know we think we think we didn't we could be more proactive and kind of maybe noticing when there's a change that a person might be at higher risk for a fall certainly if a person is in an area for a long time that they're not typically And there's no move men you know that might signal things but you know unless you have a camera but even with cameras and you have to have cameras and not locations to be able to detect that person has fallen the biggest technology they use are things that people wear by the nurse on high false positives so those so you know depending on how a person falls could make a difference as to whether registers fall of.

[01:04:54]
The. Yeah there are only together our feet can stomach those small sensors and I talked about that we stick on things that actually do you have some sensors I think can like stick on walls that will like tell you like distance between things we haven't really started working with then but those are other options.

[01:05:17]
Yeah I mean there are so many different types of sensors out there are now finding ones that you know if you provide a whole day enough and can integrate with your system that you so we have to integrate all these different things together our head to make everything make sense.

[01:05:34]
To do it. With. The words reduced and really with the program. You know we're not creating any universe for medication for the digital memory you know folks just that's what they're doing so we're you know they can program in an alert but we're not prompting a thing like badly if we notice that they didn't take it we're we're not cutting or anything like that you go to your.

[01:06:08]
Tree. Yeah. Yeah yeah yeah the tool is really for them to use as a weapon their everyday life and so for any sort of difficulty after having been in some of our. New work we are going to be tracking medication here and it's with those a good son serves and we think that this is going to give us an opportunity to chat to track more real world medication adherence because a lot of the bottles you have to put it on a file on a person might not use a bottle you know they got their pill the signs are in so and they might have a cold the sensor you know on their counter in the morning and at night they've got a medication somewhere else and so we can get that information in a more naturalistic way.

[01:06:58]
Well the host who says you know these are those 8 Uncensored site you can stick on different objects that we're going to stick around on the different medications that they're using different places and. There was a there from the border just sort of the move you know there were didn't do something to yeah I mean it's like that it's still the same thing like unless you count the pills or you have them in just something you're not going to know for sure that they took at but by looking at the time that they you know how much movement there it is how long it is we can on kind of take it maybe even medication bottle to stop functors something.

[01:07:40]
Get good for a certain just. Mysterious rumors mushrooms for me and for everybody I was going to go on the street you were. Really beautiful at Harlowe or those who could I saw the tax lawyer on seeking Yeah yeah yeah yeah we actually do you have. The capability of doing voice there's all sorts of limitations but head above because if we do voice then who plays out well has access denied them for me send I'm so far experiments are pretty much having to use the keyboard but I know all the adults have actually preferred the key part because they're in there use to I mean some of them like to use for days but how that hasn't been a problem like the keyboards big enough for them to use they definitely prefer it over the keyboard that comes up on type screen and they prefer it over writing sort of like our individual with Parkinson's Disease because her writing misplaced getting so small and also M was difficult for her she loved my keyboard it was a great many compensatory strategy in that cell for her to be able to record them.

[01:08:50]
Yes. No For that matter to you that is there's no reason whatsoever it's a system that's due to all of the money so the person just continues normal everyday life as they would so you know maybe they would have to put the sensors in it they caught it like on the shelf right now it will put the sensors and.

[01:09:22]
But then after that they just continue with their everyday life. The sensors were what they're doing now one of the things that we've been hearing experimentally is having people use like you and I then it takes them to the tablet and then they just have the cross you know the answers to questions that are pretty simple you.

[01:09:44]
Know they never you know quote There is sun on treating past security at the high but like I said the thing that is really nice about it is that you know the 10 to 20 suction is now pretty 5 sessions of very high and so that's that's a huge difference in.

[01:10:04]
How much. I. Mean. The. Question I thought for you this question is great you know we're not doing a search functions for us on a clinical trial but I think that's something that I'm going to look really closely because it could be now that individual's not going to use an exception at all but if I had a guy named Eric earlier and kind of found what he said like really hope to use it to support his everyday functioning I think he had music of all I'm also interested and as we're you know working with people in the House of Justice Carney.

[01:10:51]
Complaints versus versus really. Are they going to be using the different aspects of a digital memory a different way to support or you know where they're hiding problems in her everyday life and maybe that would change over time so maybe someone with the adoptive American place is going to be using where the suction taking more pictures.

[01:11:11]
As they start how do you or maybe that might become a bit more difficult for them but they can still use other stuff to support their. Well thank you.