[00:00:05]
>> Hi Ron welcome to the ideas Ai seminar series today's seminar you have any piece to have Professor D'Mello from the University of Colorado at Boulder He's an associate professor in the state of cognitive science and the Department of Computer Science there his work lies in the intersection of computing cognitive affective social and learning sciences and he's interesting the dynamic interplay between cognition should.

[00:00:32]
Co-edited 7 books published more than 300 journal papers. Get multiple of arms at international conferences and his work has been funded by numerous grants most notably he is also the leader of the Institute and as a young spirit on student the I feeling which was recently and we're very pleased to welcome Sydney to our seminar series and we'll be talking about understanding human functioning and answering human attention through competition.

[00:01:02]
So well I'm glad to be here. I actually have a project with some folks in Georgia Tech and spends some great time there. Would always love to be that person maybe sometime in the future so I want to talk to you a little bit about. Some of the work we've been doing and how these kind of ideas of my research program align with the the institute so.

[00:01:31]
Generally speaking let me just start with some kind of central claims one we're interested in human functioning from the perspective of cognition emotion and social interactions so this is not so much about physical health or even mental health it's more about thinking feeling doing it interacting. And you know we've learned a lot about these complex phenomena from you know observational an experimental mothered that's kind of the bread and butter of cognitive psychology at a living approaches you know observational studies where you corpus studies and so on a great deal from instrumentation and also traditional computational methods for example that model that you see read there is amazing model of I move means you're in reading right and it's actually a very mechanistic model in fact it's basically worked on all the equations and the model is really just a computational instantiation of the question of how popular much motor control was so we we've done a bunch I think so that when we get into really the only phenomena and in an ecological context.

[00:02:45]
You know the central claim is that we can computational models on a specific type which I want to talk about is more like a machine and competition model. Where you kind of have a very good idea of the theory but you're nowhere near being able to specify mechanisms.

[00:03:02]
Or even I don't even know what the partner variables are to the point at which you can actually you know run a tradition. So it's all these models essential when there's no adequate erratic of a mechanistic account to give you an example. One classic example is how do you know how do how do your eyes move when you are how do your eyes move in the context of social interaction right so there's just too many variables.

[00:03:33]
Second sometimes you just have too much data or data is too complex and I'll show you in a couple of example if you even consider 3 people interacting or 2 people interacting the number of degrees of freedom if you really consider all possible channels. Becomes just computationally intractable so it's not even possible instantiated proper model without making major simplifying assumptions.

[00:03:57]
And that these models if they constructed correctly can provide some insight with cats of course into the underlying phenomena so you can use the models to learn something about what you're interested. And they can promote change to dynamic invention right so if you have a model you have something physical it's a tangible object whereas a theory is more abstract and now you can do something with it you can you know you can bet it in a robotic system you can be embedded in a computer and so so it's not just the this is lovely blend of both science and engineering.

[00:04:31]
And of course how you construct them is really important and just 2 points I'll get on our question instructing these models is one we really want to study the phenomena in relatively at the logically about the context and that doesn't mean you always have to go out in the lab and in a while you can you can construct pretty much reasonable situations in the lab when you are going to.

[00:04:53]
Draw a distinction you know I'm going to talk a little bit about I track in reading later you might be surprised to know that maybe 90 percent of the research and reading doesn't involve reading actual text right so you're reading words are you reading sentences though that's something we would you know consider not not the context we want to study that you want to be people engaging in authentic material.

[00:05:14]
And 2nd you know we're not interested in the proper bridges in you know getting the most accurate model or the most generalizable model of soul you know running 3 weeks of g.p.u. you know a 1000000000 layers and because the point is that the models should have some constraints by and be guided by a theoretical understanding of phenomena but but not not into restricted.

[00:05:42]
Well and the reason is because the theories that we're trying to understand a very incomplete so it's it's this challenge of striking this balance between you you want to do you want to stay consistent with theory I don't understanding I'm going to sit here a year I mean psychological theory or what you're interested in.

[00:05:58]
But but it can be quite constraining because our understanding is very limited so so when understanding is limited in people then the national inclination is to simplify things but then you're not really studying the phenomena anymore so it's this kind of tricky chicken and egg problem though I'll groundings with some examples so the kind of research approach we take is what I like to call it I'm not apologetic Lee part of the stick and the sense that.

[00:06:24]
You know we try not to look at a problem from the perspective of our methodology like as a theorist of the machine learning person they're going to various is an experimentalist but really look at the phenomena and sugar and analyze it some different way than what is what's needed so this is all or do quite a bit of observational and experimental research so very basic stuff in the lab in a wild controlled experiment sometimes they are like a physics experiments it doesn't matter but that is you know whatever they are we collect We feel that you know I think this is you know speaking to the choir you are preaching to the choir but when you when you instantiate a model of some sort it really offers you explanations of different ways to test the theories that you can ever do it's the basic experiments right because because model to follow parsing you can simulate things so so we always there when you have your data or you have an understanding that could end up construct a model and you know that to run and then again you know you have a model and you can do things with it so we like to embed the models in some real time close to dollars in technology.

[00:07:31]
So that said I would say we're a kind of blend the psychological and computational sciences and in the psychological sciences the area of interest now I'm cognizant psychology you know how people think and make decisions I think to science but emotions are running sciences so people learn the toxicology of how people interact a lot of work in teams sciences and discourse other people perceive the world in the narrative form and in the computational passes the areas I've worked on are affected computing computing involving emotion attention computing focusing attention multi-modal interaction Europe is the logical sensing this is where you express a couple of brain imaging with behavioral and physiological signals.

[00:08:19]
And the man areas are human computer interaction and machine learning but I would say going back to this diagram kind of the work on you know the top 2 observational experimental research and then instantiating model so that would fall more into the cognitive science you know from a traditional disciplinary perspective and this part of the model and the in the technologies and interfaces that would fall more into x. the human to the interaction so really it's the model that kind of connects the base the more the most foundational basic research and the more like use cases where Coke disintegrate overview and I know it in the end I must just but be so we've done work in a lot of areas starting with computation models of cognition attention on my computing eye tracking and each of these are you know several your research programs but what I want to talk about today is just to illustrate examples are what is really understanding I'm against what they tell us what people read so this is very much a you know one person and a text right and though at 1st blush you may say Well how interesting can that be.

[00:09:31]
Actually it is a shockingly interesting because people people have to get a text is kind of fascinating and the variability you get is is absolutely amazing which you don't get By the way with richer media So for example if you look at how people are you know watch film you get very very a homogenous patterns because these people have manipulated your attention to such a point that there's absolutely no absolutely no.

[00:10:00]
There's absolutely no of variability so that's that's one piece and then the 2nd thing is going to different direction is looking at. A model multiparty modeling of collaborative discourse how people come together over zoom like all of us are doing today to solve some kind of complex problem or actually do some goal directed activity and what is the data we can collect in these kinds of acts and how can me kind of understand it navigate.

[00:10:28]
The case so that says let me just get into the 1st kind of question the 1st the 1st project and this is work that I started when I was a noted a member of the team to the left and then I moved to Boulder and that's the middle team and then now it's now we're getting into more of a kind of working when you're a scientist and that's the kind of rights to tell you believe you we'll i don't we are really right if you look you know typical reading study and you're looking at patenting reading you know you may see initially something like this where people are.

[00:11:05]
You know able to pretty much downtime ask. But then but they're a little later on you know it gets harder and harder to focus attention and you know this is quite common right it's really difficult to sustain attention without a lot of external stimulation and. So one way to think about this is how do you figure out what somebody the tending to is so just just so you know attention is attention is basically the way in which we figure out what part of the world we focus on in the moment so in any given moment in time your brain is trying to solve just 2 problems one what to attend to and 2nd what do I do next though it is believed that every 150 milliseconds you're kind of making this decision in what sometimes we call a cognitive Adam or a cognitive So it's just it's just occurring outside of the awareness but that's happening all the time so you can look at the direction of attention I say Ok am I focusing on the focal activity and let's assume in this case it's looking at a computer screen of working with a computer screen or is it elsewhere so that's rather an objective measure but then the question becomes what are you thinking and that is something that's very hard to measure and you could actually be having it gold or really it thinking about the task that you're doing so if I'm reading for example my thoughts about the text right now you're listening to me or you're watching this this talk you actually reflecting on the words I'm saying or it could be my something else and it could be gold underlaid and to make a long story short there are some easy cases later you're thinking about the content of the goal and you are looking to what stimulus and we call that over tension of the different types and then you could just be distracted enough to have the harder case of the author I mean diagnose when for example you actually are thinking about the goal but it looks like you're not so this because many times when people than when students or people are you know looking into the sky and it's really impossible to tell Are they actually daydreaming I think you very.

[00:13:06]
But what I want to talk to you like today is a different time when all signs point to the fact that the person in the tent but in fact their thoughts are. To get away into off task thinking so completely unrelated and in colloquially you know because daydreaming and zoning out but they're kind of charming you scientifically mind one so why focus and I'm wondering so.

[00:13:32]
In the context of learning which I'm going to talk about but it has implications for human performance in all kinds of military those are vigilance that's a kind of context this is a matter now says of studies we've done in our own lab so it's just a matter now to basically a single study tells you nothing so you really figure something out you have to compile studies of many contexts and then and then you can estimate the columns I mean in fact this is a minimum and also because this is not a big look at the whole literature these are just studies we've done over the last 10 years and mind wandering and this analysis looked at 25 studies from about almost 2000 people in a variety of learning context taxed video technology video game audio book like you would anything you imagine and essentially the finding is pretty consistent.

[00:14:19]
That on average you might wonder about 30 percent of the time and the way my mother expected in the study by the way is through what's called a pro cut method as you're performing a task you get a pop rock and you respond so now they're not going to self report or it is what's called tough guy but be all time that you may you met many times in this talk just realize you talk just totally drifted away which is totally normal let's call it soccer and we can and the reason we rely on initially rely on some reports for this is for one there's like no objective measure there's no area of the brain you can measure and say This person is mind wandering That just doesn't exist and 2nd we don't know of yet and 2nd because my mind is actually a subjective conscious experience it's really about what your.

[00:15:06]
We're conscious of this is why actually stop reports can can be quite reliable and there's and there's a lot of data validating those but in addition to occurring frequently actually negatively correlates with learning outcomes so that's a negative point to a correlation across the studies and you learned and wanting to note in in psychology in the psychological sciences that's like a moderate correlation you didn't really ever get a correlation that exceeds like points 4.5 And so typically correlations in this field go from about you know point one to about point 5 again quite there are very.

[00:15:40]
So we actually came up with a theoretical model to explain why why the mind would wander and we had a lot of the brittle studies to support them. So I won't get into those the purpose of today's talk I'll just tell you like one study just to give you a sense of what happens so much study there read this book it's like the 2nd most boring book on the Internet it's about surface tension and liquids I think it's actually kind of cool and then they also watches balloon this movie sorry this is a very classic movie in narrative comprehension studies where it's about this boy in this red balloon and so I tried to doing that to discover what the what the stimulus looks like the idea Reaper 20 minutes and then watch the movie 20 minutes of ice over some and it's a very simple design they say they have no interruption in their reading they're watching and whenever they cast themselves in my wondering like you're like wow that's what happens they basically immediately type out what they thought right so and then actually you can analyze the content of those in many ways to really really figure out what's happening and use an example the 1st thing is they you know that was the lure So that's something that's so that's was something in the in the text that that words are going to memory and the memory was somehow lost and it was in the lurch through up or down the monolith so so a word in the text basically activated a memory that was unrelated to the text and then to people and that's what we called wondering because these are just zoom around without really any direction and right so and this is one example of many many things that may occur and why it's so really hard why it's so difficult to concentrate so this model kind of explains the different sources in different ways and what happens in and so on and so forth there's one thing I want to point out though and that's this I don't know if you can see a mouse but there's this when you.

[00:17:40]
You might wonder you engage in what's called perceptual and cognitive The coupling So what happens is your processing the stimulus if you track it will die movements which I'll show you in a 2nd it looks like everything is fine however what is being encoded in the mind is nothing so it's you completely the couple So you're moving in this automatic fashion but nothing is actually being coded which is why you actually don't really comprehend anything this is why you've got this negative correlation so I just gave an example of a classic study not a classic one of my favorite studies they had people read one word at a time and then you know you read a word the space back at the next word word by word reading and at some point the text would just turn to jibberish So you just reading nonsense over people mind wondering it took them 17 words on average to realize that they are reading jibberish right so this is exactly the idea of IP coupling.

[00:18:37]
The one challenge of mine wondering and these are not in a very difficult to measure so to illustrate some videos of yours computer like Jan 2nd clips and in some cases you can see it's pretty clear so in this case if you look at her you know most people can look at this and be like Ok that's an easy case where she was on doubt or something this was another easy case where he's pretty much focused Lee focused on reading currently.

[00:19:10]
And by the way these are all objectively made verified by asking them questions about the text when they read so they said in mind wandered on you know this page we later our last question on the page and you get that expected negative correlation yes a little trick your example in this case people have a hard time figuring out what happened with that Q If you look into the camera and about half the time they get that they guessed correctly or incorrectly and you're going to the example where it's a little difficult to figure out what's going on.

[00:19:43]
And so people have a hard time understanding how to interpret that gesture so yes with a computer there's a message or in many cases the enteral to use the people who exhibit actually not very good at reliably detecting whether somebody is owning up to I'll tell you later there's a limited utility to looking at behavioral measures so another sense right because why I really kind of evolved to hide these cues Why would I want to show you that I'm zoned out a and b. you don't know you're zoning out yourself sometimes educated just opening up the eye tracking is a good way to figure out what happens because I talking reveals kind of the locus of this all the tension right and the and the and the understanding is something like this when you're when you're actually engaging in anything normally reading or doing whatever they is a dance movement between what's happening in the world and how your eyes are moving right but when you're decoupled you that association should break down because you're not actually engaging in processing so your most of your i.q. ocular motor control systems cannot actually tell you where to move your eyes so just get an example here is a person she's watching the film is going about and that Green Dot is actually high movements and in this case you can see she's kind of tracking that balloon that's called a smooth pursuit movement because you're kind of pursuing an object.

[00:21:15]
And so I will look at I talking to look at mine when it is one of the most reliable measures but it's also very noisy so look in this case you can see she's Yes You know they're you know they're chasing the horse and and so on so here's a kind of a simple pipeline that this will all work to maybe 5 years ago to just build a model of mind wandering so we had of course we had some people read about 132 people read about or read a text page by page and when I was a cup of mind wandering they had a space that was it and they are actually.

[00:21:49]
Using They did that about 30 percent of the time in the data at the same time a record I movements and it goes through some time basically cleaning where you extract successions altering it all kinds of steps and then it's a very simple machine learning pipeline you're but they weren't dating is that actually in many cases you don't have enough data because it's actors knowing the you're there with an update on the page and I don't know it's like 40 percent of the cases but you can actually upliftment your model predictions with a probabilistic approach that tries to figure out how do I What is my best estimate given missing data that actually greatly boosts accuracy.

[00:22:29]
Here's the main findings the models are moderately accurate so we have a correlations about point 4 with respect to stop records and the graph on there on the left shows you the across a given subject across all subjects the distribution of stuff reported vs model based predictions and importantly the model trying to generalize across people right to their validated and proselyte in a way that.

[00:22:51]
They give generalizable actions precision and recall at the same importantly the models predict letting up so in the study that a comprehension assessment afterwards and you can actually see that whatever the marvelous predicting correlates with. The robust emitting data and the signature in this case we were one of the 1st people to show that having fewer fixations game situations but longer take stations actually was the signal of mind wandering and the reason is that when you are when you are reading and or when you're normally reading your eye movements a very a rhythmic because you're responding to the text ha ha you're constructing an understanding of the text is guiding what your movements but when you're turning out you're just moving at this very rhythm a kind of stash.

[00:23:33]
I mean it's actually built similar my morning detectors across many other domains and sensors So for example we looked at watching some interact with the tutoring system text video that just gave some physios and Paschal expression then in an average we get about 20 percent about chance accuracy which is which is I would call it fair to moderate.

[00:23:59]
So Ok Well can we do something so now we you know had the basic research and experimental work and we generated some data and build a model that could be used a model intervention so you're kind of the general idea is as you're reading you know one page to the next we grab your eye gaze and run it through the the model I just showed you and if you're not mind wandering you know you just move on but if you are by monitoring then it actually ask you a question on that page and that question is used to decide where and if you answer correctly you can move on but if you don't you get an opportunity to kind of read and we answer 2nd question and the whole idea is very simple one you want to bring attention back when somebody is going out and 2nd you want to call correct any comprehension deficiency if you're missing something you want to address that immediately and the reason being you know a text or film or a story you know it perceives sequentially so if you're missing a piece you know in the you know 2nd page you actually will have a week an understanding of what's happening as you're going to lock in reality it's a much more complicated than that but this is $11.00 we've done many of these this is one of the interventions in this case they read that the text was divided into 15 chunks actions that were coherent and then the different let me track my wondering in real time and the aggregated section and if you decided that you know the probability that is running out is pretty high we actually asked them to construct a self explanation so you had actually constructing a response based on.

[00:25:37]
You know up front right and then moves kind of score that in real time using very simple natural I was practicing and based on how they responded allow them to you know move on or have them reconstruct it was right and that was kind of the intervention with a lot of you know about it c.r. do you get right to make this work effectively.

[00:25:56]
There's an experiment in testing out an intervention so we so you have the kind of experiments so we in one group we had an intervention so as I said you know the interventionist stuff explanations which by the way is the most effective thing you could probably do when you're reading so just having the south explanations independent of whether that tied to my 100 is expected to have learning benefits so in a control group we do with all your control where for example if I got some explanation I got the intervention some concepts you know 13 and 5 based on my my wondering I would be paired with a control participant would also get them on pages 13 and 5 but it was independent of them I'm wondering so it's a really really tight comparison and so they read the text and either condition and then they get to take an assessment of what they read you needed me after reading what's going to freshen their mind i'm a week later and this is really what you what you kept to regulate comprehensions I did really remembering it will be Claytor And then there's 2 types of questions like that that's irrelevant up to the current purposes.

[00:27:01]
And this is a measure on the fact that we're just going to see. Positive numbers mean higher for the intervention with the control though we saw no benefit immediately after reading. But we actually see good benefits of the intervention on both Western types a week later and in this case the fact is this is like a success of my point 35 you know which is kind of small to medium effects so this is this is kind of promising a lot of this is done in the lab with us like you know 40000 I a tracker and it's very unpractical though can we take it into the wild right so we said Ok let's do that then it's case we worked with.

[00:27:42]
These consumer are you know cheaper I trackers $100.00 we actually use I try this on longer available. There was like $99.00 and we actually took it into a biology classroom so this is a high school biology classroom we have a fishtank here and you know kids worked on it use computers and it actually actors you know and intelligent tutoring system that taught them you know biology content was going interact.