[00:00:06]
>> All right so thanks for the direction and thanks for the invite out it's been a fun morning so far and I look forward to meeting everybody in the afternoon and I even actually discovered that I went to elementary school with one of your faculty half way all the way across the country actually so.

[00:00:21]
It was kind of a cool discovery. All right so. You can get this work. So you give me one soak here. OK. Are good. Though as they would have been behind the podium the whole time thanks OK so. It was mentioned so we study. Quite a few different kind of directions in the lab so we do a lot of work on selective attention of perception decision making kind of the interface between those 2 things and then more recently we've been studying working memory and also how those relate to things like selective attention or selective information processing and we use a bunch of different tools we do a lot of work a lot of F. M.R.I. work and a lot of basics like of physics and then try and kind of link those with some simple models.

[00:01:37]
To kind of bridge the levels of analysis so today I'm going to focus on kind of one subset of this work that focuses on working memory using a summariser kind of major tool here and before I get going I'd actually like to emphasize that clarification questions are very welcome So there's a couple of spots during the talk where I might drop people and if I do that please stop me and just ask me to try and explain it in a different way or do we can explain it to you for me and hopefully that way we can try and not not drop too many people here as we're going through but that's definitely welcome so please just speak up.

[00:02:13]
OK So when we talk about working memory we're just talking about remembering visual information in our case it could be auditory but we focused on the visual domain but remembering information for a very short period of time so typically on the order of a couple of seconds and you want to be thinking about this is kind of a model system for how you find your car keys on the couch when you're in a hurry in the morning and trying to get out the door and so that is you know how do you hold information in mind like an image of something and use that to guide behavior and we do this using pretty simple tasks and so this is just kind of one canonical paradigm that's used in the field and in this type of experiment what we do is just ask people to remember a bunch of colored squares or it might be wine orientations or something like that we take that away there's a blank period.

[00:03:02]
Let's say a 2nd or 2 and then we put up a test or a after that and we just ask the subjects if those 2 displays are the same or if they're different and so in this case the answer would be different because this purple square here changed to a Red Square OK but the subject house again it's pretty straightforward is saying same different on these 2 arrays.

[00:03:23]
Now what you see is that behavior varies very systematically in these tasks so if I ask you to remember one colored square and I ask you if it's the same or different that's trivially easy right I ask you to colored squares also easy 3 colored squares pretty easy but then when you start getting above 3 colored squares performance in this task tanks pretty quickly and it turns out that this is actually a very stable trait for a given individual so I could you know run the simple memory task on you take you back a year later and your performance would be largely unchanged and furthermore what we see is that there's a lot of variability between people so some people you know really struggle to remember let's say half an item down here I'm not sure what's going on with those folks but other people can remember you know 4 items easily in if you were to expand this out even farther you see that some people go up to 56 items a case and some people can remember a lot of information with a lot of these and one of the reasons why this is so interesting and important and we think it's a good topic of study is that performance on those really simple tasks like I just gave you like how many colored squares Can you remember predicts things like general intelligence success at high school during scores pretty much any outcome measure you can think of in that domain.

[00:04:42]
Even more interestingly from my point of view is that it also predicts things like the efficiency of attentional control so how well can you select relevant information and ignore irrelevant information that's very strongly linked to how much you know colored squares silly as that sounds you can hold in memory at a given time and so we think that using these very simple tasks allows us to tap into something that's really a core cognitive function and that relates to a lot of your other capabilities in terms of kind of higher order cognition.

[00:05:14]
OK so today I want to walk through a set of studies that's actually taken place over the last 10 years or so and we're going to or I'm going to start by setting up kind of 2 hypotheses it either end of a spectrum here of hypotheses about how you go about remembering visual information for short periods of time.

[00:05:34]
And on the 1st kind of on one end of the spectrum here I want to talk about what's called the sensory recruitment hypothesis and this is just the idea that you use the way you remember things as just by using the same neural machinery that you use to see them in the 1st place OK so in the visual domain Let's say you process information in early visual cortex you would just kind of re instantiate those same patterns of activity that are present when you're seeing things and hold that in a kind of tonic fashion during a delay period or during the time you're trying to remember stuff and then rely on that to support your ability to find the thing you're looking for in the environment and so that's one kind of idea here and one of the interesting kind of corollaries of that idea is that the way you remember things is by using or co-opting these sensory mechanism so that means the neural code for a memory is the same or is very similar to the neural code associated with seeing it in the 1st place.

[00:06:35]
Now another idea. That's especially gained prominence I mean like all ideas and neuroscience and psychology like waxes and wanes over time but this is this ideas come back into fashion recently and this is basically the opposite of the sensor and it says OK so you're using your eyes to see right and you get some information you want to remember but then you're looking around say for your car keys and every time you make us a cot every time you move your eyes you get a new batch of visual information coming in it's going to flood primary visual cortex that activity and if you're trying to remember stuff using that same chunk of tissue it's going to get overwritten and so this idea of sensory recruitment is just kind of silly and shouldn't really be the way that the brain's engineer and instead what you should be doing is you know seeing things taking that information in recoating it into a non-sensory like format and storing that information via persistent activity elsewhere in the brain and where that elsewhere is you know that depends on who you ask parietal cortex prefrontal cortex and so on but for my purposes the key kind of notion of this model is that you're recoating the information out of a sensory like format and into a format that's not likely to be interfered with by new information or processing.

[00:07:54]
And so I just want to really emphasize that point that you know distinguishing between these 2 ideas is more than finding out whether you know memories live in view one or they live in wherever you are your other favorite brain areas it's really more about how the code or what the format of the code is that supports your ability to remember information.

[00:08:13]
Is. So temporal Yeah so that's a good question maybe I'll defer on that for the moment it but there is some evidence that your predictively re instantiating things that are really visual cortex to support memory chip. OK cool Any other questions before we kind of launch into things here at the moment OK.

[00:08:44]
OK so what I want to do is start by providing some evidence for this hypothesis and then all kind of provide some evidence for this hypothesis and then we'll kind of wrap back around and see where we end up at the very end. Right so anyway about 10 years ago or so we got started down this pathway of trying to figure out whether the center of this was viable and fortunately makes a very simple prediction right so if you're using the same cortical areas to see and to remember stuff then we should be able to look in early areas the visual cortex and we should be able to tell what you're remembering based on activity patterns in early visual cortex even after we take a stimulus away and so just as kind of a simple example here let's say I ask you to remember the orientation of a line or a greeting or something like that we know that view one is specialized presumably for processing that kind of information or for decomposing that information out of the environment so we would expect to see early areas like primary visual cortex that have a lot of orientation selective cells maintain that activity across the late periods during a memory and or.

[00:09:51]
So to test this idea. And variant of the task that I showed you the very beginning it's kind of more complex in some ways and simpler in other ways but let me walk through it and then we can ask questions if people have any at the end here so the beginning of each trial subject see a display that looks like this and so it's got a grading That's either an angle right around $45.00 degrees like this or an angle right around $135.00 degrees like that.

[00:10:19]
And it's some shade of red or some shade of green. And then we ask the subjects on a given trial to either remember the exact shade of red let's say or the exact orientation year and by that I mean you know is it 46.5 degrees or is it 46 degrees that kind of thing.

[00:10:37]
We take that away there's a 10 2nd delay period during which time there's no external stimulus coming in. So there's no more sensory drive and then we put up a test in us at the end and we either say is this the exact same colors of the exact same shade of red for example or is it the exact same angle that I showed you at the beginning and so they're basically just doing a 2 alternative forced choice test only orientation for color is relevant on a given trial and we just kind of cue them which feature to remember on a trial by trial basis.

[00:11:09]
And doing this we can also adjust the difficulty of the task so that it's matched across all the conditions so we can you know change how subtle the that we can manipulate how subtle the changes between the future that they're remembering and so forth to control behavior in that way as well.

[00:11:30]
So 10 years ago but it's either right before the start of a trial or right at the beginning of a block but they definitely know going into the trial what they're going to have to remember and I'd have to look back to see if we did a block by block or trial by trial.

[00:11:44]
But yeah that's a really key point things so any other question about paradigm Yeah. Yeah yeah so but before they see the sample stimulus they know if they're going to have to remember the color or the orientation and I'll say a few more words about why we did that in a few minutes when we get the data.

[00:12:03]
OK. All right and so right so you know given the simple task then what we're going to do is say OK so if we read out the information using F. M.R.I. in primary visual cortex can we tell whether they're remembering you know an angle around $45.00 degrees or a tingle around $135.00 or whether they're remembering a red or a green and so that's kind of the basic where we're going with the analysis at least.

[00:12:28]
OK so. That analysis itself here maybe some of you have gone through this but if not all kind of walk through the logic so in primary visual cortex right so we've got Columbia organization you've got orientation columns and some color selective stuff kind of in or mixed in there but these are very very small so the columns are submillimeter with an F. M.R.I. voxel So a single like volumetric pixel here we're sampling somewhere on the order of like a cubic millimeters oftentimes more like $27.00 cubic millimeters but much much bigger spatial much much coarser spatial resolution than the underlying kind of functional organization that we want to tap into right.

[00:13:10]
And so what happens though is that if you look and you make enough measurements and you really kind of begin these F.M.R.I. voxels will often exhibit subtle feature preferences presumably due to the composition or the distribution let's say underlying future selectivity at the neuronal level or at the corner level and so for example if you know all of these columns go into that voxel that Vauxhall may exhibit a slight orientation preference for something angled about here OK and if the single voxel level these effects are not selective It is not huge but it's reliable in alt kind of explain how we can extract that information in a minute here but that's the basic logic.

[00:13:50]
And then the next step given the week's like to video of these voxels is just to exploit the fact that we have a 1000 of them in primary visual cortex and so what that means is that we can show a stimulus like this let's say like a horizontal grading and we can learn the pattern of activation across all of the 1000 voxels in the one and we can you know say like OK that voxel you know response like the horizontal horizontal we can show another angle like this we get another pattern of activation and we just train support vector machines or any other kind of simple machine learning algorithm to recognize these patterns associated with chain.

[00:14:28]
Which is in the orientation of the stimulus. All right so questions about that are good so for OK. OK And then once we've learned those patterns of course we can then guess based on a novel pattern of activity what the subject was remembered and so here's the data from the 1st study or half the data here so this is how well we can classify the orientation of the stimulus so is it 45 or is it 135 degrees when they're told to remember the orientation or when they're told to remember the color.

[00:15:02]
And what we see is that we can significantly classify the orientation they're remembering when that future is relevant but not when the color is relevant. And we see the opposite pattern when we try and classify color we can only classify it when they're told to remember the color and not when they're told to remember the orientation and so what this means is that our ability to classify that information is not just due to a passive sensory response right because on every trial you've got a color in the north.

[00:15:33]
But it's really yoked to the feature that they're trying to remember during that 10 2nd delay period because that's the only thing we can pull information out about in visual cortex and it. OK So questions about that so far before. Same way so same general logic we can train classifiers to recognize patterns of activation associated with different colors and the one has actually you mean even in the classic textbook it's got a lot of color selectivity but it's probably even more prominent than we've realized in the past so.

[00:16:17]
So in this case the model was trained based on the whole like a whole one out procedure using the memory data itself and then the next thing I'll show you here is that we all. Just show you and go and so. Right so that brings up a good point I actually kind of glossed over this but these models were all trained by learning the patterns of activation during that memory delay period on some training data and applying it to help out test data during memory test.

[00:16:46]
But you know again this intrusion ideas that we should see the same patterns of activity during you know viewing a stimulus as we do during a memory delay period and so the next thing we did was we trained another classifier based on the presentation of one of these colored gratings you know and it could be red or green or 45135 etc They were just blasted at the subject for 10 2nd blocks and so we just present this over and over again then change it change it change it and so forth but anyway so far based on this you know Pearson's retest no memory demands at all and then we generalize that to the memory delay period and that also works so I'm just showing you orientation here it works for color as well but again this demonstrates that there is at least another similarity between the patterns of activation evoke during a pure sensory task and during this memory task that we can pull out the future that the subject's remembering.

[00:17:41]
So just kind of. Please summarize this one so you know what we've done is taking kind of the 1st step toward supporting the sun through an idea right and so you know showing that there is sustained activation patterns in early visual areas that resemble the patterns that you see during sensation or you know during actually seeing the stimulus itself.

[00:18:03]
And so this suggests that it's at least plausible right that this kind of a mechanism might be helping to support your ability to remember things for short periods of time. OK So anyway so that was the 1st step so the next thing you want to do is see if this is actually functionally significant right and so F. M.R.I. you know of course is not a causal tool at all but the next kind of thing that we can do logically is try and link the quality of the activation patterns or the amount of information let's say in early visual cortex with behavior during these memory tasks and so to do that we designed again even simpler tasks are just kind of one twist on it in this case subjects just have to remember an orientation so just a single grating and they are remembering it's told there's a delay period again no sensory stimulation here and then we give them a recall task at the very end here so by that I just mean we put up another grading that's randomly oriented with respect to the 1st grading here and we have them rotate it until it matches the remembered orientation of that stimulus.

[00:19:06]
And reason we do that is it gives us a much finer measure of the behavioral precision of the mnemonic precision of the subject right and so if they perfectly match this grading that means they remembered it you know with perfect perception but to the extent that they're off that gives an idea about how accurate they are across all the trials.

[00:19:26]
OK And then the next thing we did since we're now dealing with a kind of a continuous stimulus space here so any orientation we could ask and remember any orientation we also adapted the way that we're analyzing the activation patterns in early visual cortex to produce a continuous output or a kind of a continuous estimate.

[00:19:43]
Of what the stimulus is that they are holding in mind and so the way we do this is by modeling the response of each voxel in the one using a set of 9 the exact number doesn't really matter a lot but let's say 9 different orientation tuned filters here and these are just kind of meant to superficially mimic what you might see at the underlying population levels we've got these kind of circular Dowson functions that are ready to cross different points of the relevant space that we're trying to model here which is orientation and so basically what we do in the 1st step here is just estimate the selectivity or the responsivity of each box sold to each of these different orientations using a simple linear model kind of framework.

[00:20:28]
All right and so we go through and we do that for every voxel in the one in so you know Box One might have some weight profile voxel 2 of a different one and so on we just do that for ALL 1000 voxels that we've got in primary visual cortex.

[00:20:44]
And we do this also of course using just part of the training or part of the data so we've got a training set that you know we hold that we just use to train the model here are now in the 2nd part of this analysis what we do is we take a pattern of activation across all the voxels in the one so that's what we're looking at here we've learned how selective each voxel is for each orientation right in the 1st part and that allows us to then say OK we've got an observed activation pattern we know the selective of each voxel what must the stimulus have been that evoked this pattern of activity so we're basically like converting the linear model and going back into stimulus space and this is actually I mean you can think about this is a reconstruction of the stimulus of our estimate of what the stimulus was.

[00:21:32]
Now we do that on every single trial and of course you know sometimes they're remembering that angle sometimes the single and so forth so we then take all of these reconstructions and we re Center them based on the remembered orientation here so what it should be peaking at we're not taking a max rule here we're centering it on the actual orientation they should be remembering and then by convention the data will show you are all centered on the euro here where 0 is the organisation they were supposed to be remembered.

[00:22:02]
OK So questions about that this is actually one of those spots where if I drop people here the rest of it's going to unravel really quick so. That kind of makes sense I mean you could think about this is just kind of another representation of the stimulus that they're remembering and it's our estimate of that stimulus.

[00:22:18]
OK And then I guess the last thing I would say that's kind of important even mind is that if the one had perfect information about the remembered future this would approach like a stick function right and if there is no information at all this is just going to be a flat one there's going to be no relationship between what we're reconstructing and what the actual remembered stimulus ones.

[00:22:37]
Are. OK So just to remind you here we've also got a recall task right in this case we also have a continuous kind of graded measure of how precise each subject was and so here's the data from a subject here and basically what you can do then is say OK if the subject was perfect on every single trial then their responses their error in this behavioral recall task would be clustered right at 0 right and to the extent that they are not remembering and let's say they were doing it with their eyes closed this would be a uniform distribution right they would just be randomly guessing.

[00:23:11]
OK And so now what we've got is a kind of graded measure of behavior and our stimulus for our reconstruction of what we think they're remembering and then we can try and link those 2 things together. Right and so what we do that is we just take our orientation reconstruction so this is you know our estimate of what subjects are remembering based on activity I guess and actually in view one to 2 I suppose and then we can estimate the precision of this or kind of the fidelity of it just by Actually we can do it a bunch of different ways I think in this study we probably fit a circular data into it measure the bandwidth of it but basically we just want to know how flat this is or how that is right is our measure of the pursuit.

[00:23:58]
Or measure of how much information is in the one. And then we can also back here we can also do the same thing for this witness measure you know the standard deviation of this distribution is our measure of behavioral precision. And if we link these 2 things together what we see is that there's a strong correlation across subjects here so we've got me an absolute recall error here on the bottom which is big numbers are bad here that means you know you're more dispersion about 0 in your responses and then this is the dispersion of those reconstructions where again bigger numbers here means flatter functions or less information and primary visual cortex.

[00:24:38]
And this also works within subjects on a trial by trial basis we can go through and we can say like OK what's the precision of the reconstruction of the remembered thing how well do you actually do in the behavioral task and we've also link this in other domains like spatial working memory in a few other times where measures of behavioral precision have been linked to the amount of information we can extract from really visual cortex.

[00:25:03]
OK So in this 2nd study right so you know kind of replicates the 1st in that we're finding information early visual cortex about remembered features which again is consistent with the sensory and I bought this and then in the 2nd study here we also took you know kind of one additional step in terms of relating in a correlational way the activation patterns of the quality the activation patterns in the behavior of the subject so again you know we're not we don't really have the tools to properly do this and that kind of a call is a way but in terms of F. M.R.I. This is about as good of a link as we can get in terms of correlating those 2 things with each other.

[00:25:41]
OK so put into. Yeah that's. Quite phrased that way but. Yeah I'm not sure actually so intuitively what I suspect is going on in driving a lot of this is that and this is totally anecdotal we've got no data about this whatsoever is that some people report that they vividly imagine the orientation during the task and other people say yeah I said that was like $215.00 on a clock or something and we try and discourage the latter strategy explicitly bide randomly drawing the orientations we're having there are members of that's it's actually a very ineffective way to remember is this 46 degrees or 46 and a half degrees right that is an wind itself will do a verbal code but I do think there are probably a lot of individual differences in your ability to use mental imagery and that may be one of the driving factors I'm not sure.

[00:26:53]
It's something we're definitely interested in there's actually a population of people going you know the name of it maybe you can help me. But there's a clinical thing that's been defined where people report having no mental imagery whatsoever and actually really cool to kind of get in touch with folks like that but I don't know we'll see.

[00:27:25]
Yeah so so they're random with respect to each other so there's no consistent information from one to the other I suspect and I'm like 90 percent sure this is right after one of my students that when the grading lands close to the target you're probably a little bit better but that I mean what it has to do with better memory or reduced motor or things like that I'm not sure but in the experiment itself they're just randomly generated so.

[00:27:54]
OK Well any other questions for you go on here OK. OK So thus far you know I spent the 1st half hour or whatever building up the sensor hypothesis right and presented some data that it's like you know activation patterns and do you want to related to behavior in all this kind of stuff but there's something that's really wrong with those 2 tasks and that is we just had this big you know 1410 or 14 2nd blink delay period right now and you know if you imagine back to the example I gave you at the very beginning of looking for your car keys that's not at all what's happening right so you kind of hold the image in your head of your car keys or whatever and you start looking all over throwing the kids' stuff on the ground and you know you're getting all this visual input coming into your brain right and so.

[00:28:39]
Recently people have pushed back on that idea in a very logical well reasoned argument and they just said look it's really inefficient and probably bad engineering to use the same chunk of tissue to see into store information and memory because usually you're making sick odds you're getting that new input and it's going to just wipe out the memories that you've got stored OK And you know again that logically I think that sounds totally reasonable and.

[00:29:06]
There was a study done by Betancourt into a couple years back where they actually directly went in and tested this idea right and so they designed a study that. It's basically the same thing we've been talking about where they're asking people to remember and worry into grading they actually do it a little bit differently where they're present to gratings and then they post Q U and so in this case you get grading one grading 2 and they say OK Remember grading one here OK but essentially it boils down to pretty much the same thing you're just remembering a single item single grading and then there's an 11 2nd delay period here and then they ask you OK does that grading match the thing you were a member in or thing you were supposed to be remembering and so same kind of delayed much a simple task.

[00:29:51]
But then they introduced a new manipulation here and they either had a blank delay period like everything I've been talking about so far. Or they presented a bunch of faces or a bunch of dizzy bows during that delay period and I'm not sure what's up with the choice of stimuli here but you know that I think the point is they're just you know blasting away at the retina with new input to see if they can overwrite the memory representations in early visual cortex.

[00:30:19]
OK And then the other thing that they did in the study is they also looked outside of really visual cortex and they localized the region and superior in surprise because again I don't want to make a whole lot about where it is but I just want to mention it because it's outside of kind of the primary sensory cortices here.

[00:30:39]
OK And so the results of their study are pretty straightforward Right so in early visual cortex so they're looking I guess in this case of the one through before. Just like I showed you before when there was no distractor during that delay period they can reconstruct or not reconsider they can decode in this case what the subjects remember and so are there remembering you know that grating or this grating.

[00:31:01]
However when they present a distractor So either a face or a dizzy bow. Then that information appears to be wiped out completely in early visual cortex and so you know consistent with kind of what we all probably find very logical if you're seeing stuff and trying to remember stuff at the same time to tissue it interferes with each other on their account OK now.

[00:31:23]
If you look up in this appear upright Will area that they've localized you find that the difference between classification accuracy when there's no destructor present and when there is a distractor present. Goes away largely And really the important thing is that even when there is a distractor present you still can decode what the subjects remember and in this area of protocol or text they've identified and so based on that they put forward the argument that early visual cortex is not in fact important for remembering information and instead what you're likely doing is you know you see stuff that passes through a visual cortex then it's recoated out of the code transitions let me say it that way from a sensory format into a non-sensory format and it happens that you know they're finding this evidence for this code up in spirit if yes but what is this transition of the format of the code into a code that's more resistant to distract or input and so you know even though you get a new stuff coming in that codes not overwritten and you can still find that pneumonic information OK All right so I'll stop again your real quick for questions if they're.

[00:32:29]
Really good OK cool. OK So you know we saw that come out and you know seem totally reasonable we wanted to kind of get in and play around with it some so we designed a task again same basic thing I've been showing you the entire time I've just which the time axis to be annoying here is you know Time runs down that way.

[00:32:49]
And on each trial those subjects remember grading there's a very short delay like half a 2nd here and then we flicker either for a filter noise which is meant at the same spatial frequencies as the grating here or we figured what better to interfere with the grating than another or integrating and so on a 3rd of the trials will you know have them remember this and they will just put another grading up and we'll flicker it during the delay period for 11 seconds.

[00:33:19]
Minutes make a one quick note about this the orientation of the stimulus and the orientation of the structure independent across the course of the experiment that's important for a reason I'll tell you in just a 2nd and then actually on other trials we just emitted in the destructor here and so it's basically just like the 2nd experiment I told you about.

[00:33:38]
OK And then they do a recall task at the end and that's it. All right so 1st thing is to actually verify that you know the introduction of those distracters is causing a big since revoke responsive it will cortex and to no one's surprise it's causing a huge response and so we just look at kind of the average F.M.R.I. signal in early primary visual cortex in this case when there's no distractor you see there's an initial uptick when you get the remember the memory item right and that rapidly drops off during the delay period it's pretty much flat.

[00:34:10]
When you present the grading distractor during the delay period you see this nice sustain profile and then the flickering filter noise really drives a big response out of you one with the bold signal but anyway the point of all this is just to say you know compared to the notice tried to condition your were driving the one super hard in this case we're getting a huge response out of a huge sense revoke response.

[00:34:35]
My that though the T.R.'s are $800.00 milliseconds so this is probably going out to say 16 seconds but the delay period right here is about 13 seconds so that's going to be you know from about here to about here probably in there's a temporal lag of about 4 to 6 seconds to go to kind of shift everything but you know this is well into the delay period out here and actually that brings up a good point too that all of our knowledge is all show you're focused on the point of these times here is out here.

[00:35:06]
OK Any other questions about Oracle. OK So when we do this task You know we know we're driving the one super hard when we actually go in and we try and reconstruct now the remembered orientation using the 2nd kind of method I talked about there is we need a continuous output.

[00:35:26]
What we see is that there's absolutely no difference between the 3 conditions so if you line them up they fall right on top of each other and importantly going back to what I was talking about about the fidelity of these reconstructions or you know kind of the bandwidth of them there's no change in the bandwidth of the reconstructions either and so what we're seeing is that even when we present this noise distractor that again drives this huge response in view one the memory signal is not corrupted by that OK And so we're seeing you know in our hands at least that we're able to reconstruct the remembered features again even though we're driving the system with another grating in this case with another light feature.

[00:36:05]
OK And then that's true also in area before and pretty much every area of the visual system that we can identify. OK so what this suggests at least in our hands right is that there's not you know having new sensory input come into the eyes doesn't automatically wipe out pneumonic representations in view one.

[00:36:23]
And so the next question we wanted to ask in this goes back to that little side note I was talking about the 2 destructor The remembered in the distractor orientation being statistically independent from each other that provides us the opportunity to actually ask whether we can simultaneously reconstruct both features can we reconstruct what you're remembering and reconstruct what you're seeing during that delay period and it turns out that that also works so using the exact same chunk of data during the exact same temporal epic of that delay period we can also reconstruct what the subject is seeing in the form of that grating distractor.

[00:36:59]
That also works fine in view one before again pretty much throughout the visual system. And so what the suggest is that right I guess kind of 2 key points is that there isn't an obligatory overwriting of information of pneumonic information and the one when you see new stuff and 2nd that you can simultaneously represent pneumonic information and sensory information in the same chunk of tissue.

[00:37:25]
Part So actually let me pause there for one second and see if there's any clarification questions. There before. We're taking the average pattern from T R's likes you know well into the later and I have to ask Rosanna but you know 713 seconds or something. You can also use on time point by time point basis.

[00:38:03]
Right. Or do you see the memory. Well. All. Things. Are. I don't know about remembering more in a temporal sequence but you certainly can like this present continuous sequence of greetings You know one every couple of seconds an M.R.I. cause we've got to go kind of slow and just tell them that every new thing you see is now the memory item or every 3rd thing you see or if anything like that and then have them kind of search dynamically for something that matches that and we've done that a little bit also and that shows the same kind of signatures here.

[00:39:00]
Which is it actually kind of nice because it's also further evidence that seeing an intervening item doesn't automatically white stuff. But I'll say a few more words about the 2nd. OK So anyway we've got these 2 results now right so you know this other group found that when you present you know faces Evos or whatever during the delay period information in early visual cortex drops out right and then what we're showing is that if you just present you know gratings or filtered noises to different stimulus classes things are just fine so there's no obligatory overwriting and so that kind of left us in the spot where we're wondering well what's going on you know how come we're getting these discrepant results.

[00:39:41]
And even though our logic going into this was that nothing should interfere with the grading more than another grading it could be actually that gratings in filter noise are really easy just to ignore OK so we know they're driving a big response because you know we can look and see but it could be that people just aren't engaged in that whereas if I flashed a bunch of faces or maybe people are just super and gauging that interested in them and they kind of forget what they're supposed to be remembering right.

[00:40:08]
And you know if you think about this I think at least all of us got to put this out there but you know the idea that you can interrupt a pneumonic representation in the brain should not be surprising to anybody you know you can be sitting there thinking I could.

[00:40:21]
No I won't do it but you can make a super loud noise or is going to startle and forget what you were thinking right so we know we can interfere with stuff the question is you know what's the key ingredient that makes something a good distractor the white stuff out or that makes it you know something you can just ignore and simultaneously represent with pneumonic codes and.

[00:40:40]
So what we did then was he figured OK we're going to use the same design we've actually done faces and busybodies to and we get pretty much analogous results here but I'll just show you one of the 2 experiments here where we did the exact same things you have gratings or no distractor and we either had the subjects ignore the distractor or we had them tell us if there was a slight tilt in the orientation so couple of times per trial the grading would just jump a degree or something and they had to press a button.

[00:41:11]
Or there would be a little contrast change in a little blog somewhere in the greeting here and they had to tell us when that happened OK So they're either basically attending the contrast of the grading they're attending the orientation of the greeting or they're ignoring the distractor greeting or.

[00:41:26]
And what we see in this case I kind of changed the format of presentation because we're still apologize for this you'd never do this in a talk but we're still analyzing the data so instead of showing you those reconstruction curves what I'm doing is distilling the reconstruction curves into a Fidelity metric and we're basically kind of projecting it down on to the angle that should be remembered so anyway big numbers here mean that the reconstructions more narrow like this in small numbers mean that it's flat essentially.

[00:41:56]
So if we look I say maybe just maybe you want to start off with here when they're ignoring the distractor we see the same kind of thing that I just showed you right so our ability to reconstruct the remembered orientation is pretty fine OK. But when they actively pay attention to that distractor let's say they attend the contrast of it and we get a big drop here in the Fidelity those representations and when they're attending to the orientation of the intervening distractor things pretty much the pneumonic signal pretty much disappears and that's true across pretty much all areas so there's going to be 4 but resent is looked out through I.P.S. and this is also true throughout the visual system basically.

[00:42:35]
Now simultaneous with this so this is now reconstruction fidelity for the remembered item you can also then go in and reconstruct the since died over that distractor orientation during the delay period and not surprisingly what you see there is that if you're ignoring the distractor our ability to reconstruct it is OK like I showed you before but not great if you're paying attention to the contrast and you can reconstruct it even better and if the subject's paying attention to the orientation of the distractor you can reconstruct it with pretty high precision and so basically what's happening here is there's just a trade off so when you are paying attention to this tractor our ability to reconstruct the memory drops out and our ability to reconstruct the destructor goes up and now one of the key things that happens here also is that behavior gets worse.

[00:43:25]
When you're paying attention to the destructor so your ability to accurately report what that remembered orientation was goes down when from ation goes down in this condition when information in early visual cortex drops out about that remember it and so you know again this is just showing kind of this push pull relationship and you know I would actually say that this doesn't so much support the kind of 1st experience but it's just kind of a logical necessity right because you know again going back to what I was just saying I know that I can interfere with your memory somehow right and this is kind of a hammer approach way to do it but what this is showing is that when pneumonic information drops out of early visual cortex behavior suffers as a result.

[00:44:10]
OK So here's the light does the total speculation point about all these data and how things might work and this is actually where I'd love to get feedback from people so in addition to clarifying questions the floor is totally open now to telling me how you know stupid this is or maybe it's on the right track or anything would be useful at this point.

[00:44:31]
But we've been trying to figure out you know how these pneumonic signals and sensory signals can co-exist and kind of compete with each other in a way that makes sense with the data that I've presented so far. So one of the ideas we've been playing around with is based on some women are recording data from Peter roll some of the lab in Amsterdam and basically what they do is read so doing laminar recording that working memory task is a little bit different but the spirit of it's largely the same the animal's trying to remember kind of the trajectory of a line it's a spatial working memory task just for simplicity but the important thing is that when you present the stimulus to the animal now or when they present the stimulus to the animal you get a lot of multi-unit activity in later years later for in primary visual cortex right but during the working memory delay period which is shown here.

[00:45:25]
You see a sustained multi-unit activity in the feedback layer is so superficial and the players here and there's not much happening in these early or in these input layers are. And so if you take that data and kind of do a massive kind of extrapolation back to what I've been talking about.

[00:45:45]
So if I ask you to hold something in mind or I ask you you know to find your keys or do any of these tasks that commonly engage working memory right do a math problem. That's a purely top down feedback so it's got to be coming in because there is no sensory signal especially by to say OK you know imagine your keys imagine what your car looks like no sensory input coming in at all so no driving input here any activation we see has got to by definition be due to reenter feedback in the early visual cortex and so if that's true then we might be able to simultaneously reconstruct the contents of memory and the contents of what you're looking at because of this layer division of input and feedback activity here and so presumably you know that sensory distractor especially when you're ignoring it right is going to be driving massive responses in the input layers but maybe that can simultaneously co-exist with these reentrant in future selective signals into other cortical kind of targets here.

[00:46:43]
Now the interesting thing about this though is that if you ask subjects to ignore the distractor right that's just passive kind of sensory input coming into the input layers here but if I ask you to simultaneously Now try to remember something and pay attention to something else both of those 2 things require simultaneous top down input into early visual cortex and they might be competing more in these feedback layers and the signals might be mutually interfering with each other and wiping each other out.

[00:47:15]
OK So that's kind of you know one potential idea and like I said we have absolutely 0 data ourselves about this but it's just something we've been kind of 20 around with. OK So just kind of wrap up and to make sure I kind of well I'll try and like explain a little bit about where we stand on this but.

[00:47:37]
You know I started off with this like since recruitment idea and the memories live in view one or however you want to say it and then you know walked over or walked through some very logical pushback on that which is you know that's silly doesn't make sense you shouldn't be using the same junk tissue to do 2 things at the same time.

[00:47:56]
But I think you know the data I think are pretty clear on this and then logically it might also be apparent that things can co-exist right you could have kind of passive sensory inputs passing through an area and those could co-exist with feature selective top down modulations to that same region.

[00:48:14]
And I also really want to emphasize that you know we're not I'm not certainly you know saying that memory is in view one and that's right I mean I think that would be ridiculous by asking like I give you my phone number you're not going to remember that in the one in so you know the types of codes that we're looking for when we talk about memory types of signals whether it's to stay in spiking or silent codes and things like that.

[00:48:37]
Are going to move around dramatically depending on the format of the information you're trying to remember what you're trying to do with that information whether it's a really precise task or a course kind of code and so forth but I think there's a lot of really cool room here to explore when and why information is transformed from the sensory format into non-sensory formats how competing Top-Down demands might mutually interfere with each other so it might be a good model system to kind of understand why some tasks interfere with each other and some don't.

[00:49:08]
And then also you know of course we want to seek out a better kind of mechanistic understanding about how coexistence of memory and sensory signals might actually play out in early visual cortex so. Again you know any ideas are totally welcome on this front and I would love to hear we have to say and then.

[00:49:26]
If I wrap up I just want to acknowledge the people who work in all these projects especially Rosanna here she's a post-doc at U.C.S.D. she did all the 2nd half the talk basically and a whole lot more. Really fantastic. That's all so thanks thank you. Now. So people have tried that.

[00:49:59]
And I would say the results are a mixed bag. It's also harder than it sounds I mean it's fairly easy to get the full view of confluence with the most I think actually maybe you can speak to this even better than I can but to get stimulation in the peripheral representations at all is much more challenging and you know again the results I have a sneaking suspicion there's a big file drawer on this and there might be a lot of no results out there but the results have been published I would say are largely in favor of the idea that that interferes with memory but not entirely so I don't know open question but yeah great idea I wish it was just like you could just you know zap somebody and go away but it doesn't seem to work so.

[00:50:41]
Yeah. Yeah. Yeah so I was glossing Yes I'm glad you brought that up so the memory yeah OK. Right so it's going to go a little bit into the weeds this are. So all the models that I trained here that actually Rizana turn that into anything. Were done that I showed here were done using a sensory task basically So there's looking at a grating you know it's just blasting away for $510.00 seconds at a time so if you train the model that way what you see is that that transfer is a model train that way will generalize very nicely to remember to pneumonic patterns in early visual areas not so great I.P.S. if you take the memory task and train the reconstruction algorithm on that then that generalize is beautifully an I.P.S. And so what that means is that there is no want of information it's basically like out it was arguing there and yeah totally on board about it so you know if you train it on the memory task I.P.S. works beautifully which means that it's got pneumonic information but it's not in a sense reform.

[00:51:56]
If that made sense I don't know sorry it's kind of hard thing to explain but. For. This. One. You're. Right. Yeah OK. So this is a great great question so what I would argue. Is that probably what's happened so if you know in advance that you're going to have to make a really fine discrimination on something then I suspect the one stays online or involved in that the entire time and if we can knock out the you want to activity by you know in effect being making a loud noise doing something it interrupts your memory I mean if we can get the one to drop off line your performance suffers OK so you can still do the task B. can't do it as well.

[00:53:09]
And so what I suspect happening here and this is actually exactly the direction we're going is that if you take the information in it is being recoated we find evidence for that too I didn't really show it here. But if I take the high precision representation offline you're going to suffer and if you just have to rely on that recoded information performance is going to be worse and so I don't think necessarily that there is you know it's a strictly serial process like you know you get high precision sensory coding you recoat it and then if you are asked to do a high precision task you code it right I don't think that's what's going on because behavior suffers and people don't seem to be able to undo that I think what makes a lot more sense is that there's information compression happening and you're getting a coarser and coarser representation as you compress that code more and more do more and more recoding.

[00:54:00]
And if I force you to rely on that compressed code and then you're not as good at the task so. Most of the very 1st one I showed was match non match the rest a recall. In our hands it yeah it is experiment was a match not match.

[00:54:32]
That's the idea right and so you know like if so this is actually a really interesting thing to do because most monkey neurophysiology is like remember one of 2 directions of motion remember one it may be for directions in motion but if I ask you to do that you wouldn't use a visual code at all you just say it's up it's down it's left it's right and so that kind of memory might be qualitatively very very different than the kind of thing we're doing here because we're really trying to force people to remember the visual details and not rely on a verbal recoating scheme or some kind of digital code in this case where our cool Well thanks very much for your.