Without further ado Dr you were
excited to hear about bio.

Well thanks for stealing my thunder I
guess my I'm just going to give a talk now

that everything you say so
before I get started Thanks for

having me really happy to be here for
a variety of reasons one of which is

a thermal reason it's much warmer here so
thanks for inviting me now

I think this was actually scheduled that
way so I could enjoy the weather here.

Before I get started on what I'm
actually going to talk about let me give

you a little bit of background on what we
do in the lab which I think is going to

sort of shape the Neural Engineering
going to be talking about later

what we do falls under the rubric neural
coding I'm going to be talking about touch

all use that as an example the ideas when
you grasp an object use a spatial temporal

pattern of defamation produce
in the skin which I'm sort of

represented with a skit squiggly line and
that creates and

that pattern of defamation depends on what
you touch and you and how you touch it and

it produces a pattern of activation
in the nerve which is going to

then culminate in a pattern of activation
in the central nervous system a different

stages we you know in the Kenyan
nucleus than in the foulness and then

at various stages of processing in cortex
ultimately culminating in a perception

of the object and when we hold this object
in our hand we have information about its

size its shape its texture if it's
moving across the skin we have

information about that too and somehow
all that information is included in these

patterns of neural activation and what we
spend most of our time doing is figuring

out what aspect of this pattern of
activation in the nerve and in the brain

conveys information about texture what I
expect conveys information about motion

want to get aspect in basic information
about shape and week we create these

mathematical models where we can then if
you give me a you know two patterns of act

of nerve activation I can tell you
all lot about what causes them I can

tell you which of these two had
the rougher texture I can tell you what

the shape of the these the grasp
object was whether it was moving and

if it was moving in what
direction at what speed so

we try to come up with these and
these and these.

Models that link not just
the stimulus to the the response but

also to the person that
is ultimately elicited.

That's what we're going to talk about
today but it sort of frames what I'm going

to talk about today instead I'm going to
talk about how we can leverage what we've

learned about how the intact nervous
system infos information like how your and

nervous systems in code information
about objects grasped in the hand

to restore that information to
patients who have lost it and

we focus on two populations of
patients one in our amputees so

they've lost their sense of touch because
they've lost their arm right and I'm sure

most of you have seen stuff like this
around where one strategy is to replace

the function of the the last arm with
a bio on a canned that they can control.

You know volitionally usually through
the signals from the residual muscles.

But as I will show you in a few minutes
you can that these hands are not going to

be clinically viable so they're useful
things they're pretty cool lookin and

they do cool looking stuff but
they're not going to be useful until you

restore the sense of touch because really
there is any time you use your hand not

only are you moving your hands but you're
getting this barrage of signals back from

your hands that it's critical to using
them and so one strategy to restoring or

the strategy really to restore
touch in these patients is to.

Interface with is with an electrical
interface with the peripheral nerve so

the nerve that used to intervene
the hand is still there right so

we can do to create an sensory

perception is to to stick some some
electrodes in that nerve chronically and

there's a variety of different
technologies that I won't get into.

And then we you know it's been shown many
times now that if you lexically stimulate

this nerve you can create sensations that
are that are projected to the hand and

if you're an electrically stimulate
this nerve in an intelligent sort of

informed ways informed by what we know
about how this nerve includes information

you can actually convey
a very specific and very.

Sure sensations and that too is not what
I mean talk about today although I'm well

I'm happy to take
questions about it later.

If that will be focusing on is
another population of subjects.

To patients so these are people usually
with a spinal cord up or spinal cord

injury who are paralyzed and insensate
from the neck down after these patients

you can interface with a nerve because
it's no longer attached to the brain so

you can stimulate interview was going to
have some going to evoke and he can't

any sensation and so with these patients
because the nerve is no longer connected

to the brain you have to interface
directly with the central nervous system

there's a variety of different places you
could stick your electrodes the union

nucleus will look into that but it's
very difficult because it's in the brain

stem in near a lot of vital functions
found this which is hard to to access and

the hasn't really been taken seriously yet
and the most accessible place

is the primary some out sensory cortex
which sits right there on the surface of

the brain and so that's what I will be
talking about so the idea is to restore.

These these patients to or
equipped with by on a cancer they control

without alone and
I'll show you that in a minute but

the idea is to convey that
sensory feedback by electrically

stimulating the brain directly but before
I go into all that I need to kind of

try to because most of you haven't
really thought about this I assume.

Trying to explain to you why you would
go through all this trouble to restore

the sense of touch so this is a movie I
made a mug of me doing a bunch of stuff

with my hands most of it Monday stuff that
you do want to daily basis like picking up

things writing with utensil typing on a
keyboard and these are things that I could

not do without the sense of touch because
again any time you interact with objects

you get the continuous signals coming
back from yours your hand that inform

us of the consequences of these movements
and without the signals you would stop

using your hands let me get to work
took my word for him to prove it to you.

The movie was made by a colleague of
mine at Omeo university your role in

your hands and of a lady lighting a match
moments later saying Ladies lighting and

the doing the same thing the only
difference is her fingertips have been

done with a little bit a light a cane and
if you look at this kind of excruciating

because this simple task is think she
could do effortlessly moments earlier

she can now she struggles to do like that
she she drops a match she can see this

match is neurologically intact she's sober
she can see the match she just realizes

she's not going to be able to pick it up
without this fact checked out signals and

eventually she gets she picks
up the match and lights it and

she's never going to get better this half
an hour later she's still struggling to

match is right and again sort of
illustrating and again she is just

her motor system is intact most are as
a matter sensory system is intact

the only thing is her fingertips as
a nun and she struggles to do this task.

Now this next guy is.

Doing something even easier than lighting
a match he's just trying to pick up this

coffee cup and
bring it to his mouth now his he has

suffered from a rare peripheral neuropathy
that has left his motor system intact but

destroyed all so
medicines you signals from his hand.

All touch opera preception gone and

now he struggles to do this even more this
even simpler task of grabbing this cup and

bringing it to his mouth even though his
motor system is intact and if I showed you

this guy and you didn't know anything
about him and I say What do you think

is wrong with them you would think he
has a major motor impairment right and

that illustrates the C.N.N. Yang between
smell sensory system and motor systems and

the motor system basically cannot work
with other some medicines or system.

So that's true of your bodies your entire
body if I take away from yours years

from our sensory system you are going to
struggle to do anything with your body

now imagine now in this case so
this is Jan has a touch of pretty patient

who has been equipped with an array
with a with a pretty cool.

Robotic arm.

And she controls this arm via a revel
actually the place in the primary

motor cortex so this is the part of the
brain that sensing also the muscles and

my colleagues appeared in there one
of many groups that have done this

can infer intended movements from the
patterns of neural activation in this part

of the brain the idea is any time you
move your arm in a specific way that is

predicated upon a specific pattern of
activation in this part of the brain and

they have models and they're relatively
simple models that sort of do a mapping

between patterns of activation in motor
cortex and desired movements and then they

implement these desired movements in these
arms if all this was word salad the take

home message is she just thinks about
moving or are member there goes the robot

and she can basically control this robot
by thought so if you see this movie for

the first time you're like whoa that's
pretty cool what a great amazing

scientific and technological achievement
this person can control this relatively

complex robot just by thinking right
if you see it for the hundredth or

thousandth time you're like I'll be
way better at this and she has proof

we will but she's about as good as you are
you would be if you didn't have a small.

Sensory system if you see the last guy
showed you if you were doing this task he

too would struggle he too would be super
slow and so the idea is if you want to

create a bio on a hand that is in
doubt with the dexterity that we

are endowed with with our hands you would
need you need not only to restore the same

as the motor system but you also need to
restore the sim our sensory system and

what I focus on on in this talk is
are a first resort the sense of touch.

I'm sure I'm sure this will because I
think it's cool I in no way am I endorse

in this remake of Robocop But
if you want to play guitar

with prosthetic hands you can need
a sense of touch and we're there for you.

So this is just a kind of a little
animation to kind of give you a sense of

how this works so the idea is anytime you
touch something your skin to forms that it

creates a pattern of activation in
receptor as in your skin that depends

on what you touch when you touch it these
receptors same signals to your brain via

the nerves and comment in a pattern
of activation in the Samantha sensory

cortex so this is the part of the brain
that receives input from the head and

that pattern of activation depends
on what you touch when you touch and

how you touch it these
prosthetic hands have sensors

in them that replace the native
receptors in it in the hand and our

goal is to preach is to create algorithms
to convert the output of these sensors

in real time into patterns of stimulation
of the brain to recreate to the extent

that we can these natural patterns of
stimulation which I told you I started

Small Talk about telling you that's what
we spend most of our time doing is how

does this part of the brain respond
during interactions with objects so

this whole thing started as a DARPA
program and I mention this because well

DARPA has is very goal oriented there are
two years you have two years to do five

years' worth of work and at these two at
the end of these two years what we need is

an algorithm that converts the up of the
sensors into patterns of actual simulation

of the brain right it's not like hey
here's some money just do some cool stuff

with it let us know what you discover and
so because of that you sort of have to.

Find a problem that is tractable and so
you know rather than trying to restore

touch in all its glory which is you know
we don't completely understand really

the central basis of touch in the first
place and second by even if we did

we could not resist figure out
how to store it in two years.

We decided to focus this in the house
with a focus this project on the most bit

the necessary information to grasp
an object right we wanted to

endow the patient with the possibility
with the ability to grasp and

pick up an object so we need to what
kind of sensory information you need for

that where you need to know that
you're touching the object right

you need to know where you're touching the
object minimally you need your thumb and

one of the fingers to be touching the
object if you if you hope to pick it up

and you need to know how much pressure
exerted on the object you want to

exert enough pressure so as to pick it up
but not so much pressure as to crush it or

just even strain on necessarily so
we wanted to focus on these

two dimensions of touch contact location
where you touching the object and

contact pressure how much pressure
exerting on it and this we're super good

at the these things we know obviously when
we were touching something and we know we

are just enough pressure on things to
pick them up and no more right and

that's thanks to our sense of touch so
we wanted to convey these

through this brain interface this is
the arrow is to remind me that I am.

Kind of a big believer and I will defend
this today if you challenge me and

I'm happy to be wrong but
I doubt I am on this particular point.

That what we try to do is we try to
reproduce natural patterns of activation

in the brain and you know if you have
never doubt about sensory feedback and

brain machine interfaces
you're like the IS Or

another way right the other way is to.

Just say you know what the brain it's
like a plastic thing it will learn

just give it a new pattern of activation
it will figure it out in time.

And there's some experiments on single
anyone out in particular that show that

yeah you can learn new sensory spaces in
you know for a few weeks or a few months.

But there are various and

simple sensory spaces like two dimensions
tops no one has been able to do anything

beyond two dimensions the issue with that
with this the brain is plastic hypothesis

is that the hand is super complex you
know the hand in the arm together have

many many degrees of freedom they can move
in a bunch of different directions and

the space of all the possible contact
events is basically for all intents and

purposes infinite right so imagine
either you or a touch of patients and

I'm stickin an electrode in your brain to
restore your sense of touch and now and

I've done this kind of plasticity based
approach and now basically this means that

any time you move your wiggle your
prosthetic thumb you're going to get

this one tingling sensation and you will
you know touch something with your index

finger you get another tingling sensation
and you have to keep track of all these

completely nonsensical to
tingling sensations and hope that

eventually though make sense it's possible
that they will all make sense I think it's

very unlikely that in adulthood without
the benefit of sort of these evolutionary

circuits that are that have been sculpted
to hundreds of millions of years and

without the benefit of this massive
plasticity that we have in.

In childhood that you can learn these
kinds of of of representations and

so I don't think it's possible but even if
it were possible it's you know it would be

way better of the sensory prostheses
were plug in play you do this thing and

immediately all the sensory feedback
is completely interpretable right now

it's obviously not going to completely
natural at the time soon not as long as

we're still using electrical interfaces
but to the extent that we can sort of

approximate these natural we can tap
into these natural representation and

we can therefore approximate
these natural sensations.

These these patients will have less to
learn and therefore will be able to use

these faster and more effectively so
that's that's my position you know and

I put it out as a challenge to
anyone who to prove me wrong and

we wrote a paper by the way if you're
interested in because some of you

might have read some of the literature on
what happens after amputation you're like

whoa staggering plasticity not at
all we wrote a paper about it was

published in trends and cognitive science
in two thousand seventeen check it out.

OK.

Now that I'm going to take
it down from my soapbox and

get back to the science bit so
I told you that the decoding happens

by interfacing with the motor parts of the
brain where there's a part of the some our

sensory cortex is the part of the brain
that receives input from the body

the sensor some medicines we input from
body and as everyone in this room knows

within this part of the brain there is
a whole map of the body figure for maps of

the body because our four cortical modules
one in area three a one in area three B.

one in area when I went to area two and
within these these body maps so

that means that depending on what part
of the your body you're moving or

your toe or you're touching something with
a different part of this map is going

to become activated within this map
you have a hand representation so

anytime you touch anything with your
hand this is the part of the brain that

lights up and even within this hand
representation of this beautiful

digit representation with do you one is.

Interior and lateral to D.

two which is always interior
in the lateral D three etc.

And this is the part of
the brain we're going to target.

We use basically right now we're still
using pretty standard technology for

this so we use the electrode array which
is basically the industry standard

I'm sure most of you have seen
it's one hundred electrodes

arrayed over a four by
four millimeter area and

this is a nice electrode array because
it has really dense electrodes and

also because we can you use it in
humans as I will show you later.

But the problem with this array
is that really the electrodes can

only be one point five
millimeters long and

some of these structures are we're
targeting a deeper than that so for

those we use floating micro
lecturers that are a sparser but

can have arbitrary long electrodes and
we use these to target area three B..

So we stick these electrodes
in brain we let go and

by the way this is work the first time
we're told by work we did in monkeys.

And then I'll tell you how we we can apply
these the same ideas to to humans so

so we we we implant these electrodes
in the monkeys we let them recover for

a week or so and
when that when they're recovered we can.

Map out these electrodes right so
basically the first way we do it

is the easy way which is you listen to
the opposite of a given electrode and

then you pop at the body and figure out
where you need to touch the monkey to two.

Of oak activity activity in this electrode
by the way for those who are curious and

there's going to mass sensory nerves this
is the nerds this is the central focus

this is the parietal
Saugus the way we know and

we have been able to consistently put
our lectures in the right place is that

this is the face representation with the
I.P.S. and cross back towards the towards

the central focus and adds This is
the face and therefore the hand is right.

Here your and medial to that.

On the central focus OK so
we cover electrodes in the brain and

now well see Daisy then we can map how
would these electrodes respond to sort.

These guys here for

instance this is this you these guys
respond to the Palm these guys response of

the index finger these guys respond to
the index I'm sorry these guys respond to

the middle finger these guys respond to
the index finger I'm just kind of showing

you in sort of very hand way the way this
map of the hand that I'm telling you

exists in premise Most reports I saw show
to you in a more objective way later.

So that's our interface this is how we're
going this is the part of the brain we're

going interface with and
this is this is the actual interface

now if we were doing these experiments
in humans things would have

if we had done these experiments in humans
start with this would have been easier in

some ways working with human touch
a bridge a patient is very difficult I'm

reminded by my pick colleagues to
tell people want to get the stock but

the nice thing about humans is
that they have a language and

perception is this internal thing right so
in a human you could ask very you know you

can stimulate the brain say hey we're what
that feel like where did you feel it or

tell him this is this the task that I want
you to do with the electoral stimulate and

boom they would do it and when you get
a high impact paper with monkeys though.

Well well first of all they won't let you
do that humans in America until you prove

that it's something useful and
then then they'll let you do it so so

we had to do We had to to do for our first
series of experiments in monkeys and

the situation there is we have to infer
something about what the the monkeys feel

when we electrically stimulate the brain.

So this is a strategy we took So

this is a monkey he doesn't this is
not a Mohawk this is an electrode So

the leads coming from it from his lectures
and the strategy we took is we train we

train these animals to do discrimination
tests where we poke their hands with this

robot we design that allows us to put the
hand anywhere with Micron precision and

then we haven't had the animals do some
discrimination tests based on these posts

which I'll describe in more detail
later once the animals were trained

we started replacing some of these
pokes with electrical stimulation.

Targeting certain structures in cortex and
to the extent that the animal behave as if

it had been poked a specific way we could
infer something about what the and animal

felt right that's our general strategy and
again I'll go go go into more details.

For those into C.M.'s or
our electrical pulse trains are three

hundred hertz by phase of pulsed symmetric
by phase of pulse trains shown here

are mechanical stimuli and everything I'll
show are just simple mechanical see me and

we've done some other stuff since
I'm happy to talk about it but

the basic stuff is just like
these indentations into the skin

OK so I told you we're going to have
a contact location a contact pressure

and we were going to talk about the medic
approach so the question the first

question is how is that the brain
tracks where we touch things right.

So I showed you so this is the neuron So
this is a more objective way to sort

of map out the representation in most of
cortex so this is the neural activity

evoked over this U.E.A. when we put
the second Palmer world this is so it you

can see it's sort of hot so blue means
not active red means super active and

yellow means something in between so
the activity peaks here when you poke here

with a mechanical stimulus when you
poke here with mechanical stimulus

you get electrical system you get
a neural activation that peaks here and

when you poke the pinky you get neural
activation that peaks there right so

again it's I'm saying this now for the
third time you have a map of the hand in

the brain and so the the hypothesis
that we want to test this biomimetic

approach right was if we electrically
stimulate neurons that respond to

the index finger will we evoke a sensation
that is localized to the index finger and

really not just to the index
finger which is the if we

elect recently index finger to neurons
can we get the esteem is that is entire

or a sensation is entirely
localized to the index finger tip.

So again this would have been easy in
humans because we were stimulated and

they were and they would have reported
the location location of the sensory

experience but
instead we were working with monkeys so

this is how we inferred where they
felt things we try to train them to

do a task where we might put them on one
location the finger in this case the ring

finger of the left hand and then poke
them on another location of the hand that

was really disparate displace in this case
the middle finger of the left hand and

the end most as was to say was
the second poke to the left or

the right the first in this case
the the second part was to the left.

After six tedious months the endless.

To get out what it is we were asking right
if it's a one pair they figure out after

a few days but the ability to generalize
anywhere any pair of locations on the hand

takes them a long time but
eventually you can do now take two

random locations on the hand and they
can report which of the two locations or

whether the second location
was the level of the first.

Once we're trying them to
do this mechanical task and

there perform stopped improving we.

We started doing the following tasks
where in we might pope say the animal in

on the ring finger on the middle finger
and then electrically stimulate it through

an electrode that we had established had
a receptive field on the ring finger and

the correct answer in this case was right.

And to the extent that the animal
could would consistently.

Or the animal's response matched
the predicted response given the relative

location of the poke and the receptor
feel the occasion we could infer

something about where the animal felt or
where the animal felt it rather important

methodological points to
avoid that the animals

might learn you know if if I feel if if
I feel a touch on my index finger and

I feel some kind of weird sensation I have
been rewarded to go left every time so

I should go left and then rather than
learning these stimulus response

contingencies that had nothing to do
with location we wanted to and we.

Had on any given electrical block up
bunch of different skin locations and

a bunch of different electrodes so
the animal had no chance to learn

anything about the specific stimulus
response contingencies the only way

the animal could do this task is
by judging the relative location

of the two sensations the mechanically
intellectually induced ones and one block

it was one set of conditions and another
the next bucket was a completely different

set of conditions so none of that would
transfer from one block to the next and

another words the animal had to
judge the relative location.

So it worked that's one talking about
right now this is the mechanical.

Performance of the mechanical trials and

you can see on some trials the animal did
relatively poorly so each of these points

corresponds to a specific pairs a pair of
locations so some some of these pairs.

Poor performance when they were close
by other pairs near perfect performance

when they were far away right so
the import So this is the you know this is

the baseline when both similar mechanical
when we replace one of the mechanical

seamy with an electrical
stimulus applied to area three B.

which is really primaries ministry cortex
proper We found that the performance was

nearly identical to that it was
actually significantly less but

only because we could do a paired T.

test because we matched the locations
of the the mechanical pairs in

the hybrid pairs but really performance
was very similar for area three B.

and that for area one it was less
significantly less than Area three.

B.
but

still well above chance which if you
know anything about the most as you

say masses cortexes will not come as
a surprise to you because Area three B.

is primary so massive the cortex
proper has small receptive fields and

area one is basically one synapse away
from that has larger receptive field so

it's not surprising that
the sensations of vote by area one

stimulation would be sort of more diffuse
than those of A by area through B.

stimulation Regardless though in both
areas we found that the animals were well

but above chance which can
only mean one thing and

that is that electrical stimulation
of sensory cortex evokes these highly

specially localise sensations and I should
mention because you might be wondering.

If you distrust me which you should that
the animals never got better on the hybrid

test so we got them as much as as far
as we could with a mechanical task and

then we started replacing it may chemicals
to me with electricity really they were

good and they remain good and they never
got any better so it's not like they were

learning anything new with the when the we
started the electrical stimulation trials.

OK The other thing I said we were going
to do is try to convey information

by contact pressure which is a little
bit less straightforward So if you

touch something lightly or something
touches you lightly with a low pressure

you get this faint sensation if you if
you jam your fingers on something or

something jam is Jim the crime against
your skin you get this much stronger

sensation What does the brain do what is
the what is the neural neural correlate of

that well what happens when you touch
something lightly is you have this faint

activity that is highly localized and
as you increase the pressure exerted on

the skin you get an increase in the firing
rate of neurons right there in the hot

zone of activation as shown here and
you get an increase

in the volume of activated neurons as
shown here right I mean you can see.

In these plots and

this is across all skin locations both
these phenomena happen so we're like

well OK if we want to convey information
about pressure we're going to do both of

those things we're going to increase the
neural activation of the neurons near or.

In the hot zone and

we're going to increase of the volume of
activated neurons we're going to recruit

more neurons The nice thing about that is
you can basically do that something like

qualitatively like that by just turning
up the juice I mean the electrical

current right if you have as we current
you're going to weakly activate neurons

just near the electron that's it as you
increase the juice you're going to Morey

more strongly activating neurons
near the the electorate and

the current is going to spread out and
recruit more neurons right.

OK so so now so
that's our that's right DHEA But

now we have to test it we have to test the
sensor correlates of increasing the juice

so it's train these animals to do another
task in fact we train them to do two

very similar tasks which is like this is
standard to all turn forced choice physics

where first we teach them to do a
detection task so in this case first with

mechanical simular parts of the skin so
here they are two intervals like interval

one interval to their indicated by a
visual stimulus in one of those intervals

of stimulus attacked an indentation
is presented a very sweet indentation

ranging from a twentieth of a millimeter
to third of a millimeter ish and

the animal has to judge which of these two
intervals contain the stimulus right so

they respond one way if it's in
one response in other ways for

the interval to then
we had another version

of this house which is really a pressure
discrimination task which now we're into.

The skin twice on these trials one
inundation is a standard stimulus

of constant amplitude one is
a comparison stimulus resample to train

as from try this is a method of constant
stimulus for the sake of physics nerds.

So we used you know five or six different
values for this comparison stimulus one

two three four five five different
values for the comparison stimulus and

the animal had to judge which of
the two was more was more intense.

By the way I talk about pressure and
then I give you microns in so you're like

confused because those are not pressure
units those are units of indentation

depth which is monotonically related to
pressure as I will show you later OK So

we have so the animal so there's a low
center condition and then there's a high

standard condition where two millimeter
indentation is paired with these.

And these variable comparisons right and

all these conditions are all mixed
together forcing the animal to

to really pay attention to both
intervals in order to get the answer.

OK So this is really standard physics.

And this is what the data look like so
this is proportion

correct as a function of amplitude I'm
going to all this because it's going to

become important later I promise on
the board for the sake of boring you.

This is the proportion of times
the end of correctly identified which

interval the stimulus was presented in
in the detection task as a function of

the amplitude of the miss and
of course as the amplitude increases

it becomes more detectable and the animal
is more more easily identified which in a.

Contained in this is the mechanical
pressure discrimination data

where this is the proportion
of times the animal

selected the comparison as
being more intense and as the.

I'm sorry no the proportion of times
the animal correctly identified

which of the to stimulate
was more was more intense.

As a function of the amplitude of
the comparison stimulus and the reason why

this goes up is because as the comparison
amplitude increases it becomes more and

more different from the standard amplitude
and therefore the that task becomes easier

in this case it goes down because as the
comparison amplitude increases it becomes

more similar to the standard stimulus and
therefore the test becomes easier.

Or more difficult I'm sorry OK.

And then we did the exact same experiments
with electrical stimulation of the brain

right so now the detection performance
obviously has become more detectable if

you turn of the juice discrimination
Then we want to create gray and

all kinds of other parameters like
the effect of pulse with of on the tech to

Billy the effect of frequency undetectably
so pulse with it's completely predictable

frequency as you increase the frequency
detectability improves up to a point and

then plateaus duration again increasing
the duration improves performance and

then plateaus and anyways and I'll tell
you why we did all this in a minute I

guess that the big picture is this is a
new sensory modality right we're trying to

figure out how the detectability of these
stimulate or dependent on the various

parameters of the stimulus as you
would any other sensory modalities.

This is what the discrimination data
look like so you can you have for

the low standard you have up we're
going second but your function for

the high standard you have done
we're going one just like we saw for

the mechanical stimulus there's no
effective frequency there is an effect of

duration of to a point so again we won't
want to characterize how the sensory space

depends on the on the studious stimulation
predators This is why this was useful

it was really boring I'm sure most of you
are kind of bored right now unless you're

big cycle physics nerds in which case you
might people but what we were able to do

so that no one has to go through this
stuff anymore is we took these data and

we created a model like the simplest model
we could think of to try to predict how

detectable or discriminant will a stimulus
would an electrical stimulus would be so

the way we did it is we created this
volume of into green fire neurons

they're not even connected together
that's how simple this network is or

this this and this model is then we can

sort of model an electrode in this
volume we know that the proportion of

firing this is an empirical result.

Decreases as a function of the distance so
this is an empirical function

of probability of firing as a function
from the as a bunch of distance from

the tip for electoral stimulation we pass
the population response to an integration

time window and then we do an ideal
observer analysis on the real resulting So

we simulate the neural response of
vote in by a given stimulus and

then we use ideal observer analysis to
determine how detectable discriminants you

would be the details of matter
Here's the take home message we can

predict with ninety six percent accuracy
how detectable or discriminant will

how detectable a posturing will be or how
discriminate will to pulse trains will be

using this very simple model that just
you know it's almost laughably simplistic

but it does what we need it to do and
this is published

in The Journal Neural Engineering not this
is not a science result this is a neural

engineering result where we can
actually now speedball doing this kind

of work don't have to do this tedious
tedious stuff they can use our our simple

model that has like a dozen parameters
that can get implemented at home and

figure out how detectable
discriminately will be OK.

Now back to the sensory including our
group I told you DARPA doesn't really care

about this stuff they just want to argue
them is a give me a set of numbers that

maps input to output how
do we go about bringing.

Developing an algorithm from these data
Well I told you we had our detection of

discrimination curves based on
mechanical simulation of skin

we did the exact same thing on the same
monkeys with matched locations on the hand

you know in the brain with
electrical amplitude.

Detection discrimination and then we made
the following assumption that if something

was eighty percent detectable discriminant
with mechanical simulation of the skin and

eighty percent detectable
discriminate will with electrical

brain they were
perceptually equivalent and

then we could do that with all of that but
with each level.

Of detectability or discriminate bill
the two and we there are cheap these

functions which are called Signature with
Quins functions which map electrical or

mechanical amplitude this is exactly
the kind of algorithm we want

mechanical amplitude is sort of the
operator sensor it's related to pressure

this is the amount of electrical
stimulation that you want to choose to

to pump into the brain and basically you
know here let's use perhaps as an example

this means that if you
want to create a sensation

that is equivalent to a four hundred
micron indentation of the skin you

need to actually see the brain with sixty
micrograms right so the idea is in time

you just take the up of the sensors which
tells you how much pressure is exerted and

this tells you how much to electrically
stimulate the brain to create appropriate

sensation so that if it's a little bit
of pressure you get a weak sensation

if it's a lot of pressure and station so
there's a little bit of variability so

each of these lines correspond to
a different electrode each of these

colors correspond to an animal but there's
a little bit of variation there but

really the shape is always the same
it's like a sort of power function with

with an exponent less than one OK.

So we've got these functions there's a lot
of stuff that has happened the last five

or ten minutes and maybe I lost half of
you it doesn't matter because I'm going to

show you this is the proof of the pudding
is a part of the talk where we took our

algorithm and we tested it OK and we
tested in a in a prosthesis with a monkey

right so here we have we did we had
the monkey do this pressure discrimination

task I showed you earlier except instead
of feeling the stimulus on their

skin we delivered the exact same stimulus
using the exact same stimulator on

the prosthetic finger and took the up part
of the pressure sensor on the prosthesis

in real time converted it to a pattern
of electrical stimulation to the brain.

To the brain so that you know so this is
a low pressure stimulus evoking a lower or

electrical stimulation This is
a stronger pressure of being stronger

in the brain and the animal had to do this
pressure discrimination task based on

information through the neural interface
this is the figure that shows you that So

these are different levels of
indentation and these are the the upper

the pressure sensor just to show you that
I've been using amplitude as a proxy for

pressure and
this shows you that it's a good proxy for

pressure in that you crease indentation
you get greater pressure and then and

accordingly you get greater
electrical stimulation of the brain.

So then we.

Compare the performance of
the enemy when they were doing this

pressure discrimination test to this brain
interface to their performance when they

were actually feeling the thing and the
purple trace shows the the performance on

the pressure discrimination task when you
when the end of the stimulus the magenta

traces show the animal's performance
when they were doing the same task but

through the brain interface and

you can see that they were doing it for
two different electrodes and

you can see that the end was just as
good in that case right with validating

our psychometric equivalence function
that that claim that that's exactly what.

We had to cheat.

And by the way I couldn't make it make
the animals better I could just turn

of the gain and boom the way better than
we are or than they are with their skin

with a lecture with with this prosthesis
the issue with that is that you have

a limited range of electrical similar that
you can apply to the brain before you

start burning holes into it so this
allows us to really extend the dynamic

range of the sensations making them sort
of quite commensurate with the sensations

that we experience in everyday life and
and so and hopefully make it

make the signals more interpretable to the
motor sort of planning parts of the brain.

By the way this was a cool
result to just this one time for

a single day because we didn't want
the animal to learn anything we

had the animal compare a standard stimulus
of four hundred forty micro microns or

almost half a millimeter indentation which
we had determined using our second metric

equivalent function to be equivalent
perceptual to fifty micrograms and

then we paired this mechanical stimulus
with electrical stimuli ranging an Apple

two from twenty to eighty and all
the stimuli there were less the electrical

Similarly the there were less than
fifty microns we're perceived as being

less intense in this mechanical stimulus
and vice versa for those more intense and

than fifty microns just again validating
our function and also showing

that these mechanical intellectual
stimulation are on a continuum on common

intensive continuum because they were able
to do that immediately they had never done

that before compare mechanical electrical
on the intensive continuum OK So

this is all or our animal experiments
we said all this stuff to the F.D.A.

along with some super boring stuff
where we chronically implanted

electrode arrays in chronically stimulated
for six months to make sure that

we weren't burning holes in the brain or
damaging the electrodes and we found.

In short that we didn't and so the F.D.A.

approved human trials of
this electoral simulation.

And this was done at the University
of Pittsburgh by my colleagues

led by Rob might by.

Injure and the shorts and
Gen Collinge are.

So before I get into that the idea though
are our animals are completely intact

right so we can map out where they're at
the electrodes go and we can then we can

do some certain things that we can do with
the touch of the patient on the other side

hand though human touch of patients might
be paralyzed in one sense a from the neck

down but they can talk so the idea is
to do these mappings we electrically

stimulate through an electrode the same
lecture thirty five we've shown with our

monkey work that stimulating electro
thirty five would produce a sense of very

highly spatially look at us and station
let's say on the index finger tip so

now we can connect the sensor on the index
finger to the prosthetic index finger tip

two electrodes thirty five any ten sensor
touches something you electrically

stimulate through electoral to produce
a sense of sensation on the index

that will be and therefore kind
of convey information of where

the contact happened with a prosthesis
in a sort of intuitive way.

And how much is them similar to
a lecture of thirty five we have or

are mapping that says given this amount of
pressure this is how much you to stimulate

to give an appropriate person to get so
shoot I forgot there's some on

this I'm going to describe what's going on
because there's no time to try to fix this

can you hear this and I don't what's

happening is this this is Nathan another
patient who's such a political so but

it was now two arrays of electrodes one of
his motor parts of the brain again Mosher

your earlier and one of the small sensory
parts of the brain where we could deploy

the algorithms that we've discussed and
in this in this test basically run

he's Nathan is blind folded and Rob got
here is touching different fingers they've

done this mapping that I just described
and Nathan can actually perceives

any time you rub touches his index finger
Nathan reports a sensation on his index so

we this mapping that we
found in the in the monkeys.

Sense of humans we also did that a little
bit because we didn't have to do

a bunch we did a little bit of the second
metrics on on the human patients detection

threshold discrimination threshold and
found that really the sensitivity to eye

C.M.'s in a microscope mutation in humans
is very similar to its human to its

monkey counterpart and so all these models
that we developed from monkeys can be

applied just wrote in humans right so you
have to go through this tedious work and

you know in monkeys you can just get them
to do this task over and over again from

Month after month after month after month
but in a human touch and they already have

enough problems in their life that the not
going to do this extremely boring thing

on top of that So this really kind of
allows you know every time we submit

a grant now we have to split that like
a page to explain why it's important to

do a monkey work because there's a lot
of stuff you can do in the humans.

So this is all well and good but really
in the end I start this whole talk well

one of the things I said early on this
talk was we're doing this to make

people better using these prosthetic
cancer and it does so this is.

We're showing Nathan doing this
task moving things around with and

without touch.

With within with other sensory feedback
because the nice thing here is you can

turn it off and see how good or bad he is
and you could probably see just from this

movie that is way better because he knows
you know if I'm grasping this object I

don't even know when these figures touch
it right I just have to take it on

faith that they're touching and you sort
of Whereas the instant he touches it and

has this tactile feedback
just to give you an so

intuitive feeling of why he's
better here the minute he

touches something he knows he's made
contact with it he can start moving it.

Here just them finish them like
one side away from finishing.

And this is just a qualification it takes
him longer without touch than with touch

and if you use this is a clinical sort of
assessment of of limb function and you can

see there's a significant improvement with
touch then with a touch on this clinical

system and this is just to show that
progress marches on this is Jan

doing this tasks this is without touch and
this is Nathan doing this task

with touch even with four fingers
because the middle fingers is messed up.

And you know it's obvious
that that Nathan is better so

that what's happening here is not only are
you incorporating touch but also improving

your decoding algorithms and this starts
to look pretty fluid on the right.

So these are the people that did the work.

Terry has been collecting
gods the data as it Greg and

then these are my engineer is who
got the all this working and Greg

consumption have been really instrumental
in analyzing the behavior data.

This is a big consortium of
people who worked on this

called Revolutionizing Prosthetics
the main players that are relevant to me

are the the group at Pitt.

I got that I mentioned earlier and
this whole Revolutionizing Prosthetics was

running out running run out of the Applied
Physics Lab The I was my clock on and

then my handler or
whatnot was Francesco to nor.

And these are my.

Funding sources and most of this work
was about DARPA thank you for attention.

Thank.

You.

Yes.

Yes this is the way.

This is really a series of did
you come up with where people.

Were quite sure yeah I believe for.

What you report if you go
this is the first guy who was

controlled by it if you do it right.

Where you are or
is he just you easily see a little those.

Taxes and they probably you
know this is illegal it's so

easy to do really this
is all over this right.

There right next.

To fish so they did.

Their.

Best to buy a.

Story.

To be the.

First so
here yes this is you are first of all

they buy every day is
significantly better than.

Our Were the thirty years here are.

Three and three.

So this is all that we're using
do you know that I get to yours.

I thought this is a real one yet
you did that you.

Were is that was.

Really here this is a track.

That is not over.

With.

Your contact here is it
has to do eat right.

So those.

Bill that is right and
here's the real problem I did and

we've done right the week on the end but
this is a broad.

Brushstrokes by by creating
groups that are now trying to.

Turn cheese.

Most ingenious little busy
do good to the city but.

The issue though is that these robots
are not very good and the sensors are.

An now we're getting at the point
where the falling no longer touch for

its robotics is such a she
says assure you that

this is a high priority for the sensor
the scrap sensor has to do that.

Even though you have one person.

Here.

Anyways so most of them are now how
do you know in order to benefit

manage Yong Well we know we
need to appropriate such.

Good.

Old everything is a big
here is the best of.

What it.

Was.

Right.

Away so that.

Pretty much so I believe I was and still
to be the start of it would like to see

is it the prince spreading the figure
feedback loop that a significant part of.

The problem is that you wish you
know you can just make things

deliberately that and this and
show that it is from the book so

I think those are the kinds
of this we're busy to do it

where we had weeks to achieve and
see what our lives he lives he.

He's to me useful verse.

I'm sure.

Just.

Like all.

Right.

Out.

Here.

Right I think we're the first person.

Who knows.

So the first question right so this is a
question and it is it's one that we really

struggled with until I met this
Professor of ours for works on.

And this is actually a paper for
a surprise.

It was pretty hard to work
with our suppliers so

here's the result I think the theory of
it if you I mean if someone gets hit.

That poor grade that Mr Sparks of
the hour now can you have to be face or

some other bar.

And as well as did I think
mistakenly interpreted as.

This is amazing cause for a right.

Good old paper rather than talk about
Buddhists I'll just say to this

one just look at a prosthetics.

If you take a sheet.

And you put a little bit of
the religion of his or her her.

And this is that is the next twenty years
or so down and up suddenly from there

they're they're there for years and
allegedly still is instant T.

Is it possible that they did that after
that was going to prove that you know

her from the birth to ensure
it is just to hear the word.

We are.

In the same as a corner you know so
well as a dishwasher there's a lot of

spontaneous activity in his insides
were pretty good the visit I live just.

Ahead sensation over the box but

when you start writing mystery of
a great actor always pulls disappear and

you get it and you have an idea what
the sad part is based on MASH right.

Imagine we're going.

To shoot this show this hammer edition
was stimulated produces a sensation.

All this all this so-called
plasticity I mean it is like this but

all this so-called massive cortical
realization one is caused by

homicide if a city agrees
that this beautiful erosion.

Here in Bangalore are all that.

And do.

It in a sense because you know because of
all your sacrifices city your Japanese

worship Cortez all good measure you get
stronger by this is not the kinds of

initiatives Cartesian and or
fall into the kind of competition you.

That's why you know we have to worry too
much about it and no proof is a part of

these business needed it was ten years
ours is that it's a leap rule for us so

you're happy with Purvis Where did
twenty years and pull that were.

For us we were more you see evidence the
open loop control gives better with just

exposure to through a few people who know
that is a very personal Fortunately we.

Tend to push for all this stuff
is all wired in for the stuff so

we're not if we have it a position where
we can let them use it for a good idea.

In the be.

Worked for her for a year she gets.

A group with the whole thing back on
the table and the moral of the book.

But we need to do.

My my my.

Writing.

For a while before I got.

To talk to you.

Thank you.