Very.

Very curt.

Very for
quite some time welcome ceremony you're.

Very fond of what really
pulled off mission.

Right.

Side of the medical.

And there you go Wally.

The Associate Director of your.

Medical robotic.

Expanding the scope of modeling.

Collaboration with you on
the medical billing all right and.

I was thinking.

That you're.

Precisely right thank you Henrik.

I assume everybody can hear me if you can
hear me in the back wave your hand just so

I don't work in excellent.

So it's really a pleasure to be here today
you know Georgia Tech has really blossomed

I think in the last few years and I have
to give credit to all the colleagues have

been seeing today Henry are fine and
Jim Frank and whoever else I'm forgetting

the list goes on forever but
it's really a pleasure to be hosted.

I thought I'd start out by just kind
of framing where a lot of the work and

we talk about today set so so my lab is
part of a larger effort at Hopkins called

laboratory of competition all sensing
in robotics We're about ten faculty and

about one hundred grad students I
specifically have been interested in how

you frame the interaction between people
and robots in the last few years but

I do continue and let me run the video I
continue to do computer vision work as

well so we do a lot of work in medicine
with what we call quantitative and

be three D.
reconstruction of an atomic structure.

In navigation from video

I do have a student mostly a little
bit on our scene analysis work and

also we have been looking a lot at how
people collaborate with information so

this actually is a facility we installed
in the Hopkins library couple years ago

now it's a large scale interactive
display it's got five connects on it and

it is really used as a weighting gauge
the university community in information

technology interaction and data and by
the way I'm looking for somebody who just

wants to play with a really large screen
with a lot of cool tools attached to it so

if you're looking for a position send me a
resume I would really like to find someone

for that project but today I'm going
to talk mostly about our our languages

surgery project which has been going
on for about a decade and also as Henry

said Bridge that a little bit into
human machine collaboration so

I thought I'd start out with this notion
of competition Hanson of human skill and

so maybe at start we should start
by saying well what is skill and

I think we all have a sense in

our head of what that means you know skill
the ability to do something well expertise

we tend to think of skill being associated
with doing a difficult task you

know skilled people or people who've
had to train to do something that

the average person can do and you're
synonyms dexterity so you know the ability

to use your hands ability mastery the fact
that you've actually put significant

time into something and you've actually
mastered this topic and again if you if

you think of the pictures that come to
mind you know sports is probably the first

thing I have to be giving this talk
right around the Winter Olympics so

there are plenty of sports pictures
on line from from there and

we think of those athletes as probably
being the pinnacle of skill but of course

really an early day lives for lots of
other forms of skill that matters so

you talk about a skilled craftsman someone
who has studied how to in this case

you know for example build
furniture learning not judge.

The manual dexterity but
the cognitive aspects of a task and

of course today are going to talk
a lot a lot about medicine and

once again we always talk about a skilled
surgeon a surgeon who has great hands and

so there's really a notion that skill is
something that is it's an eighty human but

it's also something that really
in our lives has high impact and

the question really is how do we start
to think about translating people

acquiring skill into something that might
be augmented or assisted by robotics.

Now one of the things I've
been alluding to is this

question of well how do you become skilled
and many of you probably remember the buzz

a few years ago around this book called
Outliers by Malcolm Gladwell and

so one of the take home message is kind
of the sound bite version of that book is

you need ten thousand hours of practice
at any one thing to become really highly

skilled at it that's
a little bit of kind of the.

Popular Press view of becoming skilled
It's probably a little more nuanced

than that so there's obviously
also a literature in psychology

where people really try to study the
question of how do you become skilled So

this is a piece of work by.

And go bury talking about deliberative
practice necessary but not sufficient

that is to say it's not the case that
you probably can just go into a room and

say I'm going to become the world's best
surgeon without anything else happening

you have to have other surrounding factors
and there are other things that you will

learn along the way and so it's again this
kind of framework of saying well what

is it actually take to become skilled
that we've really started to to think

about how we could study both the process
of skill acquisition and then eventually

how they get rich gets reflected into
systems acknowledgement individuals so

I always like to start this story
with William Hall stead so many.

You probably know if you say Johns Hopkins
probably the the word that completes that

phrase is medicine it's probably
the I think it is still the largest

medical school in the country it's one of
the oldest and head for founding fathers

also it was one of the four founding
fathers one of the things he's known for

is actually creating the the system
of training that's now prevalent

in medicine today so it's called
the graduated responsibility model or

the short version of it is
see one do one teach one

the idea that you have a mentor that
mentor is skilled at a particular task

you get to observe the mentor
see how a task is done and

you take over parts of the procedure
that's the graduated responsibility and

eventually as you gain facility at
the task you become an expert yourself and

soon now you can go and
teach the next person down the line and

so skill is handed off from
generation to generation.

It's interesting to note it is known
widely as the person who introduced this

it turns out it was actually his colleague
also lawyer who imported it also

travelled in Germany he saw this at work
in German medical training he imported

it to Hopkins but he wasn't really
interested in promulgating it he just kind

of thought it was an interesting oddity
that they tried out in the hospital so

how did actually borrowed the idea
he pushed it through Hopkins and

then he actually rolled it out as
something there became known nationally so

really it was an idea borrowed from
the Germans imported to the US and

then virally again and
moved out into practice.

So many of you probably aren't familiar
with what it actually means to train

as a surgeon so this is a pretty typical
path that you would follow if you want to

become a surgeon so you go to med school
and you go to college you get into med

school you do about four years of medical
school again it varies by institution but

the first order maybe half of it
is book or and you go to classes.

You learn anatomy physiology.

So on and so forth and then you get
about two years of clinical exposure you

actually observe doctors at work then what
you do is you start to specialize in so

you go into residency and again depending
on what your specialization is you

could spend anywhere from two
to six years in residency so

at this point you're mostly in
the hospital you're actually working in

the hospital you are a doctor first
order you see patients you can

prescribe medications you can do many
things but you're still you're learning

you're still under supervision this
point you made choose to specialize so

you may become a surgeon at this point in
your residency then it might take place

in surgery and in fact you can then
obscenely beyond that do a fellowship and

that might be some specialization beyond
which you would get as a resident so

if you want to become a retinal surgeon
for example you will actually do a follow

on fellowship if you spend many
many many years just acquiring

the skill that it takes to be a surgeon
Interestingly there is no final exam for

surgery you know they
don't we all are patient

the room and say OK go you have two hours
and they give you a grade at the end and

that's your your Whether or
not you're a surgeon the fact is it's

a it's a training program you go in at
the beginning you come out at the end and

at that point you are a surgeon and you go
off and practice and so obviously there is

a pretty large variance in terms of both
the inputs and the outputs and if any any

of the surgeons I've talked to it happens
will acknowledge that there are people

who are much better as surgeons when they
come out there are people who are not as

good as virgins and they come out and

in fact there really is very little
quantification in the end of how

highly skilled you are it's certainly the
case that and it go to your subjectively

there's a notion that there are people
who are very very good surgeons and

they're very highly skilled at what they
do and so the thing that we really started

to think about was can we come up with
a way that we could quantify what it.

Means to be skilled and
again talking to my physician colleagues

this is something that they would dive for
to actually be able to go in and

actually look at someone in the training
program and have a sense of OK what

are their what are their deficiencies What
are they what are they mastered what have

they not mastered and how can I build that
into their training what does that mean

when they go into the O.R. and they start
to work with people for the first time and

they haven't ever been in front of a
patient before what could I intice a page

perhaps more importantly it's not just
about that training but also ultimately

can we actually influence the practice
of medicine to create more effective or

better outcomes you know I have this
picture here of you know surgery versus

taking a pill think about what it takes to
bring out a new drug on the market right

so you create a new drug you do a large
clinical trial you have you know prescribe

dosey to you guys you know a cohort with
a drug without the drug you will for

side effects you work for
the effectiveness of the drug and after

all of that is when you finally say yes
this is a drug that happens to work for

this condition with these side effects we
don't do that with surgery even though

again many of my surgeon colleagues will
say surgery really is a dose of treatment

right you know there are and I can
actually very that dose in many cases but

we don't have a sense of how
to quantify the effectiveness

of my dose of surgery in
the outcome of the patient.

Just to bring home how this can have
an impact here a couple of statistics for

you so there was a study.

A study by Reagan Bogan where they looked
at how different types of unfortunate

outcomes unacceptable outcomes came to be
by basically setting medical malpractise

records so you can say well a poor outcome
could come from a surgeon it could come

from just the workflow in the procedure
room or could come from the factory just

order of you know a hospital they
didn't have certain facilities for

example that they needed at some point in
time and then you can go back from that.

Saywell Fitz the surgeon did have to
do with what they knew what they had

to decide in C two or it could just
be that they'd been on the floor for

twenty four hours and they had just
made a had a lapse because of that

the workflow you can talk a basically
about how the team works together and

again environment so the green ones are
one said it you could argue that training

might have some impact on and
so you can now say well what

fraction of those unacceptable outcomes
came from things related to training

the answer is more than half can be traced
back to something that's probably related

to the training of either the team or
the surgeon.

And then if you look at those errors and
you start to say well what what actually

happened so it turns out about sixty five
percent of them are actually manual errors

so something where now to me was
injured in a way that was unexpected so

you know you happen to lose
concentration for a moment and

something sharp when the wrong place and
suddenly you've got bleeding going

on there now you've got to
control the bleeding for example.

And if you look at the impact of
the errors it's pretty significant So

twenty five percent permanent disability
nine percent of the time people died and

then the rest are a mix of
different things so clearly having

better training could have a pretty
significant impact you know at this level.

Here's another study just to
give you one more data point

before we drop into things this is the
Michigan variadic surgery collaborative So

what they did is they took videos
very after surgery and they had

a panel look at these videos if it was
a panel of ten surgeons for a member and.

Looking at the video and ranking
the twenty or twenty expert surgeons

ranking sorry ten I was right ten
reviewers ranking twenty surgeons and

what they were doing was giving an
expertise score from one to five so what

you can imagine is I'll be looking at this
in essentially if the they don't know who.

Who's doing the surgery didn't know
anything they're just looking at the video

and they say well this looks like somebody
who's really knows what they're doing so

they'll give that a five someone
who's having real trouble

you could imagine you would give them
a one Luckily that never happened.

But you can see the range of rankings
was really from you know about two and

a half up to almost five so the fives are
great two and a half not so much and now

you could say well what is the correlation
between let's say the upper core tile and

the lower court trial in terms of what
happened to the patient so here's

the answer so if you were in a lower court
trial if you were having a surgery and

the position was evaluated in
the report tile you were about two and

a half times more likely to be readmitted
to the hospital you were about twice as

likely to have a second operation
reoperation you were about five times

as likely to have some sort of
complication post surgery and you were

had a very large factor more likely
mortality so two point six percent of

the cases actually did not make it through
the procedure versus zero point zero

five if you were in the upper portal so
there are significant impacts to scale and

so there are real reasons we'd
like to better understand this.

So so I've been talking really
abstractly about this and

hopefully I've convinced you there could
be some real impact for improving skill so

now the question is how easy is it is it
to tell who's skilled so I'm going to give

you a little quiz here in a little dry lab
model so the question is who's the expert

who's a novice those of you who seen
this before you know keep yourselves.

So here we go we've
synchronized the videos so

you should really be seeing about the same
things happening about same time so

time doesn't matter here it's
really just about what they do

you may have already guessed it but let's
wait well maybe it's becoming more obvious

There we go I think I think
you figured it out by now.

OK So either none of you I'm guessing
most of you are not surgeons.

But it didn't take you too long to
figure out actually what was skilled and

what was not skilled right so
you could already start to tell me and

by the way you can maybe tell me what's
wrong why why is the person on your right

having so much difficulty but

I guess that there's actually a good
technical reason why they are.

I think I heard it what's that.

Now well that actually could figure into
it now they're actually holding the wrong

look how they're holding the needle they
have no control over the needle at that

point because of the way they've decided
to hold it if they held it ninety degrees

off they would suddenly have much better
needle control and that's what a mentor

would do by the way as they would look at
this they would just say hey you're doing

badly they would say by the way here is
how you can correct that and you can move

more rapidly along the curve and
that's really this difference between just

practicing and practicing with supervision
so the interesting thing is you know being

quantitative people this is a dive into
the robot we can pull data from the robot

so we can give you the hand motions
this what the hand motions look like so

on the left you see someone who
is a an expert at the procedure.

And you can see for clearly defined throws
so there's a force for running suture

one two three four on the right is
someone this is actually someone who has

trained in surgery they've done surgery
before they just haven't used the robot so

they're really learning
how to use the robot and

you can they're actually accomplishing
the task but you can see the organization

is completely different here and so this
really is the thing that got us started

thinking well how do we actually measure
the difference between this and this and

then how do we basically pushed people in
this direction you know how do we diagnose

what's going on and how do we make them
better so we're going to talk about for

the rest of the time is project
we call languages surgery

which is specifically focused on how
do we address this problem how do we

model patterns of activity particularly
for surgery actually only so

how do we basically recognise what's
happening recognise the stages of.

The pattern of activity how do we then
use that to start to quantify and

measure skill and then ultimately
how do we actually use that also

is a way of providing feedback so
that we can make you better faster

by addressing what are the deficiencies
in what you're doing.

So let me just tell you briefly what data
in this case looks like the world of data

is like many other areas changing
rapidly so that eventually robot

surgical robot commercially available
they're in their fourth generation the X.

I just came out they do a little over
a half a million procedures a year now and

if you can get inside the robot you have
your complete record of a surgery you can

get high quality stereo video you can
measure all of the tool motions you can

actually measure things like tracking and
voice in the O.R.

and what the surgeon saying if you want
to augment the council a little bit so

you get really high quality data so
the sort of data we work with is typically

this sort of data it's a dry lab model
benchtop model needle passing suturing.

Kind of a nasty MOSIS type task we
have a couple of data sets a smaller

data set we started with a larger data set
that's our current data set you can see

they're not huge a few hundred trials and
counting on the order of twenty surgeons.

We also there is a simulator.

Into it now has a simulation system
we can pull data from that so

we do have what we call a warm up
data set it's actually a simulation

data that's paired with
someone going into the O.R..

And then we do have a live surgery date
as well so we have hysterectomy and

prostatectomy at this point these numbers
are out of date I think we've got about

eighty cases so we have about three
recording systems at Hopkins going

pretty much full time now
dumping cases as we go so

our data sets are growing quite rapidly.

And then.

We also do some work with the owner and
Gallucci So

we do have a lot of data around and
a lot of diversity now that we can start

to study not just robotic surgery but
surgery in general.

So the first thing we started to
think about is this question and

now that we have data how do we actually
come up with signatures for learning so

if I talk to somebody who studies
motor learning in particular.

You know most of the motor warning people
kind of think in this axis of how do I go

from basically no muscle memory to muscle
memory to solve the task but of course we

also know that there's a cognitive aspect
to acquiring skill knowing what to do and

so you know motor people would probably
say you go this direction there's some

argument that at least in some domains you
actually kind of think about what you're

doing in the motor skill comes in behind
but for all we know you go like this

you know you could be you go back and
forth we don't really know the path so

we don't have a lot of guidance really
from the literature as to what to look for

so we took a couple of
different takes on this so

here's the first I'm going to talk
about the idea here is to say let's

not worry about the real
structure of the task.

Taking it and breaking it down into
a primitive components in gestures but

let's just kind of look at it as
a string and see if we can just find in

the actual motor behavior
itself indicators of skills so

you saw that multiple needle pass right
that's really what tipped you off so

how can I find those sorts of
indicators in the signal so

this is work of Nargus Imedi who is
actually just finishing her Ph D.

she looked at the question of
taking this continuous signal and

trying to turn it into first
a discrete representation that

has the right sort of invariance
properties that we'd like to have namely

we don't really care if your suturing up
and down and left and right or forward and

backward we care about the innate motion
so the way that she set this up is she.

Took the initial curve a continuous curve
you build a coordinate frame on the curve

a so called for in a frame and then what
you can do is you can use that local frame

to encode direction of motion and so
now you're riding on the curve you

now coordinate invariant all you care
about is how the direction changes not

with the absolute positions or
velocities in some cases are so

what this means is we now can take
a motion signal and we can turn

it into a string once we have strings of
course being a computer scientist I'm I'm

suddenly very happy because I've got
something I know how to deal with and so

the way that you then looked at this is to
say well let's think of these strings of

different tasks or different skill
levels or any category you want as

in terms of building categories
specific dictionaries so

if I take a grasp of motion for example
you can imagine that there's going to be

certain canonical movements that are going
to characterize a grasping movement and

what I want to do is to pull
out from that string from

multiple repetitions What are those
common patterns of movement and

essentially build a dictionary of the
movements that characterize grasping and

I could do the same thing for pulling
the suture I can do the same thing for

rotating the needle through the tissue so

pick your favorite categories will
just build a dictionary out of

these strings by finding a long and
longest common substrings.

And then we'll look at classification as a
string similarity problem so just like you

know when you index the web you go in new
data mined the web and build dictionaries

and you get a new page and you say OK what
category does a belong to in our case we

get a new trial we turn it into a strain
and we say OK let's go through and

figure out which patterns appear and
which dictionaries they belong to and

we'll build a similarity metric around
that and it's going to basically look for

what is the likelihood of seeing
a particular string in that dictionary

we want to favor a long string so there's
a term that says longer is better and

then Location does matter so something's
happened towards the beginning something.

It's happened towards the end so
this is a little metric you train it up

if you take no strings in this case from
those three categories grass pull and

rotate you can see it's actually starting
to separate the performances into those

different categories and
then what we do is we simply put in S.T.M.

on top of that and we ultimately do
a single class class fire out of that

how well does this do well turns out
at a least on the benchtop datasets and

we've run it on it does just about as well
or better than anything we've tried so

if we have seen someone before and we want
to categorise in this case both the skill

level and the gestures so what are you
doing and how well are you doing it

in this case we've got nine categories so
chances all the percent if we've seen

the person we can get about ninety five
percent accuracy you can doing skill and

gesture we haven't seen the person at
least in this data set it drops down quite

a bit and this is something we've
we've seen a lot it probably shows us

data's deficient that there's a lot
of individual style in how people

do things and that it takes a lot of data
to bring these two numbers together so

that you don't start to learn the patterns
for a person not the patterns for

a particular skill level.

Interestingly one of the things that we
tried just to see what would happen is we

took exactly the same method that
we had trained in the lab and

we pulled some suturing data from the O.R.
a small data set but.

What surprised me is this is a classifier
they had never ever seen live surgery

before and at least seventy five percent

of the time it was able to tell
us whether it was an attending or

a resident operating and this is a great
number to two class problems but

at least shows that there really is signal
there that's not just in the benchtop but

it's actually showing up in the live or
as well and so something worth pursuing.

So that's one model of looking for skill.

And again the idea here is that there's
not really a lot of structure so

if I were to think of it in that graph
it's it's really assuming motor warning is

the dominant thing and
when I'm going to look for

patterns that show up when you're
not skilled versus skilled.

So another way that we've looked at this
is to say well what if it's more that

cognitive path what if it's more
you learn some basic skills but

it's at how you put those skills together
that actually matters so we approach

this very much like a language model hence
the name language of surgery we said Look

why don't we try to first parse what's
happening in to in this case a variant

of a hidden Markov model and then we'll
look at the pattern of gestures and

decide if the pattern of gestures tells
us something about scale now it turns out

that if you just take a VIN hidden Markov
model like a speech person would use

that's not going to work particularly well
the data we have is too high dimensional

it's continuous data if you look at
what most people do there's a big

encoding pipeline that happens before they
ever even get to the modeling part so

we've got actually a family of
methods that we developed from

simple hidden markup models and we're
dynamical systems and eventually switching

when you're dynamical systems which
you can think of as a combination of

a continuous model that
models continuous motion but

then switching between
regimes as you go along and

actually a couple other variants beyond
this that I don't want to spend time on.

So these are results of
doing just classification so

taking so you just saw a classifying
gesture and skill this is just trying to

go through and classify gesture in this
case and you can see there is a whole

family of methods there's a paper for
each one of these so if you're looking for

something to do just start pulling papers
and reading to find out about each one but

the main thing I would take away is if
you look at the best system say ten.

To be the switching winner dynamical
systems do about the best of all of them

you can see we're doing about
eighty percent classification

I forget what chance is for
this case I think it's around.

Twelve percent of our berthing
it's eight or nine jesters So

we're doing well above chance not great
not perfect so you know we're saying

you know four fifths of the time we know
what's going on the other thing you still

see is this differentiation between set
up one is we've gotten to see a surgeon

before we do the classification and set up
two as we've never seen this person before

and you can see there's still a pretty
good disparity between these two so

you still see that we need more data
really to get to the point that

we can see someone we've never seen
before and understand what they're doing.

I just mention one of the things that
we've started to look at is how to

make these models a little bit more
precise one of the things we've realized

is the market models they're beautiful
model they're easy to optimize because

they have this Markov property that is
to say what happens in the next time and

soon it only depends on the time incident
before it it doesn't depend on some

arbitrary history further back but we're
getting our data at about anywhere from

thirty to one hundred hertz depending on
whether it synchronized with video or not.

And if you think about your hand motions.

Probably what you're doing you know from
time into Time instance pretty highly

correlated it's really only when you go
back a second or two that you start to see

the interesting interaction between the
gesture you're doing now and the gesture

you're doing before so we started to look
at skip chain models that actually have

a longer time history to them and
it's kind of interesting if you introduce

the skip chain models and you say what's
the optimal time history to look back to

get the best performance it turns out that
the skip length is about a second back so

if you actually want to look at predict
what's going to be happening looking.

Second back is actually a better predictor
in terms of where you're going to go

the thing the hidden market dot model
does is it actually is more like

a regular riser it says how long will
you keep doing the same thing so

these have some real power in actually
starting to help us do more prediction and

in fact we found out that doing using
script chain models we actually get

better results this is actually
a little bit of a different.

Test this is testing segmentation
as opposed to classification so

this is saying now instead of I'm going
to give you a gesture tell me what it is

I'm going to take the whole running
line and just going to break it up into

the individual gestures so as to classify
him and say with the boundaries are so

you get these little diagrams like this so
this is the ground truth of what gesture

was taking place and this is the
prediction of what gesture is taking place

so the same colors means you got it
right and this is the difference and

you can actually see that the differences
lie mostly just on the boundaries in fact

are getting the sequence accuracy almost
perfect one case here we actually missed

where we predicted a gesture that
wasn't actually there so we're actually

doing very very well at predicting
sequence of gestures and if you look

at the actual accuracy of labeling frame
by frame you can see now we're actually

up in the kind of in the eighty percent
range again but now we're doing a much

harder problem we're actually saying every
single frame what exactly is going on and

this is this is actually pretty much state
of the art I don't know of anything out

in the literature that does a better
job at doing frame by frame labeling

of what's happening so I guess the
takeaway of this is I think there's still

a lot of room to grow from eighty to
one hundred still a long ways but

we actually a pretty good tools now
that we're starting to see have promise.

So let me just do the last little piece
of this so I set this up by saying.

Do gestures matter if I just had
the sequence of gestures could I tell

you something about skill so
if you look at this data the.

Colors actually indicate gestures This is
a vocabulary we built with a practicing

surgeon so we went through we
label all the gestures and

now you can say well if I
take these gestures and

I just lay out what the transitions look
like what's what are they will quite and

you get something like this so this is
an expert you can see an expert pretty

efficient right they done this many many
many times before I actually I just was

recently talking to a surgeon friend of
mine and I was saying Man that must be so

stressful to go into surgery you know
a person's life in your hands and

all they say is you know actually it's
kind of boring because I've done this so

many times before OK occasionally
it's stressful but by and

large I've done this a thousand times
before it's not that stressful for me and

it's because they've really they've got
it down right this is that novice and

novice is being really inefficient
they're repeating things going back and

forth changing the sequence so

you can see there really is information
also in the pattern of gestures.

And in fact if you just collapses down
this is using multidimensional scaling

you can just pot every trial
that we've got on a plane and

you can see there is a pretty natural
progression from novices to intermediates

to experts so it does seem like the actual
pattern of how you use these gestures

carries a fair amount of information.

Now interesting Lee if you do
a bake off and you say OK so

what's better is it that first
approach where we do motor skill or

is it that second approach where we do
patterns of activity Well the motor skills

version just looking at low level motion
beats it but quite honestly not by a lot

I don't know that I think of those numbers
as statistically significant and so

the jury is still out and in fact probably
the right answer is that both of these

matters so you are obviously learning
motor skill as you become practiced at

this but you're also as you were in
these motor skill units you're become.

Better and better at applying those
markets go units efficiently and

really we do need to have methods
that that can account for both and

so this this D.C.C. method is actually
very good at picking up the low

level motor skill the H M M work
is actually much better at picking

up this kind of higher level
organization of how you do the work.

So I'd like to do just a little case study
on this just to show you some recent work

we did in this space so
this is septoplasty it's actually open.

So these docs came to us and they said
you know this is something that it's

it's called an index procedure it's
a procedure that you have to do

in order to satisfy your residency for
in this particular surgery

you're evaluated on how well you do it and
it's really hard to teach because what

it is is it's raising a small flap of
skin on the inside of the nose and so

once the surgeon is in there you can
kind of see the the working field appear

once the surgeon gets in
there whoever's watching and

can't see anything in fact even the person
doing the procedure has a hard time really

seeing a lot about what's going on so it
means if you're a mentor you can't teach

it very well because whoever you're trying
to teach can see what you're doing and

if you're doing it and you're the trainee
your mentor can't really tell you what

you're doing right or wrong and so there's
just a lot of verbal communication and

a lot of guesswork as to what's
happening he said is there some way

that you guys could could
actually help us with this so

what we did is we added some tracking to
their tools so this is Cottle is the thing

that they actually you do the manipulation
with it's a reference sensor and the head

and then again a sensor on the tools who
are getting patient relative to a motion.

Are this is the study cohort are going to
talk about is about fifty surgeries we've

just about doubled that now we don't
have the most recent results here but

you can see it's faculty
those fellows I talked about

a bunch of residents that
are all doing the surgery.

So one of the first thing to realize
is they had no way to tell us how

this surgery how do you
think about this surgery so.

The first thing we had to do is we
actually had to structure the data how do

you take all this data and get something
out of it and so the way to think of it is

this is the so-called septal
plane this is the the outside of

the nose itself all the work happens
basically relative to the septal plane and

the key thing is you're basically doing
these strokes to free the tissue and

in fact one of the first things we had
to do is we created a method to take

the continuous signal and
break it down into these strokes and so

you can actually see here the strokes here
going from blue at the beginning to red so

we pull these out automatically now we
take all the data and we have a way

of actually segmenting it into the strokes
and now we can start to look at how these

strokes relate to the actual
procedure itself and

this is what you start to see so
once again here's an expert so we start

with red and we move to blue so an expert
started to hear and you can see they've

got this pretty efficient pattern of
coverage they kind of free the edge and

then they're kind of working their way
back and eventually they've got the whole

flap elevated there's an obvious you can
see there definitely they're not nearly

as efficient and clearly they've been told
they know cognitively they're supposed to

start up here so you see these and
then they kind of work their way back but

somehow they drift down here occasionally
at the beginning and then they go

back because they're been told no no
go back up there and finish and so

on so forth so

you can really see there is that same
sort of pattern starting to show up

where we're starting to see the you know
this interesting difference in activity.

If you actually just
take one simple measure

just basically looking at the stroke
structure you can actually

start to really see
the differentiation between.

The We've got the fellows in the middle
the attendings are up here in the novices

are down there just looking at a bit of
coverage I think is the measure we used

here so now here is the question so
again can we kind of set this up as we.

Just takes some bulk measures of the
strokes like how many strokes they use and

what the structure of the strokes are we
could look at that motor activity level or

we could look at higher level patterns
of activity using H M M's and

so we just did a bake off and
here are the results so

this this is if we just look at
the structure of the stroke so

we're looking at things like rate of
coverage the consistency of the stroke

patterns and so on and so forth you
can see this is a two class problem so

these numbers are really not
particularly good sixty's maybe seventy.

We run the man mad at the H M M Actually
in this case just did not do very well at

all it wasn't able to pull out
a lot of interesting structure and

in this case the D.C.C.
method beat everybody hands down and

so this is a case where it seems like
this low level pattern of motion behavior

actually is the thing that's carrying
a lot more of the information than any of

the higher level structure that we saw
before again in preliminary results and

so this is kind of the space that we're
exploring now is to try to figure out

you know how for different procedures
how we want to think about this and

is it in that motor side
is the cognitive side or

are there other measures that we can
use but this is an ongoing study so

I checked these numbers will change.

So in this case you know I think we have
this tool to motion relative to patient so

we can basically say
here's how you're moving.

That's the extent of it now there are some
other things we could potentially work out

like they do flip
the tool occasionally and

some things like that that we
we haven't made use of but

it's a it's pretty limited I mean the
procedure itself is very limited and so

in some sense the data itself
is also somewhat limited.

You mentioned data I will say
one of the things that we have.

And working on is trying to make
sure that we're not the only

people that can do this and so just
recently we have gotten permission in this

case from Intuitive Surgical to start
to release data that we've acquired So

this is our first open data
set release called jigsaws.

And you can download it from the web
we've got the your all right here and

you know our only request
is if you do cool stuff and

you publish it that you reference right
now a workshop paper that we published

we actually plan to publish or we're going
to try to publish a broader paper that

gives all the comparative results that
we've ever achieved for this dataset and

the code as well so you will be able to
get some of the code some of the tools and

all the annotations that go with it so
hopefully we can actually start to

increase the potential of working
in this space going forward.

So I wanted to switch gears just
because I want to show one other thing

we've been doing recently and
this is kind of the bridge which is

I've been talking to you incessantly about
skill and recognizing patterns of activity

I've talked about actually doing anything
but a couple of years ago we realized what

we've got all this data could we
actually use it to do augment.

Surgery So this is a work in nickel
a priori where we we've built

a pattern recognition system we can market
model that can recognise the stages

of suturing and then what it's doing is
it's doing handoff between the robot and

human when you see the blue past showing
up that's automated It's actually

pulling the suture line on it's own
based on data that it picked up and

trained on and then what it's
recognising is also when it should

take over versus when the human
itself should do the procedure so

there is this handoff between
the human and the robot and

it actually makes a measurable difference
in terms of for example how quickly you

can do the suturing because you don't have
to do clots for our transport motions.

So I worked at that and I said Well geez
do we have to just think about surgery

there are lots of places where you could
imagine if you could sit down you could

get a lot of data and how people do is
skilled activity so this happens to be

an aircraft refurbishment
procedure spirit era systems if we

could get data on that could we start to
think of building augmentation systems for

these cases as well and
in fact we literally took this task and

you can see we sat down and
we headed to do the same task.

Just to put in a frame of reference I
think it takes a person about twenty or

thirty seconds to do that tie the Davinci
It took about three to five minutes for

us to get it so it's not as efficient yet
but we have hopes but it's not just here

is all sorts of places so you think of
manufacturing you think of rehabilitation

for statics they're all cases where what
you're trying to do is you if you're

trying to somehow capture the structure of
a particular movement or task that's being

performed and if you can capture it you
could either replicated in a robot or

you could augment a person or you could
collaborate with the person doing it.

And so we really started to say well how
do we think about putting these pieces

together in a broader sense so
Henry said I'm kind of a classic sensing

of robotics guy I kind of understand
how those things fit together.

What the project really just talked
about at it is this notion of data so

I would just surgery really grew out of
the observation that we can use robots to

get data and
we can learn things about what people do

of course in the sensing world you know
machine learning took over years ago.

Never to be lost again and so you know in
computer vision we're pretty we're pretty

used the idea that we have large data
archives with labels and we do machine

learning on it the interesting thing is
you start to add people to the mix so on

the robotics side this is called robotics
so it's part of the National Robotics

initiative on the people side you have
to start to think about reasoning about.

How people view context when they do
tasks like this and how the robot and

the person and the person basically share
a representation as well as sharing

the physical work now I just happen to
have students working on all these things

and about the beginning of the summer they
complained to me that they had a sense

that in my mind this all fit together but
they had no clue how it fits together and

they would really like to have a clue and
so we went on a retreat together

we had a bunch of good times we drew a lot
of pictures we talked a lot about it and

said you know what we want to do a code
Sprint this summer we're going to code for

two weeks and we're going to have all
these things fitting together well

the codes print turned into
a code mild to a code five K.

to a code jog through the woods.

But in the end they actually did it and
so they built a system we called COSTAR

which we just have started to
publish about but it really

does embody all those essential pieces of
how do you interact with people of how do

you include perception how do you actually
train systems to do interesting tasks

the core costar ends up being
fairly symbolic behavior tree

model of execution we like this because on
the one hand it's an executable model we

can actually program things and they will
actually work and on the other hand since

it's symbolic it segues well into be
able to recognise the semantic level of

what's going on in a task so we've built
this programming environment at the core

we have a symbolic interface for doing
programming we also have a perception

system that kind of standard object
perception also occupancy sensors and

a number of other things uses data for
that in fact you can see

here are you are five and you see there is
one of the earth to be the center is so

it's looking at the workplace environment
so we did this over the summer and

I said well look for
going to do this it's got a word.

For real so we have a small.

Manufacturing town employs about twenty
eight people does agile manufacturing

small lots of steel products called Marlin
Steel and he said you know there are all

sorts of things in my shop for
if I had a robot that my guys could

train in a few minutes to do a task for
they be doing it the reason we don't use

more robots is because it just
takes too long to deploy them so

that was the robot throw down I said OK
guys you're going to go down there and

you're going to figure out how this
robot can do something useful and

so they picked this machine tending
task this is a wire bender and

right now what happens is it's not worth
it for them to have somebody sitting there

holding the wire out and putting it on the
rock the machine itself can't do it and

it's not worth it to try to
program a robot to do it so

they just let the wire fall in the ground
for a few hours and then they have this

guy who is paid actually by eighty dollars
an hour come over and pick it up and

stack it on the rack and
that's that's how it happens today so so

I shipped them down there one day it was
about eight in the morning they arrived I

couldn't be there and I figured this would
be shall we say the typical grad student

experiment when they go from the lab to
the real world expected to come back with

their tail between their legs saying
it was really it was horrible

you know we got there and nothing
worked but in fact about eleven thirty

I got a video from them and
they'll show you what they saw So

this is just morrow and
some of the things that they do.

A fancy dancy wire bending thing doing
wire bending there's a person doing wire

binning because it's not worth end to
try to actually program this particular

wire bending task so
he just sits there all day ending wire.

You know one of their presses
pressed is the hard work of the SO

got to have a guy there that shoving
all the material into it and

then taking it out and
there is that little sensor there to make

sure his fingers aren't in
the machine when it comes down.

So there's costar and.

Oakmont.

Really.

This is in the lab just doing some set
up before that there we go there is our

robot at work so about eleven thirty I got
a video from them they said we impacted

we set it up took us about ten
minutes to train it for this task and

we let it run for half an hour and it was
just sitting there basically tending there

were bending machine and
I got to say the guys in the shop floor

were just basically they almost wouldn't
let them leave with the robot there I

just leave it here we'll
figure it out we'll use it.

It's not perfect there are a lot of
things that aren't ideal here and

to be honest it's not a hard task right so

there are other robots around that could
do it but I thought it was a need example

of being able to take this thing set up
a program in five minutes and have it do

something actually useful what's a little
hard to see is what's actually going on so

it's using occupancy sensing to
synchronize with the wire bender so

it grabs a wire when it shows up
not based on a time and all of the.

All of the different objects
have these markers on them so

if things move around we actually
can react to that as well.

So this is really the beginning of a
project as I said you know we're plugging

a bunch of things into it for example
we're building our own data set for

object recognition that's industrial tools
turns out most of the vision stuff doesn't

work well on industrial tools so
we've been developing things to do.

Another neat thing about
this is I showed it with you

our five which is a human safe
robot we also have developed

a way to program robots are human
safe using virtual reality so

this is a system we call Ivery where you
can take your favorite robot This happens

to be an American robot in our lab
that we train pretty routinely and

we just train it basically in V.A.R.
or an augmented reality you know

we can blend in for example point clouds
in the real world and go through the.

Training process so it's still very
much a platform in the making but

I've got to say it's really exciting
to see how quickly we've actually

been able to progress into building
something that is a coworker on the floor

where someone who may not know a whole lot
about programming better knows a lot about

the task they're trying to perform and
start to teach a robot how to do things.

I'll just let this run just a little bit
more just to see some of the different

modes of interaction this is just

it actually doing a task in the lab
that we've programmed it to do.

So let me just close and I'm running out
of time by saying you know I think you

know probably the take home message to me
is I think we've started to enter a really

interesting space to explore I think you
know used to be robots in vision and

then we thought about you know some
robots and people but when you start to

think about robots and people in data the
interrelationships in the things that you

can do really are remarkable in the fact
that this sort of data and these sorts of

interactions are just coming on line means
I think over the next decade we're just

going to see an incredible incredible
proliferation of these sorts of things.

And you know I think also it's
going to be interesting to see

how this teaches us more about how people
actually learn as well as how humans

learn because as we start to bring robots
on as apprentices the natural mode of

operation will be to say well how does
a people how do people learn a task and

have to get robots to do the same thing so
as I said I think it's a very

rich space to think about exploring not
just in manufacturing and surgery but

health care home assistance construction
and so on so with that let me just thank

my mafia to surgery crew the human machine

Calabria systems crew both again they're
the ones who actually do the work I get

to have the fun of coming out and bragging
about all the great things they do and

thank you very much for your time and I
really appreciate it and be happy to take.

A few questions before time's up thanks.

Yeah.

Doing this.

For.

My next trip.

So that one it was just her and
skills so we're it's identifying

what you're doing and
the skill with which you're doing it.

Yeah.

Yeah actually where we're inferring what
you're just what your skill level in each

thing is from the total thing so

there's a little bit of
a question mark however correct.

That's one of the challenges we have
actually is what level can you actually

do skill evaluation so
I skipped some work.

We're looking at maneuvers now
it's kind of an intermediate And

that seems to be a good sweet spot
between looking at a whole procedure and

saying When you're good you're bad but
we don't know where you're good or bad and

gestures which are pretty micro they're
good for the pattern recognition but

they're not good for
the human communication side.

You're very.

Right and so this maneuver a level
seems to be about the right level

we've done some work actually
crowdsourcing it shows that

actually people are pretty good at doing
skill evaluation at that level too.

Yeah.

So we have talked about that incessantly
of how to do feedback the only reason we

haven't done it quite honestly is a purely
scientific reason that up until now we

haven't felt like we're in a position that
we could actually say anything about that

we could certainly field something that
would give you have to feedback or

we could field something that
would give you video feedback or

pick your favorite feedback we
wouldn't be able to tell you if you.

We're actually getting better or faster at
it what we're doing right now is we have

a system set up in our training facility
and we've been collecting data for a year

now trainees going through the program and
the goal of that study is is that

to actually now introduce an intervention
where we would give you feedback and

see if we can actually see that there's a
change in the learning curve for doing so

so you're actually right we should be
doing it we've just we haven't done it

purely because we haven't felt that we
could say something about it if we did it.

First.

I don't think you'd ever
replace the person entire We

were at least not in my lifetime.

For the simple reason that there's always
variability that requires a lot of

context and judgement and the sorts
of patterns we're looking at at least

right now we don't have enough of that
surrounding contacts that you could make

adequate judgement so we can we can
certainly automate little bits and pieces

it's kind of like the manufacturing thing
right you know there are there bits and

pieces that are fairly
easy to automate because

you don't have to make a lot of
decisions it's fairly road versus

cases where you actually have to take
a lot of factors into account and

decide this is a course of action I'm not
going to take the other thing of course is

the question the question in my mind
is you know you get half a million.

Procedures a year it sounds like a large
number it's actually not if you think

about these sorts of issues but
of course that's been growing.

If not exponentially rapidly and

you start to accumulate that over several
years now you start to talk about millions

and millions of procedures there probably
are a lot of interesting things you can

mine out of that data just as we found
there are interesting things you can

mine out of the web once you
get a billion pages online and

they're things that are hard for us to
think of until we actually have that data.

Thank you thank you.