[00:00:05]
>> So Dr Our study bit of overlap on the almost 3 decades. Going to. Really go back to. Where the results were up the Silicon Valley for a while back and look at that it could be a good year again. But actually I think one of the joys of the book it is not just by floodwater do it with him and he will probably look.

[00:00:31]
Here's a photo of the guy tripling. Also. The American Institute for Medical biological engineering. And. Those are very expensive work grasping. That. Human life has them because of those of history were. An addition to that. What a person wrote he's been thinking it's all right I'm glad. You know what yes it's working in some of the most notable tech companies.

[00:01:13]
That try to write him about it. And I guess he could think of a better speaker borders for political. So we're very happy to have you here we both look forward to his talk on uniting the box office no record there. Were Thank you Frank I'm I'm impressed I can't remember what stuff about myself without looking at my notes so you who were very generous in that introduction and I thank the organizers I have never been to Georgia Tech before so I'm glad to.

[00:01:50]
Break that ice and see the amazing work in robotics and biomedical engineering here. So today I'm going to talk about some of our work on cardiac surgery and in particular trying to solve some tough problems in cardiac surgery so this is trying to do repairs inside the heart of things like fixing atrial septal defect this is a hole between the left side in the right side of the heart appears at both in a fair fraction of people and it can be repaired by simply sewing a patch over this hole now the sitting on the patch takes 5 minutes but there is hours of surgery involved because you have to open the chest.

[00:02:34]
You have to connect the blood vessels to the heart lung machine that circulates the blood and oxygenates it during surgery. And then you can stop the heart make your incisions and quickly fix that that hole and that process of opening the chest stopping the are using the heart lung machine is much more invasive much more destructive of healthy tissue than the actual repair itself so we've been asking the question how can we do that kind of repair inside the beating heart without the need to do all that invasive process and there are 2 main challenges you have to overcome 1st the blood is opaque you can't use the video cameras that have enabled so much minimally invasive surgery and other parts of the body but the solution to that is ultrasound guidance or ultrasound it's been a useful clinical technology for decades but in just in the past roughly 10 years there's been some interesting developments true 3 dimensional real time ultrasound so these are clinical scanners that can give you all 33 d. Volume 30 times a 2nd and that 3 d. information we believe can be used to guide surgical robots the other part is that the tissue motion is too fast so as the heart beats the tissues moving up and down centimeters and we talk about how fast it moves in a little bit but that's too fast for humans to keep pace with and so that's where robotic technology comes in and we're able to do that motion compensation even though we're working through very small incisions that do minimal damage to healthy tissue.

[00:04:06]
Ok sail over to a sound I mentioned is potentially useful here but we discovered very quickly that using ultrasound for guidance incurs some some real problems. So this is an ultrasound image it's a very simple. Situation although you probably can't figure out what it is because of all the artifacts so this is the situation the scan head will sound probe is imaging the sector down here and we simply have an instrument a stainless steel rod about 4 millimeters in diameter inserted into the field Now this doesn't look like that at all you do have some sense there was a surface down here because ultra sound works by emitting pings a little little sonic pursues they travel around bouncing off the surface and then the echo returns back to the scan head now.

[00:04:55]
That scan head then recognizes that the speed of sound is about constant so the travel time is equal to the distance and it then shows that intensity at the appropriate distance so here we get a nice return this is a mirror like surface but over here the surface of that instruments missing and the reason is it's merely like so the ping travels down here and then heads off for left field and they return back to the probe you also see this sequence of right lines down below the actual surface and that's a reverberation artifact so the matter of the instrument absorb some of the ultrasound and then every time it rattles back and forth the acoustic bullets goes back to the top surface it emits another little pulls that's going to delay it so the machine decides that must be a little further away same with the next one and the next one and the next one and the next one there's also problems here at the tip you can see there's a deflection artifact passing in this direction so it's very hard to insert rigid instruments into this field and get good images unless you're very careful and we've looked at this and come up with a number of techniques for dealing with this we can do things like apply surface textures.

[00:06:05]
In rough on the surface or instruments then you get a speculum out reflection and so things like this missing surface reappear You can also add coatings and do image processing to get rid of artifacts and it's interesting how we can do this we started working with this we some of these we talked some acoustic researchers who said yeah that's very familiar we have seen that before insane are so brave Thanks for the acoustic energy in clinical to sound about a millimeter in sonar it's about a meter our instruments are roughly 10 millimeters in diameter Russian submarines are about 10 meters in diameter and said the same kind of acoustic artifacts we see in this situation also appear in San are of submarines so we ask what the solution is and they said we can't tell you it's classified so we have to come up with their own ideas here and these happen to work pretty well you'll see later the images look a lot better looking forward.

[00:07:05]
So then let's see if I can get this video to play. It's not going to play for me so we then proceeded to do some of these simpler procedures that we show that we could in fact. Put a patch over one of those atrial septal defect holes using minimally invasive techniques and that proved to work pretty well but then we went on to the real challenge of fast moving tissues and in particular repairing valves like the mitral valve so this is a 3 d. ultrasound image you can see it slices through the middle of the valve and the module that was a flapper valve blood comes from the left atrium from the lungs goes in the valve opens up it goes into the left ventricle and as the Left federal squeezes to force the blood through the body that fell slams closed and that closing of that valve is actually the heartbeat the majority of the sound you hear for the heartbeat Ok so the valves are very important if that doesn't close successfully instead of being forced out through the body some of that blood is regurgitated as they say back towards the lungs so it cuts efficiency leads to pressure imbalances can lead to remodelling of the heart and eventually heart failure so the way this is typically repaired involves one part of it is fixing the mitral annulus So this is the ring above where that flapper valve closes if the valve annulus has spread out with age or disease then the leaflets don't need any more so they can reattach this ring on the top here you can see it sewn in using a very amazing manual surgical technique in this case and that tightens up leaflets so now they meet the valves seal successfully so we'd like to be able to do that using our minimal miss of robotic approach we've developed during deployment instruments so this is a hollow tube this ring is in reality it's pushed up inside the tube once we deployed inside the heart we can push it out so it's now ready to deploy and then this is an anchor driver it's basically a stapler it can come in staple this into position around the annulus.

[00:09:09]
And that technique looks good but the problem is fast motion so this moves up and down something like one and a half centimeters in something like 100 milliseconds so it's much faster than humans can track so this is where robotic assistance becomes useful. So this is a block diagram of the prototype system we've put together to address this challenge and starting here this cartoon is supposed to be the heart this is the mitral annulus that's bouncing up and down at a high rate and here's our instrument coming in to access that our 3 d. ultra sound system is taking sound volumes 30 times a 2nd and pushing out into image processing computer we need to be able to track where the tissue is going we need to be able to track where instrument is with respect to that tissue and then we need to have a controller so that as the tissue moves the instrument keeps pace with that and we do motion compensation motion matching So let me quickly mention that motion compensation in surgery has been around for years but almost all of it has been concerned with coronary artery bypass So this is the heart bypass surgery application you'll notice on the outside surface of the heart you don't have to work through the blood pool you don't have these minimal access problems so it's really a very different problem than what we're trying to undertake.

[00:10:34]
But let me go back to our system here and start by talking about the tissue motion and this is an example of some of the ultrasound frames and we needed to accurately 1st detect how that annulus moves it turns out this had not been previously quantified because nobody worked on beating hearts and so you didn't need to know how it moved so we had ultrasound sequences of volumes we had to go through and accurately track how that annulus displaces and we used a very inexpensive very accurate method for doing this it's called undergraduate research assistants so we would get them to do 2 or 3 datasets before they insisted on doing something more interesting but we got good data and I'm afraid this is got a kind of a bug let me see if I can restart it here not so much Ok so this is the resulting pattern of displacement that annulus and here's what the data looks like.

[00:11:38]
Once we've distilled it down and let you start this video. That works Ok so this again is the mitral valve you can see it flapping open and closed and these spots are where the annulus the ring at the top of the valve intersects this imaging plane and you can see the good news here is that it's basically linear motion if we do a principle component analysis we find that about 90 percent of the motion is actually within a single linear trajectory there's a little bit of side to side motion but it's pretty small that makes a robotic task much easier our fast robot system only needs to compensate for one degree of freedom.

[00:12:22]
The bad news is the speed this is the. Motion pattern for that one degree of freedom and you can see from this point here to this point here it moves about 15 millimeters in 80 to 100 milliseconds so the bad news is that we have to have a very fast robot in order to compensate for that.

[00:12:46]
So now we know what we need to do to build a robot to compensate for that motion Now let me talk about how we're going to track the instruments inside these ultra sound volumes. Ok so this is a tough problem you've got a messy ultrasound image there are.

[00:13:08]
3 dimensional they're delivered very quickly but unfortunately they're noisy ultrasound volumes are intrinsically a very noisy imaging modelling. We also have all sorts of structures moving around all over the place but there's one feature that helps distinguish these instruments and that's that they're long bright and straight they reflect very highly Here's one with the tip artifact I mentioned earlier but you can see that if we can find a way to detect bright straight lines we're able to locate these instruments and the approach we've settled on uses projections so here's the situation here's our instrument shaft we take our volume and we projected into a 2 d. image if we look at it from the side Ephraim instance we see a straight line but if we look at it straight down the shaft of the instrument we see this little crescent shape and the reason is the ultrasound acoustic energy is coming down it bounces off the top surface and returns back to the scan head and so we only see the top we don't see the bottom it's in shadow acoustic shadow now if we can detect that shape in the images then we'll be able to know which projection is the one that aligns with the instrument shaft within our volume to do that we use rate on transform Now you might have run into the rate on transform in the context of medical imaging namely c.t. scans where the inverse rate on transform is used to go from various x. Ray projections back to the 3 dimensional reconstruction Now the to the version of the rate on transform says I have a line I can parameterize by 2 numbers one is the distance from the origin the perpendicular distance and the other is the angle for that perpendicular connector back to my x. axis of my coordinate system so here you see 2 lines these are 2 strings in an ultrasound image we took a water tank put across pair of strings and took some images and each one of these then in our rate on transform space represents a single bright spot.

[00:15:16]
So now I can run this algorithm on my 3 d. volume and locate where the lines are however 1st I need to come up with a slightly different algorithm remember we're working in 3 d. if you take the 2 d. rate on transform and use the traditional 3 d. version finds planes in 3 d. were just in point so we had to change things a little bit but at the end of the day we can compute the location and find this now we use a course to fine approach in a g.p.u. so we can break it up into a set of parallel problems we take each volume we break it up into 32 voxel cubes and within each one of those we do our projection trick at 10 degree increment so we do 10 degrees of longitude 10 degrees and latitude and we look for bright spots we then select the sub volumes with the brightest projections try to line those up and then we come back and do a localization operation where we do one degree projections until we find exactly the place where that shaft is located and then the final step we do a correlation we know that we should have this crescent shape for the reflected pattern and we are then certain that we've come up with the the shaft of the instrument so as a demonstration here are 3 in vivo images this is with a straight shaft inside the heart you can see if you can locate where the instrument is located and that's where our algorithm said it was located and in fact it is highly accurate it gets essentially 100 percent of the time successful identification of up to about 5 instruments within the heart in real time.

[00:16:56]
Ok let me now go on and talk about the robotic piece of this motion compensation instrument and controller and again I mentioned we can basically just work with a single degree of freedom a single axis and we started out with a handheld instrument you see here this is kind of Black and Decker for surgeons it it gets inserted through the top of the heart to look down on the mitral valve so this would still require opening the chest but it doesn't require the heart lung machine patients are usually most worried about the large incision the sternotomy in order to get to the heart in fact that a real danger in open heart surgery is due to the heart lung machine that has a high incidence of not a high incidence but a significant incidence of stroke and cognitive impairment as they say meaning number of patients are not quite as smart after surgery as they were before now the heart lung machine is a life saving device that a lot of people around who wouldn't be without it nowadays but of course if we can do this without that we'd be happy so even using this approach sorry.

[00:18:11]
I don't know what's going on you have to give me a 2nd folks very sorry. Just a Power Point crashed in India. Ok. Don't ask why to speak a lot of came back Ok so we have this handheld instrument and the idea is again you'd insert it you'd open the chest in sort of through the top of the left atrium to get down to the valve pretty simple idea here it's a.

[00:19:29]
Voice coil motor a linear permit magnet motor linear displacement sensor low friction slide and then we attach this instrument this is our anchor driver here is the actuator surgeon pushes the button with the thumb the staple is driven into the annulus. And later we converted this to a catheter based device using essentially the same.

[00:19:52]
Instrumentation the same drive package but in Splice of the rigid rod we now have a catheter that we can insert through the blood vessels avoiding the need for the sternotomy we ran into the problem then of how do we aim this and initially we thought well we can do image processing on the mitral annulus will be able to detect where it is but then we realized we'd need to know a lot about the procedure which part of the emulous does the surgeon want to staple down how would we figure out what they're going to do next how do you figure out if it's safe and so we realize the smartest thing to do is just have the surgeon point it themselves which works out fine but then you're well left with the problem of how do you define the surface that you're tracking so in this case we have this idea of the instrument shaft remember we've identified that using our Radon transform approach we can then detect it's intersection with the tissue in the ultrasound image and then we control the motion of the robot to follow that.

[00:20:54]
And here's the results you can see here let me talk through the picture so this is a slice through the ultrasound volume here is the tip of our instrument you can see there are some artifacts here at the tip and this is the motion of the mitral annulus and you can see that as it moves up and down our robot successfully moves up and down as well.

[00:21:20]
In Vivo results so this looks at manually segmented tissue motion and our ability to track it you can see we're very close the error here are a mess areas under one millimeter for doing that tracking we've also used for sensing we built a 4 sense of that goes on the tip of the catheter and now we're able to do force control to better than about half a Newton one Newton half a Newton sort of range to give you an idea of the benefits of this here are results from a couple of trials for and in vivo large animal trial.

[00:21:56]
We drove one acre using the motion compensation so the robot servo weighing were able to make smooth contact we drive the anchor works well here's another one where we turned off the motion compensation and asked the surgeon to try to drive that anchors best they could and they did drive an anchor it wasn't exactly where it should have been it should have been next to this down here but more importantly there is a rough patch up here.

[00:22:19]
And. That region is where the instruments scraped against the annulus of the mitral valve repeatedly while trying to get oriented correctly in order to drive this so using this motion compensation system we're able to get better results. We've demonstrated that 3 d. ultrasound can guide procedures and that robot devices driven by the ultrasound can match that tissue motion and now I will say that going forward safety is one of the big challenges so we have software written by graduate students that's near a fast actuator which means it has to be a strong actuator near life critical structures inside a closed chest what could go wrong so figuring out how to make this kind of system robust to all the different problems that that can happen is going to be essential before this becomes useful in a widespread clinical setting.

[00:23:21]
Ok I'm going to shift gears here and talk about some other more. Some work we're doing right now sound and the heart and this one's a little different so instead of catheters that perform work inside the heart we have catheters that have otro sound imaging probes in the tip of the catheter and this is important because there are only a few places you can get good images of the heart you can try to go between the ribs very narrow access regions the muscles tend to aberrate the image you can try to go underneath the ribs that gives you a viewpoint from the apex of the heart or there's actually a point where the esophagus from your mouth passes right behind the heart and so there's a trance esophageal probe technology you can put that down the patient's throat park it right behind the hard to get a pretty good image but again it's only one viewpoint the event is to these imaging catheters is you can introduce them into the heart through the blood vessels from from the legs from growing and you can get fantastic images of anything you want in the heart it's near field it's right next to the imaging and so you can do well.

[00:24:30]
Now. This will run. One problem is and it's not going to work for me let me just talk through this so these catheters have been available about 15 years commercially they're used in some procedures but their use is limited because they're so hard to aim they're very difficult to point I've seen a cardiologist spend 10 minutes just getting a single clear view the way you you run these things is there are 2 knobs one of which bends the tip of the catheter in the imaging plane the other of which bends it in the perpendicular direction now in addition I can grab the handle and rotate it that'll spin the whole catheter in its imaging system and I can insert and retract and that will move the imaging back and forth now you can imagine the situation a surgeon is holding the handle of the catheter here down by the patient's leg it's threaded up into the heart looped over in some bent tangle Meanwhile they're looking at an ultrasound machine on the other side of the operating room and they're getting a slice through the heart of some arbitrary angle they don't know they can't see through the chest so they don't quite know what's going on so this is an incredibly frustrating complicated process of aligning this imaging plane with exactly the tissue structure you're trying to look at so robotics to the rescue here is our ultrasound catheter robot and now in the middle of all this there is one of these catheters these ultrasound catheters.

[00:26:00]
And we built this machine to interface with this catheter because this is a commercial f.d.a. cleared product it's used every day and rather than building a new catheter which would be much easier to control we elected to go with a device that would interface with the current clinical device saving ourselves a lot of that trouble.

[00:26:21]
And there are 2 tasks in particular you'd like to be able to do with these catheters and one of them is instrument tracking so in order to treat a with Mia's electrophysiologist these are cardiologists who do procedures to get rid of a rhythm Yes Will a blade tissue that is the program figure out where funny signals electrical signals in the heart of coming from and then they'll use radio frequency high energy discharge electricity in order to burn that part of the heart to shoo so that it doesn't generate erroneous harpies very common procedure done hundreds of thousands of times in the u.s. every year the problem is they don't have a very good picture of how that instrument is interacting with the heart tissue they can use alter a sound like this interim cardiac Ethyl ultrasound probe we've been talking about in order to image that but it's so difficult to align that they only do that for the most challenging most critical parts we'd like to make that automatic so that as the clinician moves around their ablation catheter the ultrasound imaging probe automatically follows and keeps it in view this means they can see every single burn that they're getting it in the right place the other thing we like to do is panoramic imaging again you only have a few small windows where you can image the heart we'd like to be able to put our catheter inside the heart and then move in one degree steps and build up a complete 4 dimensional panorama 3 dimensional plus time the heart cycle of entire volumes of the heart perhaps half the heart at a time now that's not too hard to do if the catheter is straight you simply rotate the catheter but remember it's bent up and around it's at some funny angle so now in order to keep the probe in one position and get these images I have to rotate it then I have to do the bend angle to get back to where I want to be so you have to coordinate all 4 degrees of freedom in order to do that.

[00:28:22]
In addition we discover there's a major challenge with our. Approach we can easily control the tip of the robot on the laboratory bench we have control of those bending degrees of freedom the rotation and all that but as soon as you put this into an invisible situation a large animal or human everything changes there are 2 problems my bending I can control what happens at the tip of the catheter but I have no control over what happens with the shaft of the catheter between the handle and where that bending section occurs and as a result it moves around so we can think of this is a robot but it is one of the worst robots you would ever want to control the mapping between what's happening back at the controllers the motors and what's happening at the tip is very difficult to describe it changes because it's plastic it ages it responds to history a response of temperature changes so this is a big challenge for you know how to control this in addition there's physiological motion the heart is beating the lungs are breathing things are moving up and down so that's another challenge for getting that tip and our imaging plane in the direction we'd like it to go.

[00:29:38]
The solution we wound up with is to use multiple sensors we have little electromagnetic sensors these are little one millimeter diameter probes and they can report the position to within better than a millimeter and the orientation to better than a degree within the human body and these are used regularly in catheter based procedures in the heart but what we had to figure out how to do it was to add a 2nd one so now we have one of the tip it tells us where the tip of the robot catheter is and we have one at the end of this bend section at the tip so now we know about the motion we can't control so if the base gets moved by physiological motion or the path catheter imperfections we can compensate for that by moving the tip the kinematics in control get pretty complicated here so in.

[00:30:33]
Robot kinematics we talk about joint space So again these are the motors or in the manual case it's the clinicians hands the do this in our robot the robot Motors move these around and that gets mapped to this configuration space that says how much is the tip bent and what's the bending plane and then there's also this task space I'd like to control x. y. z. position and where my image is pointed at the end of the day and we can work out the relationships between these but these imperfections get in the way we have what's called Shaft wind up so I have the catheter going in through the blood vessels in the grind snaking its way up the Nikkei of a bending around the corner coming into the right atrium for instance.

[00:31:23]
I rotate the handle the tip doesn't rotate until suddenly snap it moves 10 degrees at once when I've been moving it one degree at a time the the wind up of the elastic shaft polymer shaft causes that problem the other problem again is vessel motion so I I think I know what the configuration space is but then breathing motion heartbeat motion moves the tip of the catheter so the tip isn't where I originally wanted it to be and went to a lot of trouble to figure this out I'll spare you the very involved technical details of how this controller works it uses a combination of kinematics and various corrections at the end of the day though we're able to get it to.

[00:32:06]
Obey commands so here for instance this isn't in vivo experiment you can see there are little disturbances due to the heart rate we are going to worry about those but we have large scale disturbances and here we have the step disturbance we initially get some air and then you'll see we're able to return back to the target position here we have a change in the commanded position and once again across the course of a few seconds we're able to reach that new target position it's not designed to be a fast controller you generally don't want to move these very fast within the heart since you're just doing imaging but at the end of the day we're able to do this and now.

[00:32:45]
Let me show you a video here of trying to show you video here we go of this in action so here we're doing respiratory compensation Well we're doing instrument tracking. So here is the. Breathing of the animal These are the motions that we can expect to see. We have our ablation catheter inside the heart and we're imaging it automatically we have an electromagnetic sensor in the tip of that catheter we know where it is to point the imaging plane through it here's the imaging plane this blue blob is a trace of the location of the tip of that catheter that's what we'd like to be able to see now the clinician moves it and the controller automatically recovers it you'll see this happen several times here there it's been moved there the controller caught up to the catheter again it's given another minute there it's been moved and once again the controller catches up to it so now we're able to keep that catheter in view during the procedure we're able to see all the errors that can occur Ok So in conclusion we have by using a combination of sensing at multiple points in the catheter and kinematic and error based controllers were able to compensate for a blood vessel motion and respiratory motion and we're able to get about one millimeter and half a degree precision in doing our control.

[00:34:25]
Ok So with that let me be sure to. Before I mentioned summarize the accomplishments here I want to be sure to give credit these are complicated experiments complicated systems and it's been a a large collaboration involving clinicians that children's hospital and. More recently at Beth Israel Deaconess Philips Healthcare has been instrumental in helping us with the ultrasound technology and of course my students and postdocs who worked hard on this as well and so to summarize the end of the day we're able to show that by combining real time ultrasound and robotics we're able to do sophisticated repair procedures inside the heart of the cellos you have to overcome our very fast image processing needs they have to happen in real time as the images come off the scanner and the images are very low quality which means a lot of headaches for getting the image processing to work well the robotics has to be very fast even though it involves minimal access this can be done but the big outstanding challenge in all this is safety going forward and with that I like to thank you for coming today and be happy to take any questions.

[00:35:57]
Really. Yeah. Yeah it's. It's a good point but let me. We made a decision so there certainly are ultrasound researchers who take the megahertz frequency acoustic data and yes you can do more sophisticated things than what the ultrasound scanner the machine is able to do. That is a huge overhead and we made the decision in these projects that we wanted to work with the clinical technology we wanted to work with the kind of images that come off these machines f.d.a. cleared machines understood accepted by clinicians and well it may be possible to get some leverage by going back further up the pipeline to the acoustic signals we thought that that would incur 1st of all much more work but 2nd of all much less clinical acceptance would make clinical acceptance much more challenging.

[00:37:45]
So. That. You know you're good so I didn't talk about. A lot of the predictive control work we've had to do so fortunately the heart beat is pretty repeatable not 100 percent but largely repeatable and so we use an extended Coleman filter to do look forward estimation and in particular we need to do that because we have the process of scanning out the volume shipping that over either net from the ultrasound scanner to our image processing computer which is also known as a video game computer because it's got nicely integrated g.p.s..

[00:38:34]
And I don't know what my students do with it after hours I don't need to know. And then do our algorithms detecting instrument detecting the tissue and all that that kills about 70 milliseconds and during that time the tissue can move one wonder want to have centimeters so again what is the extent of common filter to do an estimate and to drive the robot to where we think that issue is now based on 70 millisecond old data it works pretty well I I had an odd bit of luck in this the students felt un who undertook this.

[00:39:10]
Was very familiar with doing this kind of estimation using extended common filters he had formerly worked at. Mit Lincoln Labs doing radar tracking of ballistic missiles and it turns out exactly that same algorithm for tracking ballistic missiles with radar was useful for predicting heart motion with ultrasound some Sometimes good luck.

[00:39:36]
Yes. Yeah so we put a bunch of work into demonstrating robustness So we showed that. The heart variability could be up to something like 20 percent and we would still accurately track. The beat to beat motion and we proudly showed this to our surgeon collaborators and they said.

[00:40:08]
Well if that was a problem you should have told me because we can administer drugs that suppress the natural heartbeat frequency and then pace it so it will fire exactly when you tell it to fire every time and we can get rid of all a very valid. Which is fine I think at the end of the day we're happy to have this capability 1st of all robustness is a good thing in a life critical system and secondly.

[00:40:36]
What what they describe this automatic pacing in all. Is more challenging in a catheter based procedure when you open the chest expose the heart the pacing is simple but doing with a catheter would add another layer of complication so I think at the end of the day it wasn't wasted work at least that's what I tell myself now.

[00:40:58]
It's. Like geez. You me. Yeah so it's a very good point. We do we deal with that in a couple of ways one way is we put electromagnetic trackers on the outer surface of the chest and so that tells us how. The physiology is moving and we can then use that to adjust our targeting position as well so well.

[00:41:36]
Simply trying to keep the same position in 3 d. space means Yaz the breathing occurs as a vessel motion occurs we're not able to keep the same physiology in target the same anatomy and target by take into account these external sensors we are and we're able to keep and it's not precise you know the good news is the imaging plane is about 2 millimeters thick and so if you're off by a millimeter or so it doesn't matter you're still going to see the same anatomy at the end of the day.

[00:42:03]
So. You know every now that you have it as you. Yeah so when they're doing is a blatant procedures. I won't say they're crude because my clinician colleagues who kill me let's just say they're imprecise they don't rely on a high degree of precision. The idea being that when you do a burn it's got maybe 4 millimeter diameter and so if you space these things a few millimeters apart you'll you'll succeed in killing the tissue where you need to kill it so that degree of precision is not so important what we're worried about here is is relative motion between the catheter the the ablation Catherine the tissue they try to push against the tissue as the heart moves they start burning these burns take 30 seconds to 2 minutes.

[00:43:02]
If there is sliding motion if the contact isn't good then instead of all that energy being delivered to one spot it's now spread out and you may get a temporary cessation of the a and the Mia patient goes home though that section now heals you haven't really killed it and the a rhythm you know recurs and for some of these procedures they only have about a 50 percent success rate like left a trail of blood and so they get a lot of people have to come back a 2nd time for procedure so that's where the real leverage is we think is that we'll be able to see do you have good contact as the tissue moves to the ablation catheter keep upper isn't sliding and if it is just the ablation catheter so you get a good contact.

[00:43:47]
No no they have you're inside a pretty good sized vessel so you have room to move in there yeah yeah. With. The base. There. Really. Yeah it's of course extremely complicated you know it's hard to generalize. I know that some of the 1st surgical robotics projects I worked on I solved the problem technically and they've never will be adopted clinically and there's a culture there's a workflow there's a reimbursement mechanism and if you don't hit all those things technical solutions are meaningless and so you learn to sit with the clinicians and talk about what they're willing to change what they're not willing to change what's expensive what the insurance company will reimburse you for if you want to change any one of those things what they like to do the way they're used to working with their colleagues the way they get paid it better be a remarkable improvement in either efficiency or an outcome in order to undertake that heavy lifting and I'm not brave enough to do that I now adapt to that what the clinician wants to do.

[00:45:37]
I think that's pretty much surgeons. You know understandably they're doing something incredibly hard. It's it's life critical you know they have a bad day at work and it's bad for the patients. And so their natural conservatism is warranted and you know you learn to respect that and work with it.

[00:46:04]
Although there is some interesting example so pedophile Nido chief of cardiovascular surgery at Children's Hospitals are collaborating a lot of this beating heart stuff. And he has been a huge collaborator advocate of changing these procedures from open heart repair to catheter based repair now the catheter based repairs are done by cardiologists they're done by interventional cardiologist electrophysiologist and nationally over the last 20 years cardiac surgeons have seen their case loads decrease as the intervention of cardiologists and inner and electrophysiologist have been able to do more and more with catheters and were not taken away another big chunk of their practice and turn it over to well I won't call him competitors but but not their own service but he is a huge advocate of that he recognizes better for patients it'll be cheaper it'll save the health care system money so even though he's kind of undermining his own caseload he's willing to do this there there are a few exceptional people like this who you want to collaborate with if you can yes.

[00:47:24]
Yeah so there are 2 different projects the 1st project the beating heart project there we have to have these predictive models these extended common filters. That we train up on previous cardiac cycles so they know what's coming for the next one. And if you combine that with the p.d. that turns out to work Ok.

[00:47:43]
For the imaging catheter where your robot is a plastic noodle 80 centimeters long that is an incredibly complicated controller there's a little p. i.v. built in the middle of that but there's a lot of model based control out of air compensation multiple sensor inputs I spared you the details of how to go over them there in our papers.

[00:48:07]
But but for that one no id is not accurate enough it's fast and it's just doesn't work. Right. Well. Yeah we do about a minute so 50 cycles. I'm happy to point you at the 1st author is un y u e n who's now head of r. and d. if it.

[00:48:46]
So he's still tracking ocean. Yeah. I think the. The catheters that are purely polymer most catheters are just plastics they're very sophisticated they're they have many concentric layers they have fiber diet fiber stiffening elements for rotational stiffness they're they're very sophisticated devices. And even so they're really not adequate for robotic control I mean they're right on the edge we can make it work as far as an imaging catheter goes where there's some you know forgiveness about where the imaging plane is for actually doing interaction with tissue I think we should be going to stiffer rather than softer catheters for these kind of applications that it's it's really tough to control a wet noodle and stiffer is better.

[00:50:04]
It will. Hopefully by the time I need cardiac surgery. So the beating heart surgery is very interesting. Our technology licensing people and and me have talked to a bunch of companies to talk to surgical robot companies talk to catheter companies talk to medical imaging companies. And the response is pretty uniform is like well it's a really good idea at all and that's a great approach and the 2nd thing they say is we can't do that so it's it really is this very.

[00:50:36]
Tightly integrated combination of real time medical imaging and image processing of catheters and fast robot control there is no system like this today in medical robotics. And this tightly integrated system is really beyond the capabilities of any of the current players in the Medical Imaging people the robot people any of that and so they look at this and they don't see the people with this kind of capability that they have on their staff who could who could develop is one of them said something about that looks like a $1000000000.00 development project and I have no idea if the market is nearly that big so where we're going in the meantime is we're taking the real time 3 deal to send image processing and we're trying to develop standalone clinical applications so the images are garbage surgeons are not good at looking at these 3 d. ultrasound images and understanding what's going on there echo cardiologists the radiologist who specialize in this there they're good they know what they're doing but they that's what they do all day surgeons look at this for 5 minutes before the case and then they go in the o.r. So we're trying to develop tools for instance that during the procedure or just before can present a cleaned up image showing them exactly what's happening with the anatomy so they get better insight into the path ology and into how to do the repair so the hope is that once we get that technology real time robust clinically accepted f.d.a. cleared all that stuff that piece will be in place and then there's some hope to start bringing in robotics maybe not close loop maybe not the fast beating heart stuff but where you can now start coupling the image guidance to the to the robotics in order to start to close that loop and then again at some point hopefully will be able to do this where we can do the repairs on the beating heart with a close chest minimally invasive you know have your valve repair and go home the next day and it.

[00:53:01]
Yeah so I agree with you that there are real safety issues and there are regulatory safety issues and they intersect but they're probably not the same and I will say I have no expertise in this that certainly we have not made. High reliability a priority that's not something you publish it's not something you get research grants for.

[00:53:26]
So it's kind of beyond our scope but we recognize that before we could even start to do clinical trials with this we have to address this in a in a pretty serious way and there are academic researchers who do this you know safety engineering for a living some of the specialized medical devices so that would be the next step were we to try to get this into clinical trials.