[00:00:05]
>> So I want to do. Just touch on a bunch of different topics as covered so that spanned the space of the things we do so it is navigation and mapping So we'll be talking about planning talk a little bit about some of the global control through we're doing and also be thinking about some of the work we're doing on slammin data Association.

[00:00:24]
And I see here many collaborators you'll see the names as they go along the students actually put some q.r. codes in there as you mentioned some of the work does take a snapshot should take you to the paper. What you do want to do to start off those talk about the things I'm not going to say so there's actually a ton of work that we're doing and over the past couple of years and that's focused on a bunch of different things you might have seen actually published already so this was like planning in a way to drive a robot in a pressure environment so the question here was not so much how to do collision avoidance you know if you have a good light or availability is pretty easy with the things or there's always a question of where is the stuff going to go so there's a lot of intent prediction prediction for pedestrian paths things like up part of what we're doing here with the socially acceptable navigation so it was thinking about what should a robot be behaving in a world that doesn't really annoy the other people that are in the world and so one of the things that ends up happening here in this video is we actually end up coming up to a coffee station and there's a woman there actually putting coffee in her criminal coffee and the robot sort of essentially going through the conversation or the question of should I barge past her or just wait till she's essentially finished and essentially had at the end up working through that whole question what part of here you'll see is just you know trying not to do pinch points for robot for other humans in the room with things like got.

[00:01:45]
This workout right up here this is actually. They were showing me the lab you guys have downstairs for flying around indoors without a large bike on space over the years were sort of try to wean ourselves off like on a sort of like having g.p.s. indoors is fantastic it's great for doing experiments but on the other hand you kind of get lazy and it be great not to have to rely on it so you start thinking about vi Here we're doing sort of a vo swarm some of the top work is going to you pan so each of these vehicles now is actually trying to look at the ground do visual inertial Adama 3 figure out where they are but then actually now it's shear.

[00:02:21]
Between the vehicles themselves and you'll see the huge what looked like a fix on the ground is actually to make this ground very interesting when these you realize when you're doing this visual base lamb is if you're not looking at something is very interesting it's very hard to know where you are and it turned out we did a fantastic job of making the floor of the building beige and interesting to look at so the students went down and put the hard Guilford tape and started ripping the tape up ripping the tape in the paint and completely ruining the paint job on the floor.

[00:02:52]
So this was. R.l. So now we're actually explicitly sharing learning things so what are you doing Arielle you can learn things about the description of the world is what what I've seen what have you seen Let's fuse that together I'll talk a little bit about that as we go along but one things we're learning about here was actions and so it sort of I might partially know how to do something you've partially learned how to do something so we're now thinking about things on the policy side and the question then becomes how do you actually share knowledge on the policies how do we take partial learned knowledge on how to complete a task and the merge them together and that's part of what's going on here so there's actually one team learning on this side one team learning on the outside there's a literally 2 halves of our room there's a camera sitting in the middle so to split it with there's a net in the middle splitting the 2 across and then actually them sharing knowledge and actually combining it together so those of the so spending the space of sort of single robot low level planning multi.

[00:03:44]
Asian oral sort of multiagency sort of visual of elementary those are the types of things we've been doing here you want to see in this one from a while ago this is and I think you guys plan a similar thing what's unique about this is the projectors on the ceiling now putting a lot of information on the ground game what you learn about these Robot Labs is the 10 reported boring but really not a whole lot going on in the lab so what we try to do is to actually run the video so there's actually video running in simulation pumped through the projectors we have about 12 projectors in our lab putting it image onto the ground the vehicles are now flying around using visual like perception looking at this video this now so now we can sort of stimulate them from a perception perspective the navigation is hard to do because that it's hard to actually get anything as navigation interesting but we can actually make it a perception really interesting problem but do.

[00:04:37]
I'm sure we're if this one memory let me jump on this one. The. This is the classic video of. A drone landing on a quad which. I don't sterilize downplaying that's a shame. You know you've probably seen lots of videos or drones flying over landing on things that are moving what happened with this video the small plane is this thing drives across the room quite comes over lands on it but you can barely see of these orange things here.

[00:05:10]
There is a leap blowers what I said to the student was this is all really boring make it interesting how do you make landing on a drone and on a landing on a platform interesting make it windy so and if you've ever used a Lee blower you know they're pretty strong Isabel and so if you're landing on this thing about a metre behind nothing you have to leave blowers it's howling away at about 7 meters per 2nd so you basically to fly into that wind find the landing platform in the land on it while you're actually doing doing that so it's a shame that did run but that says here the wind speeds on the order of about 6 or 7 metres per 2nd.

[00:05:44]
Which is pretty impressive to be able to fly in that type of wind and of course as turbulent as how this is the kinds of gusts and things like that going on so that's ongoing work but it's sort of a sense of just the low level planning and how does that how do you account for the mean when which is relatively straightforward but also to how to handle the storms the turbulent wind so then you get in a robust n.p.c. and slightly more control and all kinds of the things up in there Ok And this is us doing the collision avoidance on a there but now with a bigger crowd and if you're a grad student would recognize this video if it a Played this is students trying to get in it for a video with to a game in the morning.

[00:06:22]
And it's like I need a crowd you get your friends you get them up off the desk they're busy writing their own Nick repapered So they're all wandering around the lobby looking like complete zombies just trying to a crowd of like but anyway it's a shame these videos aren't running and they Kasik they work this morning about working out for summer.

[00:06:40]
But that's the robot just driving around is doing what it was doing up here in the in the sort of the the wild per se but here in a more controlled environment that we're doing in the lab Ok anyway so that's sort of a background of the types of things we work on you take a look at the web page hopefully as recently up to date you get of the sense of the kinds of things that we're trying to do what we're looking at though in a big picture is we do a lot of work in both the robot I would say is distinguishes our lab is certainly most of the students in my group are working to robot so we're thinking about things like the navigation in that environment.

[00:07:11]
Tractability safety is going to be a key issue how do you guarantee or at least get some sense of this thing is going to be safe so that when someone says well what do you know about how it's going to behave you can say something really concrete rebuses the perturbations So as you start bringing in learning methods how do you know how to get in the confidence about how they're going to work how do you understand how to plan on relatively long time scales beyond what you can currently see and do that in a world that still is fast can handle the dynamics of the sort of dynamic environment and the last thing we've been thinking about we'll talk about this at the end has to do with just fusing information across teams so it's a sort of each agent developing in a classic situation I Some wrote one robot off in the world that learned some things and then the robots in the world learned something I need to start fusing that information the question is how do we know that the human agree on what the language is that the talking about how do they know whether or not they've even agree on you know the you know item number one of my list was the same as the same thing you're talking about over here so that alignment problem and up being a classic issue and so if any people out there working on things like consensus this is the problem that was right before that because consensus pretty assumes knowledge about a state vector ordering and you now sort of just fusing across a state vector of this given what this does is try to make sure the ordering of that state vector is actually correct otherwise you're just using what apples and oranges and as you know that the top way to do it Ok so I'm going to try to touch on each of these topics is a lot to cover in this period of time so it seems like this is high level of just trying to get you the sort of the big picture get a sense of what we're trying to do why we're doing it what we've accomplished and there's lunch the papers and student names and things like got in contact me contact them and then have a conversation.

[00:08:51]
So this is the work by his use on those 2 papers here there's some code that he's released and the papers Well the faster is the algorithm and what he's thinking about in the classic sort of I want to fly a high speed in an environment like this you've seen drone racing you see in the drug basically and I know this competition that you guys were involved in we didn't get involved in for a pee issues but I mean it's one of those things where.

[00:09:14]
You know these are the kinds of problems you want to get into so you can fly faster the classic sort of situation though is that in this sort of pictorial diagram of the world we have things that consider you know this there really optimistic view which is that I'm going to know everything and I just sort of fly through it which is of the classic path planning problem it's great for textbooks completely useless for the real world right in a requires that you know the knowledge should know everything which you don't know and furthermore someone else figure out how to explain it to you in a set of constraints and that does never happens either it's very time consuming the reality is something like this which is there's a whole bunch your own known parts of the world and there's bits that you've seen and you plan around but you've seen in this just then it's a question of what you assume about the rest of the world there's the pessimistic view which is I'm never going to go in there as planned in the world that I've seen which is like you just bought a long and every slow moving robot is essentially just making those assumptions there's the optimistic which is will the hell with out all the fly into the world and then hope for the best and of course those are the robots you see crushing the walls right so so that's not a great strategy either the question is how do you generate things that are fast and safe and the net result is something ends up looking like this so you plan the performance path optimistically So you plan the red path is if you do just do what I want also doing just say optimistically I'm just going to optimize my objective and create this great looking path even if I don't know what the rest of the blue world looks like.

[00:10:41]
Well what I hold in my back pocket is essentially the backup plan and I will design a slice of both of these at the same time and this is rescue point are the point at which you vote is the point at which you have to decide what you're going to do next and that is explicitly some distance in time where some time ahead of you you bifurcate the to pass without point you design both of them you designing it choosing are so the only thing in this picture the fix is a that's where your everything else here for you finding what you then execute is neither of you know what you publish is basically a door to f. and that's what you are going to execute the premises by time you get to or hopefully over there are some part of the world's being revealed to you and you now know in the standard receiving horizon so what you're supposed to do next if that hasn't happened and or your computer is too slow and you just can't compute fast enough you basically just bail and just fly the f. and effectively stop Ok but the idea is you want this part to be long enough that it has very little effect on a to are because you just don't want this to be something where you were to begin to slow down because you're worried about safety right but the idea is that or you basically had no effect of the safety path.

[00:11:46]
And so then by doing this combination you can now be fast and say Ok So that sounds fantastic but I've taken a problem which I wanted to be doing one optimization and now I'm going to write because I have to find both the redline the performs awful and the rescue path and design both of them simultaneously so it took something I was hard to made about one and a half times harder but so far not on not good a good step forward what you want to be doing then is thinking about essentially a whole a topic the composition of the world is especially convex a vacation planning and non-complex world is hard yes you can pose it as a mix it is a linear programming but no you shouldn't because I've written many papers that said you should and then I start writing papers and said you shouldn't people apparently read the 1st ones haven't yet got into the 2nd batch right the problem is that it's a language that explains how to do it is not a language explains how to solve it it just is very difficult to solve or cut.

[00:12:40]
So what you wind up doing is you convection fire the world and then you just planning a path in a complex world the question then becomes how do you get that complexification and the guys that you Panna done a ton of work on this but basically it ends up being these great green polygons essentially you just come back to find what you know so what you end up complexifying is basically this chunk in here that you got to plan that right path in which is why this is a red color that's great but then you've got to do essentially the allocation of these thoughts along that path into the green blobs ends up in a huge technical problem and the result is something if you don't come up with an efficient Grizz to get just dead in the water this is not going to be competition truckle Ok so that's what we want to accomplish so this is the general framework just blown up a little bit so you can see that we're generating this safety path which is our af we're generating the off the mill path what we publish is the a r f which is the think of are actually going to send That's what we commit this is what goes with the robot if you never hear from bin again that's what you should execute and what that means is you will determine even the safe path and then I just reseating rise and do this all over again Ok Now again this requires you spend a lot of time thinking very carefully about the pipelines all of the work that has to be put into making sure you get fast perception data into this system and so you go to streamline everything because you now down into the 10s of milliseconds right I mean if if you have a 5 step process and each thing takes 15 milliseconds that's way too long so everything has to be quick and everything has to be set up to be to be very speedy So this is the giant mixing is your program you would pose if you're trying to solve this problem right it ends up being in this case it's going to be convex because of the fact we've done the come back to the cations was a planning in these tubes but the difficulty right where the complexity comes in is you need to make sure that each of these dots which are points along that path are actually in one of these points in one of these green polygons the polygons are designed to overlap and so the it isn't just as simple as saying put a blue dot in each one of the polygons that's so useless you're istic right that doesn't work because you don't know how big these polygons are.

[00:14:39]
And the delta these blue dots are fixed Delta t. apart so you have to deal with the Delta t. is and you have to find it so in the classic sort of way of doing path planning 1st will you find the geometric shape of this path and then you do time allocation which is what the path be along it because you've done this before you know the time allocation translates in the dynamics in the dynamics constraints could actually be violated if you pick Delta to be too small Ok.

[00:15:04]
So then ends up being this problem but basically what you're saying is I have a path I now have to pick the Delta t. which is the time between the group the blue dots it ends up being the socially maximisation of all this junk here and again we're taking problems we're trying to make them fast and it seems like we're making them slower by just having lots and lots of pieces but it turns out you can actually do these calculations of things where you essentially just doing a binary search over a Delta t. value but we can do them on a time scale here shown of about 15 more vote for milliseconds so we can do which we have a path we checking to see which Delta t. works we can do that checking for milliseconds and experience has shown that you only need to do about 2 or 3 of these checks on what the value would be which of these red dots the too small you need to make it bigger and then to find the Green Dot value which is in the case of us you succeeded so this is the indication of how you solving the problem over time here I didn't really know what the value should be I had to try multiple cases but even so that were the things that worked at about 15 milliseconds so what we're able to do is to take a hard problem in terms of path planning around objects convex afire by getting the polygons design the Redpath in it do the time allocation along it of the whole thing is getting done in about 30 milliseconds which again is not super fast but is pretty fast compared to the hundreds of milliseconds that exist in the I would be 4 o'clock.

[00:16:20]
So then you can start comparing how we do against a bunch of other algorithms and this is the classic forest just a bunch of cylinders you need to stay horizontal and to fly through them but again keep in mind the world is revealed here we don't know very much we have a sensor on board and you just deal with what you can see there's no sense of here's the 4th fly through it it's Here's a forest and as the as you see the trees try to avoid them lots of people like our t.v. star not 100 percent sure why because it does doesn't do very well in these types of bombs again it's great up planning on a global sense that's not what this problem is Ok This problem has to deal with the dynamics short horizon rapid things so these look much more like like fast type of response in terms of the other type of algorithms down here.

[00:17:09]
So mix it into a quarter and program so you have a star you have a cost function which is quadratics a quadratic jerk so is the sum of the jerk terms but the problem statement is so linear from there on down but you have been reverbs these guys here which are imposed in the constraints that you have to you have to allocate the blue dots in the green pull it up and you have to like you have to be in this color code of the Sisk right bunch of Poto but I don't know we sort of what the allocation of these posts is so I don't really like a sort of drawn as if it makes sense like they overlap a little bit but I don't really know whether or not I can ever growing dot a cup of green dots in one but you know one Green Dot in another a Green Dot this into and soul that has to be captured in these binders that here and so you have some binary variables but not many and if you have a lot of them you tend to bloom but we just don't we're not the problem there's not that many of them here.

[00:18:02]
Ok And the point is that the this problem is solved for a given Delta t. And so what you're checking is what Delta t. value can I have that's the distance in time between these 2 things come in but you know it's again you try to decompose a time space problem into a space problem and then the time problem the couple but the because of the way the couple I can solve them quickly together but if I tried to sold them at the same time that's when the problem balloons Ok So anyway we were a lot faster and then in fact the closest to us was our previous paper so this was our paper this was a paper so we sort of started off by saying wow every paper was horrible paper been literally had a table that had everything above here except this line and then we come up with a new algorithm which is classic of a paper as soon as you submit it you know exactly what's wrong with it because you found all the floors while you were finishing the paper said it any way you got accepted in the New Year mediately start working on what's better which is than we did and the problem is it only added one line to the table so that was like so classic.

[00:19:04]
You know reviewers hate that so this is what it looks like again the pictures of the same the colors of the same what you're seeing from the perception systems the voxel being recreated and again we're creating local small maps Frank virtual understand this we can create a big map and update it from a mapping perspective is way too time consuming so we create local little mouse the planning in.

[00:19:26]
That's these orange these orange things are all the schools we've seen the blue stuff with the unknown so it's the same picture we have before the run the video again so we talk about it 1st but this is the green polo type there's a bunch of them will map together this is the path or executing and this is sort of the thing with think about in the future.

[00:19:42]
But what you get a sense I hope is is you know it's run in real time in the video so this is running flying and you see how it's sort of staking out finding its way through things the world is sort of becoming exposed sort of receding and becoming exposed to us and.

[00:20:00]
And so we're able to sort of navigate through the world pretty well but that had a lot to do with just just how far you know how how short the paths but didn't necessarily get in the aspect of time so this is a different world with a bug trap and so you sort of flying from here to here there's a giant hole that would like to trap you in but it it avoids that pretty easily the colors here indicate speed the red is faster and so the point being that when we have the ability to speed up the system Qin And again none of the other algorithms that we can actually run against this were we just try to get so selves and then it was alters we're way faster so so this algorithm the new one faster literally is faster in terms of loss of you than the multi Fidelity which is a lousy name but based on what it was doing so not only is it shorter it's faster.

[00:20:51]
Yeah. I depend so the complex of the world I go show the video here this is actually the office environment and the similar thing here we're flying down a world sort of the world's getting revealed as we fly down we here we fly into an office and so it's literally a bug trap in the sense of there's no way out of it other than the way you come back the nice thing about that local map and kind of what's going on here is you can you know we have a local map around us we can use it to actually fly around and so when you go in a room you realize there's no way out of the room you just turn around in the map you're in fly back out of the room and then sort of reconnect to the map that you were then using in the hallway and then start exploring again and so the idea of the local map is only about 3 or 4 meters in diameter but it's enough for us to plane in and utilized and take advantage of but it's not very expensive update but again you're you want this part of the update step to be again in tens of milliseconds and so that's I think a talking what happened here but that's based on how.

[00:21:48]
Ok so that's cool for his simulations but you want to get to experiments so this is in our lab So again we're Viking on knowledge for navigation but perception for everything else the quad is completely ignorant of anything else in the room other than at the location and you tell it get from point a to point b. velocities up here and so we're just sort of go zooming down the other end it seems a bit jumpy but we really crank up the out as you loop so it's in that sort of millennial Falco mode of you know if you see stuff to start doing this so it's someone intentional that it's very jittery it's has to be very agile and out of Tud we get up to rates are about $700.00 degrees for 2nd and then roll which is the only we're going to go around stuff otherwise you're just going to be now this is carrying a new look and if you know how heavy those things are there are some things you pick up on their light and they feel like there's some things you pick up in the field much heavier is in the 2nd category is a giant brick that's the name of it is the brick and it is heavy you put it on a quad and then say b I John Kerry can look that's quite impressive So this is a hex the I can only quads but it's really a hex or you know it so we can navigate so and you can see the real sense up front that's the sense that we're using We're getting depth from that so the whole voxel map that we were discussing we saw in the videos still being created using the no more perception it's just the bike on is from the word is doing by calling under obstacles is interesting because it's kind of a guessing game does that if I can actually track the thing well it's flying underneath there you go up so it seems to work pretty well.

[00:23:22]
So that's one of the topics there's lots more in the air and faster is moving in a multi. Vehicle sense as well in part because of the safety guarantees that is preserving one of the issues when you get the multi vehicle is something you will guarantee you will not fly into and so one of these were able to utilize is the safety guarantee aspect which is if I'm creating a safety path and you agree not to violate it then we as a team can move forward and come up with an asynchronous in.

[00:23:48]
Limitation of a multi vehicle path planning formulation so you can look for a new paper probably in Iraq's of this year hopefully. So the certifiable certified to have a sort of work they want to get too much into this because it's this is early days for this research but one of them is that we struggle with years when you're in a spare scum field or even like robotics and you come up with a great deep learning algorithm to go out industry and you say hey you want to fund me on this the couple of years ago used to get questions like well how good is your performance right but now you get questions like well does it work and how do you know it works right and and so it's the you know things have changed fairly subtly in the aerospace field it's quite clear that they're very interested in what you can say you know in terms of the perform the sort of safety aspects of things not just about performance and you see examples of sort of how things can fall apart and there's the classic example of you can make you know Turtles look like stop signs and vice versa and all kinds of things like God.

[00:24:46]
What but as a result of that there's a bunch of people who've been thinking about how to avoid the image cost occasion problem you just took an image cost of higher in and what we would like to do is to move that in the direction or path planning were essentially what you have is a situation where you know that I'm here and I want to get there but there's some other object is in the world and especially when they are there so I should act like an adversary and try to trick me into thinking sort of that are different place the base will be a boy around them and so if there's going to hallucinate that they're somewhere else that was essential for me as an image you're right I'm looking at the world if they can trick me into thinking they're in a different location that I'll behave differently so than they're essentially able to force me to do things which end up being bad things for me to do and or it just takes me a long time to get to my goal so the net result is that there is some action I would like to take if they were there that's what I would like to do but basically they're going to they might suddenly move to the left I think I can get by but it turns out they were actually here and then I drive into them nonsensical in the sense that isn't how it's going to happen but when you've posing one of the prime problems you sort of look at this is a time when this incident time what is the decision the red guy's going to make in terms of his actions sat under the assumptions of what it knows about the blue guys doing up here and so again the blue guy is from this perspective the Essentially again the blue guy is trying to think about what is the thing I can show you this going to convince you to do the worst thing possible for you.

[00:26:14]
So it's a really what you do and in classic sort of robust control senses I would put a boundary around the location of that object I will bounded by Epsilon and what I'm trying to do is to see a situation where. That adversary has the freedom to hallucinate or sort of convince the red guy's anywhere inside the black circle and then what I'm trying to do is to force the rego to choose bad choices and that's what happens here so this ends up being a mini max for max min type problem over the set of key values that the the guy can pick So this is the problem they're a guy solving he's going to pick an action but basically at the action that's going to be the worst one with the adversary in the space so none of this is new formulation this is not the new part the new part is that what you'd like to be able to do is to pick in fact a better action you sort of go round that the question is how do you compute this because what this boils down to is this in here people have known how to pose this problem for a while what they've not known how to do is to solve the things in that yellow box because basically you might have this q. function but here what I'm talking about is the worst case possible q. for all values of the set of this guy here in a bowl in space and that's a hard calculation very hard to actually do so the point is that that's in this is you know it's like a problem I could solve if only I knew the yellow thing but this yellow thing is impossible to compute in real time so what do you do.

[00:27:35]
And the point is that it turns out that in the computer vision community there's been a lot of time working on these what they call certified defense algorithms and want to social the bowl down to was this policy here this thing q. is be learned as a neural net and what I'm going to put into it is essentially an uncertain bound about the import and it turns out that there's a really really fast way of computing how the uncertainty as an input propagates all the way through neural net to an action set so what I can actually create through it it ends up being an over Bambu above the lower bound but it's it ends up being a fast computation of the uncertainty propagate through the neural net to enable me to figure out how uncertain the input are you created by the adversary actually translates into uncertainty about the action set and so that fast calculation then enables us to figure out as a function of this absolute and how how sort of a graph.

[00:28:25]
We think this adversary by b. we can come up with a way of actually understanding what the effect would be on our guy the rig I and then actually computing much better strategy so what you end up with a graph like this of probability of collisions as a function of the robustness and so if we crank up the rebuses i.e. if it's very small we have high probability of collisions if we crank the robust and so we can.

[00:28:47]
Probably be way down and so again from a robust control perspective and even a chance constrain type collision avoidance not a huge surprise use of the kinds of things are pictures you would expect to see the real trick here is we've gone off and learned a policy using an neural net we can now analyze it in real time we will be designing it to see how what it we're are we in this graph what is the probability of a collision occurring and then we can actually come back and give you some coffin is about how well this will be functioning as a policy that's learned from data Ok And so that's the key part about this is not a lot of machinery is known but you put all that machinery together and we can answer a question which has to do with the ripoff of the learned policy in an environment where the adversary is trying to convince us to do bad things so that I guess that's ongoing work but we present that a coral interesting experience because all the people who were done the work before our work were at that conference which then led to about $100.00 emails over night all about and I said this is Will if they didn't care about the work they wouldn't bother to email and most of the times the emails would be the form you didn't cite me but these emails were interesting because it was like you didn't cite my friend and then and then this other guy would chime and say yeah you can cite my other friend which we did but it was all very interesting very collegial ventures and group to get in touch with.

[00:30:05]
So I think it's a lot of sort of a lot of future work is required here to address the certification issues a lot of issues on learning that have to do with and have to be addressed as a machine before so I so want to talk about here is some work that.

[00:30:18]
Ever it's been looking at he's a very senior species to mine graduating this in the next couple of months. And what he's been thinking about for years he was the one that did the social navigation robots who is think about the path planning but one thing to struggle with is a basic question of you know people want to do delivery robots so I have a big s.u.v. drop off a little robot and then say drop this package off of the door and there's a basic question without a geometric map Where's the door in how do I even get to the door and so one of the questions was how does how does one do this planning and to get it with given a geometric map not a problem to far apart the star or whatever and plan the path of the work as you do with the door was because someone told you but if you don't have the geometric map and or don't have time to make one what do you do instead so because again that you know this is the classic sense of you for a robot the doesn't know very much and if you were in a structured environment maybe you could you could do a lot of mapping ahead of time but.

[00:31:14]
That's not always going to be the case so I'll skip this in the interest in time but it ends up being you know there's a problem formulation you're trying to do collision avoidance but what you're thinking about is time much more like exploring in a world because at the cost of way is all this explore but you told me go to the door don't work is all this drive around hopefully find it drive an intelligent way hopefully will come across it but what you want to do is to go out this month with doing that.

[00:31:40]
So let me show you the classic picture this is the you know there's lots of words about who's doing what and you know in the classic you know good student background comes a lot of background papers explaining a lot of people work in this field but a lot of people who worked on the end were in policy right and so you'll see this going to Berkeley which is all this take my stream of data you know real sense and I'll just learn a policy all the way to action right but what that ignores is the fact that there's about for 35 years now there's a whole bunch of people working on individual algorithms to do specific pieces so for instance you would have people working on slam and a grid so you've got all that technology you've got people working on cost to go calculations up will planners so you go that technology so you want to take advantage of the questions what's missing and what's missing is really an understanding of how to get from an occupancy grid or sort of a to to this cost to go so it's a sense of this will end up being a semantic map of the world and what you're trying to do is to is to use the semantic knowledge to begin to educate and improve your your exploration ability and that's the piece that he's trying to do look at here is connecting these 2 boxes it's a learning problem but it's always a learning problem and learning problem is really in the heart of the learning problem the more data that's required in the more difficult this gets.

[00:32:51]
So again just take a step back because what this looks like the only a complicated but But what you have in here is something that's going to look like an occupancy grid or semantic map but it's essentially a picture I mean you can make a picture of it this is going to be a cost to go which in a $2.00 dimensional sense is just an indication of any he Grid Sense or heat map sense you know is this a good direction to go in or not so this ends up looking like a 2 dimensional image this is a 2 dimensional image and you're just doing image the image mapping and the classic way of doing that of the field is the unit right and you basically just do the compression and expansion in code decoder combo and you just doing mapping from what essentially is a partial to submit to the semantic map partial because but I've only seen part of the world right and the cost to go is sort of the cost to go of the things that I've seen and you try to and effectively You just need to be learning on different data sets that have sort of different parts of the world explored and so on and reveal to you when the other hand when you're doing a real time what you have is a partial semantic mapping of just doing this and using this you net to it to make that connection for.

[00:33:58]
So then there's the question how do you get training data for this and the other problem of course is you have to label stuff so you download a bunch of things you fart off the google and. Hopefully the come back a little bit right and then your grad students because I was and I was in there with labeling things but the trick here actually was actually partially the reason there's a lot of black in this image is what he was actually trying to train the net the unit to learn was that relationship between partial things and partial things but that it's if you only see this part of the world what you think the cost to go map should look like he's all that part of the world what these things should look like there's no point training on the entire map because you're never going to see that right you're only ever going to see the partial semantic map so he was trying to train not just on you know lots and lots of full mouth but lots and lots of partial maps and so.

[00:34:47]
Anyway so that's what he was doing and that so he ended up creating a lot of division mass dropping in on the real things in the training of the right. This is a comparison of the 2. And you can you can see the satellite the semantic the predicted in the true the pretty good they're not always great but they're pretty good and this is the full map but they look like this is the partial you'll see some of regularities but you know classic it's doing pretty good job and is learning nicely how to do this so good to be clear what happens here is you're driving along and what you might see a things like the driveway you might see things like bushes you might see things like sidewalk and what you as a robot a doing of piecing together the semantics of these are the things I've seen this is the arrangement that they have so this is where I think I should be going because that's likely where the door is going to be it's not it doesn't tell you where the door is what is giving It was a cost to go for a explore exploration plan or you still need to be doing exploration but is an educated exploration based on the symmetric understanding of the world that's been revealed to you so far Ok so that's the logic behind the process if you compare what we've been doing with whatever else is doing this is us this is sort of front here we had up to be the whole lot better lot of variability depending on the environment you're in but even so it's working out to be pretty good then again this is just is still exploration is just a much smaller type of exploration than than the frontier based exploration you typically use.

[00:36:09]
There's a video of how this works it's sped up a little bit jerky I hope this one place where we have so many problems that. It stops just simply because that's the simulation we're in it's a pretty nice simulation obviously has wind. Seemed a little bit over the top record of Bush's it's now seen the connection between the driveway if you squint up here you can see the semantic mapping revealed the cost to go map it's been estimated and from there it's beginning to figure out where the door should be it gets a sense of how to maneuver the doors over here somewhere and is now figured out how to get on the path and so on.

[00:36:45]
So this is help to figure out the relationship between things which you can see obviously doing the collision avoidance of the past planning level but it's just trying to figure out what the door is now in this case is begin to realize there's a pathway here it's going to turn in this and then it sees the stairs and begins recognize what the pieces look like Ok so the point is that the semantic global you can now have a deeper understanding of how the future what the future looks like and now get a better sense of a cost to go and how to explore.

[00:37:15]
So again just you know the numbers here and how it's how it's improving the what other people have done I want to show was that was a simulation he's actually taken it he's done that you know I've been asked why it does work in real life this is the our office so much of that's real life but that's it it works on a real robot in the wild it's just that it's is not very wild.

[00:37:39]
The challenge we're going outside is you've got to go outside it's cold and is really super hard to get interesting data so that actually working with Ford on this and start exploring people's neighborhoods there's only so many houses you can go to with a robot before people start asking questions like why are you here and so it's just hard to get a large data set and so again this is done you know I mean it's the test here was you know again you can see it's pulling out semantic information putting together understand the cost to go and so on so all the technology is running in real time it's just now we need to spend some time getting a really good data set outside that's as I said you can either do in November or you can do it in March she chose March.

[00:38:23]
We'll see how that goes as a choice because I think it's called a March in the ember we'll see. All right last topic so this is most of you did Association this is one of my favorite topics and has been for years because you know we worked on the Muriel bays in nonparametric for a long long time starting in like around 2011 and that's when we really began to think about you know again you're learning things a robot and you sort of begin to label them as as you know again this is you know thing number one I've learned this is the number 2 this is the number 3 but everybody does that you go off and learn a bunch of stuff you make less and then classics that was Ok great these robots meet in the hallway and it's like that was our home conversation with thing number one it's like well how is thing that will wanna one robot I knew that was thing number one another robot how do you know the scene different things the scene lists of different plants they might have seen completely different objects and so now what you need to do is this this multi view sort of data association problem so this is taking Luke closure for a slam and expanding it into the problem of the Assumption of the generic that association across robots so in the sort of fix robot view would be like this you have multi view of the world the sitting fixed somehow the hallucinate and see the parts of the world and then you're trying to come to agreement on do we see the same things but of course you wouldn't necessarily see them from the same direction you wouldn't see them the same lighting there's all kinds of other things that could have been different about the images and so like as a you could have seen things were from different sides and they look similar but different and so you have to actually figure out whether or not we talk about the same thing over.

[00:39:56]
So typically you're going to have a bunch of outliers the standard way is pairwise right you just get a bunch of agents to agree if they agree in a pair why sense thumbs up does go with that as a plan to get a home if you're paralyzed and then does not get your data set that's sort of the classic solution to this problem but it can sync contain a lot outliers so what you're striving for is a consistency goal where everybody is on the same page we all get to this point of understanding and in a pitch a sense would be like you know these 2 guys agree the agree and then they agree and the fact that this graph ends up being this close triangle that that consistency was striving for when everybody on the same page if we can get it you can't always get it because not everybody has seen it not everybody got a clear understanding of what they were looking at and so you might have bad associations that just remain but that's part of the data you really you have to remove those using a different strategy that's not really a failure in the algorithm we're going to develop is more failure in the data or so in the multi-way data Association cycle consistency trying to rectify wrong batches of a peer had a problem every the rest of the voice is going to speak up and say that you guys are wrong you should have come up with a better result.

[00:41:07]
Ok so while this might be new to the robot community this is not new but old with a computer vision community you can imagine the been spending quite a bit of time on this and what you don't want to do is to reinvent the wheel so you want to make sure that you spend time understand the literature.

[00:41:21]
Again classically just like everything I think I'm not sure that is true that everything you are in isn't in your program but it's pretty close to that this isn't in your program but no you shouldn't be solving it that way Ok And it's really really slow so what you're looking for on a graph of precision and faster you want to be up here right so you want to be in the top right corner classically you have compact relaxations of them they get you over here high precision slow you want to be over here graphical techniques is going to give you much faster but I'm not going to be particular precise spectral the compositional relaxation is the strategy that we're going to be building on typically So in the middle of the sort of in this those sort of Prieto front what I'm going to show you as an algorithm is actually all this graph is going to be up here.

[00:42:06]
And we are actually going to be superior to the others in many ways Ok computationally efficient hyper Cision and will give you the cycle consistency of the of everybody being in the same page and the reason for the same pay everybody in the same page gets you almost like a language that the robots can now use to discuss what it is that we're talking about or that gives you sort of that you know we've all agreed on what the ordering is we can now start talking about Thing want to will understand what we're talking about if you don't have that then the robots any type of fusion is going to be a complete train wreck and go completely over.

[00:42:40]
Now what this ends up looking like you cycle consistently the thing we're striving for is essentially you're going to end up with what looks like a graph clusters are going back to this picture of the linkage is that essentially what you would have is a collection of closed objects you would in this case would be trying to as does 3 but lines is not closed and dots Doctorow's right those are not the things you're looking for so what you're trying to find is a way that you can either take a graph like this an address or track stuff Tobit that would actually make this into something which is now a collection of closed things which is this cluster graph.

[00:43:12]
So how does one do that well it turns out that you can do that using the spectrum of the graphs in the plastic. And if you don't things like consensus you know with a lot posturing is the l. that has to do with the so is the possum a search of the graph but the graph we talk about is this graph.

[00:43:27]
The difference here is you're I have to add this mobilization on here of the ways this doesn't work or one of the normalization see is I please d. we're d. is the degree matrix of the graph Ok you do that the algorithm works pretty well so that's what you have to do classically though you don't actually know how many things you're talking about is not like someone had a counting system where you went along and said Ok we all agree that there are 4 things in the world because you want to seen 4 things I'm missing 4 things that we talk about the same 4 things how do we know how many things are even are right they want to be 8 things and you so 4 things I saw the other 4 things so you have to try to estimate what that number is and that comes in from part of the eigenvalue calculation you can do from that graph so you do that calculation gives you an estimate of what that number is how many how big is this world that we're talking about you then do a calculation of the eigenvectors from this the.

[00:44:17]
And then this part is a bit strange which is that in the Euclidean sense you basically take the the elements of the. I can factor in the does graph them in especially r.m. And then what you end up with is a bunch of dots end up looking like this and then you're actually doing clustering just like the moment would using it could be in distance as as a way of measure and the result and is old with things that are similar to each other and plebeian sense on this or am graph are actually clustered together and that's the part that to me was a surprise I didn't realize how well that was going to work because it did didn't make a whole lot of sense as to why clearly in distance with matter at this point but that's the way the clustering actually works and the outcome then is that the points in these clusters actually represent the same identical things these are now the aligned elements the base of what you're now doing is saying all these objects all of these pair wise connections what we're able through this calculation to do is to figure out which of them with the same in there to show up in these graphics clusters these are now the same and these are the same with as different and what we're now able to do is a sense of the permutation of one robots on their list of one another robots on their list they show up in this location.

[00:45:25]
And now they're being permuted So they actually show up at the right place and now you can connect them with say that arc for the same right so you go to this point given the tons of people working with the great things there's a whole bunch of benchmarks out there and if you want to get published in this field you've got to do every one of them right so you go out there and you download this is apparently the hotel benchmark and so is the CMU hotel Heaven knows why we care about it but there's a whole bunch of features you close them all together you come up with a bunch of socialization here you see a bunch of broken triangles here you see a bunch of complete triangles again there's red lines because there's the problems in the data the again we can remove them through.

[00:46:04]
You know through elimination but they're not really a failure in the algorithm that just a failure in the data there's just it's like one guy saw something no one else did it still shows up. So this is not interesting but I want to use this as a benchmark to show you that graph that comparison graph so again here we have persistent we have frequency uploaded to one of our ass so again you want to be up here right you want to be fast and precise.

[00:46:28]
There's a difference when things that are cycle consistent which is something we said we won and things that aren't so red dots are not and they're down here the baseline was the orange that was just sort of the pairwise stuff and again these made it worse there's a bunch of algorithms appear to dominate us on this graph and so you come down to this graph which is perception same y. axis recall which is the standard way of thinking about this type of stuff and they're way down here we're way over there Ok So on this graph so you sort of have it's own for to look at 2 graphs to see how we're better but we are.

[00:47:02]
But when you look at the computation. Again there's a lot of people out there with their favorite algorithm match Lifton and f.m. sync These all came out in the past couple years were like 30 times faster 3100 times faster than other algorithms that existed to come out of the past couple of years so a way faster so this is all very interesting but how do we utilize this type of thing in our world so here's a here's a scenario where you have a robot in this case is going to be flying around in a forest it turns out I now understand why people get lost in forest trees are really useless as navigation tools they'll look exactly the same and I saw a tree is not a helpful thing great because there's a lot of trees out there what you end up looking at there was trees ya mystery right clusters of trees that's interesting that's where the fingerprint is the trees themselves are just big cylinders that are useless so the robot's flying around it sees a bunch of stuff it sees some things that sees more things these more things right so now you start connecting all these things together and you're essentially doing the loop closure on all of this stuff.

[00:48:04]
And then you end up with a sort of giant graph of connectivity but these are not correct or good associations and again this is just a lot of similarity in the data a lot of confusion and the result is a lot of useless things but you run clear on this and clear literally clear the whole thing up and you get this again there's a couple outliers which again we can remove quite easily I can explain in detail later or the post-docs only callous but that way but the result is that we can clear the whole thing up and get this much cleaner now I get what we're connecting here is cleaning up like essentially where we think all these and streams are and they've been all clustered together and grouped together much more currently than the word here we're not necessarily solving for where we are at this point this is just trying to close together the clumps so that there's less ambiguity as to which of the features we're talking about of the world which is essentially this tree landmark these these triangles so that's what this step is doing you would then run a processing algorithm a top of this which is like salmon or slam that is figuring out given those treat clusters I now have to go through and figure out where they are and where I am but I essentially run that algorithms a 2nd step.

[00:49:10]
And that's this is just a cleaning up just as fixes up so again you can take all there for now dooms that exist in the world and try that this is the baseline pretty crummy again rate is bad and so we're down here this is our the result we had we post-process we cleaned up immensely a bunch of other organs that exist in the world is a pretty miserable performance sense and really bad in terms of computation and we're sitting here about it a formal seconds or so fast and numerically or performance was doing superior to everyone else.

[00:49:45]
Ok so that's how we do this on a single robot but of course you can do this all multi robots now because you now actually start doing associations with the robots and that's what we actually flew down at NASA I don't see know if you can see this it's going to hopefully start.

[00:50:02]
Otherwise you will not see it. So hold on what happened there. Dude go Ok. Ok now you have to squint so what's happening here and I have to put this again so robots lying in a tree forest there's another overflowing in the tree forest you won't see it they actually never meet they never overlap or they overlap in paths slightly but when they do have options things they saw So it's like I saw a bunch of stuff you saw a bunch of stuff and then we realize all we look at the same things are actually looking at things from different directions and then what happened is that maps that we're separated actually get lumped together and now we realize that about this point that they saw the same thing the algorithm runs of them stitches the 2 maps together so what's running in the background I think of a better plot in the next page but basically what happens is you're now each robot doing its own fusion problem it is running clear cleaning things up getting an idea of where all these landmarks are and then you can actually stitch the team out together in a multi robot sense nice thing about flying in this forest is there was some street canopy that they actually true to this is if it was inside which is way cool and never had such a cool safety guy was safety guys are really strict about flying outside he said No looks like you're inside me there's no way to fly outside the tree canopy he said big thumbs up so that was the easiest flying out doors I've ever done.

[00:51:24]
All right so it's been a long talk but hopefully get a sense of the different things we do things I didn't have time to talk about at the beginning things I talked about in terms of the single vehicle planning in a sort of aggressive way in an uncertain world is revealed to us as we're going then we're thinking about how to handle some of the improvements in the robustness of the things we're learning as we're going to handle some of the cost to go calculations so we can do exploration Teligent way and then the final thing is how to actually get robots to do alignment and do a better job of fusing or lying 1st and now we confuse that we could always fuse but the question is how do we align so now we can do actually do a her job of doing the fusion.

[00:52:04]
Bunch of papers listed here so. Bjorn Michael and cover e-mail them if you want to know e-mail me I'll get you in touch with them thanks for your time and questions love the 2nd. Yes. So. So we do full moving camera moving objects so in these scenarios. Like in the scenario I showed it was static obstacles Woodstock obstacles I would like stuck at sag objects but only as moving vehicles with static things but the real challenge is how to do moving things like moving vehicles with moving stuff that the middle ground is what I describe is non stationary which is we don't necessarily need to track it was moving but we can't assume that it's in the same place it was the last time we came here so classically a live closure for a slam is like I saw a red car heard before next time I come back around I would expect to see one there that's not the case in the you might expect that red car to show up a different place and so you might you know it's a question of can you make the association between those 2 red cards even though they were geometrically different locations and so ideally you'd be tracking moving people but I mean that's you know I think right now a little bit more difficult than I want to get into is because in the whole feature tracking of what is a person and moving and so on I think right now do people looking for identifiable clothing elements to help us distinguish one person from another and then we show up in a different places like Waldo it's like you know it was here but like that if he shows up 5 minutes later someone else does that mean this to all those who resist Waldo moved right and if the new relax your assumption and account for the fact that it might be the same person in the different place.

[00:54:17]
He thought it will be. Up there you know. We really I mean it's like he's like how do you do any uncertainty bound right you I mean it's a knob it's another knob in the planner been able just to say. You know in essence you know like like for instance we might use this is it world where if the other agent in the world is saying or her agent in the running that algorithm is question how cooperative do we think they're going to be because of the super cooperative we don't need any of this that just going to get out of a way if we think there are moderate aggressive then we might to an epsilon to be sort of you know slightly bigger than 0 if we think there are you know homo sidle show for them we make epsilon you know as large as we need to make it so it's it's a question of like I don't think this is an accurate model of anything but it's a knob you can put in the problem that allows us to detune the it's actually how aggressive our policy is it's like how close do we get to anything it's actually that Upsilon makes us get for the way i do we make it too big then we can get stuck right so you don't want to get too big on the other hand and make it too small We're going to we're going to fly into things.