[00:00:05]
>> I'm going to produce Frank and start this thing and I'm going to talk until the presentation appears or until you get frustrated and leave Here's the 1st thing this seminar you should come to every one of these this this semester every one of these seminars because they're going to be good the start off is Frank which is spectacular and I'm super happy.

[00:00:26]
And week from now or 2 weeks from now saying they came from MIT the guy who designed the cheetah robot will be here to talk about cool quadrupeds robots Matt Mason is going to come a few weeks from now Wesley Caleb when will come a few weeks from now it's going to be a cool semester of seminars so I hope you come to all of them.

[00:00:48]
I'm start Frank's introduction now with the anticipation that I know enough about Frank to talk until the screens Come on Frank is a professor here he's in our enter. Interactive computing school which is inside the college of computer also I'm in that school. And when I get business cards printed I'll be able to read that to you in the College of computing Frank did his Ph D. at CMU.

[00:01:18]
His peers did visor was Sebastian Thrun he was Sebastian's 1st Ph D. student who's heard of Sebastian it's because of Frank. Can anybody name another Ph D. student of Sebastian know Frank was the guy it's been downhill since that for Sebastian. After he finished his Ph D. It's a view he came here and he did cool stuff then he went away to K. you live in for a sabbatical after that he went to Skye Dio where he was the chief scientist for a while he's going to talk about stuff today that has to do with that after that he went to Facebook reality labs where he worked with Regina Doogan for a couple of years and that's that's where he's been for the last 2 years.

[00:02:09]
Frank has done good work for a long time. There are bookends that I'll tell you about in 2017 or so a year ago 2 years ago now. A paper from 19091 of the Triple A I classic paper or ward I and Frank was one of the authors of that along with Sebastian threw in a will from work guard.

[00:02:36]
When you win the classic paper Ward from AAA I what that means it means after 20 years looking back this was the paper from that conference that one should remember that's a big deal but that's a long time ago what's he done for us lately in 2018 at it grow he went home with the award for the I probably transactions on robotics best paper award for work that he did with Luca Corleone and.

[00:03:05]
On printed Gratian on Manifolds for real time visual inertial stuff the kind of stuff that Frank does. I would like to win either of those words at any point in my career but to bookend a 20 year period with those awards indicates that you're doing quality work sustained over a very long period of time I met Frank I think for the 1st time in his office at Georgia Tech when I was visiting somewhere between 5 and 10 years ago at the moment I visited Michael Katz was in his office why do I mention that you should go and read the book by Frank and Michael about Factor graphs I think you can get a free P.D.F. off the Web Yeah I'm sure you can find one.

[00:03:45]
Or you could buy it if you get the free period off off the web give Frank $1.00 at the end of the lecture today because he should get his royalties for that. When I met Frank he told me about Factor graphs I'd never heard of them before thought they were the coolest thing I decided some years later that I would come to Georgia Tech I had no hope that I would ever work with Frank because by then Frank was making all the money at Facebook and I really didn't think he would come back I would have bet whatever you wanted to bet but last fall he came back.

[00:04:20]
And I don't know that anything has delighted me more in terms of surprises about Georgia Tech than having Frank come back to join us so as soon as possible I got him on the schedule for the seminar series which is today and with that I introduce to you Frank Diller.

[00:04:39]
One of the big things that I'll talk about so that's a set said than that stalling introduction I was gone for 4 years from Georgia Tech and the super happy to be back. And the 1st the 1st couple of months I was in the. First 9 months actually but then I got a call from from Silicon Valley from sky the.

[00:05:03]
Who wanted to they heard about and so part of the things that I will talk about here is Factor graphs and over the last 10 years my students and I have built a library called. Which embodies all these concepts and ideas from from Factor graphs and you can that's a freely available library as opposed to the book.

[00:05:24]
But if you talk to me sort of you know individually I can get you a copy of the P.D.F.. And so this is quite interesting so Adam. Emailed me and we got on the Skype call and by the end of the Skype call we had gone from you know could you be an advisor to the company we're doing drones that we're doing everything we do.

[00:05:49]
It would be awesome to have you as an advisor to highlight the board in Belgium should I join your company instead and so I joined as. Their chief scientist. In 2000 I guess 2015. And let me just show you the Web site of Scotty which has a couple of cool movies right of what this what this company's about.

[00:06:22]
So this is this guy you are one. I hope you can see that. It's it's fairly expensive because they're going for the Tesla Model right so they they want to sell you a high end products 1st probably only to a few people that sort of you know for which for whom 2 $1000.00 is a minor expense.

[00:06:46]
So you can value is full of. What they call mammals but they are middle aged men in a late. And the old cycle on the Skyline Boulevard and you know. What they want no no less than showing off you know how good they look in their in their.

[00:07:08]
Like. So so they're working on a 2nd version MILLETTE me show you a little bit about the tech of this drone. OK so here is a video of the sort of the 1st video when it came out we had an actor that ran in to it so the cool thing about this drone is that it can follow you make a movie with its fork a camera it's a gimbals camera on the front but in addition to the 4 kit gimbals camera there is 12 additional cameras all around the drone So we have full tree 60 coverage of the viewings here and we do real time structure from motion and slammed on the vehicle to help build a 3 D. model of the world around the drone that means that you can run in a forest.

[00:08:02]
And the drone bill will avoid all the. Trees and the branches because it knows VERY they are OK. And it is in fact I stand by this the most advanced autonomous device of any kind vailable today I actually believe that this is true OK because one of the things so you saw some dynamics you know videos and of these amazing machines etc But most of the time what they don't tell you is that these are completely.

[00:08:34]
Operated true remote controlled right somebody is there is an expert at guiding. The robots and so their autonomy it's quite quite limited This is a fully autonomous device and I would characterize the robots as something that. You know perceives things and acts in an autonomous and right in the world.

[00:08:55]
And if you talk to us in an example tell you that perception is the key bottleneck to making that a reality for their robots so this talk is going to be mainly about. Perception and how we make things like this drone possible. And at the end of it with my students and with Byron boots and with.

[00:09:18]
One of the. Students we've ventured into money police. As well and so my I see a cool future for myself and sort of figuring out manipulation and helping to make progress and manipulation so in the meantime while we're getting power point up and running how about the following so I just started teaching again I love teaching I'm not so hot on preparing for teaching but what I teach I love teaching and one of the things I love you know doing doing most is have people talk to each other in class so here's a little experiment I want you to do OK So talk to your neighbor and explain to your neighbor what do you think Factor graphs are OK There you go for teachers in the room.

[00:10:06]
The trick to making this work is to not wait until all the noise has died down because because that means that half of the people start talking about their weekends you know so so so you have to sort of stop them while they're in. So so I'll be talking about Factor graphs so.

[00:10:24]
Any ideas from the room might you. It's a bi part that graph Yes absolutely so so factor graphs are graphs that I think look. You know in many talks people tell you about new mathematics new frameworks etc I actually will not tell you about anything you really all that we do with the draw and with our center is all about Mum linear least squares it's really not much more than that OK So so what is fact graphs have to do with money in the nearly squares How can you think about money in your lease craze in a completely different way that you now think about since your fusion that you can invent new algorithms and invent new ways of using only new scripts to solve your sensor fusion robotics perception and even manipulation problems and that's the that's the key idea that I want to get across right.

[00:11:23]
So. Since we exported a keynote beautiful keynote for a presentation like presentation to a powerpoint presentation we're going to see some some mishaps right so blame Microsoft. Before I came to to to tech this this was this was what landed. Us this. You know test of time paper which is what the kernel localization.

[00:11:55]
Which also I didn't invent it was actually invented in the seventy's we we invented slash we discovered in computer vision to track hands by guy named Michael. And then I thought I was doing robot look at his Asian wits with Sebastian at the time and we we divided the entire world up into grids and we build a probability distribution of the location of the robot on top of these grids.

[00:12:22]
But that means it's like a Finite Element method you have to if you have a large environment you pay a lot in computation because because you have to divide up in sort of a small grid say $10.00 centimeter by 10 centimeter and the larger the environment the more storage you have it cetera so so this particle filter was a way to do this.

[00:12:42]
With samples so now the samples represent a probability distribution of the robots over time and it's obviated using the sensor measurements so that sense I think is updating its probability distribution and then act so it moves in the world according to where it is right and the nice thing is it only uses as much computation as the complexity of the probability distribution actually warrants so.

[00:13:10]
So it can also represent multi-modal densities just like the grid but in a memo much cheaper way and its biggest advantage really is that it's super easy to implement I mean you can all implemented within an hour OK if you know undergrads can do this in an hour grad students and they might take 2 or 3 hours.

[00:13:33]
So I was the 2nd of the 1st student of Sebastian to graduate to the 2nd student to graduate really is still working on Google's autonomy is driving efforts as my come to Marlowe and so he applied this idea of particle filters to entire trajectories and I mean instead of still sampling location you sample an entire trajectory and instead of the weights of the location particle be to be decided by the the sensor measurements immediate sensor measurements the weight of the trajectory is the quality of the map that the trajectory produces using all the sensor measurements across time.

[00:14:15]
So so we've shifted now from from the localization of the robot to building a map at the same time and that is what is called simultaneous localization and mapping and that's slammin So that's an important subfield of robotics. And this is sort of the state of the world in 2002 where we used a laser range finders that were 2 dimensional OK And so you get to the maps a video by right and uses from about 10 years later shows what you can do when you have 3 D. lasers so sort of still to form an image OK but you get you gets only laser points in a plane but you get laser points in an entire sort of viewing angle maybe even full $360.00.

[00:15:02]
And this is now a map of the entire campus of you Michigan created. By Nick and Ryan in fact using some of the factor graph ideas that Michael case and I developed but you don't actually need lasers to build 3 D. models right you can you can create 3 D. models using using cameras just you know we all create a 3 D. model using our left eye and our right eye and fusing the stereo but if you have any snapshots of a static scene or even a dynamic scene from multiple viewpoints you can build a 3 D. model.

[00:15:41]
And it's interesting. That this this large scale 3 D. model building started getting really popular in around 20062008. When the greats of the fields of computer vision had decided that geometry was done and we're moving on to something else all right so. He's like he's solved everything he thought and he moved on to medical imaging imaging the brain and at that moment no US Naval e said wow look there is this flicker of thing and flicker has like you know 100000000 images which was a big number back then and and using his figure images we can build models of cities completely automatically so he sort of took structure from motion or building 3 dimensional images to a whole new level and that's Spondon entire sort of thread of research stretching over 10 years fully automatic model building and that is the technology behind to Google Maps and being maps and here and all these.

[00:16:48]
These in fact Microsoft even bought a company and the only product of that company was a plane that was tricked out full of cameras they bought that company to make better maps using 3 D. reconstruction from images OK That's how important this is and so. So how does this work well so here is.

[00:17:11]
This is this is just blatantly stolen from from the US Navy and senior Uyghur Well if you do an interest in the broad make you get all kinds of images from Dubrovnik. And now the key is to find features that are in common between the images right so that there is a tower somewhere here and there's a tower somewhere there and they are the same tower and I'm finding that out is one of the key problems here after that you optimize you solve for a large optimisation problem and you can build cities scale 3 D. models which are in this case sparse points right and each of these little black triangles is is a camera tourists took.

[00:17:51]
Took an image right this kind of stuff I mean I don't know how much this is review for you and if you knew you know all this but this this still sort of blows my mind OK even though that that now at Google to have 3 D. models that are cool scale right it's still this movie that truth's me up every time actually at the same time and it was.

[00:18:18]
Very soon after I got to Georgia Tech I was working with Grant Schindler grants in room I sent him to forward him to talk him out but he still in the plans I hear. He started his own consulting business and that morphed into a company and the company got bought by another company so it's somewhere come to really make times you me.

[00:18:40]
But but Grant said well we can because you know so nobody's doing images across space and building a 3 D. model we can also do images across space and time and in Atlanta that's a very cool idea because because Atlanta has a storied history some of you are new to Atlanta.

[00:19:03]
But it was a key battleground in the Civil War and in fact there is a famous burning of Atlanta which sources differ on who started the fire right so some Some say it's the Yankees and some say it's the Southerners to prevent things falling into the wrong hands.

[00:19:23]
But what is less known is that you know following Sherman's army there was a war photographer which was one of the 1st war photographers who took photos all over Atlanta so we have we have photos across time of a 100 years of Atlanta our 150 years that as it grew over time so so it's with ground to be created and 3 D. models we don't have the time slider so now you can go back and forth in time and show what the 3 D. model of Atlanta is overtime right.

[00:19:54]
Fund. But Atlanta is just one city so so I asked him grand you can you can graduate but only if you do something. In a city that people actually really care about New York. Right and also you do it's fully automatic like you know all right so he did that.

[00:20:14]
He took a large image collection of images over time from lower Manhattan and fully automatic He created a 3 D. model the the segmentation of points into buildings we could reason about when these images were taken in time and then we created a very similar but much denser.

[00:20:36]
Sort of. Interface right where you could go from 1028 to 2010 when this work so it was a series of C.P.R. papers and so that was very exciting work. Actually recently. And Gary McMurray and my student who just graduated Jane dong we plights very similar ideas to agriculture if you think about it I think culture is also but a scene that is changing over time and the people that far I'm the lens are very interested in knowing which crops are growing doing well and which crops are not doing well and is there localized.

[00:21:22]
Problems right so. The work that grand did. To get to get a bit new work by by J. so we could do things like having large datasets that are taken over time using using a tractor somewhere inside Georgia I've never been there budging has done the trick many times.

[00:21:44]
And then you can see how the crop grows over time and we can we can actually get fairly detailed information about what happened in the field for example we had 2 years or 2 crops seasons and in the 2nd year they decided to plants all over an old roads so it was a road in the 1st season it's compact that erts but it was crops in the 2nd season and we could very clearly see the stunted growth of the crops were that roads was OK using 3 D. reconstruction or should I say for the reconstruction over time OK So all these problems OK From look at his Asian to slam to structure from motion to forwards ease torture from motion rights.

[00:22:32]
You can solve and you can talk about using factor graphs. So now we're getting tired to the meat of the talk.