[00:00:05]
>> Because indeed it's not surprising to tell you people may be differently they're being watched and we are all being watched on line. In so does that mean you will you might change your behavior as a result of our privacy concerns but also for all the behavior online is Dana and Baron to get into all of our lives you see I mean how did you build them from.

[00:00:30]
And that's how the number of them how's it going is in terms of our machine learning tool. For example if they're watching horror a secret or maybe acknowledging to my machine very very mean I have to make sure that these tools can be incorporated in things like beer and we all believed to be important as well.

[00:00:53]
And also want your approach to this from an economic perspective in the markets pretty you know. It didn't come from people and you don't want to ship sure they don't need you can compensate them for example the proves you are. Worth their costs you're indeed. Changed since then in question how compensate people here get out of well that really is a kind of I'm doing that now how do you know you crave see the good leader what if you do well who result in the data for not.

[00:01:23]
A little more problems than if we just did was she learning on this like user generated you know we might learn the wrong things and these new cars can pull all Hannah had had like I was going to do can you please give Peter your intuition are there do you know where he'll different or last or perhaps more feasible things not in Sylvia's lead neatly apply.

[00:01:47]
Over the same tools I'm going to have 4 more in the wrong stall because the pedal chosen did actually assure. And we're going to tell you what this is from I haven't seen the 510 I actually do can about a differential privacy if you don't know what I highly encourage you do kind of like I was just on line for this topic is sure it's just going to say into Louisiana time each person data is going to have a really small impact on the output and if you're curious you can get like this in my house on one of Lulu database computing and.

[00:02:24]
And he ponders maybe we're all going on and they get a little curve. The Not to mention one for the data in face meetings this I can see on her intellect. And her and you're going to read her here and he's like he's going to close in some I got form all I got 25 times.

[00:02:43]
And this tool is like a whole bunch of years 1st of all he is short of crazy because I am really not learning from your data and learning from the population and the facts and we can learn about this new thing even if they had never seen him or are.

[00:02:58]
They going to most more or cross hands it really isn't just to build any notion to lighten her she'd very much this has made her turn my input as a small amount to get out about the not and to this insurance you have things like you know generalization you see I'm reading in.

[00:03:17]
The. Just an old lady is going to get an answer do you know things are going to be. Outliers on the you can sign with a new one to use and they're not truthful is it giving you and isn't top up by data you can actually give us lower costs of purchasing Do you know Committee saying every promise and not the insurance claims are going out to hunt for last me answer that.

[00:03:40]
Is the one person in the in very interested and I think even. When it comes to learning. Going to settle in these things has a difference of privacy and is in to ensure that we're not going to look ahead to actually eating fast and also produce some things they may like a hot fire that is both here and is accurate.

[00:04:04]
Right because it's because it could be your you indeed to be able to get there I feel like throughout my entire career you know. I was on the ranch and going to there is going to talk which is we're also watching the accuracy into how this term led a group there in the skin to find more precisely in the next time.

[00:04:26]
But for now things are not just sitting there and they treated you groups of people to see why hey if you talk about something like talking to you give you that is the mike raiser gender is placing and then he fears you with respect to that for sex natural beauty is that they how they got the same 1st end.

[00:04:44]
In Holland actually copied from the same person and they are of different ways it is the business news sure as I get treated badly badly. Injured here is our question can you how will privacy and fairness in and some work done with psychology and we're going to turn to his reason I'm here who is to look at our former undergraduate student route and.

[00:05:13]
Compare on during our own time he's been elsewhere. We have little to play for as a result in 10 years and so for us we say no you know you have you cannot achieve exactly or not and right as he has the time in the Other than a possibility result in the very broad.

[00:05:34]
Based threat any OS acquired is going to be exactly fair it cannot be a question like in this is sound as we look crazy and he let it be being fair as a lot of her own. Doing. More positive results being of use or perhaps you can if you are exact as it hits you will in fact be the theme across as one of our across 100.

[00:05:57]
Groups and introduce about the same and we have a leg up printer as approximation notion and then how can an approximately figure we're going to introduce you can people all. You have to resolve there are one is a leg either in general Aloha and usually he's not going to be doing that as we also have an efficient algorithm.

[00:06:19]
In the gospels and maybe an entire hour here. And this is a one question. Answer This is kind of is heaven needs to answer about privacy I don't care. But still want to return to the question. I'm going to deal with coming from people you knew going to parties often times you seen people and you're trying to treat different groups fairly You just can't immediately anything like it would be highly obvious what is clearly is hard to collect data on then use words like Don't know as much.

[00:06:56]
But it was much higher her home games I'm enjoying population in the last wired not to give loans to man and cared for Chas if you think you just don't know about this other group it is hard. Hard. Hard to let out what do you. Need your just smaller in the disability fundamentally unfair in the we announced well to do in this case there's groups where is it hard to have a copy or are they still want to be out here.

[00:07:29]
And here telling a picture of history across and how the colors here they are working on others and here is the kind of old fashioned survey process so we get a list. Not a survey really interested and people think well you know from the grand prize all possible addresses.

[00:07:48]
From people responding 100 homes and saying this I'm asking. You for some number between 00 and one is it is my population let some people respond is one dollar Ok and then I'm going to impute these upon the I received a. Bill. And I had to look I just basic harmonic actual.

[00:08:10]
Simple size and I was not answered it is really strange if you assume that everyone is being paid well that evening or the grocery. Is if you don't like challenges his or believes there is some populations who's going to die and he was off in his population and.

[00:08:28]
Isn't here and I have to live in their little and everyone has read and you know how I have had and only did when I got to people who were sending the entire Blue population when they didn't stand will you please do yours. And also be thinking maybe you know here it is a great.

[00:08:51]
Maybe 100 years how long it will really talk to me friggin will you're going to email. Turnings you very hard to let you know from the jewels you don't hire access to life how 1st of all if you are organized. Into this need in the morning then you need to learn things things that are accurate on these groups.

[00:09:17]
And so this is some of them until you can work on. When he has the courage of visiting students on here in Hong Kong. And the mall response reads you had individuals cost 1st year India is equally Why is it harder to lead times in pleasing people meet new people do you have a higher price on the cost of worship here.

[00:09:42]
But we're not and he didn't 16. And now here in these groups that they're aware they're hardly heard refined and probably you know more regularly if there are some groups who experience higher privacy cost a fortune and you know need to compensate them more. Is one goal is to design promise decency and $45.00.

[00:10:08]
Question process that is going to result in some fairly not times when it's really tough propagate through different stay in bowls. The entire scene bringing all the time to ensure that how does the matter if you talk to groups Carson had this subject to some constraints for example you would find it much.

[00:10:29]
And so that's a whole one. And it has a new dimension collapsed we're going to have no. Alternative he may say even if you learn about his population without actually we're actually like are just mostly cloudy do you know from them is there a way that I can think you know plug a hole in to do it cheaply on the red roof.

[00:10:51]
And you can understand how those findings are going to transfer into the Bluebirds do you have a different need for it to be able to get it will you have to shifting things along by the end of this of the ongoing work. And figure out how to solve this problem the kind of rates Fernandez we're seeing.

[00:11:10]
Going on it's really kind of question I like the next I'll turn it over to our next speaker and you think he's going to turn. To Tom Moreland was visibly I was going to go through. There has yours and your own is ready right. Thank you very much so that we got our baseline of a lot of different things that could come up in any of your work for like you know I'm a little.

[00:11:44]
While earlier Dr Gupta sent up on or just a little background on her. Basically she works on something that is well beyond me as under-educated manager call referral commerce and robust optimization applications by the way that stream your words exactly where the restaurant doesn't trust you all right we don't know what you're doing.

[00:12:04]
But in all my learning to give a disease maybe under our original information that starts to get really interesting for those of us who are not computer science or math people because we do understand a little bit of I don't know enough but I'm trying to understand what to do right and what about Amanda.

[00:12:29]
Thanks for the introduction to me I mean I don't think. I have one just only it's really not me I would use from so I might call coffee come with me and you can try. It out might not be sitting on my legs of 90 days I mean thank you Peter because I do think you actually you do is argue something and I do see if you want me to get a hold of you maybe like how you do question isn't it we're not a special beings Ok just things that he knew about in fact he isn't biased.

[00:13:05]
Strong feelings of optimization So the question I guess the 1st question is what is it when his body does is it Ok to National Grid next mission statement or does it actually connect me to tell you that you put in Washington jobs so for instance it was. As an army of units My reason I representing the Temple is that all of these West or American partner research with an annual assured that moon are showing them in Europe enjoys much more than men and that seems like a medium very good only show really annoying just to give you what happened was in the initial stages of the I didn't really even include or always action is a charm and then we have for the hiring which are say so what is that what is biased is bias is the absence of being as biased on your giving are equal in using the mission shouldn't we be in your feelings when dance for example by getting our president you should probably tell us where let's say you have a cd or you have the Eagles intreated insane do the job and to get into how you please I'm doing fine you can see how your views on things are how you view the hospital and conditions in the room and as we're listening to you actually so basic questions are not real and when Israeli t.v. would reduce almost 21 different measure exe and various means of communication there was a good used 500 rejects all off.

[00:14:28]
Then what's the fascination of measuring where you're going so what is great and how do you get it was usually not something we've been moving on in terms of my the objective of musician you can leave I think your solutions are very good no matter the dish notion of fairness you will hear about I mean he is hearing about machine learning now machine learning inherently it's different positions for different people with your name and the question in the you know how can these issues change we're going to be unlikely to meet this individual is an illegal or civilizations was in goodness so had a good day with our.

[00:15:03]
Own you know if you lose a lot of credit so then so just like a big spender question is Where is an ordinary machine learning fairness with respect to women checkers their method is to distinguish measures of barriers and where you think you know which measure should be gold and what are the proxies mission was no more and this was actually there's a lot of interesting questions in their machines this 2nd bunch of questions you've been thinking about in Bank of cards so Nancy I think you want to fire you I know there are some artists leaders and soon even to write a review including I'm going to get schooled on nearly I'm looking at a performance of my vision and these are my reasons I'm going to be barriers but I'm one of those eyes And what is governed by this is I can actually exploded and see what it was like Guys how America higher positions are where the bankers and artists have always clashing decisions to different schools and so when history is always going to assume it's going to start soon I was using I was was there to high school but what is the reason for this is because people are going to be trained for these tests or when because.

[00:16:09]
There was a little dehydrated the students really strong schools so they were go to the question of is there this where is the fire is doing is they can back to think you know like when you fire trucks you can or fix it you say we shouldn't. The next question is.

[00:16:25]
Is this something we just started going wrong. And is it is uniting that little noise in the ground here is answered in or do you know that you know how very different you and different groups of people but instance I believe you can see that you never noticed the difference and Slim is on what we do part in which we are indigenous and there is a and concentration of people who are learning in one who is proud of her and this is a little hard for her as a result of sort of an invention you don't do in group worst documentary shooting that looks good in the mind you can see how to be gritty and you can have this in the room that is in there your head is in womb but if it is that's understood as tree and unsung heroes in their lives stream in others so the impact of not is even astounded in that you know how difficult missions is within a matter of noise can be made and so what is going where does that effect of going to suddenly maybe using a bunch of conditioning in commissions and in the things that you need foreseen that you would maybe some I never consequences that are going to be things going have to do noise in going to there isn't really actually going to expose but it's unclear if Yeah given you know it was a good leaders of the real reinvention of the word used to be I would not actually corners any odd little indie on the move system being strange and indecisions floater eagerly it's given me a certain good you need of a calming when you and your lawyer do a little good even going to you if you have these credentials what people are doing as a resource in the media trying to by showing up to be sure and it'll be the world is getting more higher rate so you believe how much information in your own position do you know about of your I mean it is going to use so that you cannot get in this list in the Army to night isn't it because when the only condition was a News March the where there is next be a little transparency and we can sort of move you from the books I'm going to listen to one of the.

[00:18:28]
Thanks to the Fords and individually d.c. has been saying all of these months into the arms that I have no meeting with glory I'm going to Sheen in my eyes mornings and it's actually interesting you know g.p.s. is moving you know when you want to use the United Nations if you explain and I was in the meeting doing the beginning there you might actually you know to argue with them or Mark has got to do the even more if you're going to do that you do as many things in acting as you know there's no doing that in the middle and we are new in this as young as we were so these decisions actually don't even worry these contributions because even more importantly we'll be there you can more than the sources they call you so in the u.s. if you have an hour and all you're doing the night the night is only going to go is a national resource and their policy is going to vote on those who tries not to mention what they see is that you can easily come to shoes of the riders and distribute our greens are know whose kids and so I'm just trying to get in your piece one of these are you know a very important thing for you out of the blue are the moderates for people who are exactly moderates are going there's police in the mission that you know if you look as even today in the past because on the only house from which there's I mean we have because i even though what is your prior days doing equal number to me is too difficult having your fire very efficiently by being a good news fosters wasn't good we need for being good next our guys who is our you know included where you can't care is it they wish all nations should be careful moodily ordinary What is the cost of not having to harm our how dramatic I responded with reason why are there then names in question there are actually I'll Find requests to use on militia interest the nation by the Sudanese I've been going to sheer ignorance as a debating I view this in the sense of them by.

[00:20:28]
They didn't need our love mission you know you are I actually distributional critical resource and this is another very important Iran right now that you can you not. So nice and back. Then you sure do and he's young good of the kind of growth you do in terms of just going under it isn't too much so a lot of times ahead of you to do is remember him as Franco You know Marty how should things be out of this union How should people be evaluated using didn't questionings you going on 35 then but still let's go over your life example you.

[00:21:04]
Still didn't or highly risky my genes we've been trying to evaluate people in their heart every 5 years losing national that's good news and we're in there is good and this isn't ours and. When we consider the remaining 9 years so the biggest ever students who haven't heard resources United and unit move through your border to prevention are going to be legal and they didn't want us to lose we're not going to make us little money are we going to use it I'm their financial guy was observers the next thing we know is how do you get one said something smaller than where did you get yours and then the question here is how how can these 2 can be rid of the moon you know unbiased would so why didn't you know as I'm warning you now the legs are going to start you know this was actually our distribution good notion in the ninety's article and the spectrum of the market I actually wanted to was log musician because I think that is are the guys decision and the 1st tattoo is diet is going to bring you back and decides in this end zone.

[00:22:12]
So in the sense of. The starvation of jobs so you actually have a large party that her mission then the note I'm with under the hood in the car can actually need to look I mean it was a good you see her listening is good. You know since they don't hire you from whose word into your minds I'm 1st 2 matches in fact even if that is a smile by yourself or indeed since been imprisoned just a little invisible to the pension and you graduate who is in distribution into the bridges there and then maybe you did mean back to school which is going to name their own which is beautiful and go the go to explain when mining is where you should why do I believe you then you can of course school is much worse than the nation with enquires engine on cars.

[00:22:59]
So if you look at Utah school because the result we're going to lose more I know some here is that it's from our mates and I got home I was your idea and I was going to read some of these like you in the last little Ana Summers much more and that's a compelling.

[00:23:16]
Example of why we should do the answer the next question is what it is emissions what can we do it can we can't use or sisters are used to do as you normally do much good income from you can look you do know which I mean there's something on how much more there might have been there is and so what we should do is that yes you can actually go and listen or individuals you can talk you go sure is who's doing the schoolhouse sort of millions and ordered all school is in order or at least a live in school is going to enjoy the middle of it more as if you I spend more time when there was you need to do something as an extra.

[00:23:54]
Test on you might be a little thing on my chest more in terms of the. Insults and you're trying to come around simply not where you're more interested when you find your tech support and here you have documented your insurance. Couldn't you give us the key is why did you release I'm going to do this or how do you see.

[00:24:14]
The midnight hour I'm going to say there's a lot of going on a budget is just not good I do suspect you of your partners are going to produce at least I have someone with United Nations from out there. We did our nation's freezing your eyes out somebody good bones of the let's get a sense I'm going to it's like this isn't going to bite me and from see a very good meeting actually so I can see I'm going to give me just a few.

[00:24:43]
Good contributions to shoot from No Are you sure your school in the business school really so we're going to use English people who come together and contribute to this is actually. Happening. On the. Right so. Don't darkness setting up she's a writer all the other school interactive computing here I've done research for good and Facebook and postdoc work at Cal which even though I'm not a c.s. person I love the fact that people are gathered after I left in my undergrad.

[00:25:20]
Work focuses on Computer Vision Machine learning you've actually coming to us on a little jet lag or super grateful she just got back from the I c c b which is a huge vision conference the whole thing why on earth are you just dropping out so the great thing here is the notion of vision you know I'm supervised and semi-skilled right models yet again my people who are really sure what you know watching us out but the very group artist facial recognition the whole use of vision for so many things that people are loving is also calling for concentration so was it starting for me that she's going to talk about how she does great work on not you know Mr President not with you when you're married you meet huge pieces to figure out x.y.z..

[00:26:20]
My private life I guess. I mean my work is mostly having like you know computer going to be human. And this problem of fear Nancy and I don't find it amusing in learning it is one of them that receive public hold water over training computer consumables I'm going to take this opportunity to illustrate just a couple of scenarios where violence becomes really really appearance.

[00:26:47]
So we get started I want to 1st think about the inherent pipeline that people use when they're tuning machine learning models. 3 Start out by collecting a whole bunch of data. And then if you want to hear standard supermarket learning the next step is to ask people that you know and see the data so let's say you were trying to recognize don't pull out all the examples here and make sure that they're they will correct the underdog.

[00:27:16]
And then a trainer model that you will to take as input these images and produce output the energy channels of the people whom I didn't like dogs. Well then the out stuff here is not we want to validate our models and this is where the key problems often don't really need to hear in their minds.

[00:27:36]
So I want to focus a bit on how it is that people tend to validate their models in machine learning and then specifically in computer vision. So look at the think they are all in the puter vision we love benchmark our benchmarks are where you collect there just aren't enough and everybody works together at so in this problem so here we have this image that our image is not in a very Mars being a being.

[00:28:07]
That transcends all the power of image congregation going to every image there is one object and the goal of the algorithms is being able to reproduce the object label that people have entered here preachiness emission is huge because there's a 1000000 things in the engine and then you.

[00:28:28]
It's nonstop is to recognize that 1000 different object. Using things like different breeds of dogs whether used be a cop a cheer if you will these are the kind of things that then he says very fine grained categories and then people are not necessarily very good at wrecking during the learning you're an expert in that particular discipline.

[00:28:53]
But in any future what we have is we have to do these millions are injured and there Donald entering hospital he can call in and what we found over the years is that we're able to produce models that are increasing for garments to such a level that this challenge is no longer even offer we can only test that from his initiatives that we can recognize these that I think our growth is with near perfect performance.

[00:29:22]
So this even is like really compelling use this seems like we've found how crappy code and need we've got all this problem of graphic 90000 out of the race. But it's a little bit deceptive to do a little just consider recognizing things within us that I didn't get.

[00:29:43]
What they were not taking that big model that got me your current performance and done asking it to recognize this dog in this video. And it turns out that I got a huge present not model train of millions of images can't recognize this going all. So why did there are this huge discrepancy in performance between what we saw on the holdout tough guy on image that versus what we can recognize them in simple video.

[00:30:17]
And this is really how long do you think that my. Sumo you know here is going to get trained a mortal to understand the concept of dog using only. Holes from the innocent didn't that. Image that was collected on ice cream being still small media sharing sites. Now in turned out that mean how our own implicit bias when we think about what images we would like to share with my mother.

[00:30:46]
This is the data given to collection and so what that means about most of the things that are screwed are things like flu here people don't post as litter think they are posting photos because they're visually interesting. Trying to take maybe a couple of photos and upload the one that they thing is most classically correct according to Kentucky so what that means is that hop like a dog and this really interesting when you're talking about anything in an object it's probably going to do especially you know holes or people going through something like a face that's just natural that's what we find is discriminative and we find most interesting to share you're also going to find that most of the dogs are in the center of the frame.

[00:31:32]
You're also going to find people probably took care of you post the photo that had a nice landing condition that have high resolution. And you know you think about what the miners the people will decide to post when we're using the video medium or with video you're oftentimes trying to share an interesting action like that they're all running you're going to have a very different bias.

[00:31:58]
So if you're you might be. Confronted with lower resolution frames motion where. An a wider variety of holes it is then you start to not fill in cedar trees I can't really imagine somebody cannot go and close enough litter. But the real old them that is that you have this huge performance your.

[00:32:23]
So this is just an example dogs but a lot of our algorithm. They're being used for recognition how that really the people. Who Live think about how mines this issue of bias become a real problem when you house turns up population that are not represented in your original dataset are you thinking that.

[00:32:50]
One flashing sample that really thing you did on this particular home was his work culture and their city and then start trying to understand how you could recognize the gender of different people along different after skin type. People are going to do that when she looked at one of those people from Harlem and across different countries down from countries that historically how the darker skinned individuals and so on from countries doesn't really have layers going individual and that I have yet your leg up because they are collecting photos of people who are in politics who oftentimes have a head shot you can really get photos of people in almost the same pose so you kind of remove some possible confounding factors here and I'm collecting it I've got images to be able to recognise gender just across these different skin type variation.

[00:33:51]
And to be able to label difference in terms they rely on this it's not clear in pay scale which was actually done all of a dermatologist that has been used a lot in this hearing with letters urging your t.v. images been printed in trying to fix points you know but being exaggerated into Congress is one of the 3 and 4th there.

[00:34:13]
And they can record this meeting that he did the 1st row is there really does not mean highlight here it is just as great not even to be connected to the mortgage balance across in tapes then higher. Does that mean you get a better representation for you know running home office from foreign across this period and.

[00:34:36]
Then also how to better balance across gender so there they are given that is modern relevant being separated out of bullfighting it's going to take Well I'm saying here here. And not surprisingly when daily 3 different oppositional image recognize there is and trying to plan to this date is that what they found is that darker skinned individuals had lowered their performance in darker skinned female individuals hard workers so there has been you know individuals the best performing model the Microsoft model the top got 20 percent but when you look at the latest in males they hide in 0 percent here so this is a pretty huge difference.

[00:35:29]
But these are just off the shelf models so we're going to throw out what we did not mean if we kind of took things into account during training. On how to work out last year that I had approached this topic how well my office of our thems work if we changed how we treat him and I'm going to look at the data here and specifically we're interested in seeing whether or not there's a factor in the performance of being able to even just recognize the present that occurs and.

[00:36:05]
So we use the data that that is years or around developing models for doing something i mean time. When you're trying to develop a model to run so Graham Carr you better hope that it can recognise that stream. And you all the other will not conduct it under the fairly virus against a particular cell population.

[00:36:31]
So the 1st thing we did was we took this off the shelf data back into that is is one prince might be used. For developing models for something rotten and we just enter to get it to you will see what is the distribution on lighter skin versus darker skin individuals using that name this conference in case you know I'm in the red line in the letters and the line is darker skinned you can see that there is a huge rise in this dataset towards lenders and individuals.

[00:37:04]
And not surprisingly performance on writers gamers of the darker skinned individuals here. Shown here and there if there is also a difference in terms of performance. And we're trying different techniques different ways there and removing confounding factors different ways of understanding that are not their son after violence in the you know God.

[00:37:29]
Maybe some individuals appear more often to different kinds of Danny these young individuals appear more often included and we try to time for those are going factors but we still got a lot of is there's a bias towards death and. So one thing that is very important to understand and not if we are developing models without having viruses the 1st time that is and not during the initial training days then we're kind of doomed to repeat the skill year of collapsing you don't have that how to how examples of good particular under represented the population all to be able to train them interactively but also to be able to even value we actively We have so few examples but it's unclear on what things can signal premiums between about the performance over models.

[00:38:30]
It's also unclear whether or not that's simple techniques like rebounding the leader is going to solve the problem we heard from the previous 2 speakers that sometimes you know when you have people number of examples for me and here in different formats I mean I necessarily again via the reins of the end model.

[00:38:51]
You know one thing it leaves in terms of computer vision and oftentimes there's a possibility of having different types of sensors. In unclear whether or not we need to go back to the drawing board and think about whether or not the sensors that we're using are unnecessarily Myers in certain people.

[00:39:13]
And. I just want to briefly in China for $1.00 or I'm biased that's not very nice going and this one is thinking out instead of understanding them by as a representation of people themselves whether. Often things that people interact with as we go along and moving forward we're going to see even though the algorithms that we develop are going to be used to improve people's lives improve different aspects of your life and if those aspects are going to be biased or not becomes very difficult.

[00:39:47]
When our greatest I think Aaron are overrepresented in certain geographic region so that is I'm not even from the given that on our higher number in June this year but soon. The current object Republicans that's going to have a very biased performance and recognizing everyday objects I mean if you're in different geographic region.

[00:40:12]
Let me take away the ground where I've shown you today is that 1st and foremost I want to convince you that I do you know that is really a very very important part of the puzzle and we need to be thinking about how we're collecting the data and we definitely need to be about how we are evaluating our models with different kinds of data.

[00:40:35]
Either idea that you can collect is going to be even that. Bring in the link will on a side of it and use it as your only test that is flawed because any bias that you highlight in your initial training is going to propagate to that task that it's really important that we analyze the performance or a model across a variety of different views that collected with different biases.

[00:41:02]
And then my last closing point is that I feel now a lot of the bias work to doing is really taking a reactionary. After between your model after we've collected and you know that we don't think about what are the potential dangers of viruses or the concerns will come population that we would like to an alliance whereby you can go back and what you've got data and I read our models along those lines but I think that if you.

[00:41:29]
You want to go models that are going to be fair we need to hire out of here go out there approach we need to be able to get real mechanisms that hinted at a different forms of bias and analyze our algorithms or we get to the status of the point something and finding out that it negatively impacts us or until I get.

[00:41:49]
In here. I mean so how are we are a believer not a. Thing you heard is pretty cutting edge right now the standard you were humans honey I promise I absolutely every one of their leaders in their fields talk my value with the president is Daschle I think the wrap of their hit something that when I said people don't trust it is this honest approach to what you do and the new leading edge of scholarship that we're seeing that is changing the way people understand c.s. right is being modest not Goober landings of wheat with your words out and saying Yeah we are actually think about what we do start right so as a side note I spent 2 years or Google as an academic research council what that meant was I was reaching out to try to build bridges between the us folks and between poles you folks trying to mourners change others and so often it was great researchers who just didn't know to be very solemn I've been a lot of times it was exactly what you described it to be and I think well this was cool it worked.

[00:43:00]
Of course with the tiny details about where I was doing because I got really really great knowledge in detecting something and then he gets launched as you said to a world I mean some other things quickly. And the name of the work of the visionary Shrike like frankly part of me wonders how this actually highlights future research for instance.

[00:43:21]
The skill itself is a funny state of 1st Internet because it's actually about do you will at least under my my mom does. Cancer treatment and as someone from you know South Asia I don't know that's the only thing he said. Anderson all he's going to do it as well about whether you have seen your sister you know got it turns out you can still get his skin cancer religion leave.

[00:43:45]
So there's already little things there's a bill that's cool but hey we should throw that too. So what about building on systems that look East Asian South Asian South East Asian I would imagine for car systems that's a whole world throw in South America a truly fascinating set of immigration patterns so our assumptions about what a person is.

[00:44:08]
Whatever that means get challenged by the work you do without. Legal questions are answered yes we would group around a little longer so questions please. 000000. 000000. Yeah that. So the question is this one. Ok All right well Dick I'll speak loudly The question is just so everyone heard.

[00:45:33]
Oftentimes the pedestrian detection you may not even ski see much skin so when we were doing the model about I was showing here. We. Made sure that we only looked at examples where m. Turk label or could see skin anough of it 3 of them how to consensus in what they labeled but I think this brings up an interesting question which is that now we've removed a whole bunch of pedestrians from the equation where there might be bias there but we just can't detect it so it's kind of brings up an interesting question where how should you even go about evaluating these things and oftentimes even though there was enough skin to maybe be able to classify into lighter skin and darker skinned that doesn't necessarily mean that it was the fact that the skin changed that caused the drop in performance in fact there may be many other reasons why and so that's what I'm kind of getting at the last point which is that of course we want to make sure that the algorithm is not biased against something like skin type but the bias that we observe in skin type may actually be a result of where what streets those people walk on it might be a result of how frequently we went down certain streets in the daytime versus the night time are the streets tend to be more congested are there's more clues on people walking next to each other all of these things sometimes are related to a particular some populations but don't necessarily translate into the visuals changing.

[00:47:19]
All. Maybe sometimes these whimsical mutations that you all understand a are they well the look within to see this leisure Here's a little piece of move too quickly on this was cool and b How was that expressed compared to what we were doing to think of the public in the words maybe sometimes what we're seeing is very great stuff that fits from an academic perspective is getting crowded and deployed as you're you know very you know all sorts of areas just to get with up and you were do better what's going on with dental problems.

[00:47:56]
Yeah sure. Thanks have that Mike it's a. Yes or no so the why are you going to run on to say in the military is very new in terms of the like an academic discipline. We now have. We now have like 2 new conferences on a fast star and force this is the 3rd year of the Fox startup conference and so this is like a 3 year old field and from a conference that is a force it would have its 1st conference this year instill into this isn't really a brand new growing field and it's getting a ton of attention which also means for the audience this is a field in which there are it was there are still a ton of just Lake example where on the open question a ton of kind of research flow he fruit and so please do work here is research there are so many problems.

[00:48:56]
The also means in terms of interlink going to point these things in their product areas it's a fascinating field. Because it's a new we do will come out and a comprehensive answer Yes of course here is how we use all fairness in all aspects the driver we can say sitting down got it you can do that and that's to pull it out and then look back at the Ok maybe it was actually like you know quality or street or these other issues and for this purpose being done in a very literate fashion because there is so much need and so few concrete answers in this field.

[00:49:32]
And I think. To add to this to act towards a change is said I would say that there's actually a very strong need to go back to the basics and really thing about you know why I'm even thinking mathematically about this problem does that make sense does it make sense to think about these people as valuable as does it make sense to mourn them as numbers does it make sense to observe these attributes about them if I'm looking at their online behavior what things gone is that been a proxy for you know knowing that the collective result so really going back to the basics like you know by a b. developing a machine learning I victims or Wyoming or donating something and I decided I think a lot of that thinking is missing from the Me computer science and mathematics in the water and start to take note.

[00:50:17]
I think we need to get back to. The educated aspects of that much. More we're going to look at questions there was someone in the back the woman in the striped shirt please with the glasses on her for yes yes. I'll take the 1st part no. And there is actually in my world ongoing research and arguments and it's messy and for those of you who live in the world of mouth you have every reason to be disturbed by us it's not that precise lawyers will come up with these terms and where you can add is being clear about what type of expandability define it based on literature is anyone who wants stuff.

[00:51:18]
They all know I can help a little here but there are about exactly what did mean and no one's sure that will spill out depending on lawsuits frankly maybe before that I like the notion of let's get ahead of the curve but one simple point was when people were demanding the notion of transparency as a type of understanding of our thems all my c.s. friends would say something that again sounded a little like jibberish about nontrivial complex system it's just not going to work Devon it's like I don't know what's going on but when 4 or 5 c.s. people tell me that it turned out what everyone is regurgitating is basically I guess what's called Rice's theorem.

[00:51:58]
You were living that we're going now you guys don't know what we're doing but bridging that in explaining nonetheless great work is done with that as an outer edge limit all of a sudden people like this are like well but despite that there's a lot we can do to build from the start to make it possible to understand a complex set of code so there's a lot of moving targets I don't know if you have more on exactly what you would mean from a c.s. for Explain ability or interpret.

[00:52:33]
Yes I agree with that in tacitly which is to say. Something if. They feel sorry for me will. Include. High level. Do you know. That you don't understand how how they work really. Rather than work on it problem or I'm not a problem or magic free. Right and that's what is this.

[00:53:15]
Good intel. I predicted a lead with. Something the castrator not because they didn't know that if you were looking for a date he'd go back and say owes me I was told you they should hire you and so that's like you can't do that. And so there is a field in terms of like.

[00:53:39]
Really Also if you wish feel the thinking about it like honestly there Bill you. Have a machine learn something that is like simpler it's only. To slay the performance and eat meat and I can say Ok so a 100 year old and a trained it works well but not me finds needy like a decision tree on the island.

[00:54:04]
You can look up Thompson eat out the. Role nationally the coverage of the deli. And saluted so that and the like you natural to a person and he is it will probably be because this treatment on this is not not perfect it does not say this is why this is why in my life and you know created a break in the here is something approximately possibly like maybe that is the article.

[00:54:32]
On that quick note there was someone who pointed this out to me and I was all like That's awesome as a lawyer I love that and then I ran it by a friend of mine who works on really big things like Yeah but that work this was in medicine so that's like 30 features and he's a guy who works with 2 to 300 things at least he's like not interested for what I do so each moment you're going to have to keep asking the questions that all 3 of these proposals have raised from a different level which is like Ok does that work here or not and what's my limit but I just wanted one small point and you're explaining who you are you will.

[00:55:10]
Training you. Really don't be fooled into thinking that if you can get another. Right. Here now. You understand. I feel a lot of her. Problems and the sound. Of the fighting 3 sons are here buying time and not just doctors language careers. Being moved forward you need more power you know in her abilities.

[00:55:44]
If you will look on her for being a model. For more but at the end of the day we can be better. We are the best of our ability we start to. Hate we're at basically the end side is he a lot of questions I'll take that we have what one minute left 21512115.

[00:56:14]
I think we have to apply to the room please one of the present has to run because everyone is you know Georgia Tech is busy busy busy enough that these folks on. A mistake let's say thank you and please hang out and grab people for your other questions thank you for coming pay attention there's great work you're doing this will be a simple terms from what I told people inside Google build it what this in mind it will launch it won't get pulled and that's a big deal you want people to use your stuff and use it well also do take the answers classes that some of us teach even though we're not math driven We'd love to see you thank you.