You're basing it on historical data and
you might and

it shows you things that based on
history that you might not want that

that might not be the thing that you're
looking for so let me give you an example.

Great segue to my next slide.

So.

And then you might say
Well this shouldn't matter

because you're trained
on historical data and

like finally we understand why you
know he used a computer program

is just a homemaker before that you were
going to ask a question or say something.

Yes.

So Mark is a historian or
Hicks is a historian

who wrote a book about this actually for
anybody who is interested only in the U.K.

in the united in the United Kingdom
contests that So

here's here's a question so
you might Let's go back to hire a few.

I'm not saying they do but
I'm just just just or or Linked In or

whatever let's just like find you know
a company that that analyzes resumes right

so you might you can imagine you might
want to use where to Maddy's to find.

Keywords and resumes that are close to
other keywords so let's say I want to find

you know some keyword that's like a close
to a programmer so job programming C.

plus blah would be close to
programmer right but then let's see

there's also other things like quarterback
University of Vermont etc etc.

And then that tells you this probably is.

Mayor White Male Because University
of Vermont mostly why

quarterback is men don't
have women who play football

no reason why they shouldn't in my opinion
but anyway that's a different story and

then you have another resonate with
this is exactly the same thing you have.

One and then Spellman spell.

It is a historically black college and
then you have a softball team captain and

that tells you probably you know probably
a black woman and now what you will

you would have if you train is an allergy
detector which I mean they show in.

And out of Adam Klein his collaborators
there and that paper is that if you now

have the distance between the two
program or between the words

in this resume that is much farther than
the distance a programmer between us and

the other read to me because you've been
quoted the societal biases then you and

you use out to me to do tools to figure
out who is appropriate for your job then

you're discriminating against people based
on their gender or color or whatever and

this is just historical So we know if we
might even introduce new discrimination

that we didn't even have historically
because I mean that type of training

data that you're using and so this is
a kind of so you have these little.

Systems like word to many
might seem insignificant but

as you string them together and
are making really high stakes decisions

on whether someone can get a job or
not then it can create

problems that you might not even think it
could it could create that makes sense.

Yeah.

Yeah so in this case it's not
necessarily a limitation of the model

possibly we don't even know what the model
is like are you with us the one other

point I'm trying to make is that you have
these models of doing the things that

are affecting people's lives and we're not
they're not required to tell us anything

about their models that which will
top out later but yes so again so

if you take Jamie's class which
advertisement for her there are.

You know different ways to think
about this problem one is sure your

data set now should be
representative of of people

which is a great segue because I will
talk about that as an example too

is your metrics of classification
accuracy or whatever maybe should include

some notion of fairness what does
fairness mean the whole poor other

thing and that part I want to talk to much
about today to talk more about the data.

And yeah.

OK Any more questions.

We're at twenty minutes guys OK So

another example that was highly
publicized is this Google photos.

I don't know if you guys have heard
about this classifying black people as.

So one.

Point I want to meet is.

The types of errors you make and

the application that you're going to
use things for matters right so for

example if you were translation
system is you know making really bad

errors on Arabic and you're using it
to surveil people who speak Arabic

that that relationship matters right
again if your face recognition system or

whatever has errors I mean this is just
one example of an error I'll talk more

about other examples and you're using it
to surveil primarily people of color and

your error rates are systematically like
have you know they're more biased and

I guess people of color then that matters
because there's a double it's a double

whammy because you know you're making
mistakes on certain groups of people but

then you're also using it
more on certain people.

There is this is also another
reference that Joy pointed to me too

it's called the perpetual
lineup report and I recommend.

Read it because the talk in the United
States how one in two American adults

is in law enforcement duties right so
I'm probably in some database.

You might be I don't know and

they can look you up whenever they
can use it for for whatever now.

Research that I'm involved in along with
Joy shows that as you saw what we did

is we wanted to see commercial face
recognition systems that is it's

not a it's not a training system so you
can't tell me just change the data set or

are just fine tune the model can't
tell me that because I just pay for

this service I use it on some you
know let's say my smartphone or

some other I'm like a startup let's say
and I'm using to do something I pay for

this A.P.I. and I just use it right and
so all I have is what that A.P.I. does

what documentation they give me one speak
of me and this whole research started

because Joy was who was an undergrad
at Georgia Tech by the way I was doing.

Like she was just going
about her business.

Taking an art class and she's like you
know what would be cool is if I could

use like an open source you know open C.V.
like piece detection system and

I can use it to like paint walls or
whatever I don't know.

And she had some figuring
like trying to have those

thing detect her face that it
wouldn't attack her face and.

Get her roommate to text her
roommate tries it doesn't

get some other people to text them.

And then she puts on a white mask detect
her face takes off the way mask does not

detect her face so then we decided OK so

she's started doing the systematic
study right is like if we want to

see how widespread of a problem this
is we don't know again we don't want

to use this research work or
anything we just want to see.

Off of the commercial.

There is this like a systematic air so

we chose something very very simple
which is gender class kitchen and

there was a question of whether
you should even be doing that but

so this one was just like they had binary
class they didn't take into account

any sort of other you know generally
they would just say male female so

you show this A.P.I. a face and
it tells you whether it's male or

female and
we've what we found is that the darker and

darker the skin tone the higher and
higher their rates right so.

The darkest skin tone you have let's see
for I.B.M. is like forty six percent for.

Forty six percent for
Microsoft twenty five percent.

Remember Blake random chance
is fifty percent here so.

How do we do this analysis
the first thing we want

to do is we want to just be like hey
like let's look at existing data sets.

Where we can ask a question it's OK.

Yeah I'm going to explain that to you.

It's OK you guys can ask questions.

I thought you were like poop pushing I
saw someone pushing your hand down your

own head we were allowed.

Anyways so
how did we talk about your type so

what we wanted to do was we wanted
to look at existing data sets and

say hey let's look at existing data
sets let's like you know what we want to

see is how this does by a skin tone and
gender the combination of skin tone and

gender So let's first label them
by skin tone and by gender and

then we will do this analysis but we see.

I saw that even though so many people have
been working on face recognition for so

long all of the existing data
sets were overwhelmingly male and

overwhelmingly lighter skinned and so.

And then we found ones that were female
but even the ones that you know half

almost half email they were
like still lighter skinned so

we had to have more like diverse data set

in order to do our analysis so I'm about
to talk all about data now because.

What was your name you asked
us the question about.

Data so it's all about data so so
the first thing is you have to think about

how people collect data sets a lot
of us envision machine learning I

maybe it's just me I hate collecting data
sets this is like I spent my entire Ph D.

getting data sets and is the thing
that takes the most amount of time but

it's painful right so
what we do is we try to

get like the data says that our most
easily available use google search.

Has like celebrities or whatever we
generally don't do very much analysis on

on the datasets we use we don't we
generally don't think about the use cases

of the models that we train whatever
data sets so in this case we try to

first thing we have to do is collect and
you know so I truly did was.

She thought looked up which

countries have a high representation
of women in parliament so we looked.

Countries in Africa and countries and
in Scandinavia and to get to extremes of.

Skin tone so many countries
in Africa like your skin tone

can you know the variation
is really high so so

let me talk about this why do we use race
because race class cation of races is very

it's very weird like it's very unstable
across time and space what people because.

Black you know here versus.

Latino or whatever is different or versus
white or Asian whatever is different in

different places you can have this you
know certain skin tone characteristics and

be the same skin tone and be considered
different races eccentrics cetera so

we want to have a more objective measure.

Of differences right.

So we found.

The countries with the most number of
women the highest representation of women

and their parliaments and we got different

countries in different centers of this so
you were asking about what it means.

For the what what's the point
have to type sneeze.

So we used this thing called
the Fitzpatrick skin type.

System so they use it.

In dermatologists use it when they're
trying to figure out people's

risk of getting cancer so
there's six bins and

there's one two three four five six
now there's even issues with this

system because type one two three is
reserved for people we consider to be like

European descent right at everybody
else in the world is four or

five or six so but
you know you have to start somewhere so

we started with this
classification system so.

So what do we do we were like OK
let's look at the overall accuracy

of gender classification systems
of these different companies.

And when we look on the on our data
set when we look at it overall ninety

three percent ninety percent eighty seven
percent sounds great let's ship it we're

we're done that's pretty great pretty
good accuracy maybe but we were like OK.

Let's look at it by gender
now you like OK you know

female faces there's like you can you
start to see a gap right like OK ninety

seven percent ninety nine percent almost
perfect on bail faces and then on female

things like seventy eight percent and
they were like OK let's look at my race.

I'm sorry race I should not say race
because it's not by race I explained

exactly why it's by skin type OK
please correct me if I say that again.

Then you see you like by since
I've used you start to see.

Gaps and again.

Like here you know any seven percent
of our cases worse than ninety nine

percent seventy seven percent
versus ninety six percent

that we're like OK let's
now break it down by.

Gender and skin type right so

in all of these cases the lowest accuracy
where on darker skinned females.

And then again here you look at so
that's from Microsoft.

You look at Peace plus plus sixty five
point five percent right you look at.

I.B.M. Now after this paper came out they
have all definitely Microsoft and I.B.M.

have improved their general
systems they wrote all like

press you know stuff about it so that's
great and this I credit to joy because

when we were writing papers she was like
we should you know send an early version

of the results redacted with other company
names to all of them to give them time

to improve it before we publish
the paper things like this right so so

now there's been a lot of improvements but
this is just kind of to show the existence

of the problem because there are I
mean there's a lot of improvements but

you can imagine how so
these kinds of issues extend to other.

So the problem so
what are some of the key lessons

one is that intersectionality
matters are going to talk about.

Like I guess things of ideas for
a critical race theory so

I love this example because I saw
it on a TED talk by the woman who

who coined the term intersection
started so this example is from what

is it in one thousand nine hundred seventy
seven there was a class action lawsuit.

Or I don't know if it was a class action
also about five or six people were suing

General Motors saying they
discriminated against black women.

Their hiring practices discriminated
against black women and

then the judges were like OK so

they hire a lot of black people and
they hire a lot of women so

you cannot be discriminating against
black women but then when you

saw what they were doing at the time they
hired women for secretarial positions.

And they wouldn't they wouldn't hire
black women for secretarial positions and

then they hired I'm sorry they hired when
the first secretarial positions they would

hire black people period for secretarial
positions and then they hired men for

factory jobs so they would hire black men
for factory jobs but they wouldn't hire

women for factory jobs right so
if you're a black woman you're screwed

you're not going to get a secretarial job
you're not going to get a factory job so

you were being discriminated against but
if you only looked at gender and

you only looked at race you would just say
no it's you know this doesn't exist so

so you have to look at
the intersection of these things so

when we are using machine learning
systems to affect all of these people's

different people's lives in different
manners we have to look at intersections

of things now this gets starts to get
complicated very quickly right so

in order for in for for
the example that I showed you earlier.

Like people test things out you know they
have data you want to make your dataset

representative so you want to test on
you know I don't know short women under

you know forty with a row who have glasses
and who are you know like this and

wearing scarves you know it starts to get
it starts to get pretty complicated and

this is where I think that we
should think of different models so

I shouldn't have to see every
representation of every person to come out

with an accurate model
OK Another key lesson about data

is that we should think about where we get
our data sets from who's represented and

who's not represented right
when we're doing research.

And another lesson I think for

me is that we can't ignore social and
structural problems when we're working

on any of our technical fields right
machine learning or computer vision or

robotics or reinforcement whatever we're
working on we can't ignore it and I.

You read this paper with me right but
yeah so I love the steeper It's called

the wall characteristic of the moral
character of cryptographic work and

it's professor at U.C. Davis wrote it
it's a whole it's a long paper but

you know as it touches on crystallography
later but it basically talks about how

cryptography has sanitized just like
any political you know politics or

structural issues or whatever it's always
like what Alice and Bob are going to

you know send a message Alice sends
it to Bob and some adversary here but

you have he was talking about how
like if you were to think about

the social structural problems the ways in
which who is usually surveilling who and.

Things like this then you would maybe
come up with different solutions and

so he comes out he goes through different.

Things like let's say differential
privacy and other things and

describes their flaws you know.

If you were to put it in this context
he talks about you know physicists and

how they the new nuclear
disarmament the move toward

nuclear disarmament happened
things like this so

I recommend the book I mean this this
paper which might as well be a book.

And but it's an easy it's a it's a fast
and again so you know in terms of social

structural problems in the U.S. we kind to
be educated right we need to be educated

if two people commit the same crime who
is more likely to be arrested and why

which neighborhoods are heavily policed
and why you know because in the U.S.

for example the people who are heavily
policed look like this and

you know right now it's interesting this
last year a lot has happened right so

you have you know people
at Amazon working on face

recognition to to sell to law enforcement.

And you know the whole project and
thing that was going on at Google and

then you have to think about who is
creating these products right there is

really very little interaction between
the people the products are being used

on some of these surveillance products and
the people who are creating them right.

If I'm talking about high error rates on
black women I don't see any black women

there I also don't see any women actually
like well where's Waldo of deep learning.

I can't bought one but
my point being that.

You know the paper on gender.

The face recognition paper
that I just discussed.

It would not have happened if Joy was not
in the field I don't know if it would have

it because even I am not a very
dark skinned woman right so

I wasn't thinking of these
kinds of problems but

because this didn't work on on her and
she started and she has

a context on the social structure issues
in the country etc That's why a lot.

People are now thinking about these
issues so I mean we should think about

you know we it's all of these things are
interrelated as what I'm trying to say.

Another one another key
lesson here is that.

Talk about you know think about higher
view or any of these other companies

there's like no law that says you know
who can use our data sets and A.P.'s for

what reason you have the Food and Drug
Administration right so look for somebody.

Like a new food or drug whatever into
the country there's a group of people who

test it out look at the side effects see
if it's a you know if it's recommended to

be used in a particular way except for
we don't have any of that for

data sets and A.P. and and models that
said Right now we're just using it for

anything regardless of how high
stakes The scenarios are to get a job

to see if you you know should you should
go to prison for how long are not except.

And some laws some of them you might break
the Equal Employment Opportunity Act or

something like that that says
you shouldn't discriminate by

certain categories Reg You don't even
know you might not you don't want to but

you might be.

Breaking existing laws with

the various models you have right now
not you but I'm saying we collectively.

So so well I always advocate for
standards documentation.

You know regulation and.

And so I want to close I think some other
industries have been have been there and

there are more mature industries
that have been there so for example

I come from the hardware background
I used to be a circuit designer and

in circuits you knew I mean
when I came to see what happens

in our research and other in our field I
was so shocked because the circuits when.

I'm designing a circuit into ice
work at Apple intellect some system

I was very well acquainted with
a data sheet for each circuit for

each component it doesn't matter to are
there any hardware people here yet so it

doesn't matter how simple it is the could
be a resistor cost like once that it could

be a C.P.U. whatever everything comes with
the datasheet So what I'm looking for

a product I go to it here's the datasheet
right there that's the website and

the datasheet tells you many things like
It tells you no it's all test temperature

you know what happens not ideologies
across temperature like when you modeled

a resistor for example there is like
an ideal model but that's never the case.

What happens under different
scenarios also you know let's say or

is it take it even something as simple
as a resistor a resistor to be used for

railway applications need to have
really different tolerances and

characteristics than for
like some toy or whatever so

take face recognition you know it's
the same face recognition thing but

if you're going to use it to say someone
is a criminal versus not versus some like

playground thing I don't know it should
have very different characteristics.

And I should be able to know whether or

when I'm using your model like whether
it's appropriate for a given task.

So for data sheets here I think so
I advocate I think we need data sheets for

data sets pre-trained models etc And

I think that again we need to have
information regarding standard

operating characteristics recommended
usage how did you gather data for

example it's OK maybe you
want to just use a model for

people under ten years old right so

then if you have a did the fit of
the dataset you're really seeing that it

was trained on was just for people under
ten years old that's totally fine but

I just need to know that I shouldn't
use this model on people who are over.

Eighty years old.

But I need to have some
of these guidelines.

And right now we don't how we have zero.

And so.

And also like when you
look at these gender.

The first thing in your is
there's zero information for

example when you look at who when you
get data sets from the internet or

when you screen things from the Internet
a lot of times young children are not

are not represented also a lot
of times you should probably not

try to recognize the gender of really
young children because it's not

many times not possible right so
again we can give very simple guidelines.

And.

Again I there are disclaimers
you shouldn't use this for

high stakes scenarios such as determining
You should be a criminal because we

haven't tested all that that's
a disclaimer in electronics you have

disclaimers You should they say you know
this component disclaimers I don't know

if you should use it on nuclear power
plants or life support systems and things

of this nature so we have these kinds of
to clear this claim or isn't there she.

Yes.

The V.P..

Yeah.

It's the challenges of future work.

No seriously I'm going to talk about so
I think this is a challenge so

part of the things we have to think about
how did it for example data sheets in.

Electronics become a standard because
things you know it's much worse of a P.R.

disaster to have
a component that's burning.

Or whatever very publicly
than it is to have it

right so and
security I think gives this example which

is great in the security industry a lot of
times when people had security failures

they didn't want to breaches they
didn't want to they wanted to keep it.

Under wraps until it became more
standards just disclose right away and

fix that right so
I think that in my opinion like so for

example these issues of bias and things
like this they had gotten some some press

in the past so that it became not a good
thing for the company to ignore it and

one of the easiest ways to start
addressing the problem is documentation

right so bet what you want to can and
cannot document is a different

story right let's save you know
Company A wants to give characteristics

about the data set that things were
trained on that count Company B.

has a much larger dataset so

you don't want to say we treat
you know our model with X.

number of people because Company B.

used data that might be large or something
like that right so maybe then you can just

disclose the distribution certain
distributions and things like this so

you know that's I don't really know the
entire answer to that but I think you know

like P.R. has a little bit is something
that could help and I think in the end

academia we could encourage these kinds
of things you know in publications and

things like that like how do you encourage
reproducibility how to encourage you know

in many conferences it's harder to publish
a paper I don't even know if you can do it

really really care like a lot of people's
paper is you know reviewers will come and

say well the data has not public right so
so in academia we could also

start encouraging people to have data
sheets for data sets actually there's like

three data sets that I saw that have
data sheets which makes me happy.

When they're releasing
their There are projects so

I think I don't know we can try to move
the field the field as a whole has to move

towards I don't think you can it
can be like individual players.

OK So again I was just talking
about like you might want to

have a distribution of age skin type that
cetera fits a human centric dataset so

we have a paper with Jamie and
other people called.

If she's for datasets if you're
interested in more flushing out this idea

about what kinds of questions we should
ask there's a different section.

And so you can you can check it out.

OK.

And also a lot of different it turns out a
lot of different people I sort of thinking

about this idea even though we're coming
at it from a different perspective like.

We're coming at it from a perspective
you know what did we see like OK we in

hardware there's there she thinks there's
other people who call it like I have

a paper called the Data nutritionally
able they're thinking of it as food and

there's nutrition labels for food there
are people in N.L.P. were thinking

that like there should be so concurrently
it turns out all of us kind of were

like We need to have this this thing so
a lot of people are thinking about this.

So OK Again this comes back to
your question companies might not

want to disclose certain things and
how do we incentivize the field

already talked about it so
close I want to talk about just.

Like other things we know about
cars you know about cars and

in the US You know when cars first came
on the road there were like no laws

you know there was no stop signs you
can be a ten year old probably in drive

there are no driver's licenses there's
no drunk driving laws you know like and.

And a lot of.

Tests of safety tests not a lot all
safety tests basically were done on.

Dummies with prototypical meal
characteristics adult male right so

then people found that a lot of
car crashes women sustained and

children sustained a lot
more injuries than adult men

ran it took a long time in the U.S.
to specify that your crash test dummies.

Should be representative of like
different physical characteristics

which gets back to data right and.

And similarly.

And.

And it was this is my favorite thing
there was a court opinion asking whether

the automobile was inherently evil and
like you know like judges wrote on it and

stuff and I think that reminds you
of the current conversation about

is it inherently evil is going to kill
us all except sometimes I think it's fun

to look at the conversations people are
having on other transformative industries.

Another really important
one is clinical trials

which is a lot about because she
did allow breeding for this.

So clinical trials first you
know there were a lot of

illegal experimentation on
vulnerable people right.

God like sleeves.

You know doctors were experimenting
going to college for experiment with no.

Pain medication or
anything like that soldiers prisoners.

Etc etc Right so

there's a tension right you have to
experiment somehow to develop a drug but

you also don't want to exploit people
right when you're when you're developing

right so so actually clinical trials were
illegal in the U.S. until a certain point.

And once they became legal
again there was no requirement

of who should participate in your
clinical trial before it was deemed OK

to release the drug right so
a lot of times women.

Were not and
did not participate in clinical trials and

I just saw a Newsweek article that talked
about how the genomic revolution is going

to fly over Africa one more time because
ninety nine percent of genes that

are being studied have no
people of African descent.

Even though Africa has the most genetic
diversity right because I've been there

I don't want to get into the reasons for
that but the most genetic diversity and so

this could actually help advance research
but the other issue is you can you're

you're not going to create drugs
that will not work on a large.

Group of the population of the world so
and so

because of this again lack of data
representative data to test your drugs on.

A lot eight out of ten drugs say sure
that eight out of ten breast that were

pulled from circulation between ninety
ninety seven and two thousand and

one disproportionately affected
women negatively affected women so

so the whole point is that you
know we should draw parallels from

other things that have happened and
are happening and it took many years for

standards to be plays then for
people to you know like for

example make sure that they
were representative people

in clinical trials and car and
car crash test etc etc So

basically what I'm trying to say is that
we should learn from other industries and

that sometimes it's good to kind of zoom
out and think about things that are more

big picture what are the effects of
things we're building what are happening.

Kind of questions.

You know sometimes it's good for
us to kind of think about that.

And it's not all about equations and

you know things like that will take
more questions if you have it.

Thank you thank you but.

So this is a very good question right so
the first thing I'm advocating for

is to understand that data so

if I I as an insider I do have
some some knowledge because I was

working at Microsoft Research at the time
so I have some knowledge of the data but

that's her hire Terry so
I can't really talk much about her but

what I'm saying is that when I
let's say I get that A.P.I. from

Microsoft I should have that information
I should have information saying this is.

What we were trying to do so

that's the datasheet part right while we
were trying to do when we gathered this

data set is to represent the American
population so that I know.

Or what we were trying to do is to
represent everyone in the world

between twenty five and
thirty five great then I know so

that I can make a decision about
who I need additional data such as

to train the model is this Can I even
use it or for this task it's that or

Right now you don't know any of that
you have a black box of a model and

it's just given to you to use for
anything and that sounds crazy to me.

Internists.

In terms of like
overrepresentation representation.

I think there's different ways you
know it's statistics people like

they do a whole bunch of
stuff like you know we and

other and there's that like all sorts of.

Things that are proposed to.

Kind of look at and to take into
account an imbalance in data sets and

also not just that but
the metric you should be optimizing for

you know is it like OK to have
ninety nine percent accuracy.

Certain groups of people
versus like you know.

And then like twenty percent accuracy in
other groups of people is it OK to have

a model that does that or should you train
a model to have sixty percent accuracy and

everybody like you know
this is this is still like

there are different metrics that people
propose constraints that you know

that people propose that optimize
different pain a spectrum.

But I honestly believe that I don't
know I'm just thinking high level big

picture here if you will
swear to me that in order to

be able to classify someone
who is five years old or

something in some manner that
I just you know I have a very

difficult time unless I see
that kind of data in my

training data I think that's a limitation
of the algorithms we've developed and and

the approaches we're taking which is I
guess Judea Pearl Right talked about how

people neural networks have shown a lot
of advancements that we don't really know

causalities just curve fitting things
like this right so so I think there is

an opportunity to have like more reasoning
and you know inject some more knowledge or

something like that rather than
what we seem to be doing right now.

OK back and then we go here.

Yes the.

Statement is that's from Emily bender.

Yes definitely very aware of it I
guess I mean it is her she she.

I think now it's accepted to a journal
that when we vetted her paper

it was anonymously.

Submitted to open.

View so that we you know I know
we've talked and stuff yet

but but she's an N.L.P. and so
her ideas were from that perspective so

that's what I mean like every we all kind
of thinking we need this thing but no and

then my friend Kareena she's thinking
about it from chemistry perspective I

guess is like hazardous materials or so
I don't know anything about chemistry so

we're all kind of thinking like
we need something like this but

we're approaching it from
different directions Yeah.

Yes.

Well yes in fact Cheney can tell
you more about our conference

which is going to be in Atlanta right.

More.

People.

There not just.

Yet so it's very important to have an
interdisciplinary Persis which is like my

whole thing rightly because

you have to talk to people who understand
it particularly domain and I and

I would say this is not just in
the bias machine learning except

context I think this is useful and
any context when we're working on

another paper I like is machine learning
that matters by Kerry is that Kerry.

Forgot her name apparently it was
a controversial keynote I.C.L.

but I couldn't find her but
there is her paper not you know so

she talks about the shortcomings of the
way we're doing research right now the way

we're evaluating research.

And you know we are like optimize the one
percent on this data set you know metrics

and what does this one percent mean and
which context you know

my paper might not get accepted
if I don't show you this so

she talks about some of some of these
things and I think it's really import.

To work with legal experts for exam this
disparate impact some of the ideas and

in machine learning
fairness are coming in from

the legal domain right and and
like the book by Virginia you

things that I mention which is automated
inequality She's a social she's worked.

In the she's been a social I don't know
what I don't think I've activists for

a long time and she knows a lot about
poverty she's as she has a lot of data and

historical perspective about how poor
people were treated in the United States

there were these things called poor houses
and things all of which I didn't know

about and she knows a lot about how like
right now some of these automated tools

are I'm fairly affecting certain groups of
people she's done interviews with them and

she has a lot of insight right so we
should be working with people like her to

say what should we do
because some people say

it's not you can just have an algorithm
and optimize you know for fairness and

you're good sometimes you shouldn't have
this algorithm be used in this particular

context and also sometimes that the goal
of the algorithm should be different maybe

you know it should be to figure out what
kind of intervention you should take or

something like that mean that maybe not
to classify someone with a probability of

you're fifty percent likely you
know to commit a crime again or

something like that so we can go and solve
a completely wrong problem and just go off

here and optimize the wrong thing without
talking to all these other people.

I think OK yes.

Yeah exactly which is why we you know so
one good thing I saw is

in California there are starting
to pass a law to regulate.

Face recognition systems in use
in law enforcement right so I'll

lot of the issues also people don't well
I personally did not know the what it's

being used for are when every day a new
thing pops up that I find out about rape

and I think that's an issue too I had no
idea of a higher view until like you know

a couple months ago and they're being used
internationally by six hundred people to

figure out who's going to get a job like
that seems like something serious right.

You had a question someone
in the back had a question.

For.

Yeah this is a really good question
because some people argue particularly I

see this argument from a lot
of business school cool people

there was even like a Harvard Review
article saying that these out

one thing that I don't like the term
they say humans are noisy and

I am and yeah because we're complicate
humans are complicated right I wouldn't

say noisy and so then they say that
a lot of times these algorithms

they have less bias because humans
are actually higher view what it says is

that you can they say that
actually people who use them were

able to have more diverse groups of people
where they were able to hire people with

more diversity is what they said because
they said they removed the human bias is.

We don't know so I think this is
a complicated question because I

believe that you are right
like there might be cases

where an algorithm can help show
you your bias maybe or maybe

there might be cases where an algorithm is
is less biased in a particularly way but.

My issue is that without studying
any of that without pinpointing it

without quantifying it or
anything like this we're using

one algorithm to do to one company or

whatever to do mony things so
you might have had you know thousands of

recruiters in different this year and
not distributed in different countries and

different whatever looking
at different resumes so

maybe my bias will be different here
your bias might be different there and

now if we replace all of them with this
one algorithm maybe maybe it's fine but

it's biased in this one way there and
then like you know everybody's going to be

affected so so so
I'm not a poster however thems and

generally like I'm I'm
in this field like here.

But but I'm just saying that you
know we need to be cautious and

we need to do testing and we need to
have documentation and we need to have

regulations to figure out where we can
what characteristics something should have

to be used for something else for
a particular task I think there are.

I know within my you know all of
them from what we're partners or.

People that are with.

Something I wrote the reals.

What you see you know we.

Go through the process.

Of going.

To their.

Mass of course.

All your yap yap yap and then there.

Will be.

Yeah.

Like for example what you're asking is
I said that there is a data sheet for

a resistor with various tests and various
kind of performance characteristics

why are those tests in this case right if
it fits the data if it's a data set then

we wrote like the paper we wrote was like
outlining the questions you should ask and

they met in the things you should
measure for a data set and

the question is like for
a model for like a A.P.R.

what would it be really good since
the early stages you don't have the years.

Before coming from your own
you're building yourself

you know sort of what you know on him
like yeah yeah yeah you don't have to do.

So I think this is a complicated question
and I and we wrote a paper about it like

that's not that's not published yet
but but basically for let's say for

a very simple questions right for the for
if it's a human centric thing right now.

Like what are the false positive
cost negative rates for

different groups of people.

That's that's already a very calm.

QUESTION And then that's like what do
the different what does it mean to be

different groups of people I think so
when you're.

Doing these tests you can you
know you can start with like

you know groups that we identify
in the U.S. maybe and then.

You know do some even like
simple test like this and then.

The most important thing in my opinion
is to disclose your assumptions and what

tests you have done you know because then
you are you know you need to it's never

I mean we should never expect a model to
work perfectly in their every scenario for

every group of people for whatever
like what I what I'm advocating for

is more that under the understanding that
that's not the case so that people can

understand which scenarios you're thinking
about what are the tests that you did

we don't I'm telling you we have no what
we don't know right now like for that.

A.P.I. that I told you my first
question was OK what are they right in

the documentation nothing what model is it
not we don't know what kind it is that we

have no idea what the accuracy of some
publicly available data is that NO idea

was to dismiss the thing so

I think that disclosing your assumptions
and the test that you did and

what use cases you're thinking
about is already a pretty big that.

You feel OK maybe he can be the last one.

Yeah.

Yeah.

I know that at Microsoft they were very
interested in having data sheets and

I know the same at Google because I guess
like for us we came up with this idea

because from an engineering perspective we
think it is useful for engineers the worst

thing to do to engineers I think is like
or or other people who are in a product

cycle research is very different you know
I you know when I was an engineer and

I come back to research I thought I would
like maybe last time because it's in

a completely different feel completely you
know but now I'm thinking you know it's

good to have that into understanding of
one when people are in a product cycle and

how how things go etc It's different
in research so the worst thing to do

is go to some person and be like hey you
can't ship your product has not fair

what does that mean what's the process
what tests do I do what like what but

this is kind of like already coming from
like for example data sheets I things

that are already used within
a product cycle in engineering before

you release a product and
it's also something that actually is good

internally for companies because a lot of
times what happens is different teams and

companies share data like you use it for
this thing somebody else's use it for

this other thing but they don't
really talk about the they don't know

the characteristics of the data who got
her there for a while with the original.

A use case and so
that's also helpful for for

like an internal teams to communicate so
what I've seen is a lot of people

in industry interested in it including
my startups and things like this.

All right.

We're really thank you.