All cheer my tongue so

long to do the.

Restart.

So the third time I swore if you lectured.

I was wrong and

health policy is there terrorism

is also just so short here being real.

I need a full retreat to see if you speak.

So I'm sure one

of the great stuff in the media will.

Go all kind of horrors I
won't go into the long view

because of very few.

So I know

what it comes first one we

all know because I see the real guy.

In this insight into one of the I

read Air Force people if you find
the coverage of those long term goals.

Rescue.

And also the many applications.

I want to coop your own personal
no I was not a student.

Really.
Everyone was my hero.

Since I was a grade years.

Brooke.

You couldn't refuse to be equals.

You know ANY told get a small to learn.

And I was I was we all know but
that's not what this whole because.

He has this region unique feature
along with he's able to blend.

Theory in some We've reached them.

Such as.

These people they thought.

Honey we're chips if you knew do was
a kid you know one of these people.

I meet you.

In that he says since beauty
is going to be easy for

the which I don't let me only use others.

Now.

Thank you Jeff.

And though thanks for the invitation
to it's nice to be back here.

It's been a long time since
I've been a Georgia Tech.

And my being amplified.

And you know you can you hear
me in the back as they say.

And yeah you'll hear about baseball
players and Shakespeare again today that

this is a talk that is not aimed at
specialists a statistical specialist but

rather was billed as for
a general audience.

And a general scientific audience.

But of course it's about statistics.

So.

What is what is statistics.

Good statistics well isn't Information
Science is the first information science.

And etc concerns are learning
from experience but

not all kinds of experience
a kind of experience or.

A little bit of a time in usually in small
contradictory occasionally contradictory

pieces and so
that's what it means when the newspaper

says statistical argument show it means
that there isn't one smoking gun.

It means that there is
a whole bunch of different

pieces of evidence of them put together.

And we've all got used to
statistical arguments.

Political polls up to me logical
study batting averages the cetera.

And.

All of those are classical things that
reflect the direct way of getting

statistical evidence you want to know
whether a drug A is better than drug B.

you give a bunch of people drug a you
give a bunch of people a drug B.

and you compare the results
direct statistical inference

usually done in a frequentist form.

But there's the title of the talk today.

Concerns a different way of getting
evidence indirect evidence.

That it's some point may seem
actually irrelevant or dangerous and

yet is playing a bigger role in
modern statistical applications for

reasons that will eventually get to
it will take us a while to get there.

So here's here's a little bit
of direct statistical evidence.

We're following
Republican Roberta Clemente.

Through his nine hundred
seventy baseball season and for

this talk you have even if you
know something about baseball.

You have to pretend you don't
know anything about baseball.

I really should probably
have chosen cricket or

something like that where you really
don't know anything because actually in

real statistical applications almost all
the ones I see I don't know much about

the subject at least compared to
the people who bring the problem.

To me.

And Clemente this is one nine

hundred seventy baseball season and he's
going through the season and at first.

Keep in fact was a batting average of the
number of successes decided by the number

of tries.

And at first it's very
very highly variable.

He got one hit out of first to fifty
percent five hundred point five zero.

And it's varies wildly at first but

you get by the time you get
to forty five at bats tries.

He's got one thousand hits and settling
down eighteen out of forty five is point

four zero or as baseball of fifty and
I dos say four hundred a magic number.

And for the rest of the season.

He didn't do quite that well.

He batted there was he
got one hundred twenty

seven out of three sixty seven tries
in the remainder of the season for

three forty six during
that period of time.

And so this is a good example of learning
directly direct statistical learning.

The we're interested in how good Clemente
is a batting we put Commenee at the plate.

He tries again and again sometimes
succeed sometimes it fails and

we put it together by taking a simple
average and we get a dancer.

Here's another kind of
statistical learning.

That's more in line of what
I want to talk about today.

This is a sound like I made this up but

it actually happened I was
with this is this friend and

her husband of the Stanford student
union and we were having breakfast and

they physicists told me that
with happiness that they were

going to have twins and due to the sun
a gram they knew they were twin boys and

suddenly the mother who is
the who was the physics the.

Asked me what's the probability
that my twins will be identical

rather than fraternal twins and

I guess Dennis decisions are supposed to
know the answers to things like that and

I said so I am an experienced
difficult consultant and I stalled.

I said the doctor tell you.

No I'm not and I was lucky because
the doctor told her that the proportion of

identical twins is one third
which is a peculiar number for

genetic things usually they're in powers
of two right about I checked later and

one third is right in the United States.

So one third of twins are.

Pairs of twins are identical and
two thirds are fraternal.

For no particularly good physical
reason they but he knows.

But of course the physicist
being a physicist wanted to

wanted more information than
the general statistics and.

That's where Thomas base comes in.

English Nonconformist minister.

Devised a rule for combining.

Information from different sources.

IMO feel like the rule or
not it was published posthumously

by a fan other famous scientist
in the Royal Society.

And rules a rule for combining evidence

in this case we have from the physicist
case we have two sources of evidence.

We have those prior odds.

By the doctor provided by
the doctor one third two third's

that Favre is not identical one to two.

But we also have the sonogram.

And the son of Gram favors
identical to the one.

Because identical twins
are always the same sex.

So there is twice as much probability of
seeing twin boys if they are identical

as if they are not identical
the knot is fifty fifty and.

So that's two to one in favor of
identical and base rule as a rule for

combining evidence in this case
is the rule is very simple.

The you post your Iraq as the prior
odds times the likelihood ratio

one half times two is one fifty fifty.

I told them that and

they were very disappointed because
it sounded like I was guessing right.

And it would have it come out
of three seven four seventh's.

They'd still be talking about it.

And.

The biggest rule is
extremely important idea of

his first breakthrough in scientific
logic since the great times and

essentially it's founded statistical
inference as a mathematical subject.

A key ingredient of course is that based
on prior distribution that the doctor.

Provided.

In this case.

So we've had two examples of
learning from experience says

in our statistical way Clemente
learning from his own experience.

The kind of methodology that's often
called frequent Just as opposed

to base in.

This is a slur from her own
experience of the Sunnah gram.

But also from the experience of others.

Presumably that one third two
thirds rule is based on hundreds of

thousands of previous twin births not all
of which were twin boys probably a fourth

of them were twin boys and somehow that
information is being brought to bear.

And big methods.

When they work when they're
applicable are wonderful and

statisticians love using them.

But the in fact they're

not terribly popular in in
statistical applications.

Not everywhere anyway.

And the the trouble is that the.

These prior distributions.

The one that the doctor
provided here aren't available

in most scientific problems.

And so the kind of objective prior.

That was provided.

And so that people have to
resort to subjective priors.

And that that works fine if you're just
trying to make up your own mind that

you're trying to convince other people
in a contentious area like drug trials

the subjectivity is not called for

and learning only from only
from one's own experience

has been the dominant mode in
twentieth century up locations.

And the Holy Grail is does this is
going to is the experience of others

without needing a subjective
prior distribution.

The quote here I mistakenly said Fisher
it's our it's actually Jimmy savage

enjoyed the basically an omelet without
breaking the Basie and eggs and

that's a way of claim trying to use
the experience of others in a way that's.

Frequent just simply
solid at the same time.

And I shouldn't leave you give you
the impression that only basins use

the experience of others.

Frequent is also within

certain narrow ranges of which
this slide brings up one.

This from the frolic you lab
of Dr Myers the kidneys the.

Dr Myers is a very persuasive man and

he persuaded one hundred fifty
seven healthy volunteers.

To have their kidneys evaluated by
a rather rigorous set of tests.

And the and the graph here there's one
hundred fifty seven points the green

points and
the each one is one one of these.

And here's their ages along
the horizontal axis their score

better scores higher
along the vertical axis.

And even if I hadn't drawn in
the least squares regression line.

You would see that kidneys
get worse as you get older.

Sad fact.

Not unknown to everyone in the room and

the in fact it used to be that
you could not donate a kidney.

If you're over sixty.

But the need has gotten so great that that
rule is no longer in force and as a matter

of fact a large proportion of kid needed
ations are from cadavers these days.

Anyway I suppose.

Suppose that we had a new
volunteer come in age fifty five.

And we don't have Dr Myers
there to measure how

good his kidneys are and but we want to
get a measure so we can tell a prospective

recipient how good the kidneys
are people pick and choose actually.

And.

There's two ways.

We might do that from this data
the first way is the direct way it

was exactly one subject in the hundred
fifty seven who is age fifty five.

And that's that's that person's dot.

And we could just report that
number as a direct evidence of how

good that kidney is.

However most people
frequent a sorbets Ians or

anybody would prefer just to read off this
fitted line at the height at fifty five.

What Tookie called borrowing
strength from the other points and

borrowing strength as a learning
from the experience of others.

Kind of situation.

It.

The the idea of this scrape line.

Here is called the regression line.

And I want to say a little bit later
about what regression means the reason

this works so well here is that
the subjects the extra information and

the subjects what makes them different is
a simple one dimensional quantity age.

So essentially the regression
line shifts all the subjects so

that the age fifty five.

And that's that's a wonderful technique.

It isn't always possible to do it though.

And there's there's plenty
of regression problems

from the problems where it like to use
regression where you really can't.

And here's here's an example
a baseball example

where again you're not supposed to.

You're supposed to shelve your baseball
knowledge for the little bit of time here.

In one thousand nine hundred seventy.

My colleague and co-author and
friend Carl Morris was

looking through the sports pages of
The Los Angeles Times it's April

the season is just about one
tenth underway and he comes

across the list of major league baseball
players how good their batting that year.

And he chose the there's
eighteen of the players that

batted forty five times each including
Clemente who was one of his favorites and

he made this table eighteen out
of forty five hits at eight at.

For Clemente point four zero zero four
hundred seventeen out of forty five for

Frank Robinson three seventy eight
sixteen out of forty five for

Frank Howard three fifty six.

Down here near the bottom of the table
is another famous player Thurman Munson.

Eight out of forty five one
seventy eight much less success.

The average of all the averages
the grand average here is to sixty five.

And Carle being a born statistician.

Wondered how well he could
predict the truth that the truth.

He took to be how well they were going to
bat in the remainder of the season which

was ninety nine times as long as
he already had and you might say.

And most people would say
even most professional

set of stations that there are if you
don't know anything about baseball.

That's what I've made you believe.

You have to use Clemente's data to predict
Clemente and Munson's data to predict

months and they're bad independently of
each other and I don't think they were

even the same league that year and so
what else information do you have.

Well there is another set of predictions.

Lift labelled them there.

James Stein there are much
more tightly bunched around

the Grand mean of two sixty five.

So you only predict two ninety for
Clemente and

you predict two forty four for months.

And now.

Carl rated made these two predictions.

And then waited until
the end of the season and and

filled in the truth column in
bold face here Clemente round

up at three forty six months
in a three sixteen etc.

And then evaluated the two
production methods by

adding up the total sum of squared errors
between the predictions and the truth and

the if you do that with this table you'll
find that the observed average was made.

Something like three hundred percent more
error three hundred fifty percent more

error than the James Stein.

So if you were betting money you
would even have done very well to use

the James Di and estimates.

And here's here's a picture
of what's going on.

Here the.

The average is the arbiter
the early average is the observed

averages are plotted vertically Clemente's
the top green dot there Munson's.

The second from the bottom green dot and

the truth is plotted in
the middle in the purple square.

And you can see what happened.

The truth was much more tightly
distributed around the Grand average.

Then then the original the the original
numbers the observed numbers and this is

the effect called regression to the mean
are sometimes called the winners curse.

And we'll talk more about that.

The game Stein estimators over shrink

toward the Grand average that turns out to
be a good tactic for getting a low square.

And I want to say a little bit about
regression to the mean because it's a it's

one of those great ideas
that's again simple.

Once you see the idea.

That's great ideas have the.

Property.

Here's the original example
of regression to the mean.

From Colton gold in the eccentric
Victorian polymath.

Really for tagging as the fingerprints and
scientific weather prediction and

he wrote a famous book on surviving
in the wilds of Australia.

Which I guess was Wilder in
those days than it is now and

months everything did
seem to be interesting or

controversy over and
here he's being interesting.

He got a data set and boy data sets
weren't easy to get in those days

of a Thousand father son pair heights and
he made a plot with a thousand points

on it with the father's Heights along the
horizontal when the sun's on the vertical.

And then he noticed that
the overall average was five six.

At that point but
he noticed a very interesting thing.

THE FATHER.

As Who or.

Who is mean what's the average mean
was five eight two inches above

the population average their
sons were average five seven.

The son fathers who were two inches below
their son's average only one inch below

when he called this regression to the mean
and the term regression is stuck for

anybody who draws straight lines on
things or curved lines for that matter.

And the.

The idea is really interesting and simple.

Again once you see it is that Earth.

It's even clearer in the batting
averages than Sun's Heights

example it's exceptional early performance

is due to so we made a list of eighteen
players and we arrayed them on the line.

Though the one with the top was Clemente
he was at the top for probably for

two reasons one.

He was really good and
one he was lucky that that

year he was there through him away from
the grand average rather than toward it.

The guy Munson was unlucky that year and
and.

The luck goes away after a while and
that's not the still.

Skill the the true goodness remains and

that's where the regression to
the mean and we saw that out.

With the eighteen players.

You can see the regression
to the mean effect.

And the.

We can get back to base here if we
were baseball of history in autos and

new and knew exactly the probability

distribution of all Major League batting
average just which of course baseball.

This is you know it was do no we could
just use base rule to produce to do

the prediction and
the base rule would have done.

Would have shrunk each observation.

Toward the program Maine and the amount
of string King would have had to do but

with the spread of the true average
is that we don't know those things.

And where did the James Stein
rule come from.

The Dan Stein idea.

Stated in these terms not at
all in the terms that James and

Stein work on games as will or James.

Charles Stein's graduate
student at Stanford.

The idea.

It's rest in these terms is that
we don't know the Basie and

distribution but
we can use all eight hundred numbers.

To estimate.

Both the center and
the spread of the through distribution and

then behave as if we did no base rule.

And idea that her.

Robin is called empirical base.

And I've been trying not to avoid
doing any math at the board here.

Here's the.

Here is the formula.

Here's the James Stein estimate.

Here's the grand mean here's the
difference between the observed average

and the grand mean.

And here's the shrinking factor.

Chosen cleverly so that if the number
of baseball players you put into

this formula gets big you actually recover
the base rule or something close to it.

But this isn't what this isn't what
Stan was really basically interested in.

He and James proved the theorem.

Frequent us there was a matter of fact a
frictionless there being one that's true.

Irrespective of prior beliefs and

the theorem was that we weren't
just lucky in the case I gave you.

James Stein empirical base estimate or
always beats the observed averages

in terms of total expected squared your
expected doesn't do it in every case but

it's expectation is always positive.

And when this was this came out
about fifty six and was greeted.

By professional statistician's as.

Maybe lunacy.

Certainly paradoxical
because it does say that

Clemente's good performance should
we increase our estimate for

months and months as bad performance
should decrease our estimate for.

Clemente and why is that.

Well if there is information in
the experience of others and the eighteen

players can learn from that the other one
another as well as from their own data.

Well and I can't resist these couple
next examples because they're so

much fun in pickle babes.

Seem to be in the air in
the one nine hundred fifty S..

Here's a story that involves in
the one nine hundred forty S.

to here's a story that involves
a famous naturalist and for famous.

Statisticians.

The famous naturalist is Corbett.

Corbett's working in Malaysia that
Malaya during World War two and

he's been trapped in butterflies for
two years.

And he makes this table.

That's a lot of fun.

One hundred eighteen of the species
City caught were so rare.

He only caught one each.

Seventy four.

He got to represent it as each
forty four three twenty four for

twenty nine five etc.

And of course there were some of the
common ones we caught hundreds of times.

And he asked an interesting question.

How many new species would I see
if I cropped one year longer

half as long as he'd been there.

Yes I guess he was imagining
staying one next three years and

the question is interesting
because amongst

other things it refers to
the column of the table.

We didn't get to see the zero column zero
column of the ones he hasn't seen yet.

And doesn't see.

Like there is any
information in the table.

But he asked the right guy.

He asked Ari Fisher and Fisher
are the greatest of all statisticians and

Fisher and then later with added
work improvements by I.J. good.

And Turing of Turing machine fame and
her bra buns the real preventer of

babies came up with what I
call the magic formula here.

To answer Corbett's question one hundred
eighteen times a half minus seventy four

times a fourth was forty four times a day
minus twenty four times a sixteenth

that forty five point two the powers
of one half come from the fact that

he said he was going to stay half as
long as he was reading the numbers

one hundred eighteen seventy four forty
four twenty four come from the table.

And.

Instantly at forty five point two plus or
minus something like eight or

nine and I lose track of
the story after then I don't know.

I don't think Corbett state it was wartime
maybe forty five point two was enough

to encourage you to stay in war torn
Malaysia where there was heavy fighting.

Subsequently.

But the empirical base story went on.

And again we're back in the same quest

same boat of learning from others

experience of others in this
case it's other butterflies.

And so where is that evidence
that the magic formula digs out.

Well here you can see it qualitatively and

then you can see that there
is some evidence there.

If if more species are seen once each than
twice each then there are probably still

more that have been seen of all that
sort of a qualitative way and now if you

want to make this quantitative you have
to take one course in probability theory.

And I said I was going to make.

This technical but for those who I
suspect almost everyone in this room.

You assume that there that each butterfly
species has a number are connected

with that it's rate of it's true rate of
being caught over a two year period so

far is two point five.

It's pretty rare and only has expected two
and a half occurrences in Corpus traps.

If R.'s one hundred it's not very rare and
the bays prior if you could

if you would know this number for
every possible butterfly species

the distribution of the true rates would
be the Basie and prior to the effective.

The one third to third I called for
the twins example and the empirical

basis idea is to use the data in
the table to estimate the days prior.

And then use various rule to answer
Corbett's question and this lied.

Is for those later you all have a handout.

For those who want to
prove the magic formula.

You'll see it's real.

This is a hint slide and
if your hand isn't enough.

The references at the end of
the talk will give it to you

that did the IT ISN'T THE HARD.

It's one of those simple arguments that's
really subtle and it's well worth.

Working through and you have to have taken
one course in probability theory in which

they talked about the focus on history.

Once you get going on the missing
species problem it's hard to stop.

If in one thousand nine
hundred seventy five.

You'd gone to the Georgia Tech bookstore
which probably wasn't Barnes and

Noble at that point and
I'm sure it didn't have a Starbucks in it.

You would then you went and
bought the complete authorised Shakespeare

called the Canon by Shakespeare people and
they're very serious about this.

They would be in that eight hundred
eighty four thousand six hundred.

Seven total words

of which there were thirty one thousand
five hundred thirty four different words.

This from a computer count.

And moreover fourteen thousand three
hundred seventy six appear just once each

four thousand and four to thirty three
twice each two thousand two hundred twenty

two ninety two three times each etc and
I have this time I put in the zero column.

The question mark question mark there
stands for the words that Shakespeare

knew but didn't bother to use in
any of his plays so he has thirty

he has an observant Cabul area of thirty
one thousand five hundred thirty four.

But we're pretty sure he didn't
even use up all his words and

now we can use the magic formula to
peer into Shakespeare's mind and

see what he had left off for us.

So I'll say this in dramatic terms.

Suppose that you go to Oxford England and
he go to the modelling and

library and they let you because you're a
famous Shakespeare expert they let you go

up stairs and in the attic you find
an old chest and you open it up and

what's in there but
a whole new canon of Shakespeare.

Stuff that nobody had ever seen before and
to make the story easy.

It's exactly as the new Canon is
exactly as long as the old one

in total words how many new words distinct

new words did you expect to see in
the new canon that we had appeared.

Well the now we don't have those
powers of a half the way I set it up.

The answer from the magic
formula fourteen thirty

seven six minus forty three
forty three plus twenty two.

Minus fourteen sixty three eleven
thousand four hundred sixty.

Plus or minus about one hundred.

And well you can say we'll wait
a second but if we found two cannons or

ten or a hundred.

Well the magic formula falls of.

Hardest as you go past one
year end of experience.

But Ron says stead.

Then my graduate student I used
related methods to put a lower bound

of thirty five thousand Shakespeares
on unseen the cavalry and

we broke boasted that we doubled
Shakespeare's vocabulary.

After these all these four hundred years.

And you might say this is
all pretty hypothetical.

In fact you can test it.

You can leave out one of the plays and

redo the whole calculation to see if it
predicts the number of unseen words in

that play relative to the others or
worse remarkably well.

OK if.

If this was a movie at this point.

There'd be a sort of
ominous background music.

I have a friend who writes for
the movies and she tells me that two

thirds through any movie something bad
has to happen to the protagonist or

the movie is unbearably dull and

the fact is that empirical based methods
aren't used very much in day to day.

Statistic scientific practice.

Scientists have a strong bias against
learning from the experience of others.

And they prefer maybe

we've made them prefer direct statistical
evidence usually meaning frequent to some

which dominated twentieth
century statistical practice.

And I wanted to give an example of this.

Of of frequent to some action and

then bring back the story.

To the good modern times where
in fact I'll make a prediction

that in the twenty first century.

Learning from the experience of
others will be much more important.

And this will be in terms of this.

Next example.

The last example.

So in this example which is real data

I have a gene which is named as forty one
twenty four in a prostate cancer study.

And the activity.

The genetic activity of
the one hundred two men.

Were measured on this
gene fifty healthy men

healthy controls fifty
two with prostate cancer.

And the numbers whatever
scale they are were measured.

And the mean turned out to be minus
point two in the healthy group and

plus point one nine.

In the prostate cancer group.

And so you might say well
it's a chance that genes

forty one twenty four is over
breast in prostate cancer.

And that's an interesting question if
it's true it might be an interesting hint

as to what causes cancer
prostate cancer or

how you might cure prostate cancer or
at least tell you my early detected.

Or give a warning sign.

But we don't know.

These means don't see much different.

It is true that this one
is bigger than this one.

Here's the histogram the blue
histogram is the fifty

healthy man and
the line histogram is the fifty two.

Prostate cancer patients.

And suppose now we want to

we want to know whether it really
whether the prostate cancer.

Level is really greater than
the healthy control by which we

mean something like if we had thousands

of men instead of just one hundred two
with the difference still show up.

And the first step.

There is to is to get a test statistic.

Call it here is evaluate
the difference of the means

point to old minus minus
point one nine the vote.

Added by a measure of the spread of the
histograms point one nine six the number

comes out to be two point zero one year.

And this brings us to the most famous

of all frequent this method
the hypothesis test.

And then the pot the stats
state things backwards it says.

Can we convince of the disprove
the null hypothesis H.

not of no difference between the groups
takes a conservative point of view that.

Guilty innocent until proven
guilty of difference.

And it starts with a test to
stick which I just described

the difference of the means of the spread
of the observations of call that Z..

Set up.

And this isn't as hard to do as you might
think it's a matter of fact it's easy.

Such that if H. not is true.

Z. follows a standard bell shaped
curve which I've written here is e.

Under the no hypothesis is normal zero
one the bell shaped curve and everything

possible is known about the bell shaped
curve and it's known in particular that

ninety five percent of the disease are
then less than one point six four five.

And five percent or greater.

And according to our if if you're.

Early in the twentieth century.

If we observe a Z. value greater
than one point six four five.

We're entitled to reject the null
hypothesis and this is a.

This is just a arbitrary decision but
it's arbitrary decision based on the.

Intuition of one of the greatest
inferrable thinkers of all time.

And what Fischer was thinking is
it's just too much of a surprise

to get such a big number.

When you have a pretty
good up other explanation

which is that yeah there is something
going on and the five percent rule.

Can I think it's a it's probably
the most most used applied math.

Maddux to all of the twentieth century
it's used literally millions of

times a year in serious science.

And it's immensely popular with scientists
because you notice it doesn't involve any

prior distributions and doesn't involve
hardly anything except getting a Z.

value to start out with which is this and
the.

Calculation is under the no
hypothesis which usually

makes it easier to do
the numerical calculation.

If you're a drug company and
you have a new drug and

you've spent a billion
dollars developing it.

The F.D.A.
insists that you run two independent.

Clinical trials.

Independent of each other and
in each which each of which

does evaluate exceeds one point
six four five and if it doesn't.

You're very likely to have
wasted your billion dollars.

More likely are billion
dollars these days but

here's here's the beautiful
does shaped curve.

With the red area.

The five percent the upper five percent.

There is two point zero one safely in
there so I guess were entitle to write to.

Nature.

Journal and say we found the comportment
clue for prostate cancer and please

send a big prize in the mail right away
but I didn't tell you the whole story.

The.

Gene forty one twenty four was actually
the forty one twenty fourth gene

not a study of six thousand and
thirty three genes.

Each one of these had a micro rate
each man had a microbe exposed.

In which six thousand and thirty three
games were measured at the same time.

This is modern scientific equipment and so

we had the ability to get
the value is comparing.

Cancer with healthy controls for
each of those genes.

We had six thousand and thirty.

Three of them and

now we are you can ask those two point
zero one very interesting still.

The this is not at all what
Fischer was thinking of Fischer

was thinking of a standalone experiment
where you phrase things carefully you

decide before which you want to look at
you look at it you make the decision.

Here we've looked at six thousand and
thirty three and

that that that changes the game.

Doing six thousand and thirty three
hypothesis a tense test that once now.

The equals two point zero one isn't so
surprising really.

It's if nothing's happening we expect
one hundred thirty four of the disease

to exceed two point zero one
that is a fall the genes

were unconnected with prostate cancer.

Bone for only a contemporary of Fisher.

Said correctly that if we if
we formally want to rip rip

want to carry out pictures
point zero five surprise

rule in this case we really ought to use
point zero five zero two six thousand and

thirty three point zero eight three

refer to criminally read needs
a greater than four point three one.

Now it now.

Things don't look so great.

Are two point zero one looks pretty
anemic compared to that four

point three one five point two two is
if you're using snip study in which

your five hundred fifty thousand and
set of six thousand and thirty three.

So this looks pretty severe.

But fortunately and

both scientists applied scientists and
statisticians started to feel

that that bone from his role was
really too severe for our situation.

And measurement.

And Hochberg is really statisticians

in one thousand nine hundred five
published a really important paper and

they made a virtue out of out
of what we took to be a defect.

It is true that if nothing is happening
one hundred thirty four of the Z's

would be greater than two point zero one
on the average but actually quite a bit

more were greater than two point zero
one two hundred twenty one of them and

they called the ratio one hundred
thirty four to twenty one

sixty one percent false discovery rate.

And their interpretation was that
if we report all to twenty one as

not no interesting cases were likely to be
wrong and sixty one percent of the cases.

So you have to imagine the letter
being sent back to the biological

investigators saying here's a list of two
hundred twenty one interesting genes and

two hundred twenty one graduate
students are set to work and

sixty one percent of them are quite
disappointed at the end of four years.

And that will make you
an unpopular statistician.

However you don't have to be
quite that UN conservative.

The green histogram is the bar chart.

For all six thousand and thirty three.

Six thousand and thirty three used.

And showing there the hundred
thirty four expected

to twenty one observed greater
than two point zero one.

If we go out further to three
point four one in this case.

There's only two point six expected if
nothing's happened and actually twenty six

are observed and that makes the false
discovery rate ten percent.

And now you can see that it wouldn't
be such a bad policy to send a list of

twenty six back to the people and
now all but one percent

of the graduate students are going to find
something interesting happening where.

Well.

This is rather facile he said.

Is it really true that the ratio two
point six to twenty six determines the.

The proportions of true and
false discoveries and

mention me HOCKRIDGE
paper has a proof in it.

That in fact it is true.

Under certain conditions and I'll state
this here in terms of then algorithm.

Through false discovery rate control
value say ten percent find the smallest

threshold value such that false discovery
rate is ten percent at that point

in our previous sting threshold
was at three point four one

that was the smallest figure any further
in you get greater than ten percent.

Report those disease as
nano twenty six genes for

the prostate study and then the theorem.

Is that if you follow this algorithm
the frequentist expected proportion

of no cases false discovery rate is
indeed less than ten percent so that

what seems like just a hopeful
hopeful faith is in fact.

A theorem and

the theorem looks sort of easy the way
I say it to the seem so natural.

But in fact it's quite difficult and it's
closely related to the damn Stein result.

And as a matter of fact.

We're learning from
the others even use now.

So there are ten times as many use

life beyond three point four one as would
be expected under the whole hypothesis.

That gives us a ninety percent chance
that any one of the twenty six

is a true discovery ten percent
that it's a false discovery and

as a matter of fact you can set
this up but I won't do it here.

D. What you'd really like to
know is the basic in its.

Meant.
What's the probability of

that the no I puff this is true.

Given that C.
exceeds I should be three point four one.

That would involve knowing a lot of base
the prior distributions of the false

discovery rate algorithm essential
uses empirical based methods to

estimate that probability and and
we are learning each one you know

as learning from the experience of the
other six thousand and thirty two here.

And that makes sense.

In the mathematical way of stating
it makes perfect mathematical sense

in the statistical scientific way
of thinking and thinking about it.

It only makes sense if you believe
that the information in the others

is relevant to the anyone case
you happen to be interested in.

So if I had combined the prostate cancer
data with breast cancer study the values

you wouldn't accept the calculation even
though it was mathematically true because

you would feel you would correctly feel
that the inference was gone astray.

The learned learning from
the experience of others

always raises the question
of which others.

And that's the worm in the base
the an apple and accounts for

the continued popularity of

standalone frequent this methods
which continue to dominate practice.

But massive data sets like that in
the prostate cancer study have just so

much other information that
you really can't ignore it.

And that was why I said earlier I think
the twenty first century will be far more

acceptable amenable to arguments from the
experience of others in direct evidence.

But it's not going to be easy because of

these kinds of questions
of which cases are.

Are relevant to the one you're
interested in are deep and.

They're not just a just tickle of
course they come packaged it involves

the interaction between the statistician
and the consulting scientist and

awe how you use that information
is not obvious either.

So so steps statisticians find themselves
having to come to grips with subtle

questions of where is information
hiding in a large data sets of data

mining as started out as an insult and
I was become a profitable industry.

The well what you mine is data to

people it's easy to be fooled by huge
datasets huge data sets usually really

are composites of lots of little data
sets that interact with each other and

trying to get up that is just what the
status fissions are trying to do now to as

they learn from the experience of others
and then step fissions find themselves.

Coming to grips with subtle questions of

where the information is hiding and
how it.

How can be efficiently collected and

questions that lie in the boundary between
mathematics and applied science and

philosophy too and
it's a fascinating boundary.

I hope you agree here
some of the references.

So thank you.

OK All right.

That's a must win clearer than usual.

For you.

Just like.

You.

So you could some bit of
history would you accept.

Well maybe your methods.

The method the basis in
methodology is off the base and

methodology is fun to use and
people like using And

there's been some real progress
in the numerical aspects of it.

And the serious scientific uses of it.

Will parallel I think what what
went into the twentieth century

frequentist world that is
really good scientists and

really good cells fissions
will work toward a believable

believable compromise that will
answer questions like What.

When you are being fair about what what
what cases you're inclusive looting and

how heavily you're using it
in the equivalence others.

I didn't tell you about
the Clemente experience for

example of real baseball fans know
the Clemente was a much better

batter than everybody
else in that table and.

He shouldn't be included in

he was learning too much from
the experience of the others.

And.

The trying to get that right
is a really tricky point.

So I think we're in for twenty five or
fifty interesting years twenty five.

Let's say to give hope to those here of

trying to get the scientific basis
of Beijing and France in in place.

I don't think it's a question of.

Of technique.

It's something.

You know Mr.

Snow.

You was there you know he said yeah.

So I was I understand what you're saying
is what if I throw in some obvious

in incompatible things it
still might work right.

Well that's the same thing as combining
the prostate data with breast cancer data

or something like that.

That's a question of which authors and
the mathematics doesn't care.

What actually happens usually is.

You just get back to the observed
average because the things you throw in.

Make the game Stein has to made or
not shrink at all but it is.

The.

The danger is in misinterpreting.

Say again in squared error reduction
squared error with actual gain in

inference and and that's that's what your
question raises which I can't answer here.

Of course there were
techniques of tried to try and

do techniques to help reduce that risk but
I don't know there's no.

Right now there's no general theory and
that's what I was calling for.

You mean what he said it's your sign.

Use the term.

So that that's a good question about.

Shrinkage.

The.

The term.

The current term that's gotten
popular is sparsity which is

the extreme forces of shrinkage you have.

Possibly thousands of
regression coefficients.

And but you don't want to use them all and
you're pretty sure that most of them

are zero but you don't know which ones
are and the method like the last zero.

Shrinks all of those almost
all of them to zero and

only give estimates nonzero estimates for
a few of them.

That's an extreme form of shrinkage.

And so this is still being played out.

And again there's dangers of of

shrinkage almost always seems in if
you run a simulation experiment.

Shrinkage estimators
always seem to do better.

That's because.

Simulation experiments
aren't real data and

they aren't spread out over
the whole universe and.

It's really dangerous to be fool
it's very hard to read a really

honest experiment like that the last
of methods are really very powerful and

when they are working you do get much
better Spirit Air prediction for

example but there you have to be careful.

Jim.

Not.

One single.

Sizes larger food.

Yes that you study how do
you pick up the result.

OK And.

So so the question was if if we'd had
more people then one hundred two.

What would happen.

When it happens is you start being able to
estimate the false discovery curve better.

That is I said that the the.

The estimate of the false discovery
rate was when I say eight point one zero

in the one case but that's point one zero
plus or minus something and the plus or

minus something depends on
the size of the experiment.

It also depends on how correlated
the things are between each other.

And so this brings up an interesting
point that based in results are usually

stated as exact but in fact in this
world in the empirical based world.

The basin estimates themselves
have frequent us error and

trying to incorporate that
into your conclusion.

It costs you something.

To have a bad estimate of false ever and

what essentially cost
us in terms of power.

That's.

So.

We see you.

I mean.

Not many sixty.

Yeah when the added tell you is that
these counts were done by computer

so thought is a different word
than fought and stuff like that.

And.

Shakespeare is considered
a great corner of capital or.

I didn't tell you that this
didn't I got involved in another.

Six spirit thing.

We were pretty sure that nobody was
ever going to test our formula out

because no new Shakespearean actually been
discovered for several hundred years but

in one nine hundred seventy S. late
seventy's a somebody went up to the attic

of the bow doing library and found a poem
of length four hundred twenty nine words

which he claimed was by Shakespeare and
and.

The magic formula predicted seven.

New.

Unseen previous words there were nine and
you can make the formula for

how many words appear once twice a and
they worked out fairly close.

So we published a paper on that and that
put us in the camp of people who thought

the poem was genuine and there and

I gave a talk and six for
people are vicious they.

You don't want to go
against their opinions and

half of them thought we were heroes and
half thought we were the devil.

I think of.

It think.