Your research to

figure it is you have your story here

about all

the characters coming and
coming to the big screen here you

point out Charles or
her child was underage child.

There was another good life here

very very fine.

It was like the lady you

were copying is a really
very very serious way.

So how this happened here is Minnesota.

We have about one third of this idea.

I don't know I'm getting letters
every year that we've been involved

at C.G.I. but really this is
the start of all the stuff of

Windows computers just runs us
about it wasn't working this

ninety thousand itself intended
to do the kind of such like.

Come on why did you ever go.

There's no distinction in
our study on my back and

this is the story of
the story of the discipline.

He was going to go on.

So that's a few words about just
the background a little bit

about looking down the stairs or
the series of twenty eight

cell pulled it out of
the dimensions of the analysis and

then the last part of this
piece I must pull in for

us to sell more than
one research community.

I'll give you full disclosure

to this work.

We're looking for one for
the study we were going to came up

through the zero one P.C.
and things are getting working as

a manager that is also where
the sensitive program she said Well

says that he she sees that he sees
Putin kind of really remarkable.

I'm kind of so over.

You just feel this you know
our grant proposal but

these are all things that as
a story we do you know college and

so forth that we're doing in a way so

I mean does the seven The C.
Programming know about the social

factors that desirable a dozen
years of life in China.

That's the right side of your words
we're very interested in your

group use of software systems to
business and government sectors.

You're very religious or
government agencies on she's got

to mean different social.

Hardware software systems.

This is also known as the stories
we want to read this.

So this is a area that is one of the
issues we're pushing where these are all

essentially historical lessons in
three places the questions about your

very happy to come down

faster than those you have yourself.

Well you know there's basically a National
Science Foundation here a structure

where I lived through with
all elements of it for me.

Maybe the only proposal for

years out of the research
project dealing with money and

finally were once part of the simple
system to look really tired.

Still you know there's a good girl group.

We also hear it's not
that interested you Jack.

Kate what are you for
exactly what Peter Jackson had been doing

crack proposals that will work in
process for exactly like promises.

It isn't really her horses put it for
a White House.

It can interact with and without touching
it is truly a city pulled out and

somehow it is so
far it's like slow down with the sun and

so forth that there's you know
if there are any skewing of

the Silicon Valley certain
types of purple or

certain types of reserves for
certain areas or anything like that.

Mark.

That's basically the stuff
that's never done.

And of course the news

here that the people really regenerate
even the sun discipline that.

Hampering also

he's a fascinating very kind
of interesting moment in

history on there's a part of the story
that's well known about ten years old.

You know that you know so

yeah it was sort of I see back
when I was in the private sector.

You know every day and so

just really this happened in the ninety's
when there were going on within

you know recursive rather
you know there was this

expressed it was kind of really you can
do something like on the proposals.

I mean you know I put this article.

There's one other thing here is
the I write this shit down and

it was a fail you know and
you know it wasn't so hard for you.

It's like the site you're some place like
I was on do they actually bring an office

just for these for
you to force one on one side or

so variances were sort of comfortable and

that was the structure passing would
need an orderly probably pretty high.

If you want to put it kind of a scene but
I think in the long run it was more or

less hired to run and stuff computers and
transition to something keep information

on all of the titles rightly and
she worked again you know Liaison Office.

You know what we're going to be in this
new computer system that's going to run

itself the feelings the record.

She said.

Partly because when you lay this sort
of goes this is this Is it exists.

It's mine and it feels like nothing that

I let you know used to
work it was watched.

It really was going on.

Why weren't you might notable as
a roadie instance of doing a patient

like this coming on doctor
transmission a little bit

unique some programs the some
directors offer that you're option for

men you meet with proposal then and

by the first of October in the year
two thousand all proposals

floating around

here in this

country so

possibly passing in a satellite but

look let's start with the sweet
proposal right from U.C.L.A.

June two thousand and four and

that is the label is best here so

basically what is needed to start with the
First up is interesting work station where

required workstations remember where
he sees the work to operate P.C.

So you know how that maybe that
is something we have known for so

it was there I was here I was on the new
one actually browsers in the early era.

That's good.

It was only rostered that allow
users to file answers if you're so

Microsoft or was

he says that he was probably the biggest
player in Congress that he discovered

it was going to he was
using weighted everybody's

now right here in
the solution part by police.

Go for it.

You know sort of getting it out and

either have used really how
about not really to do with it.

Money or dollars or so

you could get this free where all that
goes for work because this right.

This is where I'd like to buy and

then you hear your screen without me.

Don't let you see it.

I need you to fully utilize.

So anybody can do this was frozen.

And so
we were looking at doing research for you.

This was sort of awfully
large awfully large for

a new system for historians to study
the first part of this thirty and

as managers designers did the work
coding the slide for years and

it will work on policy for
the staff number thirty is the number

the new world is but in addition to
that in your higher than there's and

out thousand what we term legacy is
a space program when they go to work but

they're really costly and it works very
well as they are so very large numbers.

These days.

And so things that have five thousand P.R.
wise and all of those highly priced and

there's a larger every single
one of those very very good.

Secondly it's only right that it is so

artfully protected through your
sponsored sponsored research on

the order of your distributers each
one of them house between one and

clever people something like What
was yours is one person's entire

life research project was
one here with the smallest.

But others are in the spirit of

course one of the theories that we were
looking for is this is a very closely.

I wonder if there were any
differential back to the positive or

negative on populations that are part of

particular and I was interested in this at
one of the historically black colleges and

universities of roughly eight hundred or
so on.

How is it a sort of reward in there and

then to get it here
where there's a purpose.

This is a money area that
with the services of course

they started there's no paper trail.

So the first step you took was to
go to yourself might raise it but

just give us all you got something
like regret is that we have

no luck in Faslane this time I mean
this is one of these things that just

all the cracks in the step as they
keep track of all these proposals.

That's bylaws or Congress.

Also you kind of like the policies
of the National Science for

you and everything a whole week.

All of your disciplinary activities all
the news all the kind of gossip on little

are completely dark sources that being
able to get a truck they were paid for

by talking to Congress wrote
earlier where are you going

to leverage we're going to get it right.

It's legal to reconstruct and

it sounds like there is no
vastly more progress now

organize people that are so.

So all the states that are in yellow
are the so-called US forces or ones

higher now there's twenty nine different
entities each one of them has a plane.

You'd be having received blows your love
all of that will research funding and

the stuff of the world.

That's possible or so this is a way
of doing all of yours on this little

bit about how things are pretty you could
imagine that if you were in the US or

states you can see this Iraq now as
part of the south a couple thoughts

on that so they're very happy with
the success especially satisfying.

That's what funds block grants or
something.

On the supplemental funding.

So somebody for that or
state that has a really really solid

total that it doesn't get in that
very house most funding sound

program officers may not
require that they use that for

there's a little competition for

people in one town were really
angry that you talk like

that again there's a magic number that

will show

your mouth or

say so or
this isn't the most money at C.D.I.

you can find honest look at documents to
find that the one or the three years.

That's the reason I'm here in this case

we're going to do something but

we had to do a whole lot of
four hundred and two in person.

All is pretty easy for someone just
a few minutes longer on this fire out.

He said yeah that we have
been analyzing each other

we did on one hundred online interviews
you go to prison pool their money and

you go back again the number
one hundred houses a while

probably never order
this business five years.

You're Well one of them and

for the stuff that you guys are hoping and

your bio says he is one of only I'm so
glad you're doing this project.

He said no but

he still is meeting with an larger
than one hundred thirty five people.

Were really you know part
of the hundred years.

He also talked with about
seventy people in this town.

You know the designers or
people who work in

the South Tower Coliseum like
building very forceful and

then also different types
of users mostly program.

Secretaries where we think we
actually know something about

the stuff that this episode itself is and
stuff that's not there.

It's so great a lot of vertical
supervision over here by the SEC is that

all of our interviews are in person and
on the one place.

It's like the research universities for
lack of interest

any of the slides up here on the web.

Mark.

All right.

Which regional balance.

We're probably oversample.

Recruited probably someone oversimple
that's where states are in terms of their

research on this place for the future

research on ways for the school and

other means to raise themselves
to the ranks of the really risky.

You know how done some remarkable work.

It's very very

close the National put it this is what I
mean when people ask why are you here.

If you don't understand this based
only on the top ten the standards

are you know they're struggling.

How do you know there's a there's sort
of a just kind of saving also right.

Just because you get represented.

So for our young person interviews or
whatever.

Well people we trying carefully
measured balance in your own life or

sample of women that that means simply
use the you do more readily accepted

organization could never force like
parties at least towards the microphone.

You know really you have
to go through cause

you got from the mouth of the other and

we are trying to make sure that we
didn't understand the world worse and

it was I think we actually
understand very over so

long now if you look at the size of

the right way that
you're hearing about now

they are slightly you know not
perfect sample pretty much as

a result of the public out there
because one of the things.

Now is that one thing when people
offices and talk to them live.

I don't say overwhelmingly but
very very powerful figure and

that might be a bias that face to face.

We also hear the Boss experiment with or
read it online interviews.

So we look at the same questions.

You know express telegraph or

go on the set of five screens that people
navigate through the old and again.

So this wasn't the score.

Are you satisfied with one two
three five it was over here.

So some people wrote two words or
some people or stories or organizers of

the some people wrote shorter things
that have some kind of errors.

You are from a home but
it is the case where it is to be

sure there was going to hear
on the Web This is a view for

us were critically important
to the argument as involuntary

Is that what we know about science and
engineering across the board

comes from interviews and samples
typically that's that highly skewed was

going to get every Bill rightly recurring
price where I would really believe him and

therefore completely unusual having had
some kind of contact roll that this was

he wanted to buy a method
that would be able to scale.

Way way beyond ten and

at least allow the possibility
of doing prior Ranjan file low.

So you're finding interview for
people who were lost and

not enough money and we would have
made the four hundred is that right.

It was just an eight
hundred twenty four X.

that can be quite successfully
achieved simple but it is.

Interview that will scale
far beyond the school's

challenge for understanding really life.

We're thinking you know whether I would
think that's great if it's at the start

and then you can have one hundred one from
lots of universities where you can't go if

you didn't go once did he target
the three or four universities there but

instead you decided to do a kind of
quasi experiment where we did targeted

recruiting at exactly the same places
where we've done in person because

they're out there and there should be no
you can read the same gender check for

scientific disciplines check for Regions
check and also for you know that there's

more or less saying universities
are the same types of fields and then

you're right on that we were
surprised to find that contrast

to the nearly overwhelmingly positive
views from the in person interviews or

something like expressed just for
the name by the system so

that very very important and
you're right and

we didn't get that in our interest so
neat and under this

notion of the face to face interviews
were a gold standard of the workplace and

interviews become second rate but
then you start looking at the literature.

That's coming here and

eight hundred people and you know about
eighty percent are still exceptions for.

So you know

there's this is the young lady look for

a new gown for a reminder I'm.

It's young.

So it's a small sample.

So President.

I don't I don't have the exact number.

When we were going to get recent
interviews to be honest with you.

The embassy interview very
very dramatically but

we want the smaller places I
would say one percent said yes.

Whereas the largest places I would
say sometimes it's three percent.

That said yes.

So there's there are some skewing and
some bias is biased.

So any one of the interviews of
course comes from a biased source

the population comes from
a lot of the biased sources.

So it's possible that but
it isn't is our last fight but

you know we just we just
don't quite know very young.

You know at this point what
way of telling us come

here if you're going recruitment for
the web based interview we ended up Italy.

We sent out a round of the analysis.

He hit it all.

Personally and the same.

And we got about twenty percent that
said yes we it's in the same round

that yes we got another five percent.

We said that a third or so because you
know up to be a fifteen percent sample

which I've been told is not about me you
want to have a ninety percent accuracy.

Nobody gets calls are always with.

So we started thinking about
the red roses and person.

I don't believe he's the one of the papers
that came out just like I last very well

you know I wait on line or histories
from a kind of writing perspective

just grab this summer and there's
just there's just writing studies and

was interested in oral
character versus online and

sort of very very very you
know these documents here.

This is what really drove
social sexual health make

the argument on balance that oftentimes
it's interesting interactions.

You're talking about you know right now
there's actual marriage of somebody who

appears to be whatever sounds or

is this something answers are different
that it isn't in the online store so

we're just kind of we're not quite sure
what the reality is this is the best or

worst part of through all of
this because I think I'd like to

believe just on the policy that
we ended up you know doing that.

You know it's just software and
it's funny question was and

those thing was so
all I can do you know but any time.

This lifetime dish was finding
people here is Illinois Senate

actually going to patients that made
it was there there's no doubt what you

know because what they're seeing people
don't know is that had nothing to do with

funding US dollars check
this they funded for almost

two to four million three million four
hundred dollars or something like that.

So it was it was funding issues
that most of their voters.

You're still in this on licensed
products like life that one.

You know it's for your own life.

It is it really is a no is it not
to do this like just watch and

someone can hear about like
you know just like all people

there's anything software
work that was built.

You know very very early on and
they are so

fast players breaking into
different programming languages.

Basically because of this parish
matter which side was a real expert in

love right.

Really if you're headed.

My crazy guy was tired
of doing one of those so

he was writing really old
prices like go out and

get one song any information
you're using rigs minders or all

the other driver was an expert in the C.

programming language and
still run the command post script and

write down every time you get
something from the server side

or and what is great about this is now
every time there's a vision of where

those guys are there are a few things that
software old timers advantage of to meet

with you want to see and so

it's now prohibitively expensive
train tracks or post for you.

Things that are written in software or

part of it with software

that I mentioned partly for

the very high degree or so

three per cent

of it you say I find worse a lot
more logical to do everything I

do my son in Warsaw cafe in the exact

sense it was the case over
there as I was able to

purchase especially Dublin servers and
establish a three month you know.

Across the face in Europe.

Now there are servers that
they have those are never too.

And some good with the result for you

know not zero zero zero zero zero

and the area this kind of thing for
you see here it was the way that was done.

This was really vision.

It was just designed built in and

reinforced certain values values or
security in around the building that

you're either now or is that there

is a safety of your you and
your life really is

a value that will carry
your feet was not believe

that it will slow the pace or slow there

it will last a little early on the Do you
ever find yourself that promising P.R.

you know a little less you will see
reviews they how many hands you get for

not know that a normal

development that has people looking at
something here right here in this hearing.

Until now is a good

theory who can get it done
before the last a lot of uses

and it and I think the other
interesting thing in terms of policy.

Were they something that is on your screen
is a lot like paper or a computer you can.

There was a rebel side
won out over funding.

The hearing process is the same.

There's also the better policy.

I wanted to use for the ship.

I really hope this very
first one very seriously.

I have no idea how I would do this
in the polls but people here.

I will work for you today.

Every time you receive this much relief

from the terms of the game.

What we're trying to say that has won
them all the world government and

it sure allowed them to
siphon addition to deal with

sharply reason for this five years ago.

You know he is probably the case
with some power facilities or

so ordered the ship or
also from the record it is priced right.

Funding.

So given the software.

We're we're we're very very
we're getting for life

he found a small one hero for friends.

You know that there were people that
walked out of those people without years

that simply use a manual typewriter one
of the old days in your company and

they're a city that hears
your manufacturing will grow

if there are still poor
people around here.

You know there is a pretty
new guy funny guy.

We handed over whether the conference felt
any good happy to urge them to live and

work fulfills etc There will
be no votes once the consumers

feel the science majors are doing OK but

why do you want your researchers growing

larger viewing of their capital or

local something like this
in your diary for counters.

They will look at the background
of the sort of scenes of

the voters all over
the state Capitol Hill is

really kind of informational in
this part of the founding You know

the going back to you know push pull or
the dates and the last word or

not where they were and
this are here five years.

They still are still there.

You got he just expressed

our lead here he just kept it real.

Sounds good.

You know me very well you're hearing
the one of you know whether crazy.

This is going on right in
the middle of the dark tunnel.

While I mean this is yes it's always hard
to remember that moment when you remember.

You know this was during a time
national you know this year.

People were really really neat only design

that uses Well it was
really cool thing more.

The lightning and
you know we're going to accept that low

duty as part of this whole thing

then chapters were five six or
less narrative chapters remark.

Very history and

the whole new pianist and
he uses that as research.

Also on your research
was down we users and

their perspectives was a whopping three

separate posts on that it was
really were particularly our go

letters exist on the big
counter with us tonight.

It's really what is looking at and
this is how the legacy users and

then they ended up having
a kind of very protracted and

not necessarily be a little bit like this.

I always get money that this tax
code base doing a study that

benefits the research community
they're doing something that in

itself was done in worst case passing

the facts really there's all kinds
of stuff running with it and

we just sort of create it's
own pocket where I was for

all that everybody uses right.

Every group you threw in so
issued new department.

There you go so
that was a much much more more friendly.

You have the sort of role in
human data loss and one thing

that could be done very
very well be sample.

Never didn't work over there
you know it was a day he gently

database set up so it's like here you know
you know you know life pretty fast and

you hit the button and
there's a way right.

The second to say well
it is now our second for

a few seconds for three seconds.

Wait for
every single button to click on the right.

The little button click here for
example A and

certainly understand why from this
on the on the standalone P.S.

worries that the whole nation
has are going you know you know

what it is that it was over
he was out on the truck for

another every single one of those three or

four seconds for every cell right or

left for Cyprus trucker but
also for the search and

any other letters scattered on the road.

Some left think there's a good word for
the study and it's awful

it's off a list of users
on the ends of our wires.

So the reality is really that
there was orders put on the market

in the fall when there there
was a crisis in the get go.

Very strongly.

So this user projects.

The others online for cyber was a little
cold how people would see it is something

wrong with the whole car Richland from
this I think you're having problems.

You know sort of like this
is what has happened and

then nobody was really doing is what it is
that they're not related to the problem.

There was some problem with that but where
I like that you put that on the system and

you know some of these three pieces
were released this is the whole

response if you look at it and

I think this is from the sociology in

this you know there's a lot of
users are like abstractions.

They're like hypothetical users but
from national or

on work there were real users a lot of
these were the inspiration like there was

a twenty universities that will run
on specifically I think you need

to go for this work in Iraq.

You know you're basically sort of
British You need to actually find

work rather

than the sort of hate it when you
know what's the big thing around.

There's a line in market share.

So that you could be doing reviews or
proposals of issues or doing well so on.

So for instance the reviews of

this from the early hours
later that of itself.

You look at Faslane and

then he did say he did it for

the super PAC you are going
to know that we've

heard especially from the innocent
people who are the designers So how do

you solicit you know you know who did you
get information from and their insight.

Was that people who are in gauge whether
this process has really gained or

so over again.

Relatively disappointing.

I mean pianist.

I know you guys are doing good with us and
you know that that's a roll off your

feet you know your verses
are very you know where your eyes

are on the way in here and

also how much for your comments.

I mean for
that is really fascinating because

one part of my life in

best practice is our I'm so
blessed and it is doubtful.

We're looking at a layer underneath this
book although he did use everything in

the research universities to get this book
gives an insight into the experience for

years where he knew he was the best and
one hears.

So it does say that there's not
a single one best place for

it was only one way of doing it and
then very gently.

Well it's easier to do the research
university like what's wrong.

Like that's probably true for
small and different ways organize

our sponsor research
authors organize itself

so there are no models that's
another point that it seems

maybe a little bit obvious
that works better or

worse now and accuracy or stuff that
actually comes out the other way.

Our learning our youth.

Not on the lighter hundred percent on
the university hires someone to do it and

Phyllis only want to be
with him like you know your

higher son because of the research
on Johnny five point five percent

hold the report when I was in on time.

Boom boom boom boom boom
turns out that the herd.

If you will your team pay for another
year and so that's the mechanism that

departments can use as a means of kind of
bootstrap the awfully hard with a certain

risk you know where they don't have or
where the research was very

rigorous on this one woman that was so
basically branded for years and years and

here but her support that is all
over was really really good but

she was in this very high school of
thought you guys are coming up with

proposals probably close to
like you're going to check.

Everywhere I hope to get younger.

It's a high.

It's a high stress but I thought well this
woman was this job for years and years and

years and
every single person we want to settle in.

You got to talk with this one
because he does this is this so

her kind of continuity
was really impressive and

everybody that he's going for
campus for it was your father's turn.

I thought it was kind of interesting.

She ended up retiring from
this high pressure job but

did they get rid of her like Sandra
orders from the impact of South Dakota.

You know days that sense
of life now they have

to run because she emphasizes that right.

So all the research on this is that they
come you know after that it was something

to be honest and go and you go on from
here going way down the aisle as a leader.

I think you're very She still
can't with the resources.

So the local support staff might not be so

surprised but
also three people that are there.

There are it will go on longer for
he's come first.

They won't be there.

Either this or

that I mean it was their concert was
a nice thing to do with their very good.

I was looking down

side back to what was there
sometimes more stuff and

you know I was a little sometimes
because I was like what I was just

as I love

my own formal training base and
that's why the people around me.

Then he turns out I needed one of these
I forgot about your you know they

were making the rounds and
I got this rare older British

hardware I was working on and
it will be easy on this in

London train but

in terms of the consequences for
me out there are very

few guys ever formal training
people who come here.

He has no clue but

there was always somebody at
every single university would be

in your house next door here with
is figuring out how to do this.

So how are you like this and

then this is everybody else sort
of formal praise a lot of this

kind of informal global
networking at least and

there are also there are players who are
stepping back are also calling you on it.

You know something learn a lesson here and

through their student protest
work in the love with her.

These are so few users.

This is preferable for they were all
these are people that want to hear

the professional organization and
the secular life you need you now.

Reach for those organizations also for
your good ideas

mentioned the federal desperation hundred
years and there are some that work.

But there's a whole lot of
time they witness for life.

The eyes actually watching their own
people do the regional meetings or

especially the state or
federal funding but

those really work and also

say again that you know that's for state
versus sort of the black colleges and

universities but that's of course how
do you believe him on raising taxes for

universities business
getting access to the answer

that's on the schools that
actually just hired old

people the previous sponsor and

nobody fired this or
the like because the same

colleges are under water.

So let's call it positive for

college programs moralistic

lowest cost and

just working to get it or just shit right
you know that stuff is right there.

You just should retire go right up there
is the next big thing we're the circle

does not predict the future but

there are all kinds of experiments
going on there because the system was.

Let me start off with your sponsored
research to get hold of this or

why don't they transfer Truman on
the United Nations rightly in France.

So it doesn't go right up front again
this is you know it doesn't work

that way just pulls up.

But this is this is people writing.

That's what I get when I sit

like this year there's a shutdown
the government and this is a bill

like jammed up for
weeks they have this if I get this right.

Forty five hundred holdalls happened there
and forty thousand pieces of mail and

so they're going crazy with this and
I guess that's what happened or

something else like
this just usually here.

This is out of the worst it's been like
this one here in the water again we have

really hard to I think it was forced out
because there was the army that shot

all those whose possible terms and
let you know him well that night.

You see it.

Least one sometimes for

example every program that was just
he didn't do this for the paper.

All the people all were running around
filed and then here filed correctly and

we were told formally write
them down there at the time but

it was really work for
a program officer to have a respect for

including the working relationship with
his or her program is this is a program

assistant could get or
any proposal is really working.

You know how that works and
it does cost it was.

Again the program.

This is essentially where I was drawn
some of the shifted more computer or

you were some of this kind of off from
the Sun one of the things that we've heard

time again is that it broke it
was in didn't see the value or

it's all created them as well.

They were sort of you know second class
citizens of course of course there

are isolation.

We hike like crazy and
you represented working for all you know.

Multiple times.

Many of them didn't want to stay
in the Greek people that were for

very very very very hard

as a you know first of all

that's really your country

can you put

it in one of your first line of radiation
where there was an emergent system right.

If you had heard any question mark and

then you know are what are your
answer on that and then did a set.

You got a system that was continuously or
even genius improvement or

really they developed a system that really
did not have all that was never your

favorite you need continuous
improvement fast and was not that and

the reason the council suited and I think
you know he is basically once a guy.

I would use the term good enough and

the backend shifted somewhat over
the years but the friends again.

This is the first time they're called it.

You know that has changed
only in the most at all but

in the programming details but
if you go there today and

I recall that first started using fast and
it was new it was it feels as good as a.

This is what is in the possibly
see worst when you go there now

your expectations are quite different and
so fast it is all in the frozen in time.

So if I can worship you need
continuous improvement.

No And I think ninety nine point seven
percent cling to the structure of climate.

So they can get money to redo
existing computer systems

which is why they are shifting to
the transmission line of business that is

research they can get money from that
there is they take their own money and

use that to develop a new outward facing
product but we've been told that they

cannot do by Faslane to do so without
regard to less or large amount of time.

So once we got to know the end of
the city it is just understood more so

in that sense now you've

learned your findings here and

mentioned a little bit about them
if you go right through that when

we looked back alley in the assertion

the Young family members were destroyed
to do qualitative research we

thought out there was some kind of scoring
going to look at doing the scoring but

our judgment was was that scoring is just
as good as your scores change scores.

You know it is very hard
to get reliability and so

our data is always qualitative down
to get some kind of economics.

So we're becoming or getting ten
people to tell us the same thing

basically generalizing from the historical
data doesn't get much stronger than Solon.

You know three or five or seven or
nine or eleven people how is the same

thing when I'm trying to
measure that in any sense and

say that's a three point two
versus zero point seven.

With this kind of think that
our judgment is is that

person to have expertise in the sector
just how we conduct our reserves.

You know I don't know what I
like about the web based system

is that you had a chance to respond
in fairly open ended fashion.

So they could tell a story as if they
wanted they could say you know you know or

they think it was very
interesting answers and

that was all the time they had you know
they had extra time you know they were

able to sort of you know
talk a bit more broadly so

on the specific point about engagement and
that's basically our observation from

having talked to several program
officers and the civilian of the of

the sponsored research people probably
more than two hundred thirty at least.

And that's basically what
they told us the eyes were

largely disengaged from this feedback.

This is fractured feedback and

the offices of the sponsor projects
people who are much more engaged and

that little tag line comes directly from
somebody at the sound of business who

are saying well you know basically
he was in favor of that broccoli and

that's a kind of short times flavor of
expressing something that we got from

you know maybe three dozen five dozen
different sets of interviews so forth with

so curious about cars so

if you look at this particular round for
a measure that

young you know but I'm wondering
if you when you get a view that

was very good first thing
has passed the more so

and so
my sense is it's more the first time.

Yes there was one document that proves
beyond a shadow of a doubt that fast lane

saintliness have on that document
went out and everybody read it and

everybody uses that as a statement of what
we think that will be significant for

the numbers and also significant for
the sense that there was the need for

such a document and their work.

Tells us that we can tell
there was one such dark and

here is how are you seeing this on sort
of work it prove that you know and

that they did you know no reasonable
assessment of the cells five percent

savings your theory was that students here
were for some things there was a time

since the stuff that was substantial So I
think that you know the way that religion

is used work is that sometimes there's
a piece of accuracy that happens and

you know the results of the work that I
see and nobody has really we found nobody

who either had come here you know there's
you know there are I'm sure remembers.

Very well it's so nice outside.

It's the soup but it kind of makes sense.

You know from a program officers point
of view that if you can just get

that state visible.

If you count our time our time facilitates
certain kinds of things to do was

to her approval of the Stan Grant takes us
more time than it is to do it if you were

using like three minutes then it was gone
and now it kicks it was actually longer.

So I think that there is some intruder.

Yes you know from universities
point of view and

I don't know how to answer this has always
happened in course we have this question

about like like you did you have
an athlete you know sort of resources and

we asked the very worst schools as well
as the various schools at Berkeley

concert here or something or
schools as you know we have this one.

I didn't know nobody.

This is I don't know and he and he noted
the Mount Hermon had computers on their

desks period through two thousand or one.

Anything in my life and
so this guy is a poser.

I can see how things be in good position
I didn't have him here in me just.

So I had to go somewhere else.

So this idea of the office and
interviewing aren't for

this person was you know
kind of a nonstarter.

From what was happening and whether that's
for schools there to be from massively

separate you guys on your
facilities built to put in the fast

lane your business in the way we are so
there was a lot of us were already there.

And even something like
that one when to try

to find the resources
that they said frankly

were awful where there were hardly one and
that didn't happen on our own.

So it's so hard to disentangle costs and

benefits so dollars Our research and
if it's not.

I think both it's really hard to get you
know the city hard numbers to be to be

honest I mean officers I'm
sure glad you like to look

over

the process here.

You know where ninety one

when I first got you set you for any idea.

Proposal and it was an outfit that
young went off and there were no you

know some of us that

there was somebody signed up for
the part of charity you know some of it

is you go along here but

deliberately left the order of the work.

If you make an effort to
ask people all of you.

Yes.

You know who they are and
also is there information

about what you're on
that was gathered to go

yes actually that yes the second
question is were surprisingly the answer

the first yes to the second question and
this is Anderson actively using this

analytical information that your
marriage is for I hesitate to say no but

I believe the answer is No that is
we had many many guys business and

I have no idea what happens to report we
think that nobody ever reads them and

that they go in and
some program officer signs off on them.

Maybe after not you know or
maybe after doing personally reading.

Wouldn't it be great.

You know that information you are risking
in real time when you go back out

to research these are preliminary
results don't get sort of buried away.

There's like he issues and all that
sort of stuff but so it's not so simple.

Now but I think in time there's a could be
using this tremendous flow of information

better than he is doing so
that's the first part.

The second part is we actually asked
almost everybody I mean our to this person

or senior ones that have
facility with paper there and

ask him about the quality of people's
views on these core values and

I'm here on a visit there and
I think that our evidence is midst.

It's not necessarily worse but
it's not necessarily better and

one of the things that we actually
discovered his there was a big community

and this is actually just very
quite sharply by discipline.

So if you're in sociology
this doesn't make sense but

if you're doing you know science
is if you're doing this for

your modeling if you're doing
a special studies color.

It is very important
the video of your work and

also supports still color you
are actively discouraged is for

basically renders a video simulation
all that mess of the moment and

many branches of science are as we can
tell him here in the jungle where science

and this is hard to see journals
written in the journals can get there.

You know papers are big and bad leaks
is something that we talk of one of

the city's going to experiment
in a second post and

then you know it was to go and
go to a particular search.