Thanks everyone for coming.

By computer decided to take a nap so
we get up.

OK There you.

Are Right so I'm going to talk today about
desired learning behaviors knowledge

education specifically in.

A lot of the data that we're going to talk
about today in kind of the things we're

going to allies deal with
one class in particular but

we are expanding into some
the other courses in.

Starting this semester so

that's very exciting I'm very excited
about that I hope you can tell.

Right so
a couple things one start off with

just a quick outline of the talk
overall I will be going over just

very briefly some the concerns with
online education as we know them today.

What we feel called the Omega C.S.
approach so what did I do differently and

then how can we see that kind of
carried out in the course of themselves

like the knowledge based course here
taught by the show in the corner.

And then I'll go over some
research that I've done what I'm

pointing basically is the agent survey.

Followed up by our topic which is
desired learning behaviors and

finally moving forward which is from
with the community of inquiry model and

I'll be going through all so so concerns
all in education just very briefly

there's a couple things we know about
only cation that are always cation that

are kind of obstacles I would call
them at least for one we have

to engage mentor in a lot of different
forms and models of online education.

We have.

Little feedback if any and
maybe a little to no expert feedback which

can be very difficult for students who
are trying to learn particularly difficult

material we have very little guidance and
support so

who do students go to when they have a
problem when they have like a pain points

where where do they find that
information from the move past it

cost trade offs a very common
example of all in H.K..

In today's moves right so
massive open online courses

which are free and open but then they
are and they lack those extra feedback and

all that the opposite end are you can
get online degrees at a variety of

institutions here are all
across the world but

they're very expensive right so you get a
structure but then you end up paying maybe

five thousand dollars of course that
is a huge obstacle for a lot of people.

There's a huge problem
with steam retention and

on education the rates for
retention are very low.

Only ten percent of people
who actually take that for

example nukes actually finish the move
through and that's thousands of people and

finally just the nature of online
education is that it's a secret and so

there's problems with how do
we keep students engaged and

how do we incentivize communication so
we can get that community of learners that

we want towards getting higher
performance and learning overall.

So they almost C.S.
model Georgia Tech decided that they

were going to make an online program for
a Masters in Computer Science and

they kind of did it from an approach where
they not only identified a new class of

learners you know individuals who were
otherwise excluded from going back and

getting a degree they did it in
a way that it was that low cost

they could do on their time and
had a high quality education

comparable to that here on campus so
almost Yes

piloted started I think it launched
in May of two thousand and thirteen.

And to date has over
six thousand students.

And I've just demographics is about
fifteen percent under represented

minorities about fifteen percent women and
represents ninety nine countries

in the world from people from all over
the place around the world so it's fairly

successful in that regard there's a lot
of people in it there's a lot of students

who are very happy with the education that
they're getting through it going to the.

This is higher education but

these are individuals who otherwise
had no ability to go back and

get their education who want to go back
and get a degree because they had families

they couldn't you know move to somewhere
they could get to an affordable school so

there's just so many reasons but
so we see a lot of that here.

The course that we're really talking
about today is knowledge based artificial

intelligence this is one of the most
popular courses in the program it's

often one of the first courses that soon
will take when enrolling in the program

it's different than maybe the traditional
approach to artificial intelligence I say

traditional but it's really not knowledge
based AI is more traditional because it's

modeled after human cognition where
is a lot of what people understand is

artificial intelligence today the kind of
leads more toward the machine learning and

so this this course is taught in
the program by a show go well and

Dr David Joyner and
it takes a lot of different methods and

kind of pedagogical approaches to
how do we teach people online.

Explicitly So
I say was explicitly designed right to

enhance this learning experience for
users and for

students it doctored up launch of methods
that we know work really well for

increasing student productivity increasing
student performance and then practices and

perceptions of the course overall and
then also is a great place to kind of

pilot some new technologies and new
ways of testing and understanding how we

can influence that it's that that
student learning experience and

I'll go over a couple the little bit later
but just a couple examples are for example

virtual teaching assistants virtual
research assistants and then nano tutors

which I call here with cognitive tutoring
agents but they're micro tiny little

specific exercises that are embedded
in these videos that you use online.

As part of the lecture experience and
I should go over real quick

in an online classroom what
we mean we say my classroom.

In.

Residence when you're on campus
you go to a lecture hall and

you have a classroom in
K B A I like other courses and

yes your classroom is your discussion
forum so you go to interact with other

students a kind of talking go over
different topics and things like that and

so that's where you actually see a lot of
some of these technologies coming through

particularly the virtual teaching
assistant to kind of work with students

and go through that so maybe I just one
example there are many other O.S.'s

courses that follow this model though how
do we emulate the traditional experience

in an online virtual space so
there is some motivations behind K.B.

I want to first of all up in
the first of is we wanted to kind of

you know I think it shows your idea
was to broaden participation in K B I

to bring it to more students and this is
odd This is a fantastic opportunity to do

that because so many students
are enrolled in the OEM S.E.'s program.

But also we wanted to go over using
research research methods from

the learning sciences to kind
of adopts new pedagogy for

teaching in an online space in a virtual
space a lot of learning science research.

Has issues and concerns with
online learning so how can we

approach that from new perspectives
how can we adapt those methodology.

So that we can get good results and

that was kind of our idea
here was can we do that here.

We wanted to conduct research in online
learning is the perfect time to do

it is the perfect place there's a lot
of students is a lot of data and so

we decided to do it and finally and this
will come up to do some AI research as

well fundamentally we are AI researchers
in our lab there's quite a few of us.

But there is this question here about
approaching education from a different

perspective looking at some of the
problems that we have in learning online

from a different lens and how can that
produce new solutions to known problems or

can it help us identify new problems so
that we can seek those solutions.

This is real quick but this is
a basic structure break down K B I and

it's not too dissimilar from many
other courses in the messiest program

which is it's a full term so
the full full semester course and

their lectures come through videos online
that you can go through and have and

you can go through at your leisure and
within the K.B. I have video

specifically there are interactive
exercises for students to go through.

And many of them have managers involved so
that they can get immediate feedback

on what their current
understanding of the content is so

that kind of formative assessment is in
Chris Christie goal to students developing

a sound understanding of
the material itself so

that's just again this is just
understanding of what goes on the course.

So none of those courses out
there has been out there for

a couple of years you know some
researchers and a lot of wondering.

You know is this efficient right
is this effect of learning

are students actually learning
in this in this course and

so there's been a number of research
that's gone on in our lab and

so I'm going to go over that a little
bit for a jump in some of mine.

A show David both did they work
together to do kind of an analysis

over several years under standing
How does the online version of K.B.

I compare to the on campus
version from two thousand and

four through two thousand sixteen
they were very comparable I mean just

look at this real close at one
year mess Yes just perform to the.

On campus students the next year the
campus students just barely are performed

on us yes but in general as you can see
even from the data here they are very

close if you just look at the means
online mean and they residential mean and

then the difference on the end there
they're very close and so this is good

this is to us is an indicator that we
are seeing some of the same performance on

student performance and
then we have our own child who was work.

So did some research looking at how
students perceive those video lessons

considering that's where they get
the bulk of their course material or

their course understanding so how do they
feel about these video lessons particular

because we know that they have those
extra components such as the interactive

exercises in those nano tutors and what
you found was that students were very had

very high perceptions of these lessons
that they found them very effective and

perhaps most important is that they were
they found them more effective and more

helpful than some video lessons and other
courses or other online courses and you

can see that down here both yes and nano
messiest that was very interesting to us.

We also have some work done by another
lab another member in our lab ita

she did some work on post complexity and
what she did was she analyzed posts in

the online discussion forum for
discourse quality and

for sentiment analysis those together
combined make what we call the complexity

of the post themselves and what she found
for the online section was that the posts

are relatively high quality however was
interesting was that as the class grew

that kind of that post quality started
to door the complex the search drop and

so there's this interesting question
there what is the threshold for

quality of discussion that you need in
order to have high performance does

it need to be the highest possible quality
if you're getting more interaction so

there's this idea here is that this
is opening up many doors to many

questions right that we didn't
know these things before now we do

how does that going to inform how we go
about approaching the discussion forum

section which is our classroom
in the online course.

There was also dealing with the discussion
forum from Joshua Killingsworth

he analyzed how Post
interactions on the forum

affected student Gage mint and super.

It's in a class so

the first the first graph we'll see here
is actually the student response time so.

One of posting post a question

that margin profile when
a response how quickly or slow.

Is here on the X.

axis and the Y.

access is their overall
activity in the form so

we found was that if a T.S.A. or
virtual T.S.A. post relatively quickly to

the students the interactions on the forms
skyrocket so they'll continue to interact

over time which is good we want more
interaction which as we know is as you

know as educators that interaction tends
to lead to more construction of knowledge.

The second graph we have here was actually
we're seeing student activity and

then student grade and so what we saw
was that there's a correlation between

the amount of active they're having and
their higher grades in the course so

their overall performance and lastly
we can see that the time to comment for

the time response like it was here
in this one to student grades and

stream performance and so we're seeing
this this convert this like kind of linear

correlation between what we're
thinking is response time and

student grades if we respond quick and
get back to students quicker and

we can do that consecutive
we can do that consistently

will we see a consistent raise or
rise in performance in the course.

And finally one of things that we've done
this kind of coincide with my work was we

did a analysis a survey analysis of
C.E.O.'s I'm sure all of us are fairly

familiar with the i Weston if not it's
the structure opinion surveys that are at

the end of every term students fill them
out and give their opinions on the course

and the interesting that we found was that
in two thousand and seventeen the O.E.M.'s

the students for K.B.
I were very positive about this course but

actually if they were more positive
than even the on campus students and

so that's interesting to us because
we want to know why they used very

positive polar words and
this is just briefly goes

over it but what she basically found
was that high correlation between

those positive words and the perceptions
that will go over just a minute.

So when I came into a show and
I heard about Joe Watson the virtual

teaching assistant I was fascinated I
was like wow we're at that point now

where we can interact with intelligent
agents online and they can help us get to

that more personalized education that we
all kind of want and we're aiming for but

I had a question a fundamental question
I think about the impact of that on

student perceptions in the course and kind
of what does it feel like as a student

when you find out that this today
that you've been interacting with for

the last sixteen weeks is
not actually a person and so

this question opened up with a larger
door for me which was what are student

perceptions in online education
particularly in almost C.S. and K B A I.

Do they change from the beginning
to the end what we see that change.

And then how does it look like
after we've revealed to the T A is

that it's they know who the virtual T.

is and so this was a project I did
last year so in two thousand seventeen

springtime seventeen in fall two thousand
seventeen gather data from the K.B. S.

it was reported date it was a survey so
we'll be going over that.

Survey used was adapted from the M.S.L.
Q survey which is a was

which is made by Penn tricked a group to
group this is the motivated strategies and

learning questionnaire and what a base
that goes over our motivational components

and self-regulatory self-regulated
learning components so motivation and

self-regulation self-regulation
are two critical components for

success in education but
particularly in online education and

so this was this is a very popular
survey very popular Internet for

use in surveying students
in all classrooms but

more recently has been used in a lot
of online education research so

I didn't so I use some of the identified
subscales from this survey and adapted.

To include an additional.

Scale on the perceptions of learning
assistants in the Course particular

because we were asking questions
about a virtual teaching assistant.

So the four skills that we used for
our survey were self efficacy

intrinsic value cognitive strategy and
learning assistance.

So if you can see for those who may not
know is basically your belief their level

of belief that you have that you can you
can complete a task intrinsic values how

much worth how much value put in that
task or the completion of that task

cognitive strategy this is the use of
different strategies such as rehearsal

note taking review reflection those
kinds of things such that you can

get to a state of cognitive engagement and

cognitive gauge MIT is one of the key
indicators for self-regulated learning and

finally learning assistance like I said
was the confidence students have in their

system they were provided
during their course.

There's a couple questions we had
going into this the first was is

there a change in still reported measures
of self efficacy and these other motivated

are self-regulated components from the
beginning of the semester to the end of

semester I want to know how those things
measured going and I want to know how they

measure going out the students are exposed
to a lot of material over the duration

that course but a lot of tools to a lot
of tools and a lot of the students

member this is a new this is a whole new
program so I was very interested intrigued

in how do they feel how do these
things affect these perceptions.

Question or
two here was how the students measure

their confidence in learning assistance
after learning the identity of the V.A.

And are there any other correlations
between the component measures

that we see one in every
cent of the raise.

Just real quick so this is a very brief
breakdown of some of the data I found from

the two semesters of the first mess we did
not get many students to respond because

it was paired test so I needed both
the student to do both first and end.

But because this was
voluntary not a lot did it

no longer an issue I'm happy to report
that the surveys that were we're doing now

we're having a lot of students
taking that's great but for

this data seventeen we didn't have
too many to take it took it but

was still an interesting analysis to
kind of see what was going on and

then fall to the seventeen we
had quite a much larger number.

So let's get a little bit into what is
these what does it mean right what is

the important part of this.

So we look at our P.

value here basically we
found was seventy M.

both semesters did have
specifically significant increase

from the from the beginning to the end
of term this is interesting to us

because we revealed the T.A.A.

before they took the survey and that
actually reflected in some of the comments

that we saw in the C.E.O.'s surveys which
were I had no idea they were T.S.A.

you know this is it was very
interesting you know they were virtual

very interesting to see that and so
it didn't affect their self advocacy and

that was great for us we want to make
sure that that was not the case and

the thing to point out was the effect size
here is that it was important to me to see

that this wasn't a tiny effect size right
that didn't make a difference and so

we have here is a modern effect size
that's great that's what we want to see.

We also did a correlation
test on these things.

Actually one thing on and I want to point
out I just think it's worth pointing

out is that you'll notice there's actually
another statistically significant value

down here on intrinsic value in false and
seventeen.

Now when I first saw this I panicked rate
because it was actually a decrease and

I was like why would they have less value

why would they find less value at the end
of the term than at the beginning is term

What was it that they found
less value it was that it was.

Isn't what they expected and so

students had understood this I thought
this class was going to do was one thing

it was another thing it didn't change how
they felt about the class overall but

it just had and it wasn't what wasn't
what they thought it was going in and so

it did change how they valued that
course which is interesting to me.

Anyway so
yes a correlation this is interesting.

Correlation here is that there is some
correlation across the board on all

on all measures and you can point out
the strength of the association so

you can kind of understand what this
means but the important bit here

is the coefficients value so we can see
if it's statistically significant or

not we do have across the board this
is all by the way should mention

that it's Coggan of strategy
is intrinsic value and

the teaching assistant ship correlating
to self efficacy so I was very interested

how those things related to self efficacy
specifically did they have a relation so

we do see some correlations and
the fact that they are sleep significant.

After all of this I started to really
wonder you know to what extent does

encourage student use of desired learning
behaviors because those measures that I

had measured before indicators of desired
learning behaviors both lated learning and

motivated components so
things like self advocacy and

value are usually the indicators we look
for when we're trying to understand.

Is the student getting what they need is
there efficient learning going on and

so I was wondering what are some
other ways that we can go about

you know investigating this issue can we
go deeper and I wanted to go deeper So

this is kind of where we're at now

is the presence desired learning behaviors
in indicative of effective learning.

Well let's get a let's get a working
definition of what we mean like I just

said does our learning behaviors I'll
break it down for a little bit so

working off a pinprick and
a group original working definitions of

these two components I'll get into
the motivation components that we're going

to be using is based off of
a general expectancy value model

what we have here is expectancy which
is essentially your self efficacy

yourself believes we have value which
is how are you valuing this experience

in this education that you're getting
receiving and your effective state so

your emotions your feelings about it
overall we also have so that there's

self-regulated regulation components and
that deals with Medicare Advantage of

strategies such as reflection I mentioned
earlier and then also your ability to

to regulate your cognitive state you know
are you engage or not and that deals

with management what ways do you go about
engaging and kind of managing that state.

And finally the call going to themselves
what are the actions that you take

those practices that you have that lead
you to being in that state prolonged and

is that related to the different
motivation components that you have and

we know from previous research that these
things are these things are linked right

but I was interested is are they existing

Do they exist do they manifest
in the course and further.

What's impacting them like what's
influencing them how do we get them

there how do we get them
to the levels that we want.

So I was like OK how can we measure
these things there's two ways

The first is we could just do
the qualitative approach right

which is to use validated
instruments such as the M.S.L. Q.

or other to measure these things in
students and then to crosscheck them with

to performance to see if
there's an correlations and

that's that's fine we can do that I'm
doing that currently there's also

another way which is something I'm also
doing which is to utilize a theoretical

framework the reason I want to do this is
because I believe it will provide me some

common terminology and kind of a map to
which we can use to figure out exactly

what's happening in different sectors of
the course what parts of the course what

part of the tools etc are influencing or
impacting specific practices or

specific perceptions that
we have in students so

I introduce to you the community inquiry
framework just a rough tell me if you're

familiar with community inquiry framework
OK it's a couple that's that's good OK so

creative inquiry framework what is it some
people I guess theory some really gets

a framework I tend to fall between the two
it was designed by an Anderson and

Archer and about two thousand the point of
this was to develop a learning community

that could produce purposeful
discourse right so

the collaboration of individuals
in the community towards Co

construction of knowledge you see
a lot of this in education and

right now it's probably one of the more
hot topics for online education so

it's particularly useful it was
originally argue or it was originally.

I guess it was described under three
dimensions which are social presence

cognitive presence and teaching presence
and you can understand those things social

presence is how you interact
with other peers around you and

become and identify with a community
of learners your cognitive presence is

how you use the content material and
what's made available to you and

finally teaching presence is the presence
of the instructor themselves and

their impact on the overall
learning experience and I use this.

This next model here
which is overlaid with

dinner a different interaction types
to make the whole bit more clear

these interaction types are actually
argued by Dr Karen Swan to help people

understand exactly why these things are
interconnected the way that they are and

so the way place we want to be
right is right there in the middle

right between all of them coming together
and that's when you get into community

inquiry model that's when you get that
that most efficacious of learning.

So I want to know is if we took these
dimensions and added two more dimensions

what would it look like so these are two
more dimensions that were actually argued

by two other researchers Dan bone ache and
Peter Shea The first being system.

Presence is that this is an online
classroom we're dealing with

we need understand how the system right
interactions with the system impact those

perceptions and practices as well
because it's an Integra part of it but

also learning presence and this deals more
again with that kind of self ethic a see

those motivational components that
needs to be that need to be highlighted

a little bit more inside the community
of inquiry model because there currently

are the in the it's regional state
there wasn't much of that but

it's it's a critical part
of the learning experience.

So.

I offer this which is the new model which
is basically brings them all together and

what we have here is with a model that
we intend to use to understand and

look at the courses in the less
yes course or in the program so

for example if we took a B A I.

And looked at teaching presence here
what are some questions that we could

we could think about how do we begin
to categorize what's going on with

the tools in the design of K B A I
well we could ask how does the class

design support teacher student
interactions so what are all the ways and

what all what are all the different tools
made available such that that's possible.

Does Joe Watson impact student perceptions
of teacher immediacy and K B A I so

we know that teacher immediacy for those
who missed to me but if you don't know

Teacher me to see if the presence of the
teacher that the student can tell is going

on so how does the student perceive the
teacher being there or not it does impact

how they feel about the course overall and
the ship performance and

so one thing we've wondered Is Joe Watkins
interacting with students in the course

whenever they are there and they make
a post or comment does that impact

the student sense of teacher immediacy and
social presence though two things that

most education researchers were really
concerned about when it came to online

education was how did those presences
manifest will will an artificial agent

impact those it will be a positive
impact positive change would also ask

what tools do instructors used to interact
with students this is done from the.

First one because the first one is an all
encompassing what's made available what

all can they do the second one the Basha
the third one the bottom there is

what's actually happening what
tools are being used are they using

tools that we didn't expect them to use
are they doing something new that we could

not have thought of and so these are the
kinds of things that we can think of

like a cognitive presence is another
one what are the different tools made

available to students to interact with
the course content so this list basically

this boils down what what did what came
in the learning management system but

also what did we add and
how are students using them.

So do you since find that an attitude is
helpful this is an ongoing question right

now I'm actually looking at this currently
as in today I was running the data

which is we have a lot
of survey data that.

Has been collected from students
over the last couple years from the.

Course that tells us how they feel about
the feedback they're getting from these

videos and K.B. I we know that
they're using the Nano tutor so

it's very important but
what about a course like.

In the OEM S.T.S.
program that doesn't use managers so

we're looking at these two things so
that we can kind of understand what

is nano tutors doing it if anything
right and move forward from there.

And then is there a measurable difference
between courses that use now tutors and

courses that do not like I said and

how they perceive the overall
helpfulness of these course videos but

this there's so much potential here
you could line up a million different

questions to these areas and
that's kind of the point I'm getting at

is that to really understand how learning
is happening in an online course or

even a program we need to
get a holistic view and

a map like or a framework such as can
even query model is a fantastic map for

that we can begin to put things
into in the individual areas but

then we can also move toward that sweet
spot in the middle that will help us

understand what's actually working
we know how we know what desirable

learning behaviors that we want to aim for
we know the perceptions of the practices.

How can we under how can we identify
the tools how can we I defy defy

from the teaching end what we can make
available to students such that they can

succeed there where everything kind of
Iraq's So finally I'm closing I'd like to

ask you all you know so what are some new
models of learning that we can discover

by introducing new technologies into
education so a lot of what we see and

I was developed because we approached
education through an AI perspective

we didn't just we didn't rely entirely
on conventional education research

did guide us the idea was to use
new perspective to help us make

new tools new technologies to see how they
worked in the space hopefully tour and

identifying solutions to those issues
that we know play online education and

specifically say online learning right
we really want to students to learn and

we want to find new ways to to facilitate
that and what are new domains or

some other perspectives that we can
do to do that or used to do that.

Thank you.

So open up to questions or
comments or anything yes.

Yes So let me get your first one first.

Threshold is I don't know the exact
numbers I would this was a my

this was in my study but what we found
basically was there was at some point

over time the first semester if we had two
hundred students and the second semester

we had like five hundred or so and that's
just rough numbers but as it grew so

as the class got bigger the complexity
of the post started to lower right and

so we're interested now moving forward is
exactly finding that point what is that

sweet spot but also just because it lowers
doesn't necessarily mean it's a bad thing

because if we need to know
exactly how it stands or

interacting if they're lowering does that
mean that maybe they're interacting more

you know is the quality of discussion
is in question here we need to know

you know the rate at which
the discussions going as well and

then also to discuss the students
themselves about how they

how they perceive those communications so
there's a lot going on there.

What we mean we say Nana tutor is
you when you go through a video

you'll get to a point where it will ask
you a question and you can actually

interact in put suff into the video
itself and submit your answer and

it'll give you feedback on that answer and
whether you're right or

wrong how you can adjust your answer to
get toward the right answer try again

here's what we think you did and so there
is some analysis going on there that ask

is basically looking at your answer seeing
the difference in your answer in there and

the actual answer and trying to
guide you toward the right answer so

there is some scaffolding going on
there and so there's about six or

seven of these in each video lesson that
we like micro So they're very specific

their content specific to those
specific lessons themselves but

we have seen that they really do
the students really like them and

that they seem to perceive the message
very helpful in their overall education.

Yes.

So that's one.

Right so the.

First was discourse quality so the words
were actually now analyzing the words and

there's a scale out there that you can
that is available for the difficulty of

the words put together and like so
how complex is your sentence structure and

then there was also sentiment
analysis that she use so

she can measure the sentiment of
the sentence overall to see if this was

conducive to a discussion overall or

along a long term discussion versus
just a you know like just a comment.

Or a just saying I don't like this
I do like this versus I don't like

this because X.

Y.
Z.

What's your feedback or
what do you guys think and

I believe she's going to be continuing
this further and pushing this so

we can get more information on that but
I also share the paper out right yes.

Or.

Just.

Right and so let me let me let me have
this business of our interesting and

it deals again with the community inquiry
model which is we understand what

tools are available and what's.

Are using what's at what's at their
disposal this is a good example because

over time since have started moving to
slack to have one on one interactions but

they still go to discussions forum to have
those those I guess those inquiries so

I have a question about this have
question but that logistics except So

there's a question then is is
the quality of discourse changing

because we've introduced new tools to the
course that they are now offloading some

of that conversation to we don't know but
it's an issue in question and

it's worth asking and
finding some kind of answer to so

it's really hard to identify exactly why
it's happening but we do know it is.

We're.

Right.

We're also moving forward into
recruiting students themselves to

give us more information more depth
information on those actual discourses so

that we can get that extra layer
of understanding and insight to.

Know this is this everything
that we did here was mess yes.

They we do have the research to
show versus the on campus or

compared to the on campus.

So yeah.

You know it's actually something
we can see now because there's so

for those who don't know
that the online on the.

On campus for K B A I there's now two
sections there's the online section and

there's the resident section but offer on
campus students we're all using one people

to form so we can now we actually
can get real data from this specific

interaction of those students who
who are used to taking classes on in

faith face to face but are in an online
section how do they react to appeal so

discussion forum how are they interacting
versus the students who are usually take

online courses because they're part of
the onus Yes of course right and so

there's there's a lot of questions going
on there about those interactions but

now because of this extra section that
we just cut the semester we can do even

more data analysis than comparison because
prior to it was two different populations.

And so it's very hard to make
any kind of a claim there

whereas now we know it's one population.

Yes we did so interestingly
enough I don't have the data on

hand in the show you can probably correct
me on this we were concerned that as.

We were it we thought that as a as the
Course grew that the discussion would grow

but we found as it actually didn't
grow in my career it actually lowered.

It's very hard to it's very
hard to quantify because

in all I mean think about a face to face
classroom how much of that interaction

of social right it's very hard to
save depending on the class so

what we see on the slack side is
that a lot of it is the social.

Right the kind of getting to know
your peers and your classmates but

so the discussion forum ends up being
just that could be the quarry and.

And there was a lot of concern on our team
was how this how is this going to work

How's this going to impact our research
houses going to pack the students'

education and their learning quality
however when we think about residential

classrooms it's really not that different
in terms of like what they say and

do in the classroom itself but
we have that kind of the lowering of

the response or the question asking
rate because there's a little bit but

because we don't have all the information
yet know exactly how it's happening we're

we're not we're holding off on making
any claims or anything like that so.

Boy.

So that.

Expertise.

Right.

Or.

Right.

So I can actually answer this yeah so.

We did we did an experiment last year
where we did try to develop multiple

shooters that were programming
language agnostic could

you can put into your own domain it's a
very it's a very expensive task to do that

these were not developed using machine
learning these were expertise based so

someone went through and actually made
them knowing So yes very resource intense.

However the idea was we want to give
the students experts feedback and

when you think about traditional.

Forms all indication that are large and
that scale it's very hard to do that

on a like we're actually interacting
with the professor themselves and

so they're the cost would offer us was
to create these nano tutors we could

provide them that expert feedback at scale
even though the cost was upfront for

those tutors themselves we cannot
provide them to many more students

whenever they need them on
this content specifically.

Now as for developing it further and

like using some like machine learning
to produce them it's definitely an open

question you know I think we've
talked about it a couple times.

Yes.

Well.

That's the thing is you can use it for
you know.

Just what.

Her Shoes are actually.

Like.

So we are.

Yes that's something that we've I don't
know we've talked about a large about that

but it's like basically you're
building a case library of nano

tutors that even students can but
then you have so there's questions about

the expertise of those monitors
right the accuracy of them so

that was something that we thought
about we can we consider it was like

if we had others develop these out how can
we ensure the quality across the board of

each one for students when they're
exposed these things we don't want to

perpetuate misconceptions and so
that's a concern that we had was that

if we come across a misconception
what happens if we accidentally run.

And so yeah so

there's a lot of questions I think
the question of managers is kicking off

again right now because I I mean I'm kind
of like in the middle of that research.

We really really want to understand
exactly to what extent they're impacting

the sinners experience because
if we can tell if we have some

some objective some quantifiable measure
where we can say we know now that it's X.

amount of points that they're increasing
instant performance moving forward we

cannot that we should before them but we
can now start to think about new methods

for the development of them to make them
maybe more efficient in or develop more

efficient because it is I mean I've taken
the costs right that uses these now to

tutors and it makes a world of difference
just to have that immediate for

me to feel about or the form of assessment
feedback on a concept when I think

I've got it and then I do this problem and
I don't got it you know then I can think

about it again and reiterate and fix it
and so that's really nice and really does.

Here say.

Yeah.

Yeah.

Yeah yeah.

Thank you.

So.

Yeah yeah.

I do.

Yeah.

Yeah.

Yeah well I think I think one thing.

We think about too is why you
why is there this perception

of larger courses meaning
lower quality education.

It didn't come out of nowhere I mean
a lot of the a lot of the data shows that

students do not do well in
larger courses right but

I think what the lot of people don't know
or they don't talk about who they talk

about is that the courses that they're
enrolled in are kind of one size fits all.

Think about Xmas that's what they do is
that you have one way of learning and

that's it you have to use that
that's all that's made available.

And that doesn't work right and so one of
the things that we're trying to push for

in this and we I mean Katie I's
example of scalable learning and

a lot of the tools and
methods that we use.

And one of the reasons why is because
it can be used in multiple classes and

it can still add that individual eyes
personalized manner to an extent such as

like the use of the Nano tutors or having
a teacher basically on hand whenever you

need it using artificial agent to respond
to questions whenever you have them and so

those are two things I know specifically
that were issues with larger scale courses

that lead to that negative perception
I don't think that everyone is against

online education against large scale or
scale education it's just that thus far

there hasn't been a lot of there hasn't
been a lot of evidence to show that it can

work and so our hope through our research
and mine specifically is to kind of change

that perception at least in the messiest
program because it is kind of pushing and

piloting this new area this
new domain of education higher

higher education certainly but it's huge
INSKEEP And keeps growing there's a lot of

people enrolled and I want to make sure
that they're getting qualification I

want to make sure people can see that we
are providing them a quality education and

I think us all over here yes.

Get.

This this great so do you want to.

OK so

this yet she has there's like a confidence
interval she will only answer if she knows

the answer it does happen on occasion that
she has been all she dropped the ball.

When that happens we step in right.

The thing is like this the first semester
came out people didn't know that it was.

Everyone who's in the class know now kind
of understand there's something going on

you know they don't know who it is
could be a teacher could be used and

who knows you know.

But so once the you see something that
happens I don't think it's shock students

it doesn't shock us we know it's going to
happen mostly because there are students

who go to the class now and whose sole
purpose is to try to like we did out

by having the most complex
English senses you can think of.

I think one of our I'll give you
example one of our members Bobbie

published a paper on this on the in
the AI ethics a division but

basically she said Ai or Joe Watson.

Doesn't care if you're
pregnant was that what it was.

Exactly what happened she.

Was saying that they're having a baby
soon or they're they just had a baby or

are they just another person and
they were celebrating on the board and

show Watson responded in the most
ridiculous manner demoed was.

Yes.

No limits to the answer I don't.

Know what.

The law.

Actually as to why it was.

Hard to understand the reason is that.

He was.

So that he.

Was killed.

So.

There was a cushion there.

They are right there for
the movie is going on so there was.

No real.

War.

This is what it is all the way or
the other markings are just.

What he thought he was some good in
mind this is what we don't know.

Her ontology wasn't fully fleshed out
time to be able to accommodate it so

I'm sure it will be part of order to.

Hear what you get for it all seem so.

We have a very insensitive T.

a parent.

Yet.

Just.

Like to.

See.

Their.

Suffering.

Right so that's that's one of
the reasons we wanted to so

I want to look into this further using
some kind of model to understand

exactly how that's working because it's
not enough to identify that these things

are that students have these different
behaviors right we want to defy what is

is there some is going on the course
that's fostering these behaviors that

is impacting them that's influencing them
how can we foster them in students how can

we get them to that point so knowing
them when they go into the course and

then when they go out you know exactly
what do they interact with during that

time that we can help maybe identify
this component or this element we know

is impacting or we suspect is impacting
this perception or this practice and

we're dealing with these are graduate
students I mean some are undergrads for

them yes yes but certainly for the on
campus version and because we are dealing

with now the on campus version as well we
can gather that data but I will say I'm

also now looking into the thirteen a one
course which is strictly undergraduate and

is introduction to programming and

there's also the open in thirteen a one
course right for it for anyone all over

doesn't matter the risk or not want to
take this course online that they can and

so we're trying to get as much data and
information as possible so

we can look across the board at
students at various levels of education

how are they interacting with these
components how do they measure on these

practices scales and what are they working
with that might impact those practices.

If.

You.