So yeah I'm a second year P.H.D.
and the past two

summers I've been working at the Georgia
Tech Research Institute with Taka and

we've been exploring sonification
projects largely using web audio only so

traditionally sonification
is something that happens

on a desktop computer you have your
Maximus peep your data supercollider and

you have a static data store
this can also be dynamic but

instead what if we start to sonification
on the web and how would that look like

what would make sense what sorts of
data types can we can we explore.

So that that's kind of the focus of
this talk I'm going to present to

two applications that
we've that we've explored.

Which are Internet of Things and
smart cities and this is just an image of

a future world where everything is
connected to the Internet so your

traffic lights your refrigerator
is your buildings your.

Your brain you know everything's
connected to the Internet and

the information is being transferred and
there's just a huge amount of data and.

So the question is how do we
how do we start to parse this

with how we sort of parse this data and
how can we.

Convey it in a really nice and
convincing manner to two users and

so the answer to this are is
clearly called dashboards

dashboards kind of
basically come to you with.

Boxes modules which have graphs of
various kinds Here's another one

providing information on things
like whether sunlight like any any

sort of data coming coming
to you in this very.

In this format.

And these are very effective
in very convincing and very.

Very I think it's that equally can
be very steadily appealing but

I think there's a lot missing to them and
what's missing I think is really more

more modes for communication and in this
conference we're interested in audio and

so we're interested in how can we
add audio to things like graphs so

that you can do things that you
can't do with vision alone.

So this is this is
the part of motivation so

effectively conveying information
using multiple modes.

Also there's a security interest in
this that's basically one of our

the second app which is
Internet of Things and.

Basically if it's more device you have on
the Internet the more chances you have for

security problems and
if if one device on your network is faulty

that could make actually a really big
problem affect a lot of people so

can convey this information in
a good ways important for security.

And also we want our dashboards to be
friendly and use it usable by a broad

audience so not only specialists
who are monitoring networks for.

For data purposes but also the general
public so how can can can we

provide information that's useful and
relevant to someone who has no.

Stake in a stake in the data and
just wants to experience it and

so there's some pretty.

Also important characteristics about
this data first of all is that there's

a there's a lot of it so we're dealing
with a a lot of a lot of data a lot

of different dimensions also changing in
time also different many different kinds

of data all happening in real time and
this is actually you know the sorts of.

Data properties that actually lend
themselves so well to sonification.

So what would it what are the vendors
of hearing Well hearing as ice free so

you don't actually have to be looking at
your screen to be able to hear or see.

If occasion and
be able to get the information you want.

Also sound is actually a stronger
alert system than visual you'll react

faster you'll react with more emotion
than present it with a visual or.

And then also you know a lot of.

Scene analysis and

the ability to parse many streams in
parallel these are these are reasons why.

We we want to use that hearing for
data and then also I think sound has these

really nice cultural properties so for
instance music is a really good example of

that but you know the social the social
aspects of sound are really important and

important for design especially in this is
really important for engaging the public.

And for usage and
sharing all these things so

basically consider to two audiences are
specialists and public specialists want

to monitor multiple information
streams in parallel.

And often have to balance their
attentional resources so you have a visual

you're actually many things you're trying
to do at the same time talk to colleagues

talk to whoever also monitoring all these
sorts of things tasks and to be really

convenient if you didn't have to always be
looking at your display if you could just

have this ambient awareness of something
like security level or or weather or

any sorts of data just just by just as an
ambient sound in the environment and this

would I would hope this would allow people
to discover and resolve issues faster.

Than just using visual stuff.

And also the public is a really
big audience that we want to

you want to attract to the display and so
that's another reason to use sound and

we hope to get to target audience.

So they're actually from what I've
seen there's not a lot of sonification

that's happening using web audio.

And what body you is a technology that
is developed for the web and when you.

Marsan a fine web based center from Asia
information using web based technologies

that are native here browser for instance
I like to call this website occasion and

it's very interesting and
very interesting useful prospect and

I hope that to see more and
more sorts of help patients like this and

again the desktop environments are Maximus
peeper data supercollider these

are very good and
very flexible very powerful tools but

they do not travel well they don't travel
to your mobile phone very well and.

So they're they're limited in
that respect and so for for

sonification on the web you might consider
having something like a web server where

you you met your message and that it
doesn't advance sound processing and

spits it back to the user that's that's
too complicated let's use Web audio.

And Web audio is is good but
I think personification it's

not it's not enough it doesn't come with.

I guess data Date data packaging and date
data operations that you often use and

sonification when you as part
of the mapping process and.

It also is I think for

people that are just approaching it
a little bit too a little bit too complex

for which which are interest in doing
as sounds of the system mapping and

I think from my perspective the it's will
be easier to think of it more abstractly

which is why I use the we use the David to
music framework this is actually A.P.I.

developed by my co-author taka who's
also going to be presenting on it and

it's an A.P.I. for
the cation on the web and it's not only so

it's ton of occasion has has data data
operations that you can do it has.

A built in modules for
doing synthesis very easily and also.

Basically it has musical
structures in it that you can

you can very easily interact with and.

In use.

So this is actually makes
it a very nice library for

sonification is probably
the best one available for

web audio at the moment and talk well
we'll talk more about that later on.

You can play with it it's live on
Heroku and then you can also use it and

build build with it and
from music using it have so

first application was showdown
which is a search engine for

devices and this is kind of
a scary thing when you realize how

many people use the whole passwords.

And many people do and so it's it's
a it's a rather it's a problem and

with this particular search engine you
could do something like search for

an outdated version of Apache with
with a known security issue and

this is actually something that would
be publicly available if you don't.

You don't actually do
the necessary security steps for

your refrigerator device and
someone who's malicious might

actually find your device and use your
default username and password or it or

launch an attack knowing what they know
about this particular version of apology

so it's kind of scary.

You know isn't too.

Hard.

I.

Mean.

If you like I.

Think.

And this basically displays the publicly
available information that.

I.

This.

Says about the it looks
like one started the day

fifteen minutes twenty
minutes now all these are.

So no no match showed in Canada but

I think it was actually go to
the service and check it out myself.

Looks pretty secure.

So.

Now there's a lot more activity also
the sound environment is different.

And so
basically we have a different level.

Using kind of.

What is it like one that
are happening whenever something.

Is a case where someone's
not very good about

that.

So anyway this is after three hours and

the video part is important here
because actually we can go down to

Fulton County Government Center
the capital of Georgia is in Atlanta and.

There's a lot of lot of
services that have to with.

Even more thirteen hours.

So there's a lot of a lot of data here and
you don't want to have to be kind of

looking at the need to have
an environmental awareness of that.

And so we think sounds really really
effective here specially the environmental

audio which can tell you something about
ambient threat level and also the tempo

which is using the framework which
is kind of like an ambient kind of.

You know just tempo and I can tell
you something about activity level

the second application is smart cities and
this is.

A burgeoning field where you basically
are equipping your city with all sorts of

sensors and trying to make it.

Run more efficiently run faster and
often a trademark of this would

be like this sort of sensor box here where
you're basically collecting all sorts of

data to another piece of instruction
which is like a light post and

you just stream stream data
from this particular point.

So you need a second a second second
using using the D T M framework and

it's a lot more graph based.

For instance traffic in this area.

We have five different sort
of sensor boxes going.

OK So this is something we the other
thing that wins very very effective as

well and
that was done by talk a particular.

So we haven't had very much it
very much evaluation we've done

a few public demonstrations and
guns in the informal feedback.

This is basically convince us that sound
is certainly enhancing the visualizations

some respects increases in gauge when it
allows you to look at your display and

still gets information that you want.

And also it doesn't really seem to
require very much explanation especially

when there's a real redundancy between
the visual and the auditory cues so

there's like a modality
blending going on here.

And then ray of things project is a is
a big organization and they really took

the the act of making this information
publicly accessible to heart and this.

Is basically our project interest
interested them for public outreach and

they've counted it contacted us and

we're continuing this I think on a much
much larger scale I'm not actively

involved anymore I think Lee Lerner who's
a co-author on this paper will be here on

on Tuesday if you want to talk
about that he'll be available.

So what's on a vacation is something that
is very useful for Io T. and smart cities

and it certainly has as time goes on and
the number of devices that are connected

to increase I think we're going to be
looking towards other modalities for

for information display and I think
sound might be a very effective one for

the reasons I've described like the
cultural impact and the ability to to use

sound where you're when
your eyes are occupied.

Elsewhere.

Sound increases excess ability and public
engagement with data and I think that's.

Pretty but very very strong.

And so which is the background for
all this is a very useful A.P.I.

for website occasion and
something we've made here the G.T.C.

empty happy a very happy about it and also
looking forward to future development and

talking with people and making it better.

Any questions.

Thank you.

Thank

you.

Are there any current new projects
you're working with the Tim from are.

You on these two I'm not so
the developer talk is using a lot and

is going to buy coding demos if you
want to see it in action talk to him.

I think it's I think when I was using
it you know on the one hand web audio

was was very kind of technical and there's
a lot of just like code that you have

to write to get it operational and to T.M.
I think at the moment was not

quite there was this little bit buggy.

But now I think it's a thing
it's a lot stronger.

And.

Capable for more projects.

Flocking for instance is another
environment you might consider

personification it's a little bit like
super collider but on on the web browser

is made for sounds of this
ISP It doesn't come with data.

Basically data operations so
you have to get another library for

processing data before you sign a five.

What you didn't mention
is that our vacation was.

Particularly strong to review
temperate aspects of your data work

to the timing of your data.

So it's all unfolding in real time and
I think we captured some of the events

through like one shot sounds or
like three three changes.

I think.

Except for in the last case where we were
playing through the data very quickly

the density that we're
displaying is not so

at the level where you might use
something like a lot of occasion to

hear very subtle things that are not
immediately available visually.

But yeah that is sort
of unlocked potential

there.

OK I've got a question in the very last
the web with your weekly newsletter sent

by Chris Lowe is that men your
first read I think I saw her

interest in Interesting give presentations
by a mathematician that's visualized

using web and the for the representation.

Of the four year transformation and
it was really.

Strong as a tool for
visualizing Why hearing the music and

understanding the data behind that's
what's the term transformation does so.

Do you fink about maybe.

And maybe.

You would sound like your certification
in that case I would guess for

the moments just frequency and and.

And the night of the notes and
yeah the eight of the notes so

I don't know if you looked at her for
a presentation.

So with sound dimensionality is not
very well defined you can keep on adding

streams and streams of streams and make
a very rich dense auditory auditory scene

and in fact you don't you're not actually
limited by a particular line of sight

which is which is the way that vision is
so you can have sound all around you.

I haven't I haven't looked at any three D.
graphing environments.

I think that.

You can with sound with with sound you you
might be able to display things you would

be able to in a three dimensional case for
the next evolution.