So I'll just stand here. So here we have a mike at but if you can't hear me or anything then just let me know. All right so this is a very interesting project that that we have to do some recordkeeping here first of all to point out that this is truly an interdisciplinary team something of this magnitude cannot be done by one person or one lab or one department. We have people from computing music psychology OK Bruce here in the audience is one of the the main movers and shakers of the project and we can do it without a real team effort. Also we're happy to point out that we're working with many of the organizations here that support visually impaired people in their daily lives and also the aquariums that are close to us Tennessee Georgia and Zoo Atlanta we're also getting lots of interest from other aquariums and other facilities around the country too. To hopefully expand as we can later. All right. So just to frame what we're doing here in a bit of a psychological theory. I want to point out that there are a lot of things that we do in our lives that are just to get by a lot of things that we do getting dressed in the morning feeding ourselves having a shower getting to work all of these things are are fairly basic needs that we we have to satisfy and mows low had this hierarchy of needs that he laid out where at the bottom of this pyramid in the in the red we have the basic physiological kinds of needs the things that you really just have to do to survive. And then as you go higher up towards the peak of this pyramid you have what he called higher level. I'm in the it's. Including safety belongingness that the need to have friendships and and and affiliations esteem confidence respect from others. And and this carries on up into sort of cognitive needs the needs to know things and to find out things and be curious the things that we have the issues that we need to scratch in relation to beauty and artistic endeavors and then finally at the top of this pyramid. We called self actualization and transcendence which are sort of these higher level aspirations that we all want to get to. The the fact that this is a pyramid suggests and is intended to point out that you first have to satisfy the lower levels before you can really work on satisfying the higher levels. OK. It turns out that assistive technology often tends to focus on these basic lower level needs. There is plenty of assistive technology to help people get from place to place or to make sure that their socks match or to help them get feed themselves or others and also communication this work. Some of these slightly higher level needs are generally addressed through the system technology for people who need it. I'm making the claim and this project is one effort in this line that we need to focus more and more efforts in assistive technology on a higher level including aesthetic and cognitive needs and make it more possible for people with disabilities to have these higher level aspirations to learn things to have affiliations to be creative. All of these things that are very rarely addressed through a system technology. So this project is is not suggesting that we don't address the needs that are lower in the. Pyramid. We definitely need still to to make people comfortable and make sure that they are able to get to work and so on but we want to add more study of these higher level kinds of things. All right. Now I mentioned disability and I want to bring that into the discussion here and certainly there are many more people with disabilities where disability is you know a lot of different kinds of of impairments and they're more often living in the community as an example for people with vision impairments often the students or children with vision impairments would live in a school for the Blind a residential school. Historically But now those students are living at home or living in the community and they're going to public schools and call it. Mainstreaming right. And people are living longer people are out there in the world. As as we have more people with a variety of disabilities in the world we need to serve them and in that in light of my previous life we need to serve them in different ways not just making sure that they get to a place to place but that they have access to more knowledge and more creative and aesthetic endeavors as well. Certainly vision loss is quite prevalent it's a growing problem. Mobility impairments lots of different aspects that we need to address and always keeping in mind how can we enhance the lives of these people at this cognitive anesthetic level. All right. So you look at the Georgia Aquarium or the Tennessee Aquarium or any IG aquarium or a zoo or an art gallery or museum any of these kinds of places are really at. But. Attempting to provide some information or some opportunities at those higher levels but they're typically very inaccessible. What we can talk. About accessibility on a number of levels certainly physical access. And we have all heard about the Americans With Disabilities Act A V.A. and how most buildings will be typically in compliance with. Which means they have a wheelchair ramp or they have some very basic accommodations. But the intellectual and emotional access are not even addressed in in the requirements. So is someone able to get access to the information. Are they able to get something out of visiting a museum or an aquarium from the intellectual side something that is even a well perhaps I was never addressed as this emotional access going to the to the aquarium. Or to Izu or to an art gallery can be a very visceral experience you can stand and be moved by the grandeur of nature but if you're blind you got nothing nothing on the intellectual side of it. Nothing on the cognitive nothing on the emotional side. So we're trying to look at all of those aspects in what are known as informal learning environments. So here's a case in point. OK. So let's let's experience this. But let's all close our eyes so so bear with me. Close your eyes and tell me how beautiful this scene is this is an amazing scene this is my most favorite scene at the at the Georgia Aquarium and it. It really inspires me that you agree. All right well you can open your eyes and and it truly is amazing to go and see a sixty foot wall of glass behind which these gigantic animals whale sharks and other kinds of of ocean life are cruising and doing their thing and. Here's some other. Samples you just see these are amazing things but they're behind glass so I hope that you know you'll agree with me that that's leaving a lot of people out there. All right. How do we deal with us. How do we address this one might say well for people with vision impairments which is the group that we're tend to focus on in this project. Often where you sound to help them deal with the fact that they don't see things as well as or at all in an aquarium or a new zoo or in an exhibit of any kind. Typically the use of sound is background music it's mood music it's elevator music it's not informative in any way sometimes there's a composer who has been commissioned to make the music Tyurin to the exhibit or give some mood or set some mood. So they're not completely out the music is not completely unrelated to what you're seeing but it's not driven by what you're seeing it's not completely connected. Further the music is composed at a point in time and even though the music itself might be dynamic in in that it's that's being played. It's not actually conveying the activity within the exhibit right. So what's actually interesting in many of these exhibits is that they move that they change the fish are swimming and they go slowly and then suddenly they move quickly. The monkeys are playing and throwing poop at each other and this is the this the stuff that we talk about. This is the joy of going to an exhibit again and again and again it's always different. But you don't get that unless you can see it. So we're trying to in include the use of sound in a more structured and more connected dynamic way. So maybe we can use me. To convey. What's happening in an exhibit and I just have to this. This is one example of how something so you can see there's a dynamic because it's so. Right. So our approach is to actually track the critters track the fish the monkeys the zebras the animals in the exhibit. Figure out where they are where they're going what they're doing and then convert that movement and dynamic information into some kind of audio typically music but also other kinds of narration. So this is basically just the start but I want to give you a flavor for what we're up to here. The typical components here is that we have fish videos or for a camera monitoring aquarium. We have some hardware some computer programs that track where the animals are the information from that tracking service is passed on to a music generation service typically running max M.S.P. which is just music creation software and reason is another piece of music creation software and all of this just gets passed on to the listener. So it's a fairly straightforward pipeline if you will but each stage is quite complicated in and in and of itself. All right so I'm going. To talk about a number of the the the aspects here but just basically to give you a feel for how many different questions we're asking here. The first thing that you have to consider is what do people want to listen for what do they want to know about and what is salient So if you went to the to the aquarium and you stood in front of an exhibit. What would you be looking for and what would you want to know about it. So we showed videos of aquariums to undergraduate participants and we said you know what are you looking for what catches your eye. What do you want to know about what intrigues you and trying to get at this question of saliency. It turns out that color is obviously salient and as a size behavior event so when a fish arrives or leaves or when two fish interact in some way. Those are all salient kinds of things. So those are going to be useful for us as we move forward and design a system that can do this automatically we can look for different colors we can look for fish that are unique in terms of their size all of the things that are salient to the visitor we can keep on those in our system. So that's the kind of perceptual work that we're doing in the psychology labs. Then we take that information and make some some music from it. This is going to progress a project that's been developing slowly over the last two or three years and we started by just using videos we didn't have cameras and we didn't have a fish tank and we were just basically mocking up were faking some of that stuff because it wasn't the the hard part as far as we were concerned we had an ugly simulated fish tank. It was all done in software in virtual reality but it. The job for us. It allowed us to convince ourselves and some others that this is actually possible and now we've got a full pipeline that includes cameras monitoring real fish or ants or other creatures connecting to a live computer music generation system and we can get real time feedback. In addition our visitors our listeners can interact with the system so that they can say I don't want to hear about this animal or I don't want to hear about that animal and so it's a complete system not. At this point. Our job is to make it sound good and to sound right. And these are the challenges. So I can show you some of the things that we've tried and this is a bit of show and tell here to give us a feel for how we can take the movements and map them onto onto musical instruments. So in in the video I'm going to start with you'll see that each fish is mapped onto a different instrument. So each fish is played by or is playing a particular instrument and this video you'll see that the the the fish or the instrument if the officious to the left of the tank. Then the instrument seems to come from this left speaker as the fish moves across to the right side of the tank. You'll hear the sound go with it and it should come from the right side if the fish goes up in the tank then the notes will get higher and pitch bum bum bum bum bum bum bum and as the fish comes out bum bum bum bum bum bum bum. You'll see that descending in pitch. We would also mouth things like as the fish were closer to you would be louder. If it went away from you would be softer. So there's a fairly straightforward natural mappings. Now we can bop you over the head with these straightforward mappings and the and the sounds of the music might be informative but it might be boring. It might be an interesting enough musical. So we've Well we've done some of that hopefully learn from it but we're also trying to make it aesthetically pleasing as well. And so you'll see that we've layered this on top of a musical structure so that you can feel that this is music and not just random sounds. Right so let's see. Here's an example talk over a little bit on the screens and there's the listening to the right here that has the full swing and of the top cop right. Screen so to hear the sound of the hear that. Before I can see that he's disappeared off the screen. You can learn all of the different sounds on up there with the different fish. All right so you hopefully get the idea that there can be lots going on in the in the fish world and we need to be able to represent all of that complexity in the music world as well. Immediately we start to ask questions. Does it sound good and in this case I think it sounds OK it's not the most sophisticated music but it's all right. And then is it informative. If you were a school child or if you went to the exhibit and saw the world. You were in front of the exhibit and you heard this would you be able to tell me how many fish there were how many big fish I mean small fish and and probably not right. So we have this tension this balance between. What's not with attention is just two different constraints that we want to satisfy one is the informativeness and the other one is the aesthetics. So this one. I think is is OK on the aesthetics and not so great on the on the informativeness Now what can we do. Well one thing we can do is change the design of the music or the composition. Another thing we can do is change the listener. How do we change the listener but we train them to fight when I explain to you how to listen specifically for the for the high hat and map that onto the turtle. Hopefully you kind of got it a little bit and we could have done that same exercise where I would point out maybe by soloing it through the music have some little training session and maybe after ten or fifteen minutes. I would teach you what the the the the clownfish sounds like and teach you what the turtle sounds like and teach you what all these different fish. Sound like and you can pull it all together so training composition all of these things are parts of this the challenge. All right. We also want to look at different kinds of species partly because this was of interest to some of the team members Tucker and computing for example was not so into studying fish he was studying insects and and ants. So in a collaborative sense we have to go where the people are interested in going. And so we looked at these ant videos. They're interesting in a number of ways one way that they're particularly interesting is that the ants are all effectively the same they look the same we can't distinguish them visually and look. Unless you look very very closely and they have some markings on them. So that presents a challenge. We still in this particular case used one instrument per ant. But we also did things like tempo of the music is being driven by the speed of the ant So if the ant is going slowly is just hanging out in one spot there are relatively few notes bomb bomb bomb. But if that Ant really starts cooking across the inclosure bump of a bump bump bump bump we can increase the tempo of the music to convey the speed of the end. So we can do all kinds of things here. We have a left right up down kind of thing and another aspect that we were playing with in this particular set of videos was that we can have the exact same video drive completely different music and that's going to be necessary if if any of us adults went with a school group or a children to the aquarium. I might not want to hear the Disney themed music. I might prefer you know eighty's rock or more classical music but my child would be completely bored by that. So we need to accommodate different less. Nurse and see whether different genre of music can still be aesthetic and informative. So I will show you a couple of videos too to convey this. Let's go this way. OK here the ants soon another ant will join up with your new instrument is the answer for meat top or bottom left or right there. They're insist on playing there and they're all together on our way through and this is when goes with them. The much of the piano the guy with you know just hanging out in the middle. So there's not much activity from him. So there's a few notes. Whereas the guy behind not proven quite a bit in your idea perhaps with some training. There's this thread X. and again this this idea of does it sound pleasant but also that informative. Let's look at the the other version of this. Where you can have the exact same video the exact same experience in terms of the exhibit a different music structure to have on the back I joined us earlier that it was just the luck of the banks was the end of the hour. So you get the you get the idea that we can have the same video driving very different musical types. We're never going to lose sight of the fact that that just because we can design something that sounds OK doesn't mean it's actually going to do any good for anybody. Right. We always have to be at their own biggest skeptics and make sure that we're we're adding value in that we can evaluate these things but it early in the project we were mostly exploring the space and try to make sure that we could do something and show some diversity and and push the boundaries. All right so we did some other things in this case we took the ants and tried to look at sort of met a property so instead of focusing on one particular area people said I can't follow one ant. Well why don't we map the density of the ants on to some kind of overall factor and we could divide the the space up into quadrants and and for each ant we add. That answer to that quadrant. And when the answer in different parts of the the enclosure the music would be would reflect that. But when they all come together the music would reflect that as well. So again these are just pushing the boundaries and saying what we can do in these different spaces. So let me get us to that again this is just to convey that one little aspect you'll see that and hear that the answer quite diverse than the ads all come together and you'll detect a very distinct change in this in this music so also the three of them are contributing to the bottom right or the most spread out again in the music reflects that. So again we're just trying to different different approaches here and it's somewhat of an experiment at this stage. OK so now we get to the point where we were actually using live videos or videos from from the actual aquarium and we thought aren't we can continue exploring this music space and of course we did and we still are but we want to start adding other features so perhaps narration why not add some spoken commentary and here's just a very rudimentary example of how we can define a space within the within the exhibit and if a fish enters that space we can automatically play a pre recorded. Narration You could do all kinds of things you could have the the space in the middle of the tank and when the clownfish came in. You could say Hi I'm new. Well welcome to my aquarium. Right. You could do all kinds of things. So here's an example of one thing just to make the point. This is found in the depths of I mean that's that's just proving the point that we can we can do different things here. So again we're we're trading or looking at these two constraints one is it as Thetic and also as it informative clearly by adding these spoken narrations we're increasingly in information value and we have to see whether that's going to increase or decrease the overall is that experience. OK So there you have it. Our Then when we finally felt comfortable enough if we could convince the National Science Foundation that this is it worth playing with some more and exploring and so they gave them gave us some seed funding to set up a sixty five gallon of fish research facility a fish tank and we have cameras pointed at it and computers and it's the full pipeline of interesting you know study of the movements of the fish. You can see this aquarium if you're over in other technology Square Research Building. It's on the third floor window so that's why the elevators. So it's basically Game on. Now we can do all kinds of things and and roll in the computing group and the H.C.I. people in the music and and then also start to look at the human listeners and the docents and in the fall I get you right so here's an example of our on the screen grabs not as as nice as we'd hoped but you get the idea. You know we're going to try to. All right so you get the impression of true. We can track these things in real time we can make music and again all of the possible music composition spaces is available to us. To try to find things out. It's turned out that we have a lot of really interesting music being generated now from this research facility. So we can really push the boundaries and understand what works well is that if they what works. Functionally what works for both and some other things were developing now are really well I guess they're making us all believers. We can take this now to the Georgia Aquarium or to the Tennessee Aquarium and and show them and they get it and they see that this has the production they can have the production value of the quality the aesthetics and also the functionality to provide an enhancement to their existing exhibits. So a couple of months ago at the Georgia Aquarium is diversity days we did a demonstration where they gave us some video of this actual fish tank and if you've been to the aquarium This is the last last tank you see on your way out the door. It's it's in the entrance to the gift shop and you passed by it on your way out of the aquarium and so the question was you know how would this sound. We had the the live tank there and then right beside it. We had a a large flat panel monitor set up in the same orientation. Right. The audio and the video and people walking by were just amazed they first walked by and I'm like what's that just sounds like yet more of the mood music yet more of the music that they've heard throughout but when they stop and they read the signs and they talk to us and then they see well that fish made that sound they get it they see the correlations between the dynamic movement and behavior and the music they were really impressed both the staff and also the visitors that were there and the visitors in this demo period for days a lot of them were visitors with particular functional limitations so it was quite quite compelling. So let's look at some of these things. That's right here we go and all right so this one is the music that was generated by our computer program a computer program. So just like the house we have video of a particular fish many fish are the music out of exhibit what is this really emotional. When you watch it with this with a computer came up and goes on and on just for interest. We also had one of our composers. Create his version based on the same video. He said well if I were composing it and adding sounds in manually and writing the score and not just let it computer do what the algorithm kind of came up with accidentally or or serendipitously this is what he came up with so it can support in the same general there's some think it was related to the smaller amount of the microscopic but this is actually over four days. So you get the idea of what we're trying to go through here. All right. So a big part of this has to be and we're researchers and we we can't just to show and tell we have to be doing research and publishing and this is as not just you know. So we have to evaluate what we're doing and always as I said be skeptics and make sure that we're trying to make sure we add value and that we're making some advances and you know we're we're always doing some evaluation. We're doing evaluation with a variety of listeners sometimes if. People who can. Can you know see both see the video and hear but we're also working with low vision and completely blind individuals and many of our research projects and so we're getting them to listen to these things as well. And we want to assess again the functional but also the emotional and aesthetic aspects of what we're doing here and try to do so in an really replicable scientific ways that involved in purity of evaluation quantitative measures as well as people sort of qualitative. Assessments so evaluation is a big part of this. Right now I want to point out where we're at now we've got this pipeline we have this system and we're starting to build different versions of options and try things and evaluate them. A big thing that we're looking at now is what we're calling dose of studies. If you step back for just a moment and think the whole purpose of this is to take a visually interesting dynamic exhibit and interpret it. Communicate it to someone who can't see the visual experience. So I fume and does that all the time in terms of a docent or tour guide. They look at what's going on and they communicate it so we're looking at a dose astonished on how they do what they do. We're also looking at better music different creatures work. We were at the zoo the other day and it turns out that their mere cats may be may be a very nice exhibit for us to to work with. You know we can have other other environments you know sort of three dimensional spaces. We can have mirror cats we can have solar systems anything that is dynamic that is where the dynamics of the movements is actually the interesting stuff we can try to turn that into sound will be installing this in aquarium. Hopefully probably starting in in Tennessee and then the big thing is not just where are the fish at any given moment but what are they doing. Are they fighting. Are they. Mading are they feeding are they hiding. What are those are the interesting things that when we look at a an exhibit we see some movements and we try to infer behavior. Look at that sudden acceleration. Well that's because that other fish was nipping at its tail and we can do that and and experience that by seeing it can we convey the same kinds of information through sound. Well if we have the computer system performing that behavior recognition. Then by all means we could if we see some kind of flurry of activity that suggests one animal was nipping at another we could play a musical flourish. Or we could play a pre recorded animation to just see that one fish. You know nip at the other one and just sort of point these things out. So lots of things going on. We're also trying to fit this within more of a theoretical framework and expand and extend the understanding of how exhibits are designed and how people interact in exhibit so we want to improve the knowledge in that space and a really a big part of it comes down to how as an exhibit the information in exhibit directly impacting or influencing or being interpreted by a visitor here you see a gray line and a gray arrow from the exhibit to a visitor. And how does the process of mediating or facilitating or interpret in the exhibit happen. So if there is a docent that says look at the turtle the docent is seeing the turtle and then it's communicating that to the visitor or if there were a blind visitor and that person came with a friend. How is the friend viewing the. Bit and then interpret what's going on to the visitor. There's a lot of complicated things going on here. If you think about it. For example the. The docent or the facilitator or the tour guide or the friend has to have some mental model of what the the for example the blind child wants to know about and how that person like to hear about things. Do they want to know about NEEMO OK. Do they want to know the biological name do they want to know where that fish is from do they want to know what that fish would eat. What are the things that are interesting for that person then they have to inspect the exhibit they have to look at it. Find some salient activity or behavior or events and again we had to we started from this premise of what is salient then they have to interpret all of that and convey it to that to the blind child in a way that that particular person would understand. So it's a complicated process we can be inspired from from and we can learn a lot from just classroom settings that a lot of teaching is really knowing your audience and then conveying and interpret ing some content to them. The key about this. That's different is that it's happening with respect to something that's always changing. And so it's it's. It's a little bit of teaching and a little bit of a baseball commentary play by play. It's a little bit of all of these things rolled together and we're having to really pull some different but related topic areas together to to add some theoretical underpinnings here. All right so other future directions. You can imagine that if the fish are driving some music they're generating some music we could have humans interacting with that music and effectively interacting with the fish. So imagine you went to. As night at the aquarium and you had a quartet playing with the fish as it were. Now the fish are not going to be able to react to the players but the players can certainly react to the fish and if the fish do something unexpected then hopefully the human musicians would react to that in some way perhaps echoing that unexpected musical occurrence. So lots of possibilities here and this is where some of our colleagues like a Weinberg in music are already exploring the space with the robotic drummer and they have devices that interact with human players. So there's this human technology interaction already happening. So we're just taking it to sort of a natural extension here. OK so our latest compositions I'm going to play for you just the audio so that you can experience the sounds of things and really just share with you that we're continuing to be more and more sophisticated musically and I think we're finally getting there where we where our hopes for communicating what's happening in the tank is really straightforward if you watch the video and listen to the music you say yes definitely. I hear that music represents that fish. So they're both aesthetic and functional because of that now there's the music just listening but if you had earlier we heard the fish are so in your early. Going on a visual image of this and I'm happy to entertain any questions or indeed I'll summarize the questions are we adding a layer of storytelling and what we might consider narrative to this whole effort and and the short answer is yes by all means we totally understand that when people go to go to these exhibits. It's not just a second by second recounting of what's happening which is if you listen to a baseball game on the radio part of it is the play by play and tell me who's pitching and what what they're throwing was a curve ball or a strike and some of that is that it's necessary to to to get things along the. But the expert commentaries the commentators especially on the radio they add in so much other stuff right because they know that you can't see it. They know that their listeners are on the radio and they'll tell these anecdotes back when millions will amazed at this sort of back when Fulton County Stadium before this and they have these anecdotes. They also describe things in very colorful way as I grew up listening to Hockey Night in Canada on the radio and the same thing applied there. The dynamics of who had the puck and where they were going was was crucial. But it was a special sauce is the speak the extra narrative it wasn't just the deal for had the puck. It's that he was racing up to the side of the ice and his golden locks were flowing in the wind right him and everybody knew that he was skating quickly because that would cause his hair to fly back he didn't wear a helmet and this was part of the narrative part of the story we can also adjust the narrative for the listeners and you can imagine if we have an exhibit that involves clownfish and sharks. Disney is all over that and you can say there's an emo inverse the shark and and tell a story or tie to some other story that the students already know we can do that with spoken narrative we can also do that with music and with a combination of first Peter and the wolf is a classic example where you've got a symphony. That's telling part of the story you've got the music in the instruments representing different aspects of it and then you've got spoken narrative that ties the whole thing together by all means Questions with it. I'll just turn up the music and thank you get shot.