[00:00:05]
>> Thank you gold for the introduction hopefully you guys can hear me fine so thanks for inviting you know my name is action so I'm part of the competition team at Magic leap and today I'm hoping to cover a little bit about what we do as a company and also what we do in the competition team that magically and kind of talk about some of our challenges that we have in competition and one of the things that we're looking to address and and hopefully we'll get a sense of the the things that we're working on as well as part of this work so.

[00:00:39]
So 1st I'll start with what is magic wand the we are a starter based out of Plantation in Florida but. Me and for the who's sitting here both of us are from the front of an office and be a part of the competition and we released our 1st version of the product in August of last year that's the magic leap one so I talk a little bit about what magic one is and then go into what is the one of the competition aspects of magically and what's coming next and in terms of competition so the talk after that focuses more on the competition side.

[00:01:13]
The Magic think one is the device that I have writer's So if you guys are interested you can come by later and I can show you some of this as well so what it has it has it's the a full fledged device on its own so it has a display and all where reality displays so what is all going to be out if there's actually a way for you to interact with.

[00:01:33]
Your environment interact with. Things in your environment and work with your environment. So it only physical content on top of the virtual content of your physical space so it's different from of virtual reality headset now watch where they are they had that you don't see anything beyond a display in all where they are they are external reality headset you see beyond your display and you are able to interact with content interact with the world and the our virtual content can also interact with your physical objects so that's essentially what we.

[00:02:05]
As a company believes heads that we build the whole platform so we build right from the optics to the device to the hardware software and the whole competition stack and all this runs on this processing system that we have so a lot of challenges that you will see and I'll play with a little demo as I go to the dock is all these competition needs to run in a very.

[00:02:28]
Form like process so you're thinking about something like that so it's only just Samsung or Google phones or Apple phones and all the competition that's running on. You know our world reconstruction and all of these have to run in the in a small headset so the challenge is there are arguments because now you might have.

[00:02:49]
To look at all these great work that's been published but also comes with a huge amount of compute and it gives you the accuracy of the cost of compute and here you will see that all the processing so it's pretty much in this what we call is a light back and getting all the crossing done in this kind of form factor makes the whole problem more challenging so what we have here is essentially.

[00:03:12]
6 C.P.U. cores and this includes all the processing for competition and all the processing also for applications so we have a huge amount of applications on a magical device and all the applications all to run on this delay on this process so that makes it. Makes it interesting also from a competition standpoint and also how you get these applications running on our processing it.

[00:03:38]
We also have a G.P.U. And we also support Open G.L. E S 3 point one So a little bit about what we do so these are some examples of. So this is the kind of thing that you would experience as a magical boy so it's you see some amount of content you can play with their physical space you can create your videos extremist but you get a sense of the kind of work that we do here so it's it's.

[00:04:09]
You can place virtual objects on top of the physical world so here you see objects sitting on the chairs sitting on yourself assets and interacting with your physical space and you have these creatures also interacting with their water content and those are kind of applications you can build a better magically device so and we also have a control on that we have a magical device which also helps you have made to interact or provide input to your mind you can do it so so these are different and so this gives you an understanding of how competition is important because you need to be able to understand your world for the able to interact with it and these are some of the other examples as well so we have different kinds of games different kinds of.

[00:05:00]
Applications and we are building more and more applications that go along so think about. A doctor when you want to operate on a patient and you're able to see the overlay the X. ray of the patient overlaid on top of the patient then you can use that in your next N.E.R..

[00:05:16]
Operation procedure itself so those are going to take action that we want to foster and enable and that can be only possible if you are able to see a voice well content on top of a physical space and you need to be able to interact with both of these to enable these applications.

[00:05:32]
So what is special computing So like I mentioned to the difference between. What we call a spatial computing is that the we are here our field of view is occluded you don't see anything beyond eyeglasses and you essentially you base it it's gone to the top of the world but then there's no interaction so here we also add that layer of interaction so we can also interact with you it understands you as a person and additional content also creates interacts interacts with your environment a part of your physical space and that's a kind of a pretty or doing much.

[00:06:06]
So we're building a special computing platform that will enable different kinds of digital and physical things to coexist together in the physical space and I just want to give a bit of history of how we all started the company as such was started way back almost 78 years back now.

[00:06:24]
The 1st device is what we call it the beast that's what you see here so it's interesting to go back and see one of the kind of protests that we built before we actually came to the form factor so what you see here is the actual display that's kind of what is the in this and so one of the things the word augmented reality.

[00:06:43]
Is that you want to be able to make it very easy for you that you don't wander users to experience fatigue over a period of time and this display was the 1st display of the kind which will be able to show that you can watch a movie or watch some watch some characters in physical in virtual space without experiencing any discomfort and that was this big device that we built back in 2013 and then over the years we have transformed this device to what we have voted features and along the way we've built multiple prototypes protests which look more like a helmet which took it from the point of being completely non-portable to portable and that you can experience and build prototypes and applications but also test back to see what kind of accuracy do we need of the competition side what are the challenges that we need to solve on a competition side how are we going to address those challenges to make things better and to enable these kind of interactions and so this is the working prototype so this is around the time the Nigerian magically by John magically grown doesn't own than 15 and this is a kind of device I started working on when I joined magic So here you see the whole census suite sitting on top of here and the display and the person can actually use this to create content to play to kind of see what are the accuracy what is the need from competition side to solve problems and then now we are here to be released our 1st product in August of last year.

[00:08:11]
So the main challenge like I was mentioning earlier is that you want to be able to solve this what the cause of urgent accommodation conflict so you have and typically when you see you have your left eye and right eye trying to focus on this little object and there's the illusion of the object that you're creating in virtual space now both eyes are focused on this particular object but then you actually display is right here so you're this this place where the means are being overlooked so now this is what you're trying to the eyes are trying to accommodate and look at the scene but this is a red eye is actually focusing then this distance is what Because of organs accommodation conflict and this becomes more annoyed as depending upon the distance of the object and that makes it much more harder for competitions like magic it was one of the 1st devices that which where they recognize that this is an important console we have cameras looking at your eyes trying to track your eyes to determine what is needed the turn that you're being focused at and uses that to place augmentation at the right location and that helps make the device more comfortable and you're able to battle much more longer periods of time.

[00:09:27]
And so we have these 5 different things that we're focused on from magically. So I'll talk more about the contextual computing part and that's where all the computer vision aspects of magically come. So we're going to so these are different to build a device like this on an augmented reality space requires all these computer vision got them to work together so we have a head pose which is a flash solution so as I move around in physical space I want my content to stay in the same location so I need to be able to track my location and at the same time I want to be able to get a spectrum environment and that we have to keep place in the content in the exact same location so that it appears at the content is not and that's what has had both the arts LAMP stack and so then we have our eye gaze that we talked about so we have cameras looking into I dominating where to place the content to make it more user useful for you we have a control it's a matter of input to them a device so if you can think of a V. V. controller we have a control here and I can show later with a kind of a 2 input I've got some sort of life and a lot of challenges here in terms of how how do I do with a sense of fusion of going from Stu and what kind of fusion are going to apply need to implement on the controls to make it work for the and make it robust make it accurate and I want to be able to point with my controller at a particular location and I in the watch was space so in my accuracy her pointing to a large extent depends upon the accuracy of sense of mission.

[00:11:00]
And then I have hand gestures as a means of input to the device so we suffer different kinds of hand gestures so I want to be able to convince my hands and I want to be able to recognize a different key point on my hands so that I can create a different kind of fan to shoot but we also how wise as a matter of input and we have an app on a magically on a phone bad we can use it to provide input and we can also have well keyboard scans of inputs to the device so that into this we also build a reconstruction of the world so we can recognise that this is a flat surface of the floor and use that information to also interact with them moment.

[00:11:38]
So so these are the food mean competence that I talk about in this story so the 1st one this tracking localization and not so we build a map of the environment and we try to use that aspect of. The reconstruction of the void and track your ice and place content based on that outlook and also different kinds of changes and all these things need to compute that I mentioned so that all needs around in this form prosper and leaving all the remaining processing for all the applications as well so you have very limited compute and the challenge becomes how do I get these competition to work with very high accuracy.

[00:12:19]
At the lowest compute possible so this is something that is unique to applications because. Unlike virtual reality. You don't see a physical world all going to be out you see your physical world so if I have if I play the chess board on top of the stable. Does not stay on the table and it's one centimeter I will from the table I would notice it immediately because I go closer to it as good and I see that it's no longer sitting on top of it and if I'm playing a car racing game on this floor I don't want the car I want the car to go on a smooth surface so if there is a bump on this I want the car to be after the pump and all that in was very high accuracy and fragility of my motor construction by bit my planes are gone and all that needs to be captured and in the processing power that we have so that that becomes one of the mean challenges for a lot of work but doing it.

[00:13:11]
So so does starting 1st with lots of lot of people here probably you get a family with plan and what's not missed so it's a it's a combination of tracking mapping and look like fish so it's a simultaneous tracking and look as I shall call them that runs on our device.

[00:13:27]
So the maps the and violent and the track the. Track the device with it it's better than mine when the G.M. out so it's kind of a chicken in economy because you're you want to build a map and you also want tracking is a map of the same thing so you accuracy for tracking to a large extent depends on how well you're building them up and accuracy of a mapping depends on your tracking because that's how you're creating points that's how you're trying lately and creating points in your map so and map is important because you want to connect photographed and there's all these cases where there are cameras that are crowded and you lose track or in a very low light that you lose track then you are looking at the cover the articles and how do I but I Cor can come back and get back to my localized and then that is both systems that I talk about so I want to be able to enable applications that across as sessions across users and how do I enable these kind of applications so to give an example of not so this isn't to give you an example of how.

[00:14:29]
In the is from a camera look like and you want to be able to use these kind of camera I mean it's to build them up and pluck them out so here are users going out on the circular tables and we see visualization which captures how the user is moving around and you see the map points that are being built.

[00:14:47]
And the traction with respect to the snap. So the key to see is under Phillip the of this is very important so many combat the same location I want to be able to see the same content the same location and that also means that I need to have a good look closures I need to have a very good description of thems and a very tight fusion between.

[00:15:11]
Camera information and actual information to be able to achieve this kind of fertility and this kind of accuracy at the same time you want to do is a kind of keeping in mind memory and compute so you don't want to build a very huge map because you don't have so much space on your device and that also causes a lot of computer distractions so you want to have the straight off they all there's the state of between memory and compute and the I go at them like yes so that's kind of what you see also here so it's more or less bottom up and that's just last month that they use for tracking so we have tracking mapping and relocalization for 3 because the fundamental blocks so in tracking a lot of our tracking is based on fiction extraction and matching features across across multiple viewpoints so to build this infrastructure that we.

[00:16:09]
Really Have we provide doff those so you start from the small point you move the headset I don't want anyone to create us that stuff both of the X. has that with us but the world at all in sense of time and we also want to be gravity Ellen because you want to have all these environments to be aligned to gravity and behave with respect to avoid.

[00:16:32]
Same time you want to be able to track key points trap map points and fuse vision and initial information to be able to solve this problem. So while building this we look at several state of the art approaches you'll find approaches on both ends of the spectrum so that approaches.

[00:16:51]
If you just take freedom acting as an example. These are several some of the state of the art approaches that we have played and some of them are extremely high in terms of compute and for a very high accuracy. Very low income people don't also provide a lot accuracy so you want to find the gap in between value want to find a trade off between very high accuracy and really wide B.S. man matching features the same time you want to find.

[00:17:20]
Things that can actually fit in your computer that fit within your processing step so to build this we actually build our own proprietary feature detection out of them which of them which we build our own description and the disk has. Learned to support where the Wide Bay its lights and work within our hardware constrains our platform constraints to be able to be accelerated so that I can finish the whole disk the extraction in a few milliseconds so that's the kind of challenge that we're looking at so you want to have the end to end latency but then the time frame of your camera and I want to be able to build X. factor based on that.

[00:18:02]
Yes. So that's one that we published Yes So the point is I'll talk a bit of talk point as well so pointers so. That's also Yeah it's also appropriately descriptive. But. Yes it's published. To a point is one example of a script that we love working on which we're trying to balance the trade off between computer and accuracy and the disk that we finally use is also out of the such risk but the tradeoff between performance and I.Q. This.

[00:18:40]
And that is very important for a lot of things like if you take examples like. Over the years so any of the state of the art. The saddle of them that are Vatican in that equipments and if you look at the in the attic with. The muse as if it was good and they used 1000 people and a particular thing and that is just means the amount of compute that you need for doing the slum problem so you need to extract those and key points in a particular frame create those and descriptors create all these map points and try to get a map where that's the enormous amount of compute so large that we have looking here this how can we reduce the number of people it's going to do it with say $100.00 people and so even 50 points can be reduced the amount of noise while selecting the key points so that the end of the process becomes much more efficient so it's a lot of these.

[00:19:33]
The more you able doctor might on the front end of your system then it kind of goes towards the end there. So it's a friend then no system is really optimized and is able to work well but as many as little information as possible then you'll be able to make the whole thing work with asking people viewed as a possible.

[00:19:55]
So this is to give you an example of the kind of things that we optimize toward So we have. A matching as an example here you want to descriptor to be. 2 different kinds of noise so nice comes with lighting relations as you change the light in your environment you have a different.

[00:20:13]
Look different and you want to be able to find a descriptor that's invariant of these kind of patches you have different kinds of blurring and so if as the user is moving you'd be amazed how fast a human motion and human head can move so some of the experience that we did we found that human motion can even move close to fire degrees per 2nd so it can be extremely fast jerky motion that can happen just in a very short period of time and that causes extreme amount of blood on the canvas and you want to descriptors to be able to handle those things and how do I do matching across these kind of blurry motions and this also means that we need very tight fusion because the only way we can handle extremely fast motions with noisy images and bloody made this is true and how do we integrate I'm used to be able to solve this problem.

[00:21:04]
And then we also have hundreds of patients so we have taking the original back and rotating it by different kinds of. Different kinds of orientations if you want your descriptions to be. Resilient to these kind of changes so that I can do matching across multiple frames so a lot of this involves learning that I had this book to be able to handle these kinds of beauty and that becomes one of the core fundamental blocks in which I slap systems but.

[00:21:36]
So that's tracking at a high level on mapping a part of it this would be the way a lot on T.V. or for mapping So we have. We have winds coming from different frames from different viewpoints I've been trying to get them to create them up points and then create a sparse depth of the map point and then we do a map refinement So we as a user moves around we refine them are we keep updating the location of them out so that we have a good representation of the map and the photograph from it so that's that's the the good building blocks in which a lot of the localization and mapping of Gotham's work on and now you need to start supporting different kinds of use cases so what do we need to do with them to go very large scale What are the challenges there what do they need to do it for have to support cases like might be session he was so think offered that I want to play a game in my house and I use a device on particular Monday and I don't off my device come back the next week and I want all the pointing to stay back in the exact same location so during this time your environment might have changed significantly in this case in this example you just change the curtains and that's it so but then they're going to looks pretty much very different so you move on somebody's cell phone or to change the in the carpet but you are going to be a slob system essentially is built upon these speeches that's built upon localizing with respect to the map it's built upon creating a map of your environment and as an end user People typically expect that it's the same living room and I want all the content back on the content is not coming back so it's very hard to explain to our end users and customers that are in that environment has changed and that's why it's not coming back so from a competition problem how do we solve it how do we detect across sessions that I am still in the same location and bring back the content in the exact same places that it's supposed to.

[00:23:35]
And. In a way that's resilient to these kind of variations so what are the to make changes that I need to do before long term localization and long term so that's one of the challenges that we face and the something that we have working on right now and the other kind of use cases might be using use cases so say you and I are playing a game together if you want to play about your chess game we play it just because just go on to her table and we're moving just because you so I want to be able to.

[00:24:06]
Play I want to be able to see exactly what the other uses do it I want to be able to. Move moving a particular piece I want to see the same piece more on my side so how do I enable these kind of applications in the slums of them so that involve goes to things like collaborative Maberly blazing multiple uses in the same app and tracking them together and looking at in them together how to enable this kind of infrastructure.

[00:24:37]
So obviously some of these problems also involve using the cloud so not everything actually works only on the device. We rebuild on from one. Second it was a passel information to the cloud the cloud constantly keeps updating the maps merging the maps together so that the border you live in the same reference frame same corner for it and how do large scale distributed map with if I want to support a large number of users in this particular room like every single person here has a magical abilities so then how do I.

[00:25:07]
Do all of you collaborate together but it maps together but these are large scale problems that we're looking at and at the same time keep this passage of the map so that I can run things on device and localize and track things on device. So going beyond that to the way we expose this to end users and applications is what we got to what we call the most important trait and so what this does is it's an abstraction so we essentially expose the store developer and say that you are that's content to what we call as a piece here or process from corner trade and I've the platform level we will make sure that the post incarnate flame does not move and the developer can decide that this content of the P.C.'s and.

[00:25:57]
The platform takes care to ensure that the P.C. of space in the same location and across sessions across users and that we can share content share experiences. Across sessions and across the SO that is the P.C. of the origin they are saved with them up in the content of our action the P.C. That makes it very easy from what I love about spectrum we abstract the whole competition our way from the developers and that with all of us not only focusing on how do we build content and enable multiplayer games multiplayer experiences and the same time in the competition side we hire the how these P.C. of the managed.

[00:26:33]
To Because system it goes through a huge number of variations on the map internally for instance as they move I don't know what I would prefer that I have in the map gets kind of different paradigm so we did the P.C. of based on all the corrections we try to keep maintaining the P.C.'s basement that when we go in looks we do look closure there are a lot of changing events that happens all the pain and all the math changing events in turn.

[00:26:59]
In changing the location of P.C.'s and that's all taken care of a lot this is all a structure of a from developers and of a the developers can still use it to build their own applications Yeah. Yeah. So. This can be defined so we need to quantify what changes mean so a lot of work that we do with the bill very large scale data sets to quantify what change things so change could mean changes in lighting like I come into this room the room straight through her locks I turn off all the lights come through the locks and that's one kind of change.

[00:27:47]
And then there is very large environment changed but I move the one set of furniture out but there are many set up on a terrace in the same place and then that is also things like you couldn't change over because of one of the teams that I'd say in the cockpit so we quantify.

[00:28:04]
The performance of them across all these kinds of aviation so things like lighting radiation are things that can be handled by the desktop so you have a better disk of which is which is able to do all of these kind of lighting changes you'll be able to be looking at and get back information together so they do pretty well across lighting changes.

[00:28:22]
For environment chance like this some pieces of furniture moving it all depends on how much person days if you're scene has changed so if it's like 70 percent of scene change you can still manage a public 100 percent of scene change it's hard to manage if you're in a room but everything in the room and all the furniture has moved around that's going to harden hard to localize everything but as a computer vision developer that's what they expect from a competition engineer so they expect from the computer vision that this is my same living room so I expect all the content to come back.

[00:28:56]
To God less of what has changed in them and so that makes it harder and challenging competition site to see how do we manage the scale. In practice same on the same day so so we have so it depends on interaction with others so when we start I show some examples where.

[00:29:23]
It all depends on how. Object that I want to interact with and how I'm interacting of that object it just gives one example. Maybe a marking on different pieces of the object and that could be like if there is a canon or something I thought that's a different kind of back and so that's so we provide these of functionality.

[00:29:44]
And then the developer picks what kind of interaction model he wants to use for that particular object. So there are ways around meaning that you may have it depends on if you're interacting with the OR and so then we can detect with hands that the approaching are touching the object and build that as part of peons in terms of how.

[00:30:11]
The object needs to interact with both hands the same thing. In the. Yeah. Yeah. So one aspect of talk would be towards the end as if I'm able to understand my environment then I can even go one step further and say that this is the object. Which can move so fast it's going to move or your doors can move in a particular fashion so if I have a 2nd level of understanding about my environment moving beyond just sparse points that you help me British that gap to the next next.

[00:31:34]
So say that these are the kinds of things that I can expect out of my environment and these are things that I can which can move things cannot move and I can big that in my information into my. My own. I don't know that directly answer the question but that's the kind of approaches that.

[00:31:55]
Would get to get us to the next step. Think of the I think over the. If you're in a hotel with a huge bunch of rooms next to each and every room has a different kind of every room has a family or does it give you the same location but everything else is different from every other so what is the expectation that you have if you are if.

[00:33:01]
Expect the content to come back to the same location in the same space or Each room is different each one has a different source that it has a different bed do you expect that the content should be different. Yes yes yes and interactions with objects can also change with respect to that like if I place the chessboard on top of the stable right if the table moves I don't want to.

[00:33:36]
And I wonder to move with the table. But if my if I place a T.V. on right next to say the dog here and I want is adored moose I probably one of the beautiful state so it also depends a lot on how developers want to interact with the environment but as competition we provide the functionalities and lead them to speak what they want to do right so you have to have the ability to recognize these teenagers and bring back on it and and how you bring back is part of the competition problem and how developers use it and with respect to their environment is how they want to create their own experience if.

[00:34:48]
So that he doesn't that's one of the directions that would help solve our address some of these problems and also help to identify your better go and get a better understanding about the environment so eventually you want your the wires to understand your environment really well and interact with the environment and that's.

[00:35:09]
From a long term perspective that's kind of rare that each of these are doing will have to be. OK so some of the other unique problems that we have in our devices are our environments is that. Things like what we call this map selection the master election is for example I want to I'm in the same location I come back the same location multiple times and I go to a different part of my home so let's say I am using my devices in my room.

[00:35:42]
On day one I use it in my living room and they do I use it in the kitchen and data the I walk from my living room to kitchen so eventually at the end of the 3 what I expected I haue one unified map of the environment bitch combines living room and kitchen and they want to have only my living room and they do I want to make it so then.

[00:36:00]
3 but I'm trying to bring back content I need to pick which map Do I need to choose to localize into so that I can get the content back now that's something that is unique to us to do these kind of problems when I'm looking at multi session kind of use cases how do I find out which map is more interesting for me based on my current location and what I'm after I start using for my current experience so that's one example of what we call this mass collection so we had to select the map and identify which map to use and start using that map for.

[00:36:34]
Photos estimation and also for that storing all the content but that they're starting to see if this actually of a true that's going on and back so those estimations again but that's part of the map that I shoot and then I store all the content back so now I know the content getting back.

[00:36:53]
So it's all a. Session by session basis because there's no definition of physical space so you have it's all its parts but after the presentation of the world and the 1st session this map won a 2nd section of my book and that's where the map starts and ends.

[00:37:15]
Yes So that is the goal the goal is to be able to assimilate those information together and merge them in a way that I build that information and that's the challenging part so it gets challenging as the number of maps grow. As a number of users grow and the part of the competition problem of the challenge is to be able to get all this information together in a way that I can one representation or a good representation of the would.

[00:37:50]
Like to. Me. One. By one. Yeah. So the moment you start looking at a cloud there's also the compute and the cost associated with a lot of it and you want to minimize the amount of back and forth that goes on the device employed for multiple reasons one.

[00:38:35]
You want to use the cloud as much as possible and 2nd also. But then if I'm having an experience where you and I on the same location so. Yeah so either I have a framework where you when I. Go into the cloud and then you and I are sharing processing.

[00:38:54]
Part of the merging happens in your device and part of the merging happens on my device and then becomes much much more complicated all the other way to look at it as both not information goes to the cloud and then the cloud does all the process so this is a different kind of architectural decisions that we have to take to enable these applications and.

[00:39:12]
Experience to a large extent depends also on these kind of decisions. OK So then going forward to measuring some thing we talk about these so this is to enable other things lately so I want to please let's. Hear using my device and looking at the T.V. and a person comes and FRONT OF front of me on front of my magical device I really want the person to occlude the T.V. That's when I know that the physical can be that's when it makes a real estate sense that you are most well content is my physical contact that's your interaction with the virtual conduit and so you want to be able to constantly keep measuring the environment so that I can identify these kind of objects and.

[00:40:04]
Not. Content behind obstruction. And then the physics based interactions you want your lay and then like a video that is sure in the very beginning you want the cars to go on a surface you want your balls to bones out of the objects in the physical world we want them to behave like of physical objects in a physical one another and them and sensing So this is an example of our machine a lot of them in play as a user moves around the environment he can see that you're creating a dense 3 D. model of that moment and this play the model of then I'm going to then use for all these new ones too so these are the of Lucian's and as a user goes around is the all locations getting filled up and and you create these kind of meshes of the world which also helps you to simulate your physics.

[00:40:59]
And then from Move On this can they go one step further and start looking at building planes you create the. One to how to make sure you build a plane that on these measures and that gives you the plane at a presentation that would and the different kinds of applications use different kinds of things or you want the other was to be I can rise to the walls and things the bones of a ball was and there floors and ceilings and so on so I talked about where in the conversation conflict I will talk about this video so this is an example of our.

[00:41:38]
Images coming from March I can us so it's constant looking at your eyes and determining. Trying to triangulate the Isle of action so that I can place content in that location that helps you avoid the avoidance accommodation conflict and this something that you need to magically devise because this is how we can ensure that the user does not get fatigued reading the device for a prolonged period of time.

[00:42:11]
So this is an example of how we use it in practice one way is to create applications and then they're in the night location and the other is also we have this application on a device called the chat it's a social application so you can eating and avatars and you want to interact with there so it's like they me and.

[00:42:33]
Me and my friend trying to play this game and you can see that the eyes of the other 30 min date what your friend is doing in the other side of the world so that's essentially based on how your friend playing and what the friend is looking at all that gets them later in the towers so that you're going to get a more realistic view of the industry experience a realistic social experience interacting with the delegates So moving onto hand tracking so because the board different kinds of hand gestures that's part of that some of the way to import content Seleka things interact with your physical.

[00:43:15]
Objects so things like open hand close hand OK science so these are kind of things that we support we have a bore key point where it's kind of a key point tracking for the identify keep points in your hands and then track those key points and these key points are useful creating different kinds of gestures and we classify gestures into different kinds of things so these are the examples of the things that we want to classify it will be more clear they go through a particular video so this is an example where I want to do a bunch of hyper.

[00:43:42]
The 2 people can be in literally in the opposite on the word and I tracked. Towards a magical device the book is that's where track those hands so that I can create the virtual sense of giving a hi fi in the world. So so that's kind of the.

[00:44:03]
Give you a sense of the kind of challenges that we have for a competition standpoint working in an almost reality space in a company like magic So in terms of future these look at the problems that we're looking at one as what we call a human centered. Video.

[00:44:34]
OK I'm not able to get this video play but the idea here is that I want. Like a personal assistant. So you're used to your C.V. your Google Google answering your questions so you want to post an assistant to do more and it can be a physical person in space who's standing next to you and helping you in your daily tasks so I think of a physical like a secretary and that person has to stand next to you and doing everything that you're saying so so the next thing that we also want to do is we want the person to imitate what a secretary would do on a person who would like you want this person to have the same kind of reactions the same same kind of facial expressions.

[00:45:22]
So that it becomes more realistic so you don't want to robot person standing next to you and expressing all these views so you want the person to actually express things the way your friend would express things so if you or if you're sad I want the person to be sad I want if the person is saying something happy I want this person to be happy and that makes it more realistic and it makes it more.

[00:45:43]
Natural indicator of again moment so for this we build our own platform to. Enable different kinds of facial expressions. Different kinds of facial expressions so that we can render them in real time and the user can then use this for. Me to be more much more realistic. And there are many examples that are working on so we talked I talked briefly about seein the standing and object nation so we want to be able to recognize objects in your environment and so that the go to the next level of understanding from all going to the space so that part of part of it includes identifying what I chairs books or different angle of different elements and how you interact with these objects so that also gives me an interaction model so there's a way that you interact with your chair or the way that you interact with your book and those things become interesting.

[00:46:39]
Both in my interaction perspective from and a little perspective also a competition perspective because you won't want you know that this is how we attack in fact just cannot float and made it so those things become very interesting. As we start integrating them into our competition stuff. And then there are things like object nation tracking you want to be able to recognize and track objects if I have this bottle I want to be able to crack the water at the bottom was alone in my environment and maybe I want the content out of this bottle and I want Bill Clinton to move on and I'm going to move on so that that's again a different way of looking at the whole problem and interacting with the environment and then that adds a lot of times there are multiple uses that I've been production about.

[00:47:19]
And and some of the work that we're doing is the only classification in the 3 dimensional space and some of it is more in the 2 dimensions which we want to be able to recognize and create semantics then went patients appointments and treaties because like in this example and also create and recognize objects and place objects and duties so think of an application led.

[00:47:43]
And the user is trying to repair something on his in a factory setting. So I wonder that. There's an instruction manual that tells me OK move this piece of object from here to here so I want my device to recognize a piece of object and say and as I move in that object class or object and place it on the back of patients so you want those kind of interaction with.

[00:48:07]
The space and there's a lot of these kind of applications a lot of the research topics that we are working actively so. To appoint is one such example so we so here we have an example of our own. Proprietary. Them for for. Trying to meet the computer. So the challenge is like mentioning a lot of time this is also what is the I want of things that I can do on a device and a way that I can enable different kinds of these applications I think that's that's kind of why we are.

[00:48:48]
Let me know if you have any questions thank you so much. Is that so that is the way we want to approach it so that's so that's the kind of things that we can incorporate into our modeling but that requires understanding of the environment and how the objects move and what kind of movement.

[00:49:36]
So that's the kind of learning that we need to do for enabling these kind of. Computing. So how do you kind of could put the. Right no no but. I mean in future it depends on how you want to go with that. Question. One more question Rod right share.

[00:50:28]
For $23.00 points on the. Old way. So I think that would eventually happen there are some stand on abortion that are coming up right now in their state same to unify the supercenter I think or if we're up there I think go over the next 50 as they would see that unification I think happening.

[00:51:08]
So I mean you are working in your environment and there are buttons working and has an hour late so it could it could be that you are in one corner of the world another person is working on the other kind of the land each will each one each device build it on map of the our own environment and then it's only about.

[00:51:26]
Representing the unification of the space so that you can bring those interactions together and so it's not so that that becomes a way that value plays out there in the our location and weather but the base there at that location. OK thank you so much thank.