[00:00:05] >> I'm going to introduce the session which is focused on cost of transport and debate centered around whether this is the correct metric for mobile systems and this will all features go I'm I married young for those who don't know me I'm a professor assistant professor of mechanical engineering at Georgia attack I run a class and I'll be the moderator for the session so they'll be. [00:00:33] Specific questions that will be asking all the panel s. at the start to kind of positions and allow for some of the debate happen and I'll also be monitoring the the chat and so a little bit of. Background there we are going to take some audience question or we hope that you guys will provide some questions for us to get to the panel as we do that through you in a feature don't do it through the that chat so you see the q. and a that's where we'll be monitoring for questions please put questions and throughout we'll be asking those later on and if you really like questions please vote that will do it and the ordering of how popular some of the questions are hopefully we'll get to all of them but it's all right so for the our other panelists introductions we've already seen on Mazumdar are also from Georgia Tech and and me and then next is Greg which he another one of the Georgia Tech professors runs a power lab here and sort of facts. [00:01:41] And South who the newer faster with us joining the year and a half ago. And me and lastly we've got a guest Hannah left from the University of Michigan route I think many of you know and run by fog there and also joining us for today's hot on cost transport Ok so let's. [00:02:12] I'm going to provide just a brief background to just kind of get get the idea started here so energetic Costin's humans were going to touch on this and more than just humans 1st humans when were we talk about energy costs largely the predominant measure of this is through indirect calorimetry And so this is essentially measuring the oxygen consumption and the carbon dioxide out. [00:02:38] Of the user and we can do these through these 2 systems you can see why and our lab here where basically we're measure the breathing rate and we can use the basic chemistry of converting that your oxygen consumption to basically how many calories you burn and that allows us to calculate how many watts somebody reading and to calculate cost of transport we simply divide by the distance travel and so this is a very standardized rate that a lot of labs use to measure performance and humans as well as humans augmented with exoskeletons or or other power devices prosthetics. [00:03:27] And other interventions that you and this is a this is becoming pretty standardized. Term and what the exciting if you had a look over here on the right graph this is from a paper that Greg and I recently published. You can see what people are complex with those gals and technologies and and not able bodied humans and what's pretty cool is that so what we're showing here is devices that have sort of 1st some sort where our devices actually be. [00:04:01] Human not augmented with anything and that that was a pretty big accomplishment to feel essentially wearing a device gives you a larger idea in terms of energy costs compared to not wearing the device so humans are very efficient and they're walking and usually out of mass as a day penalty but recently I've gotten to have broken the metabolic prosperity or at allowed humans to be more efficient. [00:04:27] They're not using any device that relates her 1st in 2013 in the Malcolm study with an angle exoskeleton whether it's how their device which means it was not autonomy or you know that. The power sort of thing and this was all external but then a year later in the movie Paper autonomy the the. [00:04:51] The the no after condition significantly which which was pretty exciting and Elliot Roth who's one of the panel s. is one of the co-authors on that paper and then in the very next year again one of our panel asked and that's. Not easy with the columns of the 1st author and Greg as the great the way he is the senior author managed to do this with a completely passive axis called the no no after all motor and so those are some of the big and Vishal advances all with ankle devices and then you can see as it goes I'm smother 1st like I have managed to do it and then the knee and then even a combined device that's out of her while she's loud walking and running and so as you can see the field really made big advancements been able to be. [00:05:43] The metabolic barrier and you can see that also and and and this graph is this kind of a graph of years versus metabolic change compared to not wearing an exoskeleton So essentially devices are getting more and more efficient at beating the metabolic barrier over time and what we've seen at that early on almost all of these were happening at the end all many of these kind of shifted to other joint special to have. [00:06:12] And larger and larger metabolic reductions compared to no you know unassisted human has that have been made and so Scott and technology and if you look at some of the conference proceedings from the year this was through the end of 2019 it looks like labs are set to be some of these by potentially wide margins in the upcoming years that's pretty exciting so the field is making a ton of progress and terms of this measure and what I at least want to convince everyone up to in this hot at that this is the gold standard measure is what is being used to compare devices across the feel and lots I think. [00:06:55] There were hundreds of studies that we went through and which method bought for a quarter these are just the ones that the the no actual condition significantly and so this is the Rosander measure and we really want to debate today as is continuing you know progress on metabolic cost and beating. [00:07:14] Sort of eating that the no actual condition or improving and energetic the number one goal all of these wearable devices moving forward on the same line we're going to look also at this for multiple sets this is a great graph produced by by song based lab. And what you see here is a comparison of some of the basic this is a little bit dated but some of the basic robotic walking. [00:07:42] You know by. For quadrupeds how they compare to various animals and what you see on the x. axis is a lot of body mass. Increase mass. You get a reduction in log. Are and generally swimmers what kind of following this green line wires are less efficient and following this green line in terms of comparisons their mass and runners. [00:08:09] Attract all runners mice humans which is right here following this higher lie and you can kind of see how some of the different robotics the end of the Aussie involving dog very much less official made highlighting of course as is Cheetah robot being more efficient even in the basically and I thought that I had were out this is a real key measure bat that that was featured as one of the main competition at the DARPA revives challenge and very interestingly to see some of them and maybe adults are worse on this because the extra math but things like I think her and bicyclist are superficial compared to math and I'd be something we discuss a little bit later but I think this kind of you know that's an intro this is a dominant measure in the field and I can get across that. [00:09:03] Looking at the cost for autonomy systems for humans that has developed Labs has devoted huge amount of time a huge amount of restarts as many things are measured off of this process and I think that there are significant concerns in the field that this is. Perhaps targeting this too much but we can also debate that maybe this is filter craft measurement and we should continue to make law smart gains on energy costs are right with that I'm going to start the past all the discussion. [00:09:38] And I will start with a question for you each of our panelists and. Greg I'm going to call you out as being the 1st well as new I think you know probably the most knowledgeable about physiological cost of energy a cell or child cost a good gold standard for the field where ball a mobile OS x. just go ahead and give you know what I want to see at the start of your general position and tripped up on the topic just to kind of missed out much where everybody's. [00:10:13] Ok thanks for the really good intro. I mean I'm a little bit biased I'm going to try to you know I'll give you my opinion and then also I'll kind of list some pros and cons moving forward that I think that our lab thinks about when it comes to cost transport so I think on the plus side it's a very simple measurement we know how to do it it can be standardized across labs like Erin mentioned. [00:10:38] If you can measure oxygen consumption or some proxy for the input effort that the person is using to move in the world then you can you can drive something like a cost of transport which is technically defined as the energy needed to move a unit distance. And you know again like it's it's it's simplicity I think is the simplest argument for it. [00:11:02] That it generalizes it's easier to use if it's a standardized measurement that we can all kind of share shares a common currency. It's also relatable in terms of our dissemination of our work to the general public which I really like still like when I give a talk I always use the analogy of a car and gas mileage that also makes it relatable to people who were trying to turn onto or research or. [00:11:30] Make them understand things counted in your elevator pitch or layman's terms about why it works or not for those reasons I think it's good I'll also mention very quickly that you know a lot of the work that we try to do with wearable robotics as a clinical that we're interested it proven people's quality of life and again the simplicity of the measurement and the ability to track it sort of outside the laboratory setting makes it appealing. [00:11:58] So those are all the arguments for right that's a tangible thing to work towards simplicity generalizes. Just to kind of paint a few cons that we think about it doesn't consider time so a clinical example is a good one here if I'm a person who's a stroke survivor and I need to like get across the street before I get run over by traffic. [00:12:23] Having a low cost of transport might not be the best thing for me I might need a high top speed and cost of transport but by definition does not consider time actually eliminates. And so if you take the extreme position to minimize costs of transport you may end up with very massive things that move slowly but are economical of energy and that's a lot of times not what the goal is for wearable robotics so that's one thing is that it doesn't it doesn't normally consider time. [00:12:58] The other thing the other big one the elephant in the room is always like. Who really cares about it like if I'm out again like is this a priority of a person when they move in the world are they thinking about the energy they consume or are they thinking about not falling over for example or being able to manipulate objects while moving like what is the cognitive load of a task. [00:13:22] Or so in general our lab is shifting away from considering energy costs as the only factor that we care about when we design wearable robots and thinking more about sort of. A suite of things that people may be trading off in terms of their priority per task or post per contact and those include things like stability and comfort as an especially important one and also like measurements that have to do more with perception of effort rather than straight up measurements of effort I think are becoming more and more so I'll leave it there hopefully that starts off all right thanks Greg what's going. [00:14:05] On the mobile side so I think. The next round. That the 1st thing I'll just say is to give some reef history of why which is it's not it's a very good job but I just want to hammer it out to transports actually measured by looking at how much energy used outlets certain amount of distance and then in the denominator you also have g. which is the gravitational constant and the mass of trying to normalize time so you can think of it as a unit was metric in some ways it's sort of like the coefficient of friction or moving and you could ask why we're using this why don't we use the robot and now to estimate efficiency that that you know you guys learn and you transfer or whatever and the reason for that is when you're a mobile system and you're moving out of grout you're not actually using the output right so to look at how much output working produce you're not going to get and so they have to invent a metric to actually use it NASA look much worse Aaron give credit to our sun base for that graph a lot of those a lot of the spots on the back are actually from past tuckered study animals and that's fine and so I just want to give you history of the metric I think that it's plains it's an eyesore engineering metric it's you know these are all nice things and I want to start with a little more controversial question if you know if you're in a fight right you care about your survival and you want to survive right which is what we should care about animals care about who would you pick to be next year and I would say that hasa transport would tell you to pick like Americano right some guy whose life I hope for really skinny got past that or your midst and you would pick back but I don't think that that big like a Bron James right and so if you care about survival is positrons sport really the best metric you're thinking people that are already in systems that may not be optimized for metrics that actually very real so great I think alluded to this but a lot of pieces for robots right especially in diverse environment things like agility survive not great right not to be hit by a car and I think cost of transport starts. [00:16:08] When we start use it when we start thinking about those so I just want to throw that out there not to speak for too long about it as an obvious thoughts on this religion my own work maybe that's the food for discussion. Thank you I will go back to. [00:16:25] The next. Remember question as every other cost a good old standard of. Iron say like a kind of hose or as a grown standard for a couple of reasons the kind of the 1st reason is that like I'm not sure a gold standard appropriate that I like the lead main issue running out with metabolic rate in the field of where we're more robotics is that it's really like a hyper focus on metabolism and Casa transport going like one of the only metrics that we use to judge the defendant and that I find problematic x. is going to assist with all aspects or many aspects of mobility and this is one of the leading to sort of cracks that I think we needed sort of a diverse set of metrics on which to judge actually Scout and I think metabolism sort of rose as a gold standard one because it's measurable and we're engineers and want to measure on a fire or systems or to also because excess about sort of makes sense that a system a system you were improve your efficiency and so because of those 2 aspects I feel like we kind of latched onto that and in this talk I kind of want out of this and I want to do it like go over some evidence that maybe it's not the best gold standard metric and then propose a potential new metric it's kind of out there as a topic of discussion and I have a few like slots and kind of points I want to make which maybe I'll make later in the discussion after John or initial. [00:18:01] Great things are yet Ok. With you. All right and I ask I think I'm going to comment a little more from the about excited and I think that reading as previous panelists starting to play their prose and the cones are like many different perspectives. And I'm going to talk a little bit of some thoughts from our groups and maybe my personal thoughts to any definitely this energy efficiency as it's been is that is a well received a metric of one of the like the system are there like many other metrics to quantify the locomotion performance so basically for I think look emotion work. [00:18:49] If you see like if you see the. Baby not only the mechanic with a design but also the controller implementation you in the higher level task in the motion plan or design many times we saw that of addiction problem so in this case if we want to generate stuff like optimal motions. [00:19:10] Like the system of video x.-o. system I think this can show afeard of the Activation afeard maybe always the 1st to feel terms going to come to in your mad when you design this class of options. So as a person if you are one direction and I think people have been pushing allowed this they say is really what is the past way to quantify and benchmark the energy efficiency and in what kind of context I think some are some penalties are they mention this I think one thing is basically the task versatility So how this energy efficiency can be evaluated maybe across different look motion tires different or global hardware and definitely there's a tradeoff between the task of us or to the end the energy efficiency. [00:20:03] I just feel like many times if we see a lag with robots it's maybe designed optimally for once that's a good behavior but it's maybe not necessarily the best option for others too in this case. I think how we can be dying goods I think it will pass like the locomotion system for there were some systems that the police reported. [00:20:31] And the definition of task at their worst they could be very proud and multilevel like a locomotion at different speeds different patterns and I could've had to have different working gates. And many different ways to interact with the environment so I think there are a lot of things we can comment. [00:20:49] On or I think the energy efficiency optimizing it is important but we should be clear about the context and what assumptions and Anglos we talk about this energy term Ok great things. You know I like the general and Prof You know I think one thing that I think about as. [00:21:17] A reference is from a narrow control argument. Etter g. cough is something that all biological systems do often by which I think it's really interesting to compare and think about what I'll use or. Something that gets optimized but I think most of the studies from the literature will say that it does sort of a last priority like I'll lot of other things for example if you kind of look at. [00:21:50] Can about x. kinetics the m g a and energy cost the thing that is lowest to converge and basically your energy costs and I think part of the reason is the body is optimizing other things like you know probably probably something like stability and sharing that you can do you know normalize walking again and you can achieve your calls it sort of more important than that metabolic and. [00:22:20] Philosophical perspective it seems strange that the field would think that metabolic the number one priority when biology doesn't think number one priority is that my thought is that really shouldn't we ensure our. Gate is stable I think if you put in clinical populations a lot of patients would agree with that which is they want to be able to go someplace and not fall out and not necessarily do at the minimal energy expenditure and so I devise designer control designers in many ways I think that there's an argument that there's probably other variables that people care about and that biology terrible lot more about that we would want to optimize before we consider energy costs in the equation are and human in the loop off of either so that that's one point I want to bring forward. [00:23:15] You know I do think on the sort of on the pros for what are the things that maybe wasn't mentioned that this measurement is super hard to cheat in terms of in terms of local sort of cheating well I'm good I disagree with that I don't. Suppose it is a big thing that task it is very hard to achieve and then you're saying that you're locking at speed and that's actually one of the big that it's agreed there is that this advantages of it is essentially you can only measure at during a 6 that was. [00:23:49] So that's a huge disadvantage but within that within that path it is very very hard to cheat Luckily I did and you could make a bunch of different compensation and other measurements like our joint parts or things like that you know can easily be sort of whitewashed under out by you know but metal lot can now if you allow the past to have really change this metric become very easy and all that id and get from there as to why he doesn't believe it's a good measurement and heavens of not being cheated Well I think yeah thank you for that and I encourage you guys interact and do some more belligerently going out there now and I think that that opening statement so feel free to go but I want to 1st address this this will help me address the question that the panelists I voted on Proton are the audiences of photons per job you take a look at the audience q. and any other Shawn Wilson asked a question he said how do the panel think of tasks in purpose to be incorporated into an energy cost metric for robot systems or. [00:24:57] I think this relates to something that I wanted to talk about and that's insurance so we keep on talking about energy efficiency right but let's talk about the metric that may actually matter to people right Gregg talked about you know gas mileage mileage in the current How many miles you can get and maybe offer down as an insurance but Miles as your vehicle can fall is probably more important than mpg and that's that's just in durance And so for example the 25th and our provide a challenge Aaron and that had been in durance town so they actually asked us if you go out far and you go on a single battery charge because that's important read the Iraq what other reason 5 people or whatever ready to travel on distance. [00:25:40] Response or whatever and that they say you're here he was point 2 But you didn't get me are replies that the it so and so that leads to some interesting trade off and one of those is how much fuel do you carry out because your bag because guess what if you carry a big **** battery which you know a lot of robot need those years you know if you've got to go up some people make the go go down because you're putting it in the denominator with math but it's not going to go up because you've got to bear all the loads your torso up your car yet your love to go out and I think that's a great illustration of even an energetic connection I've been Durance right which we actually care about Asa tramp if you try and offer my the cost to transport you to the small battery right and you travel 10 feet but you did it as an amazingly high energy efficient but you did achieve your mission right and so I think also that aftershock question I think I agree I think we're trying to go with more paths I think Eliot has and the box office I'm going to end it over then but I do think you know even pasta transport and games as well. [00:26:44] I'll take the opportunity of that Ali and then you did just add on to the an example of cheating though Aaron you kind of mentioned that cost the transporters cheat for a year and I think that an example from wearable robotics is are these tethered systems where people offload the might of the motors and the mass of the structure and and they don't always report they don't know as acknowledge that when they're reporting the cost of transport performance of the user who's not carrying the whole device though there are there are ways to cheat this I think one way to avoid cheating is to always compare against you know normal walking and and then also include a condition and be transparent of the device without providing say torque assistance or energy input but just the load itself of the device should always be accounted for I just wanted to mention that Elliott might be back on line now looks like he's on you to. [00:27:45] Turn right when I was going to do is to go through a couple of pieces of evidence that in my mind sure that's not a good gold standard measure for my peers to watch I will show I'm up here and you can sure screen. For the 1st piece of evidence for we're just going. [00:28:03] Never balik after wrongs are not consistent with what people prefer So this is the study that we ran recently uses a variable statistical prosthesis which is not shown here on the screen and the only thing that you really need to know is that it has kind of one definable and controllable or ammeter that's the stiffness of the ankle choice that someone is wearing for their prosthesis or you skip to the next slide area. [00:28:32] If we look at them from earlier talk about a plot to the right so when a study we measure people's preferred deafness we gave them a dialog select the stiffness that they find and the best for that and then we varied it up by past minus 15 percent impossible as 30 percent and we measured a host of different energetic and biomechanical factors that may change alongside changes the step ness but we shall if you look at the plot of the left this is not about us and the axis there is Stephanus or deviation from preferred you see there is really the color here denotes walking speed here there's no relationship really between deafness that people have a preference for and the resulting without. [00:29:18] That's kind of like one point I would say deafness is. Metabolic Rate will not consistent with what people who appear to optimize and then if you look all the way to the right lot this is an example of what maybe people are optimizing which is just to say kinematic symmetry is kind of the easiest way or when that plot that we show that metabolic rate affected by what means not by people what people refer you know there are things that are affected by what people are for that's kind of like one point I wanted to make a nobody wants to respond to that but the idea there is that the tag was not consistent with what we were. [00:30:04] If there's no response we will talk about the other points from the other piece of evidence I wanted to discuss and this is were read by live McGraw now in my research group and what we're quantifying is people's ability to perceive perceive their own changes in their own metabolic rate though in this study we let people walk with an Access Calton we vary the control parameters for that action skeleton how much assistance a provides and we impose metabolic changes to them via this controller and we let them walk for 2 minutes one setting and then walk for 2 minutes at another setting and then we ask is your metabolic rate or down and we do this many times in a row and what we can get is that the concept of someone's just moved to a difference which is the change in metabolic rate that they would perceive with 75 percent accuracy and what we're showing I have white one suggested and I can talk about specifically we showed a j n d and just noticeable difference of 25 percent but I mean unless you see in that metabolic change by 25 percent or more people won't really notice that. [00:31:12] To be fair the standard deviation on that measurement was 23 percent. Mostly due to the fact that we couldn't complete that study from the Congress shut down we've continued to collect data from whence we reengage with research we're getting similar numbers between 30 and 15 percent changes are required people to perceive. [00:31:35] Any response. I'd never say that. I'll actually I'll play devil's advocate just a little bit I think on that last point. The just noticed all difference. Like I would say who really cares if people notice they may not know what's good for them. Yeah that is a great that's a great point. [00:32:02] Here's my response to that the reason why I think this matters is because people's ability to perceive the changes that occur directly affects their decision making. We want somebody to actually wearing exoskeleton they have to have an internal sense of it's about you and I would say like perception of metabolism and that's been going to be part of that internal sense of value which maybe or maybe it is maybe it isn't so much kind of like the key takeaway here I think is that the reason why this matters because it involved in a q. decision making do I want to where do I want to pay this much money from the system. [00:32:40] I really like placing it in the context of the you like you have where the per the user's perception then becomes front and center like you say like is this thing valuable to me or my just going to put it in a closet for. Folks on the college design wearables for sure. [00:32:57] But you know it's just not up notable just noticeable perception for an autonomous robot is probably kind of completely irrelevant. Or is it I mean is there or is there some analogy to the to the autonomous humanoid robots in terms of its ability to sense the resolution over which a machine can sense its own state that is important here to what do you think about that. [00:33:25] Or anyone on the panel I guess I see a kind of nodding a little bit like it seems like you know the ability of a robot to sense that does is an important design factor if we're going to keep it from falling over etc but whether the robot has a brain or cognition and knows how well it's sensing. [00:33:47] Is there really an allergy to that's notable noticeable just distant just noticeable difference in the autonomous robot world is what I'm getting at. The I think I can just a credit card for little bits and I think sensing and the perception I think I mean of the environment is definitely very key capability for me for both sexes verbal and tournaments like robots. [00:34:15] So and I think this may be related to I think one metric I use as energy costs and every official see may be related to my robustness So this can of when this like a very complicated system deployed to the real word how we can predict the end gnomes in the environment and also maybe responding to these unknowns and maybe reject external disturbances I think it's all part of the capabilities. [00:34:48] And I mean this kind of like a gnome send the pretty Beijing's I think can either from this environment or maybe humiliate self or maybe the robot system modeling itself so I think robustness is definitely important metric we should take into account I mean if it has been widely exported in that in many report extended I think from a control matter error human like human Bell mechanics people study how to respond external disturbance repast early on the keep Allen sing and even likely in I think a measure of the state machine learning and the ai planning a lot of people talk about the concept like a safety cry can a human like verify or abilities I mean I think they are all related to. [00:35:34] I guess what what to what level we can quantify the as certain days and how to design go past the controllers all planners to reject the disturbance that impose internally and externally so for like the system I mean basically like how to keep balance a war the focus and the navigates like a very complex environments and awarding credence with an occult circles I think these are key questions we need to address. [00:36:06] I mean definitely not a very relevant but I think just a person of color have is basically. We should read about the robustness not only from the low level control I think there are a lot of work in this area but also robustness from the higher level from the task level from the decision making level think about if there is a failure occurred at a lower level. [00:36:31] For example character robot staff song and maybe an expected to most of the terrain how can we do than a high level decision maker in the sand maybe a new command to the robots to achieve a robust disturbance rejection. So or I feel like I'm proposing in more like a coherent or maybe scalable planning control framework with robustness reasons at. [00:36:56] Multiple errors I think it's too soon either a lot of. Investigation. I just feel the community doesn't have like a stance solution yet or at least the consensus is I think many words are scattered to several different a difference so I think with plenty of them all unified solution on this robustness. [00:37:21] Ok so I want to I want to actually go there we've got to get to a question there's a class I know I'm getting I didn't even. Go for All right the most popular question we want to you know and this was this was related to what I question but I asked because I think we can talk wasn't part of the 2nd part of variable that the past all agree all right is this and you know and so you know part of the I don't just cost what you know I don't know if we know they're all going to come to agreement and that's why. [00:37:54] Why why would people propose is what we should be using to benchmark progress what would be a potential alternative gold standard that people might be able to go tell what you think it should be dealt with what's you know from some people on that I'm going to give a shout out and your use if you want to go 1st Jerry I can go I'm just going to I want to talk to that last line there and you can throw them out. [00:38:24] This is very like kind of a new idea a way to establish a kind of measurement of success for Actually cell phone use and it's going to be it's really different than things going to been used in the past so the idea is to obtain this plot that you see here which is something like you know dollars per minute demanded so they're like cost on top of monetary cost as a function of time where we test both kind of when people are unassisted and then people would be wearing an extra skeleton and basically like what we're trying to show is as time goes on the cost to keep walking so I always think a step back every 2 minutes in this test you would be asked how much do we have to pay you to walk for another 2 minutes. [00:39:15] And you would be competing with some people virtually that that I won't get into for the sake of something called a victory auction which is how this type of test is done that you would they would you would state in amount to keep walking for another 2 minutes and you walking on a treadmill potentially up an incline so it's going to be difficult and then as time goes on you're going to want to do it last and last you'll get that you begin your have to be paid more and more to people walking around the idea is that the test conditions actually help in the system and actions cause an honest instead and what we get is a dollar value that is converted to how much the act itself and its growth to you know during walking or to bend and it's kind of a it's a kind of a different way of looking at the problem this is the type of method used in. [00:40:04] You know the social science and I think a kind of gets a more present like accessible metrics that converts the things that people really care about like passionately people who might be trying to commercialize That's how much of this technology. There is a creative way of looking at it and I think that's the thing that you know I like a little more broadly I think that the real value irony and the seminars is you know brightens up across social science right next week we're going to have someone from science fiction. [00:40:33] Really brilliant perspective and look at things that really caught my eye one criticism of that is a robot doesn't do that better robot that the tell you I needed to do this or to the So how did you do a robot I have a box on a different metrics of God and I just hear it or I jump in. [00:40:56] And say like I can't think of a direct analogue to the kind of mobile robotics world the intent about Natura because they're really like get at what people value and persistence acquire the kind of monetary value associated with that that doesn't quite make as much sense in the context of mobile robots or to make robots. [00:41:15] Kind of want to hear your response on that too yeah so I'm going to shamelessly steal something that my wife suggested and she mentioned that in computing or computer engineering they do benchmarking where you have to maybe a certain set of codes maybe function different tasks and that's how you evaluate the performance of a computer and so you have that you can define those tasks beforehand and you agree on and then you evaluate performance based on the ability to. [00:41:46] Somebody in robotics at least of our advice we need to come up with some canonical obstacle course maybe something that really highlights why you would use or of us maybe assets and there's that's out of incline at the time a curve that you're go over the door that no robot seems to be able to open these. [00:42:03] Things like that and maybe all the mobile robots that are sort of in disaster response and Obama's not just after complete the same course and you look at you know how many times they complete it or how quickly they can or even in that case maybe energy is Ok how much energy it asked I don't time it I think time is your forehead and it's tough to find this valuable to people and so our success rate right in some cases there may be a lot of failures. [00:42:32] But yeah maybe I'll throw that out there is something you know that's a bit complicated right after some course maybe this handles it or something certainly for the simple hopes I don't know how to do that with some folks because everything works in 70 way but. Yes I think that was a cool idea I seen with the saw from my wife and colleague but I'll give credit for that yes if you like it also anonymous asking these hard questions I know you are and will talk to. [00:43:01] All right so I'm. I'm going to propose mine by hopefully partially answering another question so a lot asked to people's consciousness we've been talking about this a little bit if you will conscious preference reflect what I actually get from them from a subconscious. And you know I think we've been talking around some of this question but. [00:43:23] People all I think that Eliot Eliot measure is a very sort of this is my conscious preference and I do agree with that we are finding that been very similar and that people do not just prefer the lower amount ot cost conditions I think that that's been shown and the columns work than are work you know there are thousands and thousands and where they do and I think it requires a giant energy change really to really you know that become the dominant factor that's why they choose that and sell I think that but I do think there's potentially. [00:44:02] The issue of people like you especially doing a clinical population they're not always choosing what sort of. Maybe that's for the what we might think of you know scientifically or clinically from a physical therapist perspective people they just choose what's best for that part completely sort of other reasons I so that's why I'm going to propose you know what I think to be the goal that but this is really again this is the problem and this is why kind of going back about thought process so simple as I would do it as symbolic and this was not the ball but I think it is the gold standard which is not what people say and not what we measure in the lab on energy costs what they actually do with the real world and basically I think the Gulf energy should be you take home this robot and you actually use the robot and you take more steps and and not just the quantity of stuff like a stock monitor but are you doing things that you were doing without the robot I think that's like are we improving people's quality of life and to do that you've got to be able to tell you know are they going out in the community where they were going out and are so basically they're like give somebody a robot let them take it or wait and see whether see how it affects their life and to me probably the gold standard that I believe the field would move for because that demonstrates in my mind sort of true value is that they don't just think it's better but they're demonstrating it by wearing as and actually modifying their lifestyle I think with an i Pod is robot kind of the same thing like that the robots that you're caught in that are people using. [00:45:45] That robot the news out in the real where all or is it just the call in the lab that never gets used I think to me that that is the gold standard measure but again I think we would all say that it's really hard to do that right it's hard to deploy devices that are robust enough to last for one week in a position that is not an easy for research perspective but to me that that's what should be the gold standard measurement is the change. [00:46:12] Ashfall daily life in terms of their their use and image quality Aaron just want to add that he gets it is a perfect topic for field trial for programs where they are looking for exactly. One thing like to just build on that and so. I agree like with what you're saying because we really want to see as kind of impact in the Rue world thank God it's kind of the standard real standard by which we should judge these systems but the problem with that approach is that those are the most difficult kinds of studies to write and they require robust hardware and when the field being operated usually regular people want and you know train engineers. [00:47:01] So Mike I like the idea this feeling of my going to be something between where we are now and number that I'm on the subject of right as we come up with a standardized course that everybody right that's like between go out in the community and past. And just basic energy measurements in the lab we don't think necessarily translate well to performance or look I think. [00:47:30] Another idea for the bridge because I you know I I love the idea that we're thinking about our technology in terms of its utility and value like taking an economics on a stance on this is I think the right the right way to think about it but like you say there's a Brit so if you lob the number of our couple a couple kind of. [00:47:50] That will have to get ideas on the logging use so for example I have like this big textbook in my house that I just keep the door propped open with a measurement device might say I'm using that in a useful way but it wasn't it's intended. So there is there is a gap there and terms of figuring out how to log hours that are. [00:48:14] That are that are actually leading to some goal of the technology in the 1st place that's number one and number 21 way to put a bridge how do you make those conduct those really hard in the home studies to what we now can do in the lab is to generalize the problem a little bit more and begin to study what are the attributes of a widget that make people want to interact with it that is a that's you know that's a could be a whole program at n.s.f. trying to understand how people you can measure value or codify value how to how the human brain manages. [00:48:53] Or an engineer who's thinking about their product so I think there's a lot of sort of. Engineering design product development cognitive and sort of economic research that could help bridge these ideas about how we think about utility with respect to physical widgets like we're talking about robots and wearable things and one more thing if we're going to do the course we should run it like at the castle and so that people who are really good at one thing don't get totally forgotten. [00:49:29] So there should be like an overall prize and then you know maybe some sub prizes think Tour de France or the Catholic style competition. Ok so what was taken over Audience Question What I think what the question caught the votes was Magnus So do you guys think one size all cost is appropriate and all situations and that you know I want to start out there with something that I think in the Important that let's touch a little bit on the clinical side of things on this one side off and. [00:50:14] Review had all that I'm guessing Greg Ali I've had similar experiences there are many clinicians how that struggle a lot with not at all costs the main outcome measure and. I guess that read it right and some ways rehabilitation is not in the way not the same thing and there are significant and started all this minimizing or reducing metabolic costs in the lead after ca and that patients may not be able to recover without using these devices because they're not going to use them all this much and this is actually you know these devices are bad actually bad for the patient because they're reducing that all. [00:51:01] And so I would like to have the panel call that a little bit about you know whether they feel like that is a valid concern that many clinicians How are that you know. Again or whether that not valid maybe we can go 1st and I think that's a like a really great point I think it's a great question and I don't think that it is likely or maybe even plausible. [00:51:34] One reason for this is I think if you assist people in the make their movements easier they're not going to do less they're doing more and that's like if you give someone a bicycle you know if they don't you know they go further and they go different places them then they do things they have been able to do and I think the same would be true for actually go on go like I'm a kind of it's difficult to know that without a broad study but I think for me that doesn't ring true with kind of what I notice you know about people mobility and technology almost all of cost optimization. [00:52:10] We're going back to the other side here I don't know about you know. I think this is more about humans think the use of actual skeletons were actually detrimental effects to people's house along the lines of where you are saying in the question but I don't think that's true in my mind that's separate from metabolic cost of a mattress so yeah I can I'll just sort of throw my full support on what Elliot says like we we are starting to kind of examine this question in simulations because a large step scale study is hard but we do we have a student named George Schrader who's going to lead sort of the exoskeleton gym study where we begin to address. [00:52:57] How long term use of facts muscular skeletal tissues etc But I totally agree with Eliot when my answer to this question is always to an audience what we don't know we need do the research but. And the bicycle analogy I think is is is really a good one so and if you think about a bicycle that has a certain amount of air in its tires as as a as an extension to that example you know if I if I drain the air out of your tires overnight and you don't know it you're probably and I left it that way and you still didn't know it you'd probably go do less things with your bike on the other hand if I if I added a little emoter to your bike without you knowing it you would probably change your behavior and do more things and access more more access a higher quality of life the other thing I want to mention is it's important to acknowledge that at least in wearables there's 2 kinds of devices but I still think it's an important distinction like are we talking about assistive technology or rehabilitation robots because rehabilitation robots were never really intended to be a crutch The idea is to build a robot that helps someone retain their own strength independent of the robots were talking it was so that whole class of robots that this question doesn't even apply to and that's still a large portion of I think the robots that. [00:54:16] People who are working in clinical areas think about this just of technology robots the ones that are meant to be worn as permanent assisted those are the ones that we need to be thinking about this question for sure so I think we're wrapping up there and want to continue a few more minutes I did want to say something before I forget because I think maybe one of you Stu there just threw out a pretty sick burn and this is what I want to encourage. [00:54:45] Eccleston's are not art should be examined like are so used to seeing this you know if you think he then people who drive. And there is someone out of the center of gravity is not a myth so I'm not offended by it but I'll let you respond to. [00:55:02] Them question you know why not special I worked with got my carpet not everyone was pretty. Well. I don't know I'm not I'm not necessarily you know voting for my boss but I think that I mean I think I do like the idea that we would have metrics that we could put a lot of you know basically out of window people that relate to people and gas mileage kind of relates to evolve I would I was kind of argue that of all costs not really all that relatable to the general public I think we need a stronger you know a stronger metric that people can actually relate to is how this enhances my life like I think a good example with some with revive technology is you know I can hear I can guarantee that you can go do stairs where you could not see stairs before if you're some sort of water right like capability I think in my opinion more important than just energy costs and yet we can we can relate actual capability where you can achieve things that you can't achieve without the real bot that that to me I think is easier way of telling this technology where maybe people will actually put this device on before they leave their house and use it in the real world. [00:56:27] And I just like real quick I can follow up on that question on the question kind of like relating the golf and the cars and I think that at least like in my field I think that might be like a little bit misleading I mean over an hour as you began like the goal of reduction of not about like rate isn't to say like you can last you know like that's not it's not about right to bring food energy to mechanical work it's about the actually intent of the Act to tell them it's to provide mechanical assistance and if you're isn't providing mechanical system successfully then sure reduce the mechanical go in the body and so it's kind of a measurement of that aspect of the x. golf I'm not were you know you can consume less food energy if you use them so I think that analogy like emphasizes the wrong part of the use case of my mind. [00:57:23] Ok here we are at time so I think that it's time to wrap up there any a palace have something burning that you're either a question that you address or some last point you want to make before we of the session. And get as want to say thanks for doing that seems such a cool format with discussions been great to chat and Mike incorporated night it's been fantastic so I'm just really impressed by the opening price when it's a song that. [00:57:55] We can get up on the question there is. There were a lot of questions but I think people were but yeah thank you all for joining us. And I think I want to give a virtual plotter Arab Spring actually organizing all that and you doing it pretty well the 1st time we've done. [00:58:15] All right thanks panel s. I was great hopefully we have some follow ups one last thing to mention if you are interested in this Eliot and I are presenting that matter I'm writing this week for this new friends here's your university series that Friday at 330 I think I'll go out to Iraq. [00:58:36] Newsletter tomorrow so you know if you're if you're in trouble. That we will be will be in Berlin for you and I are involved with the i room Web page right now robotics not gonna take them not a duty in both ears and I was there a lot. [00:58:53] Thanks Anonymous also just 2 weeks ago we got the Star Wars one so tell your friend I think I'm going to be awesome and so do you guys in a couple weeks.