Well you know I think I was right or you were right. I was one of them and then go on and. Thank you. It's great to have a chance to talk today I've been back for NASA for about a year and a half and this is being a chance to recollect with a little bit of hindsight and separation. Now that I've got back to the warm Georgia air and I'm not dealing with the sleep of a commute in Washington. And yes my talk today is based largely on my time at NASA. And just to help situate where I was what I was doing the now says centered around various our research mission directorates while I was there. We had aeronautics research Mission Directorate which on the or charges on the equal status with the Exploration Systems Mission Directorate the Science Mission Directorate it looks nice on your chart it belies the fact that aeronautics budget is about three percent of NASA. Total budget but still on your chart it looked fairly impressive and I interviewed with Mike Griffin and then his associate Ministre to her aeronautics Lisa Porter during my tenure there the new associate minister Seamus Xing came in and the new administrator Charlie Bolden. Our program offices within aeronautics are based at NASA headquarters so everything above the line is. If the nav. South-West in. In Washington and then the research that we were overseen was largely based almost entirely based at the research centers or had a strong component of research sponsored out to industry and to contractors. In looking at what I wanted to talk about today. I actually thought I'd scarf liberally from some NASA slides and in fact this was the plenary address at our last workshop. I was very grateful to the conference organizers that they let me sneak in this title for their big aviation safety plenary address and in it. It was a chance to start thinking about why the nation even needs at NASA program in aviation safety and some of the directions that we sought going. It was interesting. Being an aviation safety program director. Right next door to me was fundamental aeronautics and that's what you think of I think that's that the common stereotype of NASA aeronautics is what fundamentally are nice to us they get the bang metal into more efficient engines. When they're asked to make a picture of what their contribution was last year they can put up a picture of an air airliner that has Chevron so on the engine nozzles to reduce noise laminar flow on the wing. So they can point to one or two things. But when we point to safety. We found that we couldn't find a picture of safety. So instead we always had pictures of danger of airplanes flying through thunderstorms with lightning and mountains and engine on fire. And it became a real challenge to try and represent what does safety mean and how do we go at it. People who have thought about the serious safety and how to maintain high levels of safety have talked fairly pessimistically I thought that there's a trade to trade off between what they call safety in production and in aviation production is getting people and goods safely transported to where they want to be doing it with increased fuel efficiency reduced environmental impact. And safety is often seen as a versus issue. Personally I thought this was a little bit too negative. Sometimes you can have safety and production working together. I think for example we can look at classic aerospace designs that were simpler. And being in simpler they were less heavy. More efficient and safer if you were parts to break. But either way. Either way if you say that safety versus production or safety and production the role of a safety program is not just to address safety but to provide the knowledge about what I'll call this trade space here and indeed. Often we were asked what accident are we preventing in a safety program. I didn't think that that was quite the right way to frame it. What we were looking at was what safety research do we need to do now. For technology that we see being fielded in five ten fifteen years technology or new operations new ways of running an air traffic system. We view this as somebody who's going to have to make a decision. We've brought out this box. And we want to implement it. And indeed it's no longer just an industry decision. It's no longer just an individual decision. If I walk up with a new airplane at an airport and say hey I'm going to go fly this they'll say wait their system called the F.A.A. and certification and they have to work with you to help. Decide whether this is going to cause not just harm to yourself but collateral damage to others and so with that then. We have to help people understand whether they might be erring optimistically and cause a safety problem whether the new operation or system that we're implementing is unsafe and need some fixes in terms of safety but we also have to look at cases where we're having problems certifying systems at all. And some of it notable examples that come up here includes to cast ticks off where complex flight critical software and to you. A.V.'s in general in civilian air space. So so with this then we also found we needed a safety program that provided the confidence to make a decision to implement a new system. And that in that case our our customer here was not just industry building systems to be safer but the F.A.A.. Having the information they need to make an educated decision. So either either type of error here or there read or the orange indicate some fundamental gap in knowledge and that was where we saw the rule for our program. Now. Unfortunately aviation Scott really say we can look in the bad old days when wings used to fall off fumbling for his recline button Ted unwittingly instigates the disaster here. Here's the recline button and there's the wing stay on wings fall off but we don't have those modes a failure any more. You have to go back to should we do little to be finding those sorts of failure modes to be the dominant aspect that you're worried about in safety. And I say unfortunately because the math is so he's. If you want to find the probability of an accident. You just look for the single point failure and add the probability together but we're not there anymore. Where are we now. Well increasingly sophisticated. Good models are being developed. Looking specifically at safety is a construct. One that's well known as known as recent Swiss cheese model. He describes it as having inherent hazards. Under storms metal fatigue anything that is out there that could go wrong but in between the hazard. And the accident or loss. We also have. These layers of defense. When does an accident happen when all the defenses that you put in place each have some holes and the holes all get a light. And in that case then the probability of an accident. We could look at as a multiplication. What's the probability of having holes you know all that allow for what we call latent failures an accident waiting to happen has hurt having some hole in it combined with the actual instigating failure itself. And in fact we go further than that because we find that in systems like aviation we can't assume independence. Instead of failure in one situation tends to a great or weaken the other layers of defense and as such the probability of failure. In part one component one. Well a group a the probability of failure component two and so forth. So what we're finding then is that we're moving beyond the idea of just traditional fault trees or looking for single point for years. In the safety program we viewed ourselves as needing to be on the cutting edge of. Safety Research and defining how to make something say. I love aviation safety I hope I won't talk all day. Try to end on time here. But I always perk up when I hear things like September eleventh this year. N.P.R. was interviewing the New York Times correspondent tiny fish. How is our view of aviation changed with September eleventh. And he he noted that you have a higher. Fatality rate per hour. Of people in their houses than in commercial air transport. Now of course you have a different population staying at home but still how safe is air transport. It's safe for the interstate the riskiest part of your transport journey is driving to the airport and driving home from the airport and so in aviation safety then we have a special challenge. Not just of you. It is making aviation safety. Making evasion safer. We're also feeling ourselves as being on the cutting edge of safety reaching out to other areas patient safety rail road safety Highway Safety industrial safety in plants. And with this and we need to bring in these new models. So some of the thoughts that we've been working with included the notion that when you talk about modeling risk. If you view this as a time line going across left or right. And up mean somehow some notion of more safe and down mean some notion of less safe. Traditionally we've assumed that you just hang out at safe until something bad happens. That risks are out there. We need to model the risk because they degrade safety. So and if if this is the way that you view safety then you say let's eliminate risk. Which is hard. Think about eliminating risk in your everyday life. Think about how you've done on an exam recently with You had one hundred percent all the time whether you want to say that your model of safety in aviation is everybody gets one hundred percent all the time or whether you want to accept that you have these little data gradations. Risk models that we've traditionally use can only model the negative. They can't model what restores us to safe operations. And I feel passionately about this but I also found it a very hard thing to sell. When preparing testimony for. Or from my boss or Congress for example what you mention. You mention Colgan air going into Buffalo. You mentioned Big Bad Thing. Smoking hole in the ground because that's what we often think of with safety and yet somehow a policy that is also looking at how we restore things to safety. Is where we need to be. And so with this then came the idea of incorporating trying to get the language by which we talk about aviation safety to be one that values not eliminating risk. But valuing resilience. You have some normal variation you get little perturbations down and then you have somebody step in and save the day. And by this model risk degrade safety. But other processes restore safety. We have people on the ground people in the cockpit. Who are constantly monitoring for let any little issue that comes up correcting for it and covering for each other. And by that model then would we call this a success or failure. There was a big risk by the N.T.S.B. definition of an accident as a whole loss more than a million dollars worth of damage and yet To me this is one of these cases where we should we could focus on the Canadian geese. You gotta watch out for those Canadians but. But it's also useful to focus on what went right. And how we can make certain we build that into our flight deck how we build that into our pilot training how we build that into our air traffic operations and how the controllers perhaps. Can also assist. So this is my own personal diagram that I made. In the last six months and I know it's candy. But what I'm trying to show is when I looked at safety. I could see a whole wheel of issues going from autonomy and human the fact that aviation is a human intensive system looking at it from the point of view of complex. Eddy looking at it from the point of view of health management of systems and components recognizing environmental hazards and our remaining big tall pull in commercial aviation safety is what we call loss of control which we saw for the Air France four four seven. But another dimension that we found was also interesting was that on the outside here. The fuzzy text or all little specific point technologies that are aviation safety program was working on on the inside. What we were trying to represent here is that there is some core notion of safety. That spans all the little technologies questions about how do we analyze whether an intervention for safety improves the resilience the robustness that we've been talking about wanting to have to provide some core notion of how we model safety in these very complex dynamic interacting systems. Within the program it became it also interesting in terms of having a structured process. How do you identify what research you want your program to do. And so this was a chart that we had published about how we would identify what. A government research program needed to do some of it focused clearly on national need national need because there's no safety issue or national need because safety is constraining innovation. We can build it. But we can't certify we can't prove it safe enough to put passengers on board. Another other areas was is it some long term fundamental base that we would like to support. And then there were some very pragmatic questions should NASA do this. And there could be fierce debates about that you can imagine these are different and yes no no I think that's more F.A.A. the tech center of Atlantic City don't tread on the N.T.S.B. stows they guard their space jealously. And so working out exactly what. Where the pick tradeoff was between NASA. F.A.A. N.T.S.B. and also working out whether we saw this as a societal good that needed public support or something that was starting to look like we were supporting a specific industry or company was an ongoing thing. And then finally made I found it fascinating to be working in a government. Situation where we had a given. Workforce. So we did start up some new initiatives brand new research areas with a given workforce and that's a challenge and I'm certain schoolteachers understand well. So. So when I came to the aviation safety program I was thinking about the lower part of the chart thinking about the research that our program would do. But what was fascinating to me and what will be the next part of my talk here was the stuff above the line. Functions of the program director in interacting with the broader community and this is where the alphabet soup comes and joint planning and development office works with developing the next generation air transportation system it's joint in that it coordinates all the government agencies involved in your transportation and there's a lot of them commerce Homeland Security F.A.A. of course and NASA and T.S.P.. I do commerce. You know defense military how strong vested interests in their transportation. There were some specific working groups issue analysis teams that we would spin off at that would report directly to me if there was some known safety issue we would spin off and. Issues analysis teams that would work with them. So NASA would provide the technical expertise industry people who are related to the issue would get involved. We would serve serve on the F.A.A. a research engineering and Development Advisory Committee. So we would be peer reviews of our peer the F.A.A. but up at the top are the ones that. I'll be talking about here. The Office of Science and Technology Office of Science and Technology Policy and then some of the groups like the commercial aviation safety team the international helicopter safety team and Assayas the aviation sharing and in for me information analysis. If you can safety information Allison Sherry. Initiative. And it was these are fairly high level ones where we really saw sausage being made in a fairly fundamental way. So the first one was with the O.S.T.P.. It's a fairly research development this last decade that America has had a national aeronautics research and development policy which itself is just a thirteen page page document. And then more recently while I was at NASA. We were fleshing out a more detailed plan that articulated what did we see as the core elements of this. The. In my position and then at the F.A.A. We were two co-chairs of the group that wrote the safety chapter of this plan. And so we would get together and debate what do we think vice its way into the plan for the most important areas for safety. There's also areas of poor environmental issues defense and so forth. And with this often more debates about what should the government to. This committee was co-chaired by my boss to Munchen and then also were Robbie Samantha Roy was then at the Office of Science Technology Policy and within the meetings that they would share we would be debating what elements would make their way into the chapters and to specifically how we could articulate what role the government should have versus where we saw industry going. Likewise some big questions came up around infrastructure. Government facilities in America. Some of them are accused to be a little. Old and so there are questions of what things we wanted to maintain as a government infrastructure versus what things new tunnels are being built by your frame manufacturers Boeing and Airbus have made their own tunnels for example should we tear down the NASA tunnels and tell the military that they have to go to industry for their testing for example. And with it was also a question of how it would be costed so we waded into issues with full cost accounting whether I as a program director when I wanted to run flight tests out at Dryden had to personally play for a flight line of F. eighteen. Because no one else was paying for them or whether they were a larger resource that people could pay for the use of on top. I even had a debate with Mike George. I was having trouble paying for those F. eighteen S and he said if a team isn't a research platform it's an airplane. And I was like No the X. thirty one the X. twenty nine. Those were those were the product and the airspace was the infrastructure but for some of the things we're testing aviation safety Luhan test out a bit of avionics press the airplane with the infrastructure. And I'm sorry if you think of the head of pins that's what some of these conversations were like but but but the decisions could mean a twelfth of my budget. Right at this level. So I talked to just briefly about O.S.T.P. mentioned how the plan was developed and some of the things that I was surprised by I didn't even realize was in my job description is the next line down. The commercial aviation safety team and it has partners the international helicopter safety team and as I was leaving the general aviation safety team were all for me. Historically the commercial aviation safety team or cast was formed about ten years ago twelve years ago now and they have. Gone and very systematically looked at historic causes of accidents. Such as loss of control in flight. See fit. Controlled flight into train. Those are still to be once and used historic data to try and find out where the safety issues. The focus of this team then was let's see where the big safety issues are and work together to solve. The problem is that our systems becoming too safe this is a pet a ten year average. And many of these accidents happened about nine years ago. So so the system's becoming on our hope safe enough that we have a den of hide. Many of these big things. And now we want to look forward to the future things. Some of the thoughts we've had about this in our own research program was how we could be analyzing everything that's happening on the airplane. Viewing it not just as having some sensors on the airplane in a fairly scattered sense but instead thinking about the entire system as being a network of sensors analyzed by the cloud Nasa's Columbia supercomputer do a little do it is a little box right there that's only about five years. And. And think about system level reasoning monitoring what's happening in your airplane. At current time meat and schedules for example are determined by knowing how fast a crack happens may spread in your aluminum structure. For example. And so if you think a crack of a quarter inch up to a quarter inches say you think it can grow an eighth of an inch every six months you've got a distribution curve you can set what you want your your. Inspection schedule to be. And our side here was personalizing the inspection of aircraft being able to monitor for safety issues been able to have sensors right in the material that tell you when it was a crack is there so that you don't have to bring it in and tear it down every six months. But instead can be looking at monitoring the aircraft as it flies and seeing where the problems are. And we've taken that this is a new technology based model that some people at NASA says traditional NASA quad chart. We're very excited about. Some wonderful people at NASA Ames Well we're working with Google and Easy Jet which is based out of London and they are very excited about data mining. If we're talking about having two thousand sensors on an airplane with a sampling rate of twenty hertz and they're each being utilized for fourteen hours a day we're talking about very large data sets and from now says point of view that provided a technical challenge how to look at massive data sources and detect when there is some problem in the system the system might be an engine and isolated component might be the whole aircraft and we also had some projects that were looking at the entire airspace for degradations in traffic flows. So. So from a technical point of view this was was fascinating. But we started running into some brick walls. It's kind of fun to watch an engineer an old maid I got joined in that I saw all their Google. Relation coordinated research and we were all excited about the data mining algorithms and then they said we just need data. We just need an airline to give us all their data. And we ran into that brick wall. Very quickly who owns flight data recorder data. I don't know. We're still working that out. Now that we have this technical capability. We're still working out who owns that data is it the airline if the sensors were headed by the airplane manufacturer is it the airplane. There's even some who argue it's the flight data recorder provides a service and they'll release the data to the whoever pays them for the service. You would. The airlines believe they own and they won that fight but in general aviation you can get quite data recorders and they're fighting the battle of whether when you buy a flight data recorder you're buying a service and the data is beamed to the flight data recorder service and you don't own it you pay to get it. Another interesting thing is made the government possesses. Now if you go to W W W dot nasa dot gov. On the front page. You will see file a Freedom of Information Act request here. As soon as we get data from anybody. It may be open for you. And there's very limited clauses guarding against the release of that data. So one interesting thing was I said got to do was spend a lot of time with General Counsel. Determining whether to any data we got from an airline could be considered proprietary because proprietary does fit under an exemption clause in for you. And this was important to us because if we were going to have to release it under for you know airline wanted to give it to us. We eventually fought the battle that it was proprietary through officer General Counsel NASA and then D.O.J. the Department of Justice with the argument that it was their risk signature and the wrist signature determined their insurance rates. So in other words we said if we release their data we could make their insurance rates go through the roof and put them out of business so that was argument for so far saying it was proprietary and until we were able to do this no airline would give us data. Nor would an airline give the F.A.A. data. And the F.A.A. supposed to be the regulator. We had dealt with this a little bit before historically but never on this scale with the aviation safety reporting system which was also. Part of the program. In one nine hundred seventy five Congress put into the Federal Register the need for the F.A.A. to pay for. A reporting system that would be anonymous and confidential and for separation it had to be given to another agency nine hundred seventy six. It was written into the register that that other agency would be NASA. So in other words the F.A.A. is the regulator the people who can come and punish you as a pilot take away your pilot's license and so they put it off to a different agency NASA. Poor F.A.A. provides most of the funding about ninety percent of the funding. NASA runs it in theory but to get around for you. We actually provide to F.T. full time equivalent civil servants and give it to a contractor so technically the contractor holds the data not the government and and with it. We use human factors research ongoing studies to inform how to. Administer the overall aviation safety reporting system. It's something that you can't draw just in terms of an agency that the F.A.A. is the recipient of alerts safety reports Hoppe original reports and studies. Indicating to them where pilots have. Or others in the system maintainers airline dispatchers or traffic controllers flight attendants indicating their report has been filed. And NASA does the analysis of the reports as they come in and. The F.A.A. gets the results and of course we're working closely with the community there the source of our reports and maintaining good will and a trust with the community is very important. And to in two thousand and nine we received the order of fifty five thousand reports into the system. So we've had some history with this but our day our solutions to date have been very crude. Gives the data to a contractor so technically it's not going to the government. As soon as you do you Dennis hide the report you think are valuable you burn the original with any confidential or again a fine information and some of our interest then really is now where the title of the talk comes in. How could we start looking at taking the models we've taken from confidential safety reporting systems and get them broader used. One thing we did while I was there is so set up a space after agreements or other interagency agreements. To. With other countries. Many of these were established years ago all modeled after the S.R.S. while I was there. We did the firefighter a near miss reporting system there railroad safety reporting system. I think this came up after an engineer was texting and there is a crash and working with the Veteran Administration for a patient reporting system and taking the models that we found in aviation of how to structure all these constructs who holds the data. What balance on confidentiality can you provide How do you do. In a way that it will inform the public without apparent apparently identifying the reporter. But we also want to take these lessons a little bit broader. For example we wondered does it have to be that the airlines give all their data to the government in its full and. When we started out working with Google and Easy Jet We assumed that we would just get all the raw data. And that was where the policy issues were that was where the ownership the data the observability of a government research in the case of NASA or if these tools were then passed off to the F.A.A. regulator though that's where all those those privacy confidentiality insight into risk concerns came in. And so instead we started looking at can we modify our vision of what technology we need to make for data money. Can we view. You it as giving some tools to an airline and say Here you go. If everyone uses these tools all airlines will be using standard data analysis and out of that will come some. Far more aggregate data that had done a feisty issues without identifying the airline pilot the specific component that has the legal liability. And so with this then we changed our own research direction to try and see if we could make technology that state keeps the data in-house with the owners but provides the reports out from any aircraft operator or airline. Out in a way that NASA from its research point of view or the F.A.A. from the regulator point of view could look for nationwide or systemic issues. With safety and with this came questions of what is a great get safety data. What is that intermediate level that could be reported out. And how would we manage it. So now we have talked about in the last few slides here is getting data mining it looking for insights and then you have the question what to do with. An easy model for this and if you follow congressional hearings after Colgan Air and so forth. The model that and I'll take from that is fairly pervasive is one where you say that the role of the regulator is to maintain sufficient oversight that they can ensure safety. For aviation that is nonsensical. You don't have a check airman from the F.A.A. in the cockpit watching over the pilots at all times. Instead what we want to do is get the entire community. To buy into a notion of what is a safe operation and what is recent will to do. And to do that we found that you needed to involve the regulator dictating what they think is safe. The air traffic operator which is a building across the street. Also labeled F.A.A.. Aircraft operators are air carriers. Have a green. Yes we will implement this you say we need a new enhanced ground proximity warning system. And will be two hundred thousand dollars per aircraft are we going to do it. Prove it to us. Airports which are owned typically by municipalities or even corporations labor specifically including the pilots' unions but also with representation from the National Air Traffic Controllers Association flight attendants and maintainers the air framers and avionics manufacturers often were very much there and you could see. Debates between a an aircraft manufacturer in an aircraft manufacturer be. In some of their philosophies about will be safer but they also in this case wanted to call in technical advisors. What are some of the possibilities that will be coming up in the few years so that they are not limited to current day operations current day systems. And and that was so much fun to go there with all the weird wanky stuff I've just been know to drive in and look at what we did and stand up and say that to the rim. So so the various teams in the general aviation safety team the international helicopter safety team or the one that's most established cast the commercial aviation safety team then it took me about a year to understand what's going on quarterly meetings running for days you craft chaired by their social minister the F.A.A. with representatives all the way around some of the. Double stars meant that they allowed somebody else to serve in their stead. But technically this was a representation that we had and I'm sorry for the alphabet soup they assume that everybody knows what's going on. You can see some of the air framers here but I'll post the Pilots Union the International Pilots Union the Air Transport Association representing airlines and square. The list. I showed on the previous one. And at these meetings. We would usually start out with this. Where se is safety in him. This is something that the entire you could add a safety enhanced meant by getting on the agenda a discussion of data you had about a safety problem. And it once agreed there was a problem then they were collaboratively to figure out what the enhancement would be to improve safety. And then once they did that. We now needed to implement. The cycle on this was something that where we wanted to see that everyone in the room had Badia. And so the cycle was longer than my tenure. Safety enhanced myths that I saw coming in my first cast meeting two years later we're now starting to be implemented. And the implementation might include some retrofit on airplanes which would be allowed at an annual meeting and it's check spread over three years. So this was something then where every safety enhanced meant some of these might deal with controlled flight into train some might deal with icing. Some might deal with maintenance fatigue and the aircraft maintainers being to work in shifts or so long that they were conducive to air. And we would keep this charged up as a running track of what was going on. So there was good and bad about this. The bad was that it was mind numbingly boring. To an outsider. I mean you took years to even understand the undercurrent so why the pilots were pushing for a new cockpit system in the airlines were pushing back. You had to ask someone a state that's because it's going to cost two hundred thousand dollars per craft you know all I see. Mind numbingly boring at any I double a symposium I was asked what as a NASA program director I was going to do to make aerospace fun again and I said I'm sorry. In my position my job is to make aerospace boring. My job is to make it so that you will take your grandkids onto a flight with pure coffee. And there's a little bit of excitement but there's not fear. That's my job and how do we get there we get there by having the pilots and the air framers and the avionics manufacturers the regulators the air traffic operation sitting around in a big room and working towards a consensus that they're going to move forward in this direction. This is a very different model. From what I thought of his aerospace. I love reading about Cheney doodle he was an amazing pilot he would walk out to an airplane he jump in. He could break a speed record get the colyer trophy just him in the White House. But I take this as a slightly different model of how we work at safety. We work at safety by everyone buying him about what's the right way to go. And so to me it was a fairly big deal that the cast was selected for the two thousand and eight Collyer trophy. To me this was a a big change. I was so of course. Just ecstatic. But it's such a change from Apollo eleven. You know three guys facing danger straight on. Instead what we're seeing now are the big changes happening when you get the F.A.A. Continental Airlines. E.T.A. and so forth all working together to. To agree that we're going to make these most detailed and precise changes to improve safety. So just as my last slide here is and the summary that the takeaway points. I am with aviation safety and I've been thinking increasingly about how aviation safety I see as a leader in many To me it's what I can look at trained I can look at the Veterans Administration see the issues they had and how some insights from aviation safety can go out to so many other industries or areas. Technology is only part of the solution we're finding that you can't just put the ground proximity warning system in the aircraft on its own without buying. From everybody. So we need to be carefully coordinating these things. First off here I'm just trying to show reminiscent of chart shown before the idea that we need to look at safety research not just to prevent accidents but as an enabler of innovation because right now we don't let something fly unless we're sure it's. So that's one thing the ability to assess the safety of and certify new developments. Then there's that notion of data protection versus aviation information sharing how what mechanisms do we need to put in place whether it's policy careful data protection or data mining technologies to carefully handle and protect the data that we have going around. And then finally that idea of a shared construct of information all the the acronym showing up together in a room in a green that we agree collectively that this will be an improvement to safety that everyone will buy into and contribute. All right thank you. Dr Shankar. Yes I know now to or not it has a lot of agreements with other agencies. While I was there we were working closely with Anil our a deal are trying to set a good and exchange programme for example. We were worried a lot about I seen and it turns out the Canadians research I seen a lot. For us we need to have refrigerated wind tunnels to look at I seen they just go out with a chainsaw and throw some ice into engines that they have running outside and so it was a very relationship there were several that were looking at. We also found that there were different agencies had different levels of emphasis on safety. If that makes sense once or tending to grow their aerospace industry overall are still focused on have. In the most efficient engine some new technology that distinguishes their aircraft their engine and so forth and and so for me the safety of the people we had safety interactions with were often fairly different. I guess. There or I should say. They really weren't. There must be here and to. Yes Yes And in fact you already see that tension right now the N.T.S.B. when they come up with their recommendations does not need to consider cost but the F.A.A. when they make new regulations in response does. And so there are issues on the N.T.S.B. top ten list that will probably hang out there for a long time because the cost of addressing them is so high. How did we look at it and I kept a picture of this up in my office as well because cost and production production is enabling air transport in the way to reach its economic and mobility goals and. And so we looked at is safety holding back our ability to move airplanes in a cost effective way. And if so what can we do about it. Gerri. Three. Yeah you know I'm no longer at now. So maybe after this talk I will never be again. But. It's personality driven by. And large but some of it you could look at a person's performance standards for their annual evaluation and guess whether they wanted to collaborate with you or not. If that makes sense. So there were a couple cases at the F.A.A. where our research budget up there was a great line between our research and F.A.A. research. There was no reason they wanted to collaborate with us if we did similar research to them and made them look bad. They would look at so. So you could already see some sources of tension. From my point of view that often meant needing to make certain there was a reward structure for my own personnel for interacting with the F.A.A. if I asked somebody to go work with the F.A.A. on a specific directed issue I needed to make certain they were confident that they weren't going to get burned. In their in their everyday life from. Krishna. Yeah we actually were given two questions which I treat fairly similar one is how do you look at general aviation safety because we focus fairly on the commercial aviation. And the other was how do you look at international safety and we said you know a lot of the problems that they have there are there are known technical solutions for. And you do not need technology research technical research to solve many of their issues. So there are some exceptions where they had unique capture buttes and we thought we could help out but there are other cases where it was more a matter of their accepted standards for safety and how that reflected in their type of operation the duration of the life they would allow an aircraft to have so forth. As well. If you're. So. Yeah they're the real visionaries didn't really get along with us because safety shows up and says Well have you thought about it. Have you thought about you know they're still trying to prove that the concept will work at all. But we were trying to look fairly hard term some of the things that we were looking at a lot where What's the durability of new material. For example. Just as one example in the future if we start having automated systems how do we prove that they are safe at the level that we require for air transport safety. It was an interesting thing for us we had to run program reviews where we had to report on the quality of our internal research. Which meant that we needed to have a metric of quality and it was a being county organization we had to list it was quality and I was trying to get people to articulate their own notion of what they were trying to achieve. Tirrell one research would be quite happy with journal papers that they would view their impact changing the industry the thought leaders in the industry who were doing the initial development but we also had some people who are doing very practical developments making ultrasound technology for inspecting aircraft and tell them there they would be successful if an industry in the industry picked it up and started developing those technologies and so we had to have a range of types of research because we wanted to support them all. Well allow them to evaluate where we were and recognise that we weren't just here a one or more fundamental research that we also needed to see a transition path. Thomas. Yes you know and they sometimes they can't even get replacement parts because the replacement parts are now considered outdated by the manufacturer of those parts. You know some of the things we're looking at we start up a new area in verification validation of play critical systems where we recognize that verifying complex integrated systems specially when you start looking at is more electronics and software has a time cycle of ten years for hardware and software that has an inherent lifecycle itself over six months and so how could we get automated testing for example of software to be more so that the surf cation process was more in line. It's a it's a very tough call and some of the things that you do is you look at. Either equivalent level of safety or total of safety equivalent level of safety as we can prove this new system is this good is the ones who are flying right now the problem is that you can't do that for truly innovative technologies for that you need to prove total level of safety and how do you prove something is ten to the minus nine with any intellectual confidence. And new areas where we're seeing problems with total level of safety. Are you abs plane in the air space system. A separate anecdote that happened while I was up there or not directly related to my program where discussions about you a piece fine in the air space. The F.A.A. held a fact finding mission where they involve invited Micro you of these things that are like two pounds or less to come to an airfield and fly around and I've heard stories of how the two populations came together. The F.A.A. regulators who showed up were used to traditional aircraft or for case where there's a book this thick and the person who shows up is from Boeing and has been familiar with that book. And the people who showed up with the microwave East were from Iran. I Robot and they make room and they're like yeah we've got OSHA covered we've got will teach protection covers where you plug it in and. They were they were in shock. You know either so I could talk to each other and that's a real problem of how you prove safety for something that's so different that you cannot go after a quick equivalent level of safety. So is this your. Yes. What was your. Well first off on Note that I was very aware that I was on a term position. And that I had tenure back here and I love that intellectual freedom. I could tell that others in the room were sitting there shifting shifts Lee not wanting to tell the truth and I was like hello I'm the first with the wild hair. Well nobody said I was just stupid. I do you. So I appreciated having the freedom do that. And and so that was one way to go. I did not think that it was going to be a permanent position and that very markedly shaped what my interaction was there. I said to the position because it looks like technical fun. I wouldn't recommend taking a position if it doesn't look like fun. And the positions can chew you up and drain you out drop you back to Georgia Tech a little puddle of sludge on the floor. So you really have to want to go. It has to be something that looks like it's interesting. On a purely pragmatic note I've since then had people ask me about various levels and some of them were fairly low level and they gave them no cost of living allowance and so forth. I think it's important recognise too that some of those can be negotiated that there should be a cost of a. Living Allowance or a travel allowance or something that that that is at many levels normal because it didn't seem like that is consistently offered. Great thank you all thank you.