[00:00:05]
>> Hey hi welcome everyone the Milken to the fullest and 7 out of the i.r.s. seminar series in fall 2020 so the series will bring top ideas such as from around the country. To talk about their research mission and plans but I'm delighted to have a professor to be the government today from the University of Oklahoma and she's a lie g. and Joyce Austin presidential professor in the School of Computer Science and the School of Media ology there she has been leading the development of for artificial intelligence and machine money picnic for that obligations for almost 15 years and has won many awards for her research most recently she's not in the n.s.a. the Eye Institute for Research on trust with the regular climate Coast emotional.

[00:00:51]
Let's welcome e-mail again we'll be talking about building trust with the advent mom thanks to as he said I mean recovered and I am going to talk of the I that is a lot of things on the 1st 1st flight but I was specifically asked to talk about the new n.s.a. Iosif research untrustworthy and whether women cost less European which is only a short version of that name because that version of the name is quite warm.

[00:01:18]
So I want to just because you know about 45 minutes I want to give you a roadmap of where we're going we're going to talk about why we care about trustworthy ai for Environmental Science and then we'll talk about the is the 2 and then becoming on time I have a couple research results that we can talk about as well but this will depend on how many questions we answer of a mystery.

[00:01:37]
So I'm going to start with some motivation that I think everybody probably is going to agree with which is that as our climate is changing. So for example this is a tornado that's hit this is some hail these are actual photos that you know so I go back that's a tornado from the southeast that's hail that's actually in my hand a female a pellet my house.

[00:01:57]
As we're seeing impacts on the ocean this is a sensing system that's deployed that's looking for her McLachlan's. We have impacts on sea life so there's a turtle there that they're working on re homing in making sure that the turtle is you know not dying out as a species and turtles are very popular so you can see the number of people there.

[00:02:16]
And you know as we have other impacts such as this is a picture from last year I could well have had a picture from this year we have multiple hurricanes coming and. You know all these impacts are changing as the climate is changing and humans need to be able to adapt to the other species need to be able to adapt and we need to be able to improve our resiliency as the climate is changing and if you don't like just picture based motivation if you just want to think about money this is a picture of the $1000000000.00 weather disasters and these are only from January to June and we already have a number a 1000000000 dollar weather disasters this year if they had updated this graphic this is a graphic produced by Noah I went just like 10 minutes before the top to see if there was a new version of the graphic they don't have one out that includes July August and September but if they did you know there would be additional disasters such as the wildfires though we need to use ai to improve our resiliency to all of these events and then you might say Ok well I can believe you that we need the resiliency to the to the environment science because climate is changing but why do we need a I you know what's what is in it for ai Well one of the things that's happening is that we have a tremendous number of new sensing systems that are coming on line we have satellites we have moved the radar trucks we have crowd source data all of this data is being provided a really high spatial and temporal resolution.

[00:03:34]
And there's no way for a single human to sift through that data in real time and so what happens is a lot of that data is ignored the satellite data for example only about 5 percent of the data is actually stored and used that means what's wrong way 95 percent of very valuable data and then if we're going to have an old state economy and we're to have the i system look through it we need to be able to have the ai systems guide the end users and their end users it can be different kinds end users words making different critical decisions because they have to make many forecasters have to make life or death decisions you issue a tornado warning you not issued a tornado warning if you get it wrong people are going to die if you get it wrong in the you know in the tornado came people may die if you get it wrong and there is no tornado people are going to laugh and say you can't ever get this right you have to decide what the right answers and on a larger scale to evacuate a city for a hurricane you do that instead of a tornado warning scale which is 15 minutes in advance you evacuate that's usually days and.

[00:04:32]
I grab this picture because I gave a similar motivation to talk to NASA and you know one of the things they have to make is a go nogo for watch so that's a picture from the space x. launch the 1st space excellence where it was scrubbed at the last minute.

[00:04:46]
All these engineers are just make critical protective decisions they need to decide at different times scales to close your business or school way to get into your shelter if you get on the roads etc and I want to make a point because the trustworthy Ai that we're going to talk about really cares about the end users the end users needs differ very greatly So what we're developing for ai is going to matter about which trustworthy Ai end user which end users are using our dose of your weather which are the examples I gave here typically it's forecaster so other scientists you know forecasters are tree it's emergency managers but it can also include the public and the needs of the forecasters or differ from the emergency managers and they differ greatly from the public and then if you want to think even outside the domain of severe other which is more on the very short temporal scale long term the seasonal the subsidies and all the really look at crops and then the end users man differ for crops you know corporate corporate corporation that has millions of acres as different needs than a family owned small farm but both of them need to care about whether or not there's going to be a drought coming for example.

[00:05:51]
Del as was mentioned in the beginning we have this newly funded n.s.f. Ai Institute with a very long name and is super research on trustworthy Ai and weather climate and coastal oceanography our short version of our name is a I.C.E.'s so ai to environmental science there's an embedded master in that ios plane later somebody wants me to explain it.

[00:06:11]
Our partners are listed all the way around the slide so we have no us the lead institution we have Texas a and m. Corpus Christi North Carolina State University Colorado State University University Albany University Washington and Del Mar college and that So those are our academic purse and then we have no one as a government partner and then we have private industry so we have well and I didn't encourage not private industry and cars a federally funded research center to me to forget them.

[00:06:37]
And then we have a for private industry said video disaster attack i.b.m. and Google and they're all partners in that really helping to both develop the fundamental research and move it into the public and the needs of the public though and to us is going to be uniquely benefiting humanity by developing novel physically based Ai techniques that are demonstrated to be trustworthy and will directly improve the prediction and the understanding and communication of high impact environmental hazards and I will say we have a web page up if you want to look more about what we're doing it's not only fleshed out yet we've only been live for about 3 weeks and we have a Twitter account and again we've been y. for about 3 weeks but feel free to go and follow our Twitter account as we get results as we're spending up we will be texting it's reading and putting things on the website much more right now the website mostly lists who our partners are if you want to see any of the people there on the website as well.

[00:07:26]
But I was asked to talk a bit more about what our center will be or our institute will be doing so I'm going to talk about it I don't have results from the center because we've literally literally we started September 1st in the September 24th. But we have 4 Focus areas so the 1st is the foundational research untrustworthy Ai machine running and I want to talk about each one of these core focus areas at least on one slighty so you get a little bit more detail I'm giving you the overview now so is the foundational research we have the n.s.f. is call it use inspired research it's often called you know work that you're working with their main scientist but they call that uses fire and environmental science and then we have foundational research in ai risk communication for the Environmental Science hazards and then finally and I do not want to miss this because it's a critical part of our plan because we are an institute we have workforce development broadening participation and then all that together goes to our institute which we have a pretty logo so I like to go to the foundation research in trustworthy area and machine learning we have 3 main goals and so our goals are develop explainable area methods that are that are aligned with environmental science to mean perspective as priorities and what does that mean explainable ai is really up and coming technique where people are looking at the ai appreciate it learning and saying this isn't the black box and we really want to understand what's going on inside this black box but the key for us is that we're winding what we're doing with the needs of environmental science data which is very different than typical a I did it's not image I mean although there may be images it's not image space data it's not id it's very highly spatially and temporally are correlated.

[00:09:03]
And it has physical laws that govern you can't come up with an example a plausible explanation out of your method if it violates the laws of physics because nobody wants to use your method if it's doing that brings us to the 2nd go which is develop physically based Ai techniques for the Environmental Science terrain so we want to make sure that we are incorporating the laws of physics I know that physically based ai is.

[00:09:26]
Something that's also up and coming and people been talking a lot about it and they mean different things with it for us we mean we're incorporating the laws of physics. The 3rd goal is that we're going to be developing robust Ai techniques techniques of prediction checking send them where theoretically validated though when we say with adversarial data we don't necessarily mean that there's going to be an adversary out there but that data can be missing and it could be intentionally Ron we want to make sure that our techniques are robust so I have a little bit more detail on each one of these goals this is just some examples of some work that we've done on x a I and environmental science the sort of the preliminary work version.

[00:10:05]
The the members of our group are the leaders and element of x. ai for environmental sciences so I just have 3 examples here. The one on the left and hopefully you can see me now so if not I'm on my left hand column where it's application one we're looking at trying to figure out how to we could emulate radar imagery using deep neural networks and then once with it we've emulated that radar imagery and that by the way we can do emulate in radar imagery from satellite data were places where there isn't radar that you can actually.

[00:10:33]
Dance so that if you can learn if you can learn with the radar should look like this alligator then you can you can essentially get radar readings where there isn't radar and one of the things we're doing with our people with an x. Ai And so in this particular method it's a clear eyes relevant propagation they're looking at what strategies it is that the neural network is using to derive its estimate so that you can see whether or not the strategies are physically possible the one in the middle is looking at using a deep neural network accomplished neural network for analysis a severe hail and so predicting hail and one of the cool things we did with this was we looked at what kinds of neural kinds of strategies we're going to fight within the network so we're looking of using saliency maps to see what looked important and then we looked at those and said these are these look like super cells those are in the a little imbedded window up here they look like 3 different kinds of storms a super sellable echo in a pole star and then we craft it like where did those storms appear were they highlighted by the neural network and they match with what happens physically that there's more pulse storms in the southeast that there's more super cells in them in the southern Midwest where Black is in the upper Midwest so it's really nice to physically validate that the ai learn something that you expect and then the 3rd one is looking at learning the real time storm morphology with with c.n.n. as well and I'm going to of been that in the name of time I will not talk about that one so much.

[00:11:55]
Moving to the physically based I technically So this is just an example on the right hand side of just trying to say that we're using the laws of physics to create robust. Primarily deploring based methods that. That it that create here which is physical strain loss functions that can create a neural network that learns something that cannot violate the laws of physics or rethink it hybrid together the physical models of the ai system predictions etc so we can create novel architectures for the deep learning that use the physics and then use those to also be back into the x. Ray ISIS to make sure that what the x. ai is showing is also physically possible.

[00:12:34]
And then the 3rd one the robust ai. Ai just have some some graphics from a performance diagram so just showing differ this I'll just leave those there that they're just trying to for performance under different situations for predicting I hail the environmental data sets are often very very limited it's a strange world in the sense that there's a tremendous amount of data out there and I said that in the beginning and yet there's also limited data because there's a limited number of tornadoes for example the for those who know how many examples that deep learning likes in a really bumper crop of a year in tornadoes has 2000 for natives a year and you know just like 2 or 3 of those are going to be the really intense tornadoes we need hundreds of thousands of examples for most of machine learning and for deployment and so that limited data we have lots lots of examples of not tornado but we need to have examples of the tornado itself though we have to deal with class imbalance we have to deal with creating robust ai for so we could train on the common phenomenon transferred to the rare events make sure that it's actually still robust look at trying to label data when label when data is really hard to label where it's expensive or it's unavailable and just we want to provide theoretical guarantees for the requests this of the methods if we're going to field them and we want them to be guaranteed to say for example as an example something we have seen in the past you know you don't want to predict that something is going to be less than pal is your account that's physically impossible you could just put a hard limit Ok if it's less than 0 you go I predict 0 but even 00 unrealistically don't get to 0 tell them Do we need to have those guarantees that that tell us that there cannot violate the laws of physics.

[00:14:11]
Moving So this is a cycle of the research the ai is based on embedded with the environmental scientists the Environmental Science terrains we want to use the trustworthy I to predict actionable environmental science information and we want to enhance scientific and physical understanding of the Environmental Science prophecies but I want to talk a little bit about our our domains Our 1st I mean is compact of weather so we have so I know that the southeast gets just as many thunderstorms as we do it's a different kind of regime of the understands the last but you get a lot of convective weather thunderstorm hazards the tornadoes and hail produce billions of dollars of damage they kill hundreds of people every year and tornado warning scale really hasn't improved in the last decade though our goal is to use the c.i. methods that we're developing to improve the prediction of tornadoes and hail and other convective phenomena or focusing on the tornadoes and you know 1st and so I just have some pictures here I'm actually the last half of the top going to talk about the tornado results and then the 2nd half of the top so I have search for a 3rd let's call it the last 3rd of talks about radios the 2nd 3rd is about the failed predictions but we've got results to prove that the both here and tornadoes are predictable using the methods.

[00:15:24]
Winter weather so winter weather is a major hazard United States and you might think why is Oklahoma studying winter weather we have a lot of partners and that's part of it and part of it is that Oklahoma actually does it with weather but we're going to be doing is really using things like The New York resident and the Oklahoma maisonette to create methods that can robustly predict winter weather across different parts the United States because New York's winter weather is very different than Oklahoma's winter weather we typically get ice they get snow.

[00:15:51]
And we want to be able to provide tailored guidance to that to the end users in this case will be looking to see managers. A 3rd application area for the Environmental Science tropical cyclones also known as hurricanes they have a tremendous societal impact in terms of damage and flooding and we're trying to improve the understanding of the tropical cyclone evolution when they rapidly intensify and would understand they're pretty understanding of the forecasting of the tropical cyclones and I have a assure example there we've got some results showing that you know we've got to stationary satellites where we can get views of the hurricane or the tropical cyclone every so often every 10 to 15 minutes but sometimes we can't see the internal structure the clouds obstruct the internal structure and so if we can use other microwave imagery to reconsider that we do know and a wind a satellite images we can you train a deep neural network to be able to reconstruct the recreated injury that we can get on a much more frequent basis because we can only get the microwave energy a couple of times a day whereas the satellite can give us information every 10 to 15 minutes and we can use a x. Ray I to do that and create those methods.

[00:16:59]
So there are 5 different environmental science application domain to this is our 4th one sub seasonal the seasonal prediction. So in the in the graph down here on the left if we have the if you wanted to sort of ask the question between what's weather and what's he's not there's a gap the weather events typically can be a prediction near you know minutes to 2 weeks time frame although it gets harder and harder to do the prediction as you get down towards 2 weeks it's really really hard in the gap between 2 weeks in about 2 months that's the s. to ask out the duties of predictions and then there's pretty there's a decent amount of skill at predicting seasonal outlooks and we're looking in that gap at whether or not we can use ai to improve the prediction in that gap and by doing so try to also improve the science of the fundamental understanding of what's happening though we have an example over on the right of looking at predicting regions regions relevant for the prediction of extreme rainfall in North America.

[00:17:53]
18 days later see i notice of the graph is not real America the graph is Asia we're looking for what's going on in Asia that's going to affect North America and create extreme rainfall vents in North America 18 days later. We have coastal oceanography coastal phenomena like the tropical cyclones but there are other coastal phenomena we're going to talk about and we're going to work on our impassioned any regularly and so we can improve the prediction and understanding of these We're going to save lives and property so on and this graph we're talking about a couple phenomena in any case which are very large scale rotations that last months in the ocean and they severely a fact that oil rigs in the ocean so they have to because they're I've heard them called Ocean for natives but they're not a tornado in the sense that they're on a completely different spatial and temporal scale but they still have a similar effect so they need to be able to predict them but on the larger timescale.

[00:18:47]
And then harmful algal blooms which are they you know I've got a picture down there of where the water is washing green the harmful algal blooms kill humans as well as many animals that we need all the protect those better and then compound flooding so flooding in. In around the coastal areas and then the 2nd coastal erosion out of the area that we're looking at is predicting coastal fog and then you know if you noticed there was a turtle in our logo because everybody loves to roast turtles are cute and we're working on improving the prediction of old stunning events where turtles that are in the the bay in Texas it doesn't get cold very often in Texas but when it does get told that her journals rise up in the water and the ships from over in the dam and so if you can improve the prediction of that they already have a preliminary Ai system in in the actually used and if you can prove the prediction of that and then stop the traffic then the sea turtles much more of the sea turtles will live and so we're trying to improve that prediction as well so it's another impact.

[00:19:49]
So that was 2 parts of the cycle the 3rd part of the cycle that I think is key because it's really what turns us into a cycle of the social science research the foundational research in risk communication though we're going to be using the risk communication to understand to increase the knowledge and understanding of how the end users are understanding the trust in ai people have asked me you know how do you define trust we're defining trust based on her and his hers actually use the at we want our end users to understand it and to use it we don't just want to develop Ai that we just are when we want to be used to develop models of how this attitudes are perceptions of the thrusters influence the risk perception and then we want to take those methods and use them to inform the development of the ai so that it truly becomes a cycle so that Ai is developed for viral science test that with the risk mitigation and that we improve we generate improved i'm And so the risk communication we're going to be interviewing she users.

[00:20:46]
We've got. You know to to understand what their definition is what their perceptions are what their definitions are of trustworthy Ai and on the next slide I have a target population of some of these end users we have a variety of different users you know that range from forecasters to emergency managers and we will also include the general public through some of our collaboration with the private industry are we doing interviews of the experiments and then feeding all this data back into the development of the area.

[00:21:15]
And then I said in the beginning that we had 4 Focus areas so the 4th focus area is working on our workforce development broadening perseveration plans and need definitely are working on Deville improving the diversity of the ai and environmental science workforce the wanly part of this aspect because I don't want to spend a whole talk on and just as a want to give some research results the part of this aspect that I want to talk about is our unique so it's cool for a we're going to be developing an ai certificate but the other part you know we're working on mentorship some broad introspection and putting a lot of training online we get asked all the time how can I learn a I happen to her machine learning I've already got a job how do I retrain were putting modules in line for case or 22 that means only to university as well as workforce retraining.

[00:22:03]
For the ai certificate. Texas a and m. purpose Christie is a Hispanic serving in minority serving institution and they've partnered with Del Mar college which is also based there and serving minority serving institutions and they are developing a level one Ai certificate which means 5 classes in Ai has to be ai for Environmental Science and then that certificate we will take nationwide once we've proved that it works really well there they would be able to get the job directly from getting your such a ticket but it'll also transfer to the university so it can be a recruiting method if you've got the c.i. certificate then you know universities can accept those transfer credits and put you in into the higher into their Ai programs.

[00:22:42]
For those who do want to know who was on the team this is a picture from our proposal I would like to get some more photos of all the people at the bottom as well but the leadership team are the ones who are shown with actual pictures of the top there so if you want to ask any of us I'm all the way on what's inside when I ask any of us questions you know if you know any of these names you say are excited our partner about it you were interested in learning were partnering is something you know any of these people are good people to talk to Ok I save myself enough time but I'm going to pas and say Are there any questions yet before I actually do research results.

[00:23:20]
Actually me so meet me actually have one question are coming from Blue Light who is asking are ample signs a certainly not the will be paid interest in interest will be back to how well do you expect this research to transfer to other applications and that's a good question I agree with that there are a lot of people interested in trust for the ai and I would assume that our Since we're focusing on creating physically based prosperity I think it will work to a variety of science and engineering applications.

[00:23:52]
But I mean I don't think that we're creating anything that's that's we are certainly focused on environmental science but I think our results will generally. N.s.f. is funding us for the environment science application by the way is all of one other note on that and I have to require a domain they require these inspired research so our users fire researchers environmental science but I agree it will apply to other to me.

[00:24:17]
X. I think because the main question and. So I will what I was going to do I have to have to the research results to go through them relatively quickly but we can take questions halfway through I want to talk just so you can see some examples of trustworthy and but these examples are done within my research group so looking at health prediction and then working on radio so they actually I for tornadoes so the part that's read is they have protection from the talk about 1st and I know that most faculty start with the slide I was waiting so that it was right next to the results because this way you get this I get to say thank you to my students for helping the generators all but you get to see them right before I start talking about the results.

[00:25:02]
So my research group we work you know my specific research of yes I'm directing the giant is that you're now but you know my reply for 15 years as he said the other direction we've been working on using machine learning to improve prediction understanding of high impact environmental science phenomena and and I actually do research beside environmental science it's just that that's most of what I talk about and today that certainly to talk about so I'm gonna just leave that there the one that we're going to start talking about is hell prediction and I'm going to assume that everybody knows what decision trees in and run forests are and just leave that out there for a minute and just explain where we're going with the Health Protection model so the task that we're doing is to predict fail in the next 24 to 48 hours.

[00:25:42]
We crane to random for us we also have a decline in a person that's but I'm not talking about that one today and only talk about the decision tree one thing in the rainforest one because it's much more mature. We train 2 different trees the 1st tree it predicts whether or not it will hail in the 2nd tree predicts whether given that it will have also all the branches that say hail there's a 2nd forest will go down and say what size they have will be because it could be speedy little pieces that are going to do anything to be the giant golf balls you saw in the picture at the beginning.

[00:26:11]
And you know why do we choose decision trees around a forest Partly this is this is research that's it's mature research we work on it for a while before deep learning was really shown to work on this so we're just starting to turn in parts for hail but one of the things that reading for us to do that is really nice is that you can pretty much serve the kitchen sink at them and figure out what really matters and so that is actually good and also there's already explainable Ai methods that have been developed like.

[00:26:38]
Trying to understand which features are most important out of a tree and then out of a forest. Well our overall steps I'm going to I know that this is a very interesting very audience so I'm going to try to explain at the high level sort of assuming you don't necessarily know the meteorology but I am going to assume that the area is relatively known since that was my explanation of trees we're going to extract the data from the numerical weather prediction model train or machine learning method and then because I said this is the part we're going to talk about how we actually.

[00:27:09]
Look at trustworthy and we've actually been implementing and testing this in the hazardous weather test bed which is a unique thing that now has said Norman where you actually get to test new methods with real forecasters but they don't go live so if your method is you know doesn't work very well then you're not hurting any of forecasts but if your methods great then eventually your method can transition into the forecasting so we have 2 papers there as well as a 3rd one under a view.

[00:27:32]
So the models that we're building on for this work we're using convective allowing models and this there's a call convection the bombing model so you think that they might be high enough for a resolution to actually resolve the hail in the tornadoes but they're not in order to do a convection line model across the entire conus that's that they are current on the United States and they're doing it 3 kilometers that's still really high resolution but it's too low it's too coarse in order to be able to resolve.

[00:27:58]
The machine learning can be used to predict those missing hazards and also you know sometimes the models correctly identify there's going to be a severe storm but there are by spatial or temporal you know the machine learning to learn to correct that spatial temporal forecast error. And so for those who actually care about the details about which model that is I have the models listed down there but I'm just going to leave that.

[00:28:20]
One what we're doing that we take this is a picture from one of the models to a trust model you know you've got basically pixels of data so we need to be able to identify the storms and then if I then then extract data from it and because I said we're not very deep learning we're not using the image based approach on this we're doing just expects approach because we're doing random forest we need to extract data for each one of the storm tracks so taking a storm what we do is we identify the storms themselves using a watershed method which is just an image method that looks for pixels that are similar to each other that have been a fire storms and then within a storm you can extract about just the system about that storm so that's maximum minimum indication of a whole bunch of variables so that the wind the updrafts the wind blowing up the downdraft from down reflectivity is the intensity of the perseveration etc There are about $500.00 variables that we extract and so our model with them all here obviously.

[00:29:12]
The labels that we're using come from the retarded up there they come from the maximum expected size hail which is a radar drive product and the reason that we don't use human scale labels is that the human heel labels have been demonstrated to be highly biased toward cities and highways as in it basically never Hales in a field but we know the details of a field and so the radar drive product is a little bit of an over forecasting product but it doesn't have the biases across the whole United States so we decided that it was a better product.

[00:29:42]
Well you know can you give any of the details are going to talk about training and testing invalidation all that if you want to talk about it offline we can't but I want to talk about the human end of. The this is where I go to talk specifically about end users and how they matter we're working on this product with the s.p.c. wish the Storm Prediction Center and the s.p.c. issues forecast multiple days in advance of what they think is going to happen in a hail tornado when other storms and they put out a wide area a very low probability but you can see there are legend on their forecast for this particular day is there a 5 percent 15 percent 30 percent 4560 and significant which almost that just turns into a hatching but it's not used very often they almost never issued anything about 15 percent there probably is 10 to look like 5 and 15 percent so the 1st time that we were testing this you know our pro the lady over here on the right this is the machine learning model all the black dots are hale what you can see is up abilities were much higher base or a scam and we highlighted the vast majority of the areas that are here we did miss a little bit here but not a lot.

[00:30:49]
Of people is much higher but they didn't like it because the problems were too high and said Well I mean this is part of the trustworthy part we need to make our end users happy with our model so we looked at it in the back and said What can we do about this and the 1st thing we did so that was 280-200-1000 what we did was we calibrated our problems we didn't retrain the model Well we did because we had another your data but we didn't change the process of doing the model we did is once you had to reinforce probabilities within reach calibrated then just adding another machine learning a lot on top of them to bring them in line with more of what the Storm Prediction Center actually wanted.

[00:31:23]
And what does it actually look like this is a reliability diagram where if you were probably more if you were perfect you were going on the right was excellent. And this green line is one of the machine learning models that's been calibrated and is pretty much following that weight was excellent again there's not much up here but that's because they don't have an issue anything about 60 percent.

[00:31:44]
And then the the version it was I'm calibrated is the blue line it's Or you can see it's probabilities were were were nothing like what they should be for the reliability and that's for $25.00 millimeter hail we also have broken into the 50 millimeter which is a larger hill but it's a much more rare event we talked about rare events in the beginning and so the probabilities are reliable for more Probably but then we just don't have enough events at the higher probability and if you like verification.

[00:32:10]
This is the x. you know this is the equitable threat score and then down here is biased This is 25 millimeter gallon top 50 millimeter Halla bottom and the calibrated version was had hot much higher biased so you want to have a lower bias to the calibrated and the calibrated version at the lower bias to uncover it has a higher bias.

[00:32:28]
And it has a better Eccles for a score so it's looking good. But we were still unhappy with what it was doing and so what we tested in 2020 was actually to try to bring some of the physics into the model in the way that we 1st got to bring the physics into the models and so random force really doesn't have physics in it was to look at the data itself and make the observation that alle differs by season it differs by region and the training data is very very limited every time the weather service changes their model you have to start over because they don't retrain it historically they were just give us a new model from the from scratch really have a couple of years of healthy and so we wanted to use that data to maximize what we're doing and try to improve trust and tell what you get out of this graphic that's over on the right hand side is showing that how the hair was moving over time.

[00:33:15]
So what we did was we tried a couple of different responses and I'm not going to show them all here I'm just going to show one the one that we're looking at as we talk the idea of weighting the storms weighting the examples based on the month that they happen and so this is just a graphic of what the weights look like but this is probably the easier graphic to look at which is that you have all of the data available to run a forest but you wait if the proud the example such as if there were about getting the ones right that are more recent and so for May This is showing of the size of the dot and the darkness of the dot tell you how much it weighs its weight is for me you know you have more we did we don't have to we just did May June July and August.

[00:33:55]
For maybe have you know one Patterdale over in the upper Midwest June it moves to the further upper Midwest or why even further so in August you can see there's just not very many examples but if you want to see you know has actually compare you just created this thing how does that compare to reality this is from another paper isn't anywhere at all paper about climatology that means that they counted how often they feel happened in this is about 15 years of hail reports and they use the radar and they're just giving you a graphic of how the hail falls in the higher publishers are they more likely to see their regions and I thought back and forth between these 2 graphics.

[00:34:31]
But you can see is that our rating system research teaching time is finding the spatial patterns that also exist and if matching the criminology even though they have 15 years of data and we have 3 and so the question is Ok we just retrained our system we kept the calibration the regression system and all that that the ran for us you know how well did it work what works really well this year so this was tested in the hazardous weather test in the spring of 2020 and this is my favorite suite for this because I am circus with people who helps run a Storm Prediction Center and he said you know a I could be a game changer for this because he really liked how well they were and for us was pretty this calibrated enforcement predicting the hand and there will be we have you know more objective evaluations are coming out we have 2 papers that were working on it but they'll be more results coming soon and I did that fast enough that I could use an exception but I want to see if there are questions on how before I actually talk about Actually I for me to.

[00:35:35]
Now one of them has like of what and of quantitative evaluation do you use far I do you know such well just let me areas especially when you're using explanations like is there any quantitative evaluation Richey you know wanting metrics the Knesset's the metrics really depend on the domain that you're in go for the real work for example you know we were looking at the Go back to our objective those are quantitative evaluations here right because a little press corps is a score that they use inside the storm prediction center where I believe that games are a score that they use there isn't one single.

[00:36:15]
Trustworthy area metric it is actually something that we're supposed to work on and part of the institute is coming up with different metrics. But the metrics that we use right now are the ones that matter to the people that want to use the ad so if they measure it using one score than that's the score we're going to look at because that's the one they care about is if you give them some other score they're just not going to care Ok.

[00:36:36]
Thank you and one more from Robert Rebecca Don's And I think I believe you touched upon it briefly in the couple of flights back but regarding trust you said that you would collect information but you can be you and he was there's yes the general public and read this information back into the development of the Act So how will you do that well that's an excellent question.

[00:36:58]
I'm going to say this is I'm not the risk communication researcher I'm the director of the center I don't have an answer to that yet I would We're actually we're in the process of doing our site like it often in all of our meetings it's all virtual just like this is and there actually the risk mitigation researchers are reading the next talk in a week I'm going to defer to say I don't have the exact answers and I don't want to get it wrong I know they have plans to do the interviews and they're working with mental models and I think that that's about my level of details that they're understanding as I'm the ai researcher.

[00:37:32]
Ok sorry lot of how not to have you talk about how that will go into your area I guess well. Yes I mean that's part of the mental model work which I'm still learning about but mental models are you know based on what people are thinking about the thing and so what they.

[00:37:51]
Well I'm looking to use the feedback of Ok we didn't trust this because I mean I can go to our hill work in the in that sense that that's without the risk communication researchers because it's preliminary work but when they said we didn't trust this because it was too the probabilities were too high but I'm looking we're looking to look for things like that like here's your x. Ai method but it's not physically possible so what can you do to improve the physics are there so you can show us that it's actually going to work.

[00:38:16]
And that it's going to be based on what the end user say I mean I know one thing that's important to environmental science is users is how do things work in. An extremes to how does it not just extreme weather but like in days of extremes when it's really really hot does your model go crazy when it's really really cold is your model crazy though you know can we demonstrate that it works in extreme So those kinds of things as we do the interviews and as we show them how it works we're going to get that that human feedback and that's what we're going to use to improve the models but I don't have you know I think we've been of it we've been in existence for 3 weeks.

[00:38:50]
That's an excellent question and I will have better answers with more depth in probably 6 months. Thanks a lot here Ok I think that's are from Ok so I will. I end at 15250 right for you. I want to show you can give a bit more pain about.

[00:39:09]
That I just wanted to show some examples of actually I as well so there's that was sort of the trust and then books were 25 x. Ai and I can do it relatively quickly understands her questions on and this is just another example of how are looking at the axiom when I said we were talking about the exit at the beginning you know I said that there's 2 results I talk about more this is work for my students so that's why it's it's I can answer better questions on this because I understand this one better and I'm and I'm learning the other stuff.

[00:39:38]
So the question for the for the tornado prediction the research question we have was can we develop an ai model on the data available in real time so for a tornado forecasting and that on the day you know Bill in real time matters because it isn't so much that it would develop a model that you know it takes an hour to run that tells you whether or not there would've been a tornado an hour ago that's not useful we want to develop an e.i. method that the researchers are going to trust but that will improve for rate of forecasting in real time and I said the beginning tornado forecasting had really improved over the last decade so I grab some results mesmerism paper from Brooks and Korea and the hot one is China probably detection so if there's actually a tornado did you detect it and it's by scale of the tornado doga the larger scale tornadoes have a much higher probably the texture that has been improving but we're still not getting where we want to get especially for the lower scale tornadoes and then if you look at the lead time which is the 2nd graph this is like 985-2016 I think it's a 2700 when the because the papers are 2800 it doesn't change like we would like to increase that we time and we would like to decrease the false alarms and then we want to produce trustworthy guide to the end users and so the approach I want to talk about is the tree and the planning model and it uses the radar installations and the environmental Sonics.

[00:40:57]
So this is my diet graphic on c.n.n. and what data it took in we had them center of radar data. Across United States and we have 3 dimensional data which is actually really cool because it's not something that people are going to process in real time we have 4 different variables from the radar reflectivity spectrum with we're just sitting in divergence and for the computer scientists reflecting is basically a measure to pursue potations the spectrum with and of course a city in a divergence tell you different things about the wind and I will 3 make ups I'll leave that technical details for that earlier but but just basically you're taking a whole bunch of 3 dimensional data about the atmosphere in real time and then trying to project using a c n n whether or not it will that particular storm will predict it will produce a tornado in the next hour.

[00:41:47]
And so we have 12 different heights and because you know this graphic is already complicated not we use the same graphic we're trying to explain c.n.n. to audiences that probably have not seen deep learning or trying to show what what the convolution does to the graphics. It was can show you all top heights but we have all 12 heights for all 4 variables and then we also have something called environmental sampling and you'll see that in an example in a minute.

[00:42:11]
The data is pretty high resolution is one kilometer data point 23 by point 2 To point 022351 kilometer in the vertical and it is actually available online if other people get excited about the state of the graduate us that is available on mine. We have to process if they don't as I talked about with the previous you know what the hell do we have to process the data on this one we're taking deep learning so we're doing more of an image based approach but we still need to be able to understand you know when a storm produces a tornado when it doesn't produce a tornado and when you have just have a lot when you're effectively how do you know which blob actually produced it for me to go again we're looking at Watershed to to break those blobs into some cells and then we're tracking we have 5 minute data and so we're tracking this graphic is showing as we're tracking the storm every 5 minutes or time so that you can associate because storms change they grow they morph they disappear they merge you need to be able to say this is what one storm over time and then we also and this is an example of a proximity sounding It's if you're if you've ever seen the weather balloons go up it's what they're measuring the high though the wind and the temperature and the humidity the dew point as it is a measure of the function of the height in the atmosphere so they graph it as the pressure but it's just high in the atmosphere so it's those balloons that you see go up and they go up very very coarsely So we're talking about data and how data is available these are launched around the United States but in a very coarse sense so there are 2 of them thank you out in the in Oklahoma for example and then they will do them twice a day in Leicester during a special one and if it is special and they've done 3 terms of a so they're a temporally and spatially course but they're still very useful data.

[00:43:53]
So now in the beginning I was talking about how data is a rare event and tornadoes are very very rare events even if you break the tornado into. Each individual 5 minute segment you still have a lot of tornadoes our training set for 2000 fold to 2014 has 353575 so if we don't have better survive tornadic storms and 40000 non-symmetric stops or validation data pretty similar to chose 2011 as a testing data because 2000 was kind of a bumper crop for tornadoes.

[00:44:24]
But what we had to do so that the model would learn anything was to do data augmentation and this brings us back to the physics space needs because you can't just brand only 2 data augmentation this data means something physically and so we did a little bit of translation where you translate that into you know north or south or east or west but then we did a little bit of rotation but when you rotate a storm physically you're doing something to what's happening in the atmosphere and so you can't just take a storm image and flip it it doesn't work that way then you have a storm that's not physically possible and so you're training your network on something that's not actually useful to look at just minor rotations as it turns out plus or minus 15 degrees was possible but if you get any more than that then you've got something that just couldn't prove he created the atmosphere and so it wasn't actually useful to train him and then finally he added random noise to the data but again you have to pay attention to how you're adding that random noise so you don't make something that's completely unrealistic we did find by the way that adding all of the state of the nation definitely improved the robustness of our the ever model.

[00:45:24]
The how well does it work. I was asked about metrics a minute ago objective metrics these are some examples of some objective metrics and then you know the question a few months ago was what matters easing and I said that it depends on what the forecasters care about everybody cares about something different that's the interesting thing there by 40 different metrics out there for atmospheric science.

[00:45:42]
This one on what inside is an r.c. curve which is a typical one is cna Asir looking for the area under the curve and the area under this curve is excellent or in the point $13.00 range and that's without bootstrap or sampling so you can see confidence intervals and it's good.

[00:45:59]
This is probably less of a common diagram for ai it's a performance diagram and so instead of an a you see where you want to be perfect by going up to the upper left this when you want to go up the upper right. But the area you would still want the area to be one and so this is still a really high score for tornadoes and no point one for one on an a.d.c. diagram will be considered terrible not terrible all over here it's actually a really good score and then you also want this guy Graham also could tell you how biased you are in our data are the nice thing about this is what we are very on.

[00:46:30]
Because this is the one point on the way was excellent. And in the name of time I'm going to skip this part I'll skip this part just to say that it works well is that we did an evaluation of our hours and then we didn't have a reason for space which I will talk about briefly because I think it's interesting that this is a the upper left has the number of examples of tornadoes.

[00:46:53]
And a number of all examples number of examples of tornadoes and we have 4 different performance metrics the area under the curve which is the one I just showed you yes I would choose one that forecasters use ability to tax and which is whether or not you actually forecast a tornado if it actually happened in the false alarm rate which you'd like to be loved so you want people to be high replacement rate to be well and what we got out of this is to say that we you know we have very nice AC It's very strong the c.s.i. is pretty decent into the c.s.i. is never super high on for Nato's but it is worse in areas where there's fewer tornadoes which is not particularly much of a surprise.

[00:47:28]
But the key part that I want to talk about this to bring us back into the trust for the ai is to talk about how we're actually digging into what the model actually learned and I'm sure that others who are actually safe ecstacy fans like me have probably seen that this cartoon but you know I like this cartoon because this is the public's view of what machine learning is you know it shows up on the news Ai machine running I had a news interviewer a couple days grasping what I was and wasn't going to be Skynet wasn't going to tell all of us and I'm like No I had to explain that you know people think Ok well this is your machine running system yeah you just pour the data into this big pile and you're out to bring collect the answers on the other side what if the answers are wrong disturb the pilot so they start looking right and have not really what we're doing well looking into the model we're looking to a Model 2 different ways.

[00:48:15]
And so and I swear I'll end it like 3 minutes because I'm trying to be on time just want to tell you sort of 2 different ways to look inside the model we have a whole bunch of ways in a bunch of papers on this but I want to show you the 2 1st ones the 1st one is staring at the maps does it just basically tells you the gradient of the model activation with respect to some input value the we're looking at.

[00:48:35]
You know what is most important to predicting an actual tornado and we're going back to those graphics of different levels of the reflectivity and the cool thing is that there's. We can see things that are very very physically possible so it tells us that our model is learning something is physically possible the probability it's radio increases.

[00:48:52]
As you have a stronger rotation or just the as a measure of instantaneous circuit rotation those strong rotation in the lower levels it's increasing of the spectrum which which has to do with the spread of the winds it's increasing as the storm grows in height because the more severe storms are higher in the atmosphere.

[00:49:08]
And those are all really nice and physically possible so it's nice to be able to see those to confirm. And then the Secondly that we looked at it is called Backwards optimization or feature optimization and the idea of this is that you can take that you can create some sort of synthetic input example and you can use that synthetic and what example to.

[00:49:29]
Use to learn what it is that maximizes your model or minimize your model and what we did was we took her magic examples and we tried to turn them into non-phonetic examples so given a tornado a traumatic example what would the model have to do to that example to make a non-swimmer and it took on it we did this over 100 storms 100 best hits and it took the probability from 99.2 percent to 6.9 percent and here is the original this is the track storms this is what it created and what's interesting is it doesn't visually look like it changed all that much what it did do was it severely decreased the height of the storm so it made the storm less severe at the higher part of the altitude and then.

[00:50:12]
If you get down to the sounding and this is where it gets really interesting physically because you get down to the sounding any say here's the original sounding which is you know what does it look like well it's a little bit different that this version that I showed you was when we had physical constraints in this version was when we didn't have any physical constraints and if I just flip back and forth you see how jacket that something is you may or may not know what a something should look like but they should look something like this tooth this is not what it should look like and so when you try to show a forecast or emergency manager or anybody a scientist a model that learns and he say this is the optimal non-traumatic storm they look at that and they say asked on and on for magic storm that's junk and so it's a it's a call for trying to make a better version of this is our 1st start and you know this is an improved that was opposition with physical constraints built in but we need to continue to improve.

[00:51:05]
I think that's it until the end on time so I could be questions those are my usual go it is I will leave that out for a minute because those are some papers that you can find more. Great thank you Amy program talk of so if you have another question from Clinton call asks the your model predict hurricane chances scale and direction it heads and use any of the hearings start our weekend tornado throughout prostates to predict the extent of climate change and find the factors which explain our effect climate change Ok that's a big question.

[00:51:46]
Let me if you want me to repeat it for now now I'm just going to stop sharing and I think to see you and I can actually see questions to. Ok only in certain halves We're not predicting where that were not particularly focused right now at least on where hurricanes are going because the hurricane track prediction is already really really good the National Hurricane Center is really good at their job.

[00:52:10]
The hurricane part that we're looking at is trying to understand the Science of Hurricanes trying to build my ex a I models in particular that will that will improve our understanding of the structure of the hurricane so we can improve the prediction of whether it is going to intensify quickly.

[00:52:23]
Ok that was happy question there was another have the question was about climate change and hurricanes and hail and hurricanes it's writing a moment right can you use any hailstorm or you can draw needles through the prostitutes to predict the extinct had our limit change and find keep factors or not so how much climate is changing using the hurricanes and I feel so odd looking at all the loss immigrants yes.

[00:52:49]
Yes I think you can it's a big debate in the community exactly because climate and weather are different rates so we all agreement climate is changing we all agree there's an impact on the weather but figuring out exactly what caused that particular tornado or that particular hailstorm or that particular hurricane is awfully hard thing that that was caused by climate change is hard.

[00:53:11]
So and that's not something that we were planning to focus on we're planning more on the fundamental research of just trying to understand even the given the storms here how can you improve its understanding of that storm. I'm not saying that's not important and long term yes I would like to do that but you know right now within the scope of our funding $20000000.00 is a lot of money but on the other hand you know it only goes so far.

[00:53:36]
I guess I have another question here like. You have your course of your research and you're right your approach is have you found a situation where the machine learning performance is very good so you've got somebody you see but but somehow it is not good for. Our going exploits right maybe something missing but yeah.

[00:53:59]
I wouldn't call it something missing but the 1st person in our Hale research all the objective measures were beautiful and they didn't like it you know so the end users didn't like it because it wasn't fitting there that what they wanted and their needs and I've had other people tell me Well why don't you just tell them to adapt and learn to use the model make That's not the point they've been doing it their way and another end user might want something different right we need to develop what they need but you know objective way it was scoring pretty well it's doing better now by the way but you know even then it was doing well.

[00:54:30]
That the tornado work might fit into that you right now it's ready to work I said that our goal is to do this in real time but we haven't fielded it real time we actually just got funding to do that so after that year but you know objectively this we need to work works really really good though right whether or not it's going to work with the actual forecasters I don't know yet but that women know that you know in addition to our a.i.a. center we also got funding just to operationalize that product so Noah has a a program called the joint technology transfer initiative and we just got from let's say operationalize that that's wanted a product to will be able to know whether or not it really needs the ends and user needs but that farming started September 1st as well so.

[00:55:09]
I don't have an operational as our goal is to test it within a year right so b.b.l. another question not by you would giving that answer is the national center model based on machine learning as well the internationally I center Yeah certainly we're using ai is the products are starting and I think I don't know but they're going to mend that are probably the.

[00:55:34]
I think the probably he meant. The no OS until right now that is based on. Using machine then he asked what I think the national model I guess. Well I was maybe going to going to specify a book yet because there's a no a I center to Ok on doesn't mean I come back to that if Conan explains what he or she had in mind so region tomorrow is awesome question which is a follow up to what I had as what did you do great Dr Martin.

[00:56:05]
In the suit the end user b. I guess you have given a couple of examples if you want about right I mean I've got a couple of examples one received the probabilities of what the tornado or what we ended up doing was adding physical constraints at the beginning of the end user needs was just sharing them here's what the model learned them saying that's not physically possible so we're trying to put physics into the model to make sure that it can learn some things physically implausible it's going to get deeper than that I mean that's the beginning of it.

[00:56:34]
Right. There was another question Alexander Dobie Gillis who asks I wonder of data from other countries as relevant for creating memories in these opposition. I think cell I mean I've got a student is just starting to work on predicting our convective initiation which is the start of summer storms and the data he's using is global I mean I don't see any reason why we can't use global data as just the examples that I have the data that we have available to us is from now and no it is only the United States Ok yeah so are coming back to Conan's question so.

[00:57:07]
I have a mind to go up yeah no offered Yes Yes So there is a no way I center and I'm I'm involved in it peripherally as in I'm working with a lot of the researchers many of whom are working on starting that up but arsonist that's under there is going to be more focused on there's a very different model it's a it's a model for it's like a dispersed center and it's going to be workforce retraining and you know helping to provide expertise on Ai throughout Noah but they're going to be working with us as well we're actually in the middle of trying to sign the I know the the lawyer level agreements so that they can actually work with us as well.

[00:57:41]
Right I mean that's that's that's all the questions that are here thanks again Amy for party time and for the good doctor and I'm sure you can hear you really clapping that he's like thank you thank you for giving the take I think you are by everyone.