[00:00:05] >> Introduce Dr David Womble who's the director of artificial intelligence programs at Oak Ridge National Laboratory before joining Oakridge in 2017 he spent 30 years at Sandia National Laboratory in both management and technical roles as a manager he served as the program deputy for advance simulation in computing or C. responsible for developing. [00:00:34] And deploying models and simulation capabilities including hardware and software in support of say India's nuclear weapons program David served as a senior manager for the computational simulation group and for the Computer Science and Mathematics group he earned his Ph D. in M.S. ph even applied mathematics and in electrical engineering from Georgia Tech in 1906 his bachelor's came from Rose Holeman Institute of Technology in 1902. [00:01:11] His interests include high performance computing with contributions in numerical mathematics linear solvers scale over them so I'll. He work closely with our ease atmosphere to electrons program for the past 2 years and his recognitions include 2 R. and D. 100 awards and the Gordon Bell award in high performance computing and actually. [00:01:40] There is no such thing as a free lunch now you have to pay for it by listening to me I did note however that I'm opening for I guess Dana and Peter and I know you know you think as a speaker when you have the opening act for for Peter. [00:01:55] But. Now when Tom asked me to do this I didn't have a lot of time to prepare a few graphs I warned him he's going to get. Kind of an old presentation here I've already see things on it as also look at this morning that I want to change. [00:02:09] In advance for that but if I do my job to kind of raise you to fever pitch for the following talks and you'll remember what I said so it will be good to go there this is more problematic in nature letting you know what we're doing at Oakridge in order Fishel intelligence programs and lastly a graph or 2 will be a shameless plug for recruiting and things like that I couldn't pass up the opportunity. [00:02:34] So. I really want to start with why we're doing AI and. What and how we're going about doing it the question I got over launch was why Oakridge given everything that's going on at Google Amazon Facebook industries start ups universities what's Oakridge is role in this and so I always start with a few quotes here adapted from various sources I don't have to read on that will replace a scientist. [00:03:03] But will have to use it in some sense this gets at one of the fundamental aspects of science and that is the experiment in collecting data what do you do with that data the people who make the most effective use of that data are going to be the best scientists. [00:03:17] There's another one here we're in a arms race and that kind of gets at this perception and this fear actually that that we don't do this well we're going to fall behind whether the national security sense or in a scientific sense with other companies and it's only made worse by the fact that the Chinese for example have announced their intent to be you know actually they say a leader. [00:03:39] Interpretation of that is the leader and it gets a certain political reaction from the country as well so there are reasons to do this. Why now and why Oakridge Well 1st of all they just plentiful I mean we can collect I mean that camera back there is collecting. [00:03:56] Well if it's measured on contact a couple kilo bytes of data measured by you know. Measured by actual pixels it's going to be a few gigabytes of data or something like that and it's plentiful you can collect lots and lots of data and there are computers in every car you can record everything you're doing on Stars and get right back to G.M. If you have On Star turn on your car you are the you're the product here and finally well not funny but computing is ubiquitous I mean we have a tie some it is the smartest machine in the world but but computing is everywhere and finally as accessible and high school student with a laptop and an internet connection can get all the data they want can get all the computing they need they can do their own weather prediction models they can do whatever they want sports prediction betting you know you name it and a high school student has done it at home in or you know in their parents' garage so very accessible and what does that mean for for science and for national security mission and things like that white Oakridge Well certainly we have this mission in science and national security but we also have facilities dealy is very very good at large facilities much better than say in S.F.. [00:05:06] Those produce lots of data and they give us a kind of a unique scientific role that these very naturally to how do we use that data and. So now as I'm thinking strategically about what Oakridge should be doing what I did is kind of separate taxonomy I don't generally like taxonomies but I found this useful in setting strategic priorities I wanted at least put it up here the 1st one is classification regression and that's your basic find a tumour. [00:05:34] Image segmentation this case find defects I'll talk a little bit more about that we've discussed a lot of attention this is telling the difference between cats and dogs in in pictures on the web. Surrogate's is one that has actually found very good use very immediate use very successfully used in science and that is especially when across scale we have very expensive. [00:05:57] Modeling and models for evaluation modeling a simulation that has justified at the scale computing for for years but even so too expensive to compute can we train an AI to provide provide a surrogate in this case looking at turbulent flow into combustion is one area that's been very very successful there the next one is the inverse problem what we're doing here is we're looking at inverse material atomistic structures now talk a little bit more about that so that gives you. [00:06:34] I mean there are parallels to optimization and inversion in the modeling the simulation world and finally control system and this is another one that gets a lot of attention with the autonomous vehicle there. And that's a whole different application area with a whole different set of constraints it's not just autonomous vehicles but control is everywhere what are you willing to do in terms of a nuclear power plant for for example I'm going to cover just a couple examples of what we're doing here at Oakridge the 1st one in fact through the 1st 3 of you graphs here are from material science and we have our center for now face materials science we have a lot of electron microscopes there the S.T.M. machines a few layers thick atoms and look for defects now when you're designing parts you design around book properties but the defects of what actually controls the performance as parts get smaller that becomes more and more critical and in things like the additive manufacturing we have to understand the effects we have to locate I'm looking here a silicon in graphene. [00:07:39] And we can we can take measurements we have lots of things like this the question is where the defects and what are the material properties that result from those defects then you can of course have somebody look at images this is already been labeled essentially with a defect but looking at images but finding deep learning in order to do that the question is now how do you apply physical law physical principles to that so you don't want is just not sufficient to grab a new thing that looks like a defect you got to make sure it's physically realizable and so what you do is you build up you know libraries that you can then write your defects in terms of a in some sense a basis set fine that we have not finally but you can calculate electronic structure calculations and from there to the. [00:08:28] To the electrical properties of whatever properties we want. In terms of surrogates this is the next step here is to replace D.F.T. with a surrogate type calculation of. Course any time that you deal with physical data or most other things you've got to deal with noise the question is how do these perform in the presence of noise we have empirical evidence we don't have a theory around that you have to have empirical evidence that in fact we are well within the links of these materials and so that the data that the accuracy of our electron microscopy gives us good data for for defect detection and and creating that library here inverse problems so given a whole bunch of images that look like this can you find identify the structure including steps and things like this defects in interfaces interfaces are very interesting a lot of material sciences around the interface structure right now in this case it's meant to live in a strong in tight nation. [00:09:37] To insulators with an electron gas at the interface is conductive. Very interesting design options and things like that but you've got to really understand that that interface is so you've got a forward problem here going from the images to the structure classification the next step is to turn that around. [00:09:57] Run it through a continuum modeling the simulation code for example and then compare your with the with the data and iterate on that until you until you have your structure identified so kind of a classic optimization loop bug but using an AI in there instead of some of the traditional modeling and simulation so now one of the things to note is that these. [00:10:22] Talk just a minute about the AI that comes out of here the. The image classification this was done actually with the hyper parameters the design of this network was done with it's our local hyper parameter optimization. Tool is interesting that this is come out with something that looks very very much like an Alex net type of. [00:10:44] Neural Network. Maybe that's Imperial evidence for the quality of Alex now why a success for something like this but strictly empirical. But but again even when we optimize the network we come up with something about the same number of layers about the same size on the same construction as the as Alex net. [00:11:04] There's this is a really interesting. Problem here we funded it out of our initiative in F Y 18 but the goal is kind of just completely different from anything I've talked about before it is can you build on a Mims device using materials water in oil in this case with setting up ion channels in a contact coupled with silicon based hardware for a bio inspired spiking neural network in your morphic computing we've designed such a device our test problem here was discriminating between. [00:11:44] Epileptic seizure data and non-epileptic epilepsy data and we can do that now we've done the design with our code. Evolution or after position for your morphic system but designed a network with 66 bio inspired neurons here we can actually build up to 30 on a single device to 6 is enough to distinguish between these signals here the nice thing is these are very very low power so we're talking nano watts of material something you can get out of basically a bio generated energy the question is can we now turn this into a predictive circuit so that you're actually a time prediction for the type detecting the unset and the epileptic seizure so there's a picture of the actual device down here and the way they construct it is you know they have this interface they put the. [00:12:36] Added to the manufacture put the drop down here and then roll down into contact and that's how you get your synapse OK So some examples of how we're using. In material science space of course we have a lot of applications in. Energy Security and things like this but damn But what are we actually doing what does our AI initiative look like. [00:13:06] Certainly. AI is very applications driven you've got to have lots and lots of data that ties a very tightly to the application in a way that some of the traditional thing relation may not be. Down in the AI world you have lots and lots of things you could do what is it that sits in our Mission Space that distinguishes us from google I when I talk to our lab director and he says well how are you going to hire people I say I want to build a reputation so when I go up to somebody and say Hi I'm Dave not from Google they say they want to talk to me because of that not despite that a long way from that point yet but we'll keep keep trying so within our mission space what we have chosen here are 3 areas and actually I. [00:13:49] Got left with an application and 2 starting driver applications that drive this what we get out of it we send back into the wide range of applications the 3 are areas in AI research that we've chosen our physics informed learning and more specifically I'll talk about these in just slightly more detail reinforcement learning transfer learning representation learning scalability how do we use these big machines big facilities big data everything else that said was part of the Oakridge mission very important and finally confidence and confidence here today are business critical for good science but for but also how we're going to to use these things I'll talk a little bit about that as well scale ability 1st. [00:14:39] I think everybody has probably heard that were we House summit and I.B.M. machine biggest machine in the world has been advertised as the smartest machine in the world is probably the most complex accelerated know that's in production I mean through production at this scale only because I just slept a lot of stuff together and that presents a set of challenges here but the really interesting thing actually is that the G.P. use here. [00:15:05] Our. G.P. use they have even though we designed the machine for 200 put a flops and spectate out and got everything based on this number it turns out that it will rolling to use reduced precision mics precision things like this that we've got a 3 X. machine and suddenly we have a lot more options for for using these machines it especially in the context of learning now instead of vectorizing code like we did in 1975 we we now temporize our codes are very mention mental it's evolutionary genetic algorithms run lots of possible networks through in that sense from the standpoint of the scale of some that is easily paralyze a ball which makes it also one side effect of that is also makes a very good for studying single node performance and so what we did is we're working on some of our Gordon Bell runs is to turn the switch basically from exposition on and off and start playing with that possibility. [00:16:12] And we get a completely different distribution here single precision well double precision is going to be much more to the left over here a picked up single precision is out makes precision moves out and so if you can operate in the tail out here suddenly you're instead of getting one to 2 teraflops on a on a node you're out in the 10 to 15 teraflops you have to turn that around now and ask what does that do to your neural network is it going to change the learning rate the convergence rate or or the quality of the answer and so we have a whole other axis here in which we have to look at and that is correlating the performance of the machine to the actor see that we want out of the neural network accuracy of the of the training. [00:16:56] Good infrastructure is required I mean this is. Preaching to the choir here but I just want to emphasize that it's not just about the big machine that you have to think through the entire work flow where to get the data how is it curated How is it transferred stored. [00:17:11] How they're brought into the machine itself for the learning learning aspects of it and finally you have to get to a deployment stage computing at the edge and so on and then you have to close the loop once you're doing that computing at the edge of collecting data in the vehicle how do you get it back into an overall learning learning system from how much time do I have left. [00:17:41] OK. That's one minute per view graph. OK So 1st of all. A little bit on the on the learning one the problem that we have the scientific world is that data is very limited it's often poor quality despite the fact that said electron microscopes produce data with noise profiles that we can handle the fact is that a lot of the data is a poor quality it has very very large uncertainties it's on labeled We don't have what we don't have a crowd sourced labeling that Google uses you know the find all the signs in this picture in order to get to this website but. [00:18:26] So label data is hard and we had in the physical constraints how can we use that. Adversarial networks and reinforcement learning or the 2 ways that were approaching approaching that in terms of learning itself. The concept of the. Residual network is a good starting point and that by itself is good for a lot of things but it also captures the idea that if you're inventing one network in another adding those extra layers should not theoretically do worse and so building it up and that can they do with options in in transfer learning multi-modal learning and things like this if you extend that concept the way we're extending initially is to draw a parallel between that and it's a classic optimal control problems that we hope will accelerate learning and remove actually some of the hyper parameters. [00:19:22] That you know how many hyper parameters we need to deal with in the in the code knowledge representation is key and key in many areas in some sense what you're trying to do is go from a syntactic space borrowing terminology from the natural language processing world where what's important is order and grouping sentences paragraphs and things like this close means that words are next to each other into a semantic space where close means they are close in meaning and those are 2 very different things but how do you do that representation and these are some pictures I need to take out I actually have a story around those I'm a skip it here but I'm a come down and I need to develop I need to develop this idea a little bit more of our focus here is on graph based representations ontologies and graphics rate based representations for the. [00:20:14] For the data so many physical things are represented naturally in that way you have nodes objects things you have properties you have relationships that lend itself to a graph based structure and this is area is actually pop in popular over the last year or so confidence and certainty quantification. [00:20:34] Moving into the 3rd area. You queue is hard it's expensive you have additional forms of uncertainty in these in these networks we're developing here the Basie and variational inference methods. But I think we're going to have to look for more creative ways simply because of the computational cost even of these machines inherent in these methods validation explained ability and reproducibility Well the things I often have to do is explain the difference between them and as I explained it this way. [00:21:10] Validation is is the data the model the AI that I have appropriate for the decisions I have to make if I'm in a car what's going to give me the confidence that's not going to kill a bicyclist in Arizona OK So that's the validation explain ability says once I've killed that by slick bicyclists can you tell me why. [00:21:36] And actually that this of this is a somewhat validation are explainable is actually very important in medicine the treatment that you get depends on being able to act accurately. Explain to a human why. You were the cause or why that disease actually happened you know why would you treat all lung cancers as if they're a smoker they're not all smokers for example OK so explain ability and then this also ties into interpretability and that gets into the human computer interface and we had talked earlier today just emphasizing how important that is and and it is indeed very important I put this on a couple of axes in which interpretability is opposite of complexity humans tend to think in terms of you know local plexi high the interpretable and. [00:22:29] We the AI's are very natural in the high complexity models that we find very difficult to interpret how can we. Adjust that curve on that graph. Observation here is explain ability tends to be close to the interpretability here verification validation has to deal with the complexity on this axis so they're very complementary ideas and then of course you can pull this into the quality of science issues around reproducibility and things like this some observations that issues these are some fundamental issues that kind of guide our research in the areas that we have the for what is it they do quality is easy data or quantity is easy quality is hard and that has a lot of implications. [00:23:20] Validation validation validation and explain ability if you are going to rely on an AI to make a military decision. Do you trust it. What's even scarier is your opponent is relying on an AI that you don't trust are they libel to do something unpredictable because they're relying on an AI So there's a lot of military implications to the statement as well I think the military itself is going a lot more to the human organization you know augment the pilot on the ship so it doesn't hit someone in the South China Sea rather than making the actual decisions because they haven't gotten to the validation requirement and also gives you some ideas about how to relax that and crew validation maybe just controlling false positives and false negatives might be sufficient for for some uses in the various areas resilience and robustness are critical. [00:24:14] Highly optimized systems are such destruction vulnerability threats we all know we've seen examples in the literature where you can add noise to an image or put a piece of paper on a stop sign and it fundamentally changes the classification how do you deal with that most of what's out there is basically a kernel method smoothing and things like this you just add enough noise to the system that it kind of messes up your ability to. [00:24:37] Distinguish but. You do something there. And I may change how we do science book and also cover up bad science. You know this whole thing that we causation or correlation is not causation is I mean we all believe that we've all heard that and we can even find examples of it but in fact AI is making it easier and easier to confuse that correlation I found these. [00:25:04] On the table at lunch. And you know here I am at a crunch time and with a crunch bar there's clearly a correlation there somewhere but what's the causation. So my I will find this and that that's important you know as we get to this kind of fairness and equality stuff you know the math weapons of mass destruction. [00:25:24] And so on but what experiments could i designed to test that for the cost of relationship there or not so design of experiment is one that you need to think about here instead of others and by the way I took 2 of them talk strengthen the correlation does that make it more cause if I don't know but we will see. [00:25:47] Ethics and human impact. Enough said there is not just. If you're in a time this vehicle doesn't kill the the old the senior citizen the child or or your or the driver you know which ones they kill that's kind of the prototypical problem here I think it's actually a lot more interesting than that problem. [00:26:11] So by that scale ability but it's not just the single run you have to put in the context the entire look workflow how you know what the quality is. We need to manage expectations there is a pinch with a period of about 20 years you know 1950 S. it was all about 970 S. It's all about a 1990 S. it was all about AI and that's where you know you'll be done Kasparov in chess $2015.00 it's all about AI and it's actually winning and go now it's exactly a 20 year period. [00:26:44] And right now the expectations are very very high and I think from a if nothing else from a political standpoint we're going to get burned if we continue to let those expectations run high. I'm on the other hand machine learning those That's the consistent vestment that's happened kind of the academic continuing. [00:27:01] Investment and you see the kind of the normal progress you know this kind of exponential growth in progress and capability and that underlies the AI So this is kind of tension back there that we need to resolve from a political standpoint if nothing else human computer interface is important and I as an art and then it gets it how we work with a lot of practitioners right now what do you do you download intensive load you play around you some templates for different networks and you kind of make it work and you say That looks pretty good you download your psych it learn to Tory all and you follow you know should I be doing clustering should I be doing you know whatever and. [00:27:41] You've got to get past that and you can only get past that if you really understand some of the fundamentals so I promised a shameless plug for recruiting so there's any graduate students here who will be looking for a job or hiring everybody else is by the way but we but we are to. [00:27:59] And we're hiring broadly we're trying in some of the core areas you know the statistics math computer science not just people who have. Demonstrated the ability to use an AI but really getting at some of those fundamentals Oakridge is a national laboratory with some really. Leadership capabilities and if you're not ready to graduate or looking for a job yet still think about our summer institute or trying it here with the team based approach working it's a very interesting problems. [00:28:26] And we're working very hard now to find a right problem in the scientific community for these teams to to work on and. With that I'll breathe i think i talk too fast. You can't. Know so this is an this is actually a key to the surrogates one of the things that we have in P.T. baseball in the simulation is this concept of predictive it like it is considered 1st principles and over a period of centuries we've developed a theory that we think is extended well beyond the areas in which we have collected or observe the question is how far becomes and it leaves these questions like I'm am I going to Inter plate am I going to extrapolate and things like this in a I we're only training on a subset of data and so that leads to a different set of you know where is that fit happening essentially we're fitting a landscape to this idea valuate then make that surrogate and you've got to ask questions of my overfitting in my you know what's the appropriate model form and things like this but you can only capture what's in your What's in your data you know limited by the by the measurement uncertainties and things like this so you've got to be very very careful about how you use these surrogates in fact if you can characterize the uncertainties you know one of the projects we have is characterized uncertainties in ways that tell us when we revert to so-called 1st principles or predictive simulations of the very sophisticated sophisticated way to think about it but but you have to be different careful of different things and you have to. [00:30:15] You kind of redefine the whole concept of predictive It is no longer you know 1st principles from the standpoint if I go perhaps one lower scale lower level and scale and solve the equations there and then feed it forward that have a different concept of predicted that as an exist for the AI's if we've been doing circuits right now so there's a more complete answer. [00:30:39] Thank you. OK I guess I'm supposed to. Wait here. Yes OK. I'm going to take you. Now yeah you got the crunch part if you know. I actually had a professor here add George attack who guy named Cain in back in the 980 S. and mathematics who insisted that if you did something really interesting during a lecture they said and so it was like if you want talk of the chain rule you stand up on the chairs. [00:31:31] And they'll remember the chain rule because the thing on a chair so. The only thing you're going to remember that my talk is to crunch bars Yes OK.