[00:00:08] >> holes to this is was will pull thank you thing is things key for a introduction to the mike it's working good so at to they stock will be overly view of the work that we do and now a group and focusing on to area security was security in the chair put away i and that is main theme off what my group does and i like to name things have to myself so it might get was called a polo cup of data science which a come by artificial intelligence all kind of the best from best side and also the human intelligence him in a shy if you if for short and to develop scalable into active tools to make sense off complex data and large scale model and i'll other what that i've resigned work done by my students though the current mine up at the bottom solo need to thank them 1st snow and then i'll claim autocrat it afterwards so and so we have to a number of a area as i work at it in my group and humus enter ai's won the where makings sure i'd up people in understand what's all these latest development in machine learning cybersecurity has been now i mean application area fall on tight and since the stay sick people do care about security not only for now the traditional sense but also 1st machine earning so which is why now we also said the episode on machine your earning and what you were see some examples for we need a tech and defense machine or inning and we also have been working a lot on the graphs so a law grass mining and also visualize sation and also in the application area off social good and hell but for today we're going to focus on 2 main areas secure ai all the thing of that it more of a several machine earning more gentle and interpret so inter bit about yeah so i'm asked why these topics so why focus on them and how they relate at so they have very interesting and connection so put a secure a eyes side i probably i don't need to convince you to much you already seen some unfortunates How now I can be used in safety critical application if you don't do it well then you can kill people so make sure things don't break this important you may say well stay for the critical What about every day life so every day life now we're seeing a security problem and everywhere so there's a cartoon there is says that well in the future we have the smart toaster that may go haywire and turn out that we do have that now smart toaster to exit so can you can get one right now as security is an increasingly big problem so that only going to be even more problems like that so it's great so people working on AI security and say well use machine learning and so on but the current state is that for a lot of things that we develop a special you say I have a tech technique so we had the model or you have a defense technique. [00:02:52] Although most of the time we don't really know what happens often is because we use the machine learning black box and not really good so I believe what we do we want to do is to probably open it up a little bit maybe not completely make it transparent but at least know a little bit about what is happening so that when it doesn't work we know how to fix it so why the bricks we know how to fix it so our goal is to develop these interactive scalable and use the usable techniques to help people understand complex machinery models so today's agenda will cover some of the present some of the latest work that we do in our group to area and also we didn't make the connection among among them as we go so on the security of the examples of some of the some of the latest attack and defense technique and also if you are interested in trying those out we'll give you some examples like how you do it we all do it yourself everywhere machinery and then on that interpret the boy I will talk about some of the latest work for example work that we developed with Facebook understanding last industry industry scale model and also with Google we developed a tool to help people understand gang games and one of the latest learning techniques. [00:04:06] And also want to conclude with the summary of the current research landscape I would be interested in working in both areas where we might want to go next so we'll start with the security eyesight and understand their goal is to study the machine learning vulnerabilities and so that we develop something that's more secure more state and I'll give you some examples A into category one can defend technique. [00:04:35] And then also how to make it easier for you to try them do it yourself so we start with the 1st one. And often when we talk about secure security we need to think about both tech and defense and tech is easier in general that just being the bad guys it's easier to be in a good guise of being a good guy you just break things ones and that's IT systems is broken as a good guy you kind of protect against every possible kind of. [00:05:00] Nobilities So if there isn't a tech techniques you may ask why we're starting a tech techniques because you want to have strong defenses you need to think like the back end so that's why we also want to be back in a sense so think about how to break things so shift here is the 1st we call targets of physical office Aerotech detection so a lot of technical term there what does that mean so 1st it understand what what it's really up to the tech to I while we are checking it so you might have seen a lot of examples already about what image costs that cation as a given a make you get a label in this case what image of like this to a part of say with an image of a car. [00:05:37] But then. You are really similar a little bit and maybe I have a lot of more content in your images and you realize alone may be a. Good description for it so that a lot of things happening so a lot of objects in it as the people there there's a building that's even more cars so why would you label it just using a car so that's that the French in there if you're using up that detection technique you are trying to extract localised what other things in it and each of the things you label what it is so so. [00:06:09] Is to protect these kind of up to detection so that it's not really turning and a label into wrong label but now we're actually doing something to the individual that has turned out to be harder. And whether you are doing it because of the occasional up attacked or now we can pretty confidently say that all of these are actually pretty vulnerable especially these days when using deep learning network very vulnerable so you probably already seen those from popular news where the image image is that the image of a stop sign you can change the pixels in ways that you can human being can really see change a little bit darker to pixel dock a few pixels lighter then you can make it because as whatever you want so benign the stop sign but after some changes and to change whatever you want so in this case we can change it into a stop sign the Messiah so I can be anything very very easy so anyone want to do it. [00:07:04] But however often we assume the bad guys have access to everything almost we call the white box attack as they know it all the details all model which is why you can really optimize exactly what to change how much to change we also call this a digital logic but in practice you'll see that while that may not be a practical threat model as in maybe too many assumptions about it so for example as in if you say you are building a child driving a car is that foreign car you want a car to recognize is that a stop sign is a pedestrian and so on so we can think about how to build this system you or your car in to capture what's happening and on the road and then you do some processing within the cars use it to use and then did a recognition. [00:07:47] For and then figure all that's like a stop sign so currently all those a lot of research that assumption is that they have access to access to that whole system so that means that the whole thing you have access and then you can launch this attack to determine what to what thing to protect so. [00:08:05] In practice we know that is quite hard now you can hack into the system and then do all these then maybe you don't we then need to do to all the perturbation you can just do a lot of things you can just crash a car if you want so so that's the limitation. [00:08:22] Or so we're thinking of so instead of making a lot of sunshine What if our tech is about manipulating the physical world where we don't really need to hack into a car but then we can easily let's say put something on the going to subside or maybe replace it with in a fake one but then we put in the same place so what would happen so that is what we call the physical physically realizable effort Sarah and as a shape shifter as about so we think it's more realistic and also we can launch both target and also so before going to detail I want to show you some video of how that works so this is an example of tech so I'm not a student and they. [00:09:02] Are. Not here so he's he's the person who drove the car so the video that you going to see so if you're going to car is on the top of a parking lot here in Georgia Tech and you'll see that he put some sign on the left side and the face stop sign on the right inside and her drive to it's the car besides the site you'll see that the left one the real one consistently get. [00:09:26] Recognized as a real thing and then the fake one you don't really get to get any recognition and it would have been closer than to become any person person person person person so this possible living you can turn something into in that person so we show you a few more examples of that so that you get an idea that we're talking about detector vocalisation and so on so actually there have been. [00:09:50] Work on doing physical attacks and a lot of time they're focusing on image classified for example you may have seen they putting stickers on stop sign but that's not taking out the detector is turning the whole image label in so doing to local isolation and the tech and looking at local isolation so may say well what about we apply these to a tech technique that works for a middle class that fire can afford to detect or in turn no no so earlier work that no that's not once you do the technique applied a technique that works really well for it may cause a fire at the detector at the detector and can recover that series itself and correct person it's correct. [00:10:28] So how does our technique or so specifically we have tech the to talk to a call faster Al C.N.N. is one of the up at detectors and we want to give a little overview about how it works so that you know how tech works so how it works is that you pass an image so we have some complexity in internally and what happens is internal IT should try to figure out what are some potential we call region proposals that means what the potential objects that you would detect So actually they would try to figure out a lot of potentials and the main idea is to see if there's enough overlapping regions so let's say a round of stop sign that maybe a little there may be a detection that we can do and in this enough such but regions then we sell maybe we want to we do want to do a cost of a cation of that so you discover all up attention regions and then all the promising regions you to classification to localize classification so what that means is if you want to attack this kind of C.N.N. model you need to attack multiple regions so here we're doing an illustration there the rectangles other potential regions and they can be overlapping so that means that we want to turn to stop sign in to let the person in to fool all of those region at the same time so there's some set of this attacking and ensemble models so attacking ensemble much harder than attacking a single thing and the 2nd challenge of attacking the process the end is that you're going to be dealing with the environment so right so physical world nature we have to worry about distances how far the things that we go into or stop sign what kind of angle we're approaching even the lighting as and if. [00:12:11] Pass through the sky is going to mess up your attack so in practice yes actually it does so you need to have all of these cases so in this case every hour of a sterile technique is actually nature so we need to take care of that so what's our solution so the 1st one holding the. [00:12:28] Instead of minimizing consultation then we do optimization which pixel discern turn and so on instead of all to myself one single region now is minimized across some of the So in other words you want to work. In that combination of all that region. And also specifically take advantage of the fact that while so you want something that maybe people would still see it. [00:12:58] Is normal normal as in look at a stop sign. It's not. So we'll only protect the red area so red area. In my perception or maybe taken visualize if you see I causes you know that for a darker area you can do a lot more to it and people still can not see it that way but since we're here we're fooling the detector which is a computer program we can actually take advantage of that and then protect a lot more in the regions while still remaining less perceptible to human being so we can at that we call a deviation so focus more on the on the dark region. [00:13:40] And then what about handling the physical world so to make the tech robust to the real world distortion so potentially many ways you can a different angle different distances. Different rotation so we adapt the technique called expectation over transformation so that I will do what that means you tacking the effort after all these potential things that you can do so attack the average tech the ensemble and then combine all of these then that will confuse. [00:14:10] Technique So we asked what was so previous that we saw and a person what about making invisible so on target tech so Actually yes you can do that also. So you don't see it still don't see it maybe this one you don't see it that's one for that that find it we're so close if you go close much closer you'll start to see that it doesn't really look like a stop sign but in this sense you probably mean it is the stuff that we're talking about so pretty scary so you do work on a security problem hopefully we don't see that anytime soon but the doctor does recognize that that could be a threat and so if you're curious one of the motivating factors for the new program called car which is defense for A and how to S one of this they have the physical attack that they do one more research on to help protect against it. [00:15:06] So enough of attack and. Scary So how do we either guide and protect against it so that does what it will also work on experience. So they can become a better defense so she was one of the defense and they were working on and I'll go here is to aim for something death fast impractical that people can use immediately so that was published like 82018 and the joint work with Intel. [00:15:36] So the current landscape in ever so machinery and says a lot of techniques. Often is because it's easier to publish your student your adviser you think OK well how do we promote more paper so paper being of that tech is easy. And good on the defense side much less work because it's hard to protect against many things so as I said. [00:16:00] Being a bad guy you only attack once broke it once been a good guy need to protect it against everything and among the defense techniques even if you were focusing on fast and practical so all lot of the techniques as a dirty secret is both attack and defense I often these techniques are very time consuming compute taken intensive. [00:16:20] And as a. That we've said is that when you do this kind of research you're killing a lot of trees and which is true because you're running on all these tens or hundreds of G.P.S. for hours or days so our focus here is to want to have something that's crass and practical so that people can use it right away so what's the main idea. [00:16:39] Of the images summarize that So remember recall from the stop sign example where we say what what then the bad guys do so you can change the pixel in. In perceptible way. They can make a diagram make it lighter so that they could think of the tech and the top sites right where you where you have an image of a dog you can change you will do whatever you want in this case in the chain mail chain mail you change some pixels and if you with no protection then of course they get so I mean idea to say well if your technique claims that you are doing this in imperceptible changes to images then what about our defense is exactly removed that so you probably have already been using this defense technique that many talk about which is essentially the compression so as you want to save images an image for the web where you want to compress it make it smaller and how the compression work is they remove things that human beings usually cannot see so technically at the high frequency noise so that correspond exactly to what the bad guys try to at right because their claim is that I think that people cannot see what will remove those so that's the main idea so why we want to apply it says no matter what kind of images where they are or are all benign we don't care we just pass that into our shoe protection and then hopefully that we will correct those and then wrong things that misclassified correct things don't get hurt that much but that's the main idea so that's the middle part of the show part so how does it work is that well. [00:18:12] And we also left the different kind of compression level in compression so you can compress it more in lower quality. More or less and the high compression level here at the one of the top left compression so you see a few artifacts the one at the lower part that's more compression So generally you don't want to have a deterministic approach for defense because if the bad guys know that's how it work well they can to protect it so that means you want to have some kind of randomization in practice some more is probably better so we try to incorporate that idea so that means when we have an image that's that orange square here which was split into blocks and then for each one we would take a different kind of compression level sometimes a higher compression level sometimes a little compression level and we call this tiny. [00:19:05] Local point I station you fancy name but you already see how it works so you just tell the image of how the compression level and that is only one of the techniques so in practice often defenses you do need to use multiple approaches you what you just felt that one of the approaches we also incorporate we call model vaccination so more than vaccination here that means we want our protection to be able to understand though if there's some artefacts they introduced by the compression can account for it so we. [00:19:36] Will incorporate those as we call training to help them although the to differentiate between the artefacts and the octave and also ensemble we have all the one using one quality multiple qualities and all these put together becomes the protection technique so when you apply this attack so there are quite a few these days a technique some of these. [00:20:00] Techniques and also i F. D. S. M. and the state have P.D. and so all of those are working on it so how well does it work so with the protection that it's a great line and the bottom so accuracy drop a lot. Were there were higher the better so no protection very low so once you do any kind of compression that helps So that's the purple line. [00:20:21] But you see if you only want one compression level it can also drop quite a lot so you do our shore approach doing on sample combining everything that's orange then we can still maintain a pretty relatively high so eventually everything once you have a very strong part oppression everything is a GO DOWN no guarantee you but the question is how much can we widen the gap. [00:20:47] And the focus on. Approach as I mentioned is we want something that's practical I think that people can use right away so we come up with existing potential techniques for example using medium filter which also removing noise and also to variation the noise in this well. Is much faster actually 10 times more 20 times more faster and so that I mean this is something that people can use right away you don't need to wait for a long time and that's also something that Intel's been very interesting cooperating in this technology stack and it was so well compression really simple yes that's by design at the end because we want it to be something that may be helpful for other kind of attack any other kind of delicacy So for example for attacking speak text to speech but you can also use compression too so for example one kind of tech would be to turn a sentence as they open intel dot com into. [00:21:43] Into another and the phrase read them. Do I have to start so Intel there should be evil open evil dot com so we can change it again like images into whatever you want so we can incorporate compression also so in this case we have a system diagnosed with use the same idea compression of course for all the different kind of compression M.P. 3 and now there's another dodge it was published on the P.K. P.D. also collaborative Intel and have interactive U.I. So if you can actually play with it if you feel like you can upload your own all the example to the transcription you can try some of that technique technique and then also apply the defense and so I'm going to show you a quick video demo of how does your work so here when there's nothing there you can start to upload your video recreate a car or auto sample and load a wave file that's a. [00:22:35] Computer and you can try to do the transcription into text and now you can do attack and you can see change a transcription again you can change whatever you want it to the attack is very easy so. And also tech tech very slow which is why speeding it up 20 times can turn and do whatever transcription you want right. [00:22:59] So turn to whatever you want now you can do the defense attack 1st and then you launch a the defense defense much faster in this case we can use compression M P 3 players very similar to very very fast you can do it real up and then they can correct the mistake so that is something that you can try very easy and that's actually one of the example of what we call to do it yourself and so oftentimes we have so many tech and defense techniques out there that can be overwhelming so how do we know what we're against what or what doesn't work and what So that's our main main goal for making this do it yourself so I dodge is one example and more generally we have now the boy framework which is also collaborative Intel and open source which specifically is designed to make it easier for people or try as many as you can so we recently presented at Black a share of this and also as a key 19 showcase so at a very high level we want to abstract be able to to for researchers to provide these we call the Research Module as an attack defense on different modalities a small Jews can add to the M.L.C. framework so that the user this case could be researchers students could be practitioners they can interactively child these different. [00:24:16] Techniques and also we want to build a community around it allow people to to continue to update so that it's a good collection and also we want to make it easy to use so I mean heard often for research is that while we do all the great were published paper and that's it but most people don't we paper or they may not have the sufficient knowledge to read all the details so we but then we want people to get people excited expression for students only starting and one of them what does it do so we want. [00:24:45] To provide that kind experience so Currently we have a number of ma. Those already on you with your cell already and also some technique tech and defense technique we also have from other Georgia Tech researchers more on that I'm a web site so I. Know where and also their tech and defense by passing. [00:25:07] And I'm aware of detection if you ask me that other news years ago or so right so all these become more of us all emulous voice which then the users can use they go to a browser you can although you example it can be images can be software and so on and to support a framework and the through the U.I. You can pick what you want to try and you want to try what techniques different technique you can create your own company and of experiment and then look at that result so a little bit of technical detail we want to make it extensible and want people to actually use it so our. [00:25:41] Architecture is that we have a research tech and defense budget we called a recession to overall and then we are told that people will use and they went back in and if people want to launch a more competition intensive work they can enlarge the work or instances so other than a technical detail or they want to make it easy to install one click install Currently we want to make it even more easy and that's work in progress they want to move it to even on a WS or as your Google Clowne. [00:26:14] So we're going to show you a quick demo of the how it works so this is the entry point to Emmaus boy so that it's quite tiny the main focus here is on the right hand side these we call the line so as each little rectangle there is actually a Research Module or configuration of research modules that you as a user can select and choose what to do so that means you can string together. [00:26:39] And you can string together in the tech technique pick whatever you want and then you pick whatever defense you want and then you get that you create a pipeline of data you run your pipeline through the data and you see the result so all these without programming so and of course you can look at all the details that's what the extension of the law would give you. [00:26:59] And also I open source very very recently and also the now is incorporated into being incorporated into Intel courses so that we want to expand the education back to a more people to learn about FS or machine learning and also people are tried that so those are a few examples of the work they've been doing on the security guys side so as I mention. [00:27:22] It. Working on a 2nd defense is great very exciting but often we don't really know what's happening internally at the model when you say we have successfully attacked this model where successfully defended this model in practice just look a little how the number really gives a way what's really happening particularly a particular nuance that really you know so that's that there are 36 secret so that means we do want people to to know and we should be working on how to make that easier for people to know so that's what brought bring us to the 2nd part. [00:27:59] Interpret that the way I said it so I would say for practitioners you don't need to convince them as much about that and the need for interpretability. Security practitioners they that's the main focus of that work often when they deploy something they don't only want to know that how it works in the beginning but they want to make sure that they can maintain it if things break they want to know how to fix it so that means they want to know why and how all these things works so how do you do that different approaches one you can use while texting explanation and our main approaches is to use scalable interactive visualize ation as a medium to connect to users who want to understand this model do the actual model then we ask why interactive visualization and the main reason is we can leverage what is good powerful that human being can do so machine that is great at finding patterns and while visualizing is great and amplifying human can. [00:28:57] So we our eyes our brains are really great so we can swap hands that computers have still a problem spotting so by Create thing that into Activision allies ation we allow the user and the visualization to incrementally together make sense of the model so which of interaction and socializing. [00:29:17] So for this part of the time I gave you a few example of some of the interpretability work that we have been doing start with that work that we've collaborated with Facebook on developing a visualization too that can help internally Facebook data scientists and engineers to understand industry scale model and the 2nd part I'll give you an example which is to help students practitioners and researcher to understand how to use one of their gang technique which you learn more details so it turned out that a lot of the small Those that these things get so complex even learning about it before you use it can be difficult so again it's one of the tools that help make that learning easier and that will conclude with the landscape research landscape about and took it away I saw where we are currently and also where we're heading so we're going to start with the 1st one so activists and that's the name of the system that we developed a Facebook so now it's deployed and a process don't work with Facebook collaborator. [00:30:16] And you may be surprised so companies while they have all the resources Facebook Google and all those and visualisation tool is still something that's sort of the need it because internally they're also differentiating between the people who will engineer the machine learning and also people who use the machine learning for example their scientists they may be more interested in not so much in the internal machinery but also but instead using those to analyze data so that means a strong need for visualization but then they need to deal with a number of practical challenges especially Facebook is that the data is and also the model often it's very complex they decisis you and also they need to deal with any kind of data so images text anything you can imagine. [00:31:02] So actually if it is started it. Was developed over 11 months and it started with participatory design specialists with over 15 researchers engineers and scientists at Facebook and now activists is deployed by Facebook and. Behind it is a screenshot we look at some of the component of the system and one of the challenges is. [00:31:25] The very 1st one is about the complexity of the model so. Like many tech companies the model that they use can be big and complex so what that means is if they are looking at the learning model for example I could have many layers and could be many Terrans so pretty overwhelming. [00:31:46] Sort of that well even though there are many many layers to the actual users they don't really care about everything's they usually care more about individual things so give an example why is this so is author of you familiar with the models their earlier layer on their early feature that this effort you shows more of the level lower level as it may be detecting a line today right spot and so on one of the later layouts will be high level concepts I could be in that maybe about face is about all the hints and so on so often that understanding the book is more helpful at the data point where they can more easily interpret the lower level part may not be as interesting. [00:32:23] So that to the left that's observations so then in our activists and system it's a screenshot of it so in copper 8 we call the model overview where we do show the whole architecture but then we allow the user to zoom in pen and tend to pick the particular modules that the interested it so that we don't show everything the same time. [00:32:47] And then at the bottom that you'll see more detail about it so that it's using the famous overview 1st so my filter details on the man idea so I may say Wow OK Well that's very simple Yes we want things to be simple we want things to be to be so and you don't need to go bury fancy so you just need to be things that were the important. [00:33:08] So that's the 1st 1st main idea and the 2nd idea is that we also that data scientists often to end allies data using a variety of ways different patterns can cause qualify them into to her make area where you see this is actually a spectrum so what we call instance level and they're using the specific examples to pass into model to understand what's happening so this is very similar to a reading program it's like a unit testing if you will so you particular example you know for sure it has to be classified if not then something is wrong so those are specific instances so it's great for the bucking very specific you don't exactly reprising you know exactly what is wrong so that's 11 way and analysis but there's also the other side we call subset level the reason is that for instance level individual samples of them is not very scalable so if you're talking about Facebook scale data so you have billions of items so you can then really do instance level so that we need to support also subset level and other way of thinking as well so you want a group of instances analyzing at that level and when you think about grouping instances that means you can group at different going to lead level you can group things by let's say a class classification grouping by class or maybe subclasses so it's a smaller chunk within a class or any user defined way so you have a lot of data but now you can you want to chop it up in different ways so that we all put into a subset level in practice we need to support both because there's a reason and also the ease of debugging so we support So how do you support it was turned out that a lot of the work system work do not support that that we're all integrated that easily so what we do in activist system so we can focus on the A little part you know in the top overview part is we integrate all these kind of visualisation all the different analysis into we call the neuron activation. [00:35:04] So where its role here. as you call them here it would be it one neuron in the in the did room although and the for issue really he would be say it is tins or in since subset so the data there where looking at here is a tech state it so as in my facebook facebook feet you know thing this facebook fee for each face 0 feet you my want to classify that what kind of feet it is is that talking about as they hear about description is that a bro abbreviation is about entity and so on so at the top the top roast those are at the casas i cost of the classification so a demi said that's one concepts that but you can also have other kinds of set may be more granular so here's each dot the the saturation means how strong how strong the activation in its for the particular kos think the ghost up in that little bit so we can define your own subset for example i won't lee won techs that contain particular work so you create dose us well and you could also incorporate instance this into a to of us well so here on the right hence i weigh instance election the you where each square he s. one specific instance you miles over it you will see the actual techs and you click on it then they also get add it to the new or activation you so Right so let me set design is actually quite simple we use a matrix table view where columns on the runts the rows there could be subsets you'll be classed level stuff that could be user defined subset it could be specific instances you could even at instances that might be misclassified So here when using the border and the interior fill color to represent what is correct and once not the benefit of integrating all these together is that you can now sort all the neurons based on the string and allow you to easily compare as at the bottom so those are specific instances you know what just correctly calls if I want to misclassify and can compare the activation pennants I mean that the situation with the classes that you that should be current of classify and then you can see by doing the sorting is a wow OK so it's quite different there's a lot more gaps and not really in the in the grid ation going down so as an effective way for data and scientists to spot and compare and also doing everything in one single view is what we call the unified analysis for instances. [00:37:36] And of course Facebook we also know you care about scalability so not going into too much detail use a combination of techniques one we call them instant sampling so through sampling so dealing with 1000000000 items that we can do a sample and then also do computations so you can be from you specify in the model what Which of the components that are interesting and then we're going to compute those activation beforehand so that when they launch the activists platform they can see thing in real time and also we left Facebook's scalable communication. [00:38:12] Back and so that it's to do a lot of the computation and be able to handle the. Data and now act as the point and Facebook on their machine learning platform it's called F B learner which is used by almost 25 percent of engineering teams so now it is very easy for a face that is scientists and engineers to use it they go to the traditional view of the machine or they train machinery model they have like a summary plot if you would all the summary statistics and so on about but then now they also have a button that can click and then they would launch activists specialized. [00:38:47] So for X. if this is great for understanding a single model and probably also good for some particular data set but sometimes we want to have even more high level if you will what's happening in the model or make up for a big dataset so that some of the reason work that we have been working on one particular one system can summit at which it's going to appear at the visualising conference is 2019 which to aim to tackle the problem to. [00:39:15] Do scalable summarize sation and to help people understand if you pass in let's say a lot of images like white wolf or the particular cats into a daydream although one of the influence schol neurons in influential connection all these images are really activating So what concretely that means you want to know what the future is and what a company in the future connection and features that the modest really discovering So what are the most important one was the less important one so the main idea of summit. [00:39:48] So there's a U.I. or summit and they can't you can use some of for the this inherent describe but also it's helpful full of the bucket. Which include also understanding efforts arrow attack and defense because you can think of tech as a book as in your model you are attacking it so what's really happening so we're in the example of how they could be so in this case we are looking at a particular class. [00:40:15] Image and so a mission as a popular mission is about $1000000.00 images and have $10000.00 classes we're looking at particularly the class called Tench 10 yellow fish so we select the yellow fish on the left hand side and then on the right hand side we want to see what the features that he has learned So remember we want to have something kind of like a next network summary so it's fish class but they really mouse over some of the nuance to look at the future and look at a lot of people actually people face is. [00:40:44] And every look a little high I want then we start to see something but I also see fingers their fingers. And more people so why is that. And it gets and I guess why why so many people were talking about fish they get. Very good so it turns out that tension is one of the. [00:41:08] Kind of fish that people like to catch and then they want to show off their pictures so which is why in the dataset I said most of the pictures of people carrying the fish in their hands which is why. Surprising because if you. Mission earning a developer data scientists is why one that model's accurate very accurate but then now we've found out what $99.00 out of one to images for the Tench class actually dependent on the people feature which is still want to use it or would you rather maybe have more collect more data that's not only person with the fish but maybe actual fish most of the fish so that is one example of what you can you sum it for so we're very excited about this is the only early early work and we want to use it through understanding but I did it quality issue although issue. [00:42:03] And. Learning model as it could be like the ones on Facebook it could be like the model for Facebook that story is for someone to try to understand so those I would say Who are some of the earlier brain model and now we also have some more recent development and over time they're getting into quite complex so complex that even expert they may have a hard time understanding so that which is bring us to the 2nd part the middle part where maybe our experts also in the some help or me out student also need some help to help with the learning of these complex models and particularly we want to focus on a kind of model called generative F. a serial network and this is work that joint work with. [00:42:48] Google Google the pear group based in Cambridge and. The goal there is to help with machine earning education so you probably have seen some of these examples would be in classes on the web and they tend for the playground for example one of the very popular ones on the top tell that they're some of the early visualization and those are great but just same time also they're designed more for one classic learning model and more than they get going a model can be pretty complex so it can be many layers many components and particular one called genera episode network gangs are the most exciting interesting technique. [00:43:28] And that's even more complex so you might already have seen a like example of this before where you have these faces that look real but it's actually fake or maybe of the Air B.N. B. that doesn't exist yet images a baby but they actually also fake so it turned out to be a pretty pretty complex so why is this so complex so if you just look at a map that's already quite a bit a map but we're not talking about Map So what we really want to focus on is about the core reason for complexity I call compact reason is that in the gang that generate these fake but real looking thing is they using 2 competing and doing that work inside it which create the complexity so in particular the 2 component we call generator and discriminating so generator you kind of think of it as everyday life as I someone who likes to make want to make build the counterfeiter make the rules and then we have the good guys the police who try to spot these. [00:44:20] Of course in the gang is that this is the conceptual understanding and practice what's really happening is something somewhat different so what do the counter do how do they want to say so I start with some pretty low quality and try to fold the police then and police OK so it's not good so police know what the real thing out of the thing looks like compared to real thing and them this is fake but then the Comfort to get smarter and it's all OK I'm going to improve my things so I'm going to make even better bill so that the next iteration the fake it looks even more real and while the police also become smarter because of all of you getting more faith will it stick to think things are going to improve myself so this is that internally that's happening and gain so that in the end if the counter to is able to produce these think that that is indistinguishable from the real thing then that means you are done with training your game so as in generating think faces take pictures so when the police cannot tell the difference well that means you have enough generating so conceptually is OK That's very nice well and practice not as easy to explain this expression of real data and set up this toy data so how do we explain all these. [00:45:33] So the high level we need to handle a couple of challenges still at the conceptual understanding what you were just described and also we want things to look to be real people who can interact with all these in real time so interactive training and more importantly for education that means into be easy to use easily accessible to anyone so going back to our examples or how do we achieve those goals the very 1st question that you might have what the data to visualize So in the narrow Would you say builds but imperative when you do a visualization explanation how do we do it. [00:46:09] So we decided to use 2 D. data 2 dimensional data instead of dimension with a Y. e 2 D. data for real data images so those high dimensional so why don't we why do we go for today the reason is that even flow to the is already complex enough and also the understanding here is not so much about that dimensionality is actually that into action so that's why we decided use to date and focus on the main concept and so that we can also easily visualize the data because once you have I mentioned data you need to do a dimension after reduction which means you loose information which also me introduce and other complexity in the explanation which I want you to detail so we use green dots as real the actual real thing you want to mimic and then the purple with you generated data at the fake things so it's aeration your eyes. [00:46:56] For example so you want to improve your fake data so that gradually look more real this is the very 1st version virtually 0 point one It looks like so we're incorporating design in the finals like. So that's the generator and. The data So how do I visualized said generator iteration so. [00:47:17] In gangs I was starting point is actually you would call random data a random thing you can add to fit anything to it and improvement you can think of it technically as a transformation transforming your initial random thing into something eventually the REAL So another way to think about is you are trying to map in your original data input space into something that you want so that means you could put a good uniform grid and we're really twisting and want paying your original input space into the open space that you want to but real things so you kind of moving these. [00:47:54] All these area so how do we convey to so since this walking and thing which change over iteration so that's where we bring in animation in general if you take an visualization if class you don't know that any mission and you use it really carefully because the people. [00:48:08] So this case we think is a really good physical and we want people to most over the middle part generator and then they can they can play the animation on the man to see how the original regular can't transform can't walk that. Animation is actually really hard to understand because you only see the output you don't know what was the input like so here you can repeat it will you play it and then say OK So lower left corner and now get want to corner so that's out of the generator. [00:48:35] And what about discriminator police so that's easier if you have taken machine during class or the. Experience with the most common ways to use the heat map this is what we're doing so we use the same color coding green for real and fake for purple so we visualize that this sort of boundary so putting everything together so that become. [00:49:00] Visualization so we had real data big data generated discriminator everything sec to get it which you can turn on and off and that becomes a visualization this is a point 5 So why why is this a 0.5 because while Fishel I station we're doing interactive training that mean you need to somehow let the user through the with the parameter so it goes away OK that's good so when some sliders that was disastrous really bad and even better for expert I should have no idea what all these sliders keep in mind that we have 2 competing in the network so it's actually that's 222 copies of the same slide are very very confusing so why is that the main reason is because the mental model is very hard to capture just by having a collection of slider so in the head and people's head is what you have to do to network that computing to work in together but that's not capture using the slider so. [00:49:53] So that means we really is something that do show the over the view and connect everything that brings us to our final design of a game that where you reduce road interaction to do pick tango here I salute to the component and then also we go everything in between so I'm going to show you a demo of it so let's say we want to generate real data that looks like a ring so then we want to take one to turn into a ring eventually So that's our goal. [00:50:20] And we wanted to active training so that means that user can go in and then they fiddle with the primary to learning rate and so on up to 2 parameters they were networked and then they can start training they can press a play button and see how things got moved to Originally it doesn't look really good but that it to rigidly. [00:50:39] Training is happening the better you can turn on and off the visual isolation and sometimes you can be pretty overwhelming on and off them. And that's just the they're walking the animation that you show to see the input space going to transform into all the open space. And change the a little bit and also you notice that we speeding up the training at 15 times speed so similar to come in to learning. [00:51:15] Model it takes forever so so it was speed up and in the end you can look pretty good now if you're going to ring. And you say OK can you shot it at home yes you can actually try it that is our 3rd goal is to make it easier for people who are shot at home so conventional visualize sation if you want to build something like that previously you would have a back and powerful back end well because a lot of money you and with and while that's is not a good way not the easiest way to do it what about would push everything into the browser so that means everything in javascript specifically uses tends to power etc to buy one so that means you have graphics card all you want and a CD You have also had a component you can do all of these in your browser so you can actually just go to google for that and then you can go there and play with it in your browser and want to release a lot of interests likes and we it's so. [00:52:10] State you can try it on their browser it doesn't question browser it may just be slow but it doesn't match your browser. So that brings us to our last topic right so what I have all these. Very excited exciting development interpretively and also I have a 0 machine or anything and then there is hopefully I make the case why we want to interpret not just secure AI alone because a lot of time we don't really know what's happening so what's the current landscape for research so we did a recent survey on this which I encourage you to. [00:52:46] Learning very recent and to summarize the landscape so in particular the way that we do it is we take a very human centered approach and so it's summarizing all the all the points we want to put these into perspective why people do these into a bootable technique to visualize sation why. [00:53:04] What they really interpret saying you will see that for different use cases differences and narrow things that they're visualizing may be very different when they may want to do it is that to help with training model training for helping understanding on that's a new game that is it because they want to use it for education who is that for you will find that often the audiences although very diverse design for a student can be designed for data science at the company so different skill level so and also how and why so some key take away from doing the survey thing it's a great thing for the community is that currently a load tools are still designed for experts so we sorely need tool set design for Express for students who are starting only so how do we get them excited how to make it easy going to try things you may recall the activists of what we call instance space and the US is that that means they focus on specific example but in practice one thing to be scalable so we need more work that would support that can scale the analysis and we're also a very excited to see that a lot of the work is pretty good going in a very good trajectory is very into disciplinary so it's no longer happening only in one community but now we also across communities so that we think that's a great thing Number 4 is they action ability so currently a lot of tools are designed to help with that diagnostic and all we see something is wrong but then they don't we need to tell people how to fix them so. [00:54:31] That we think action ability is important so also suggests what people may want to do in such a stand hey something's wrong go fix it so we want to keep them people some guidance evaluation is still very hard and a lot of the tools often we propose that but then evaluation often it takes quite a bit of time to do what we think we have some good handled on that. [00:54:54] The last day of the model not very robust I think now by now you would agree with me that that also means you want to have more technique step to help people understand all these vulnerabilities so the remaining few minutes I would summarize with. One of the latest work it was as. [00:55:12] Senior student he had a wonderful internship with Microsoft Research and they also study a very important problem which not to talk about too much but we can Xscape is what is in top of the ability after all so we have been talking into a bit of a China could this you that's important what this really doing so they did a study on it and this specific design a tool to help understand what it turned out that interpretability is kind of like data science where there's no agree a point definition you talk about people from different communities give you things so they would talk about maybe about the internal discovering understand the operation looking at the data mapping and also snow agree a point definition so for that reason they built a tool called Kemet to help understand what. [00:56:03] They decided to engineers are really hoping to achieve what they really want to do when they want to interpret a model so that means they want to study what are the capabilities of interpretability they want to provide and. And they discover a number of questions that often is answered would like to to ask and which also means capabilities that you want to support be able to support if you're working and to and what specifically let's say we're looking at. [00:56:34] House data look at the house the cost of houses so what other kind of capabilities do you wish to be able to support to help interpretability So it could be something about. Instances as you already saw an example in activists though looking at particular examples of local instant explanation it could be about comparing thing the impact is very important it's very rare that you don't develop one thing you have multiple models multiple instances comparison important number 3 now is a lot of interest on it we call counterfactual very fancy name what that really means that we're asking what if questions and then well not only the data that you see but also what if you change the data a little bit what would happen so it's very accurate to say I'm out because as I am now is changing the data in the ways that may not be good for the models so that's that's one example of that finding similar thing number 4 that's that also we need not only looking at what you are going to focus on also finding a similar thing you care a lot about ARA So in practice people do say well you don't because just give me a model tell me when it would hit a brick and what was the likelihood that. [00:57:42] I also like what Summit activists do discover what is one of the future is that important what importance so how do you approach the paper was published recently 2019 very nice work. That so you want to look at more detail so the paper at the website is there and they cover quite a bit of work some on security eyesight some interpret the pull some examples. [00:58:15] And make your case for the reason I said to work security and interpret it because we're only starting to build a bridge. Convince you that the bridges sorely need it a lot of work on it secret security. Also now getting more work in debt but then we should combine them. [00:58:34] And that would be great so with that thank all the Sudan and all the all the audience think you. Are just. Here For. THANK YOU THANK YOU THANK YOU THANK YOU.