[00:00:15] >> Thank you Maria for the introduction and thank you all for joining us here today. Like Maria said I'm a Ph d. student here working with Professor sock and. I'm going to present today a current work that we've all been working on it's called robo call guard so we're trying to fight voice spam with a virtual assistant and this is a joint research done by. [00:00:38] D.c. and my advice in which the. So I always start my robo call talks this way who in the room has ever gotten a role because most of us almost everyone so this is a serious problem now almost everyone is getting these unwanted calls and although there are efforts and these call blocking apps and everything this problem is just not going away. [00:01:04] In 2019 you mail has estimated just in March there were about more than 5000000000 recalls now you could thing that they could have estimated a bigger number than usual but it's really a big number like this and there are these call blocking apps like you may through color and so on high but do these call docking act applications actually work if they do how well do they do or if they don't what other methods are there that we could implement to prevent these types of calls. [00:01:37] So most of the applications that I mentioned like Humana Truecaller most of them rely on Blacklist and these blacklists rely on historical data like user complaints honeypot information and things like that. But our previous work has shown that blacklist can only be somewhat to fictive our previous work show that it's only about an on an average it's about 55 percent effective and their effectiveness decreases with caller id spoofing the caller id spoofing is something that a phone call comes into your phone your phone and the number that it says that it's from it's actually not that number to school thing that number it's much like IP spoofing. [00:02:20] So nowadays there's a new kind of school thing which is called a neighbor school thing but neighbor school thing means is that the incoming phone number the 1st 3 or 6 digits is exactly like your 6 digits like your phone number so you're more likely to pick up that number because you think this is from my area code this looks similar so you're more likely to pick that up so a caller id spoofing is really increasing now it is higher has reported that more than 50 percent scams reported by their users rely on neighbors thing so with the increase of caller id spoofing the fact of missed of blackness could read decreases. [00:02:58] So the 1st thing that comes into mind when we think about caller id spoofing how to fight caller id spoofing is caller id authentication so caller id authentication is you authenticated the caller id so the number that the call is coming from the number it says it's coming from it is verified that it is actually that number so a number that says let's say a.b.c. it's actually a.b.c.. [00:03:22] F.c.c. proposed a caller id authentication protocol it's called says she can or stare so. It's very fine that that number is actually true it's not a spoof number but there are. No you could think that caller id authentication would reduce the number of robo calls because there is no spoofing but there are a couple of problems with this for example for this to work all the carriers every medium large small companies have to adopt these protocols and for every company to adopt it it's going to take some time plus what if most of the scammers are operating outside of us choose to get jurisdiction caller id authentication for example f.c.c. is one wouldn't work. [00:04:06] And what a victim's continue to fall victim without even without caller id spoofing So let's say you don't have caller id spoofing but the scammers are using actual phone numbers but they're churning through the numbers these are real phone numbers but these are like disposable phone numbers so caller id authentication works but still not going away so caller id authentication is not a cure all solution with this one completely go away. [00:04:34] Now let's look at our research so let's see what if you had and everyone in this room had an assistant whenever a phone call comes you're not immediately notified of that call you have a secretary or an assistant who picks up the call on your behalf makes in conversation with other party deter minds if this call is actually you want to or unwanted to you and only if it deems that this is a want to call the system let you know that this is an important call you should pick it up and if she things that it's not an important call a terrible call or an unwanted call like a telemarketer or debt collector they block that call and you don't have to bother with it so this is our idea and start off with an actual human secretary we propose of our choice of Treasury pics of these every incoming call not every Let's say your mom calls you she would have to go through the virtual assistant it would go directly to you so everything you have on your contact list doesn't have to go through a virtual assistant but every phone number that you don't know goes to the virtual assistant the virtual assistant picks up your phone call makes a conversation with the other party and it reminds whether it's a want to call or not if the call is wanted your phone rings and your phone with a transcript of the conversation that the caller had with a fortune assistant so you have a context of what the call incoming call is about and let's say the virtual assistant blocks a call like a freak your phone wondering instead you'll get a No it's not fixation that your virtual assistant just block the call for you and this is what the transcript was that this was a. [00:06:06] A free cruise call or something else maybe. Now there are some current called screening methods like recently Google had Google pics of ones had They're called screen option so what it does is when an incoming call an incoming phone call comes to reform and sort of just answer and decline you have 3 options answer decline and screen and once you click on screen this is what happens the other party like the caller would hear something like Hi this is the person you're calling is using a screening service please say your name and why you're calling and then whatever they say would be shown to you and then you might have something more to say like tell me more or I'll call you back and whatever you wanted to say and Google assistant will say that to the caller you know there are a couple of issues with this or more like this similarity is with our system so with Google called screen option it's it is not filtering any calls like what we want to do our virtual assistant It filters out don't want to cause for you but with Google it's not filtering it it's just showing you the context before you pick up the call Another problem is user intervention is always necessary when a phone call comes in the phone will ring and you will have to decide whether you want to screen the call decline it or accept the call so you would have to be in the loop with every call. [00:07:31] But with a virtual assistant Let's say your virtual assistant picks up the call but you're not made aware of every incoming call let's say you receive 10 unwanted calls each day at 10 times your phone is ringing what you don't want with a virtual assistant that is not going to happen. [00:07:47] Another call screening kind of application is called robo killer So what it does is once you've installed us. All cause maybe except the white listed ones or your contact list ones all calls are forwarded to a centralized server so the call goes through a centralized server and an answer bot picks it up perform some or you analyzes like one of them is. [00:08:12] They keep on playing the ring tone so the caller a human caller would hear that there is a ringtone on the other side so they wouldn't start speaking because they one thing that the call has been picked up but caller wouldn't know that and they would think that caller this picked up and they would immediately start speaking so that is one of the methods they deploy and there are some other methods like for example when a human caller is speaking there would be more silence there would be some discipline seize versus when there is a robo call or there wouldn't be. [00:08:43] And all the so all human callers whatever did decide that this is a huge This is a call from the human caller it's forwarded back to the user your phone rings again and all the robo callers are past 100 part and their main issue is they want to waste the time of these telemarketers or over callers so they have a they keep on making conversation with a real caller so they want to ways the rebel callers time now there are also some issues here it sounds like a very good system but keep in mind that all foreign calls are being forwarded to a centralized server so there are some privacy issues all of your phone calls from all the callers are going to a central to the real killer centralized server and they're making conversation well not a conversation they're doing some audio and I says there's no actual virtual assistant I'm in between. [00:09:32] So let's look at our contributions we're the 1st to evaluate a call screening virtual assistant that filter out your own want to cause and protect you from because even in the presence of caller id spoofing. And now with Google's called screener Robel killers these are products so they don't have a rigorous evaluation of how these things actually work. [00:09:58] Averred our virtual assistant aims to provide of the less sophisticated than an actual human being because this is not an work more like a security based work so it's definitely less sophisticated than an actual human being when the caller is talking they one thing that they're talking to a human being like Google duplex but the idea is to have a virtual assistant in between who protect you from your unwanted cause. [00:10:25] And to this extent we have developed a proof of concept voice application us smartphone app we name did call Guard and with our experiments we experimented with about $8000.00 tree alife robo calls and we were able to block all like 100 percent of the records and we conducted a user study using the app we made and the user study was to deter mine the usability experience from the caller and qualities perspective and most of the callers and colleagues most of the users who use this said that they were comfortable or thought that that was beneficial. [00:11:06] Now let's look at the design goals what are the design goals that we want to achieve so we definitely want to have an extra layer of security between the caller and the recipient of the call to something that stands between like a virtual assistant which filters out your on want of cause which deals with caller id spoofing and the 2nd design goal is when the virtual assistant is talking to the other party it already knows some context of the call so before you pick up the call you're already provided with some context it's more like an email let's say when an email comes you have a subject so you know what the e-mail could be about it's kind of like that so you have a context of the call before you pick up the call or even if you miss a call let's say or even of the robo call Guard has blocked a call for you since we're providing the transcript you know what the called was about we also want to preserve user experience so since this is an added step in a natural phone conversation a natural phone conversation is just a caller and a caller there's nothing in between so since we have the added middleman in the bit in between so we want to make sure that user experience is as smooth as possible in terms of latency and accuracy and we also want to ensure privacy when an incoming call is handled by a virtual assistant we want to ensure that this is everything is handled locally now we were able to achieve the 1st 3 goes with the 4th call there are some issues that I want to talk about later so come back to it. [00:12:39] Now let's take a look at the whole system overview so an incoming call comes and your phone immediately doesn't ring. There's a call intercept or so the virtual assistant intercepts your call and checks whether it's in the whiteness blacklist or unknown means neither of them so if it doesn't the white list. [00:13:02] Your call is forwarded to you your phone rings and you pick up and so far so good if it's in the blacklist your call is blocked if it's a known malicious caller your for the call is blocked and there is no engagement that is neither whitelist or in the blacklist so all the unknown callers like this let's call them green list the Via pix of the call v. is a virtual assistant here so the viewer pix of the call and greets the caller like hello your research will stand. [00:13:35] Please say who you're trying to reach so the intrusion here is when we're trying to filter out these wanted versus unwanted calls now the notion of unwanted call is very broad and so different for each person so we take a conservative approach here. That we ask the name of the calling the intrusion here that someone who is calling you and if it's a wanted call to you it's more likely that they would know your name if they don't know your name we assume that this call is not immediate you want to for you. [00:14:09] So and the name could be said by the user let's I'm using this phone I'm using my phone I could use my name I could use my family member's name so their call also comes. And if the name is detected let's say the caller says the correct name. [00:14:26] So you deem this as a want to call and the phone rings the user is notified of the call and we also provide the context of a transcript of the call let's say this is a caller who doesn't know your name so no name is detected we have a specific threshold of of time so after we asked the name we have about 15 seconds so let's say within these 15 seconds no name has been detected. [00:14:54] In that case there is a specific time a specific time stamp so we use 20 seconds for these at the 20th 2nd the virtual assistant to interrupt the caller Now the reason for this is the sparse and who is calling you doesn't know your name so we label them or we think that this is a unwanted caller So we're definitely sure that we're not going to pass this call now we want to deter mind that whether this unwanted call is from a human being or because unwanted call could be a telemarketer a debt collector or. [00:15:32] Now to determine whether this is a robo call or or a human being the idea is when you interrupt someone let's say an actual human being they would most likely stop talking when you're interrupting them versus a robo call or not most of these rebel callers These are making mass Cipro because they're not very sophisticated so didn't start playing the recording and that's it they don't do anything about it so you interrupt the caller if they don't stop it's very much likely that there are caller so if it's if the caller is interrupted where we know that this is a human being because they were interrupted and we block this call because this is an unwanted call did and they didn't know the name and we label them as a human caller the message is saved whatever they said to the virtual assistant So let's say even if it's someone if even if it's one to call a person a person who has called you but they don't know your name but the call is actually want to for you the message is saved so the notification immediately comes to your phone. [00:16:34] If the virtual assistant made a mistake you could immediately call them back or call them back at your own convenience now if the caller was not interrupted we block this call and we label it as a robot caller and whatever they say it to the virtual assistant is saved as a message and the transcript of provided an implication comes to the user sworn you know that this is a block phone call this is from a real caller and this is what they said. [00:16:59] Now let's talk about the threat model what are the in scope threats so these are the threats that we are able to deal with Mastro because most of the real Because that are prevalent currency are mass will because they are not targeted they don't know your name they are trying to reach as many people they are trying to increase their coverage with as less cost as possible and so they made this mass will cause the player a. [00:17:26] Pretty accorded message and that's it so with our virtual assistant when we ask the name or reenter of the caller these are because we'll not be able to pass this challenge. Unwanted live calls from humans such as telemarketers or debt collectors or even spam callers like people calling from call centers about your tech support scam so the these mass cause are not targeted so they don't know your name so they won't be able to say the correct name and will be stopped by the virtual assistant spoofed call as since we're not only relying on Blacklist unlike the other apps that are available so we have blackness and case but we also have an additional step so even if the caller id is school if they don't know your name they won't be able to pass this challenge. [00:18:20] Now even with area quipped attacks like let's say we have a sophisticated attacker like Google duplex who is able to imitate a human being now if they don't know your name of the attack is not targeted if they don't know your name they won't be able to say a name maybe they would be interrupted so it's possible that the caller may be mislabeled as a human being but the called will definitely be blocked because they don't know your name now the out of school threats here are targeted attacks if the attack is targeted and they have found association with the phone number and the name let's say using leaked private data then it's possible that they would be able to bypass the system so in our future work we are working on that how to defend against this targeted attacks as well and says This app is only for smartphone if a scam call is coming through landlines we won't be able to defend against them. [00:19:18] This is the system architecture of our virtual assistant or so I've discussed some of that in the system overview section as well with each incoming call we have a more you look call call call intercept or so this is where it checks the caller id whether it's a white list which color of the actress to call and of and or an unknown caller and all the known callers are passed to the right model the spam detector so here we have to. [00:19:49] Name recognise or so name recognizer is the part where they're trying to recognise whether the name has been spoken or not if the name is spoken the caller the call is passed to the user with the transcript and your phone rings. If not then it goes on to the robo call detector now they will call detector is where we have our interrupt or we're trying to interrupt the caller and if they're interrupted we save them as a block human call or if not we save them as a block and the transcriber is the module where we transcribe all these conversation that the caller is having with the virtual assistant you know let's go into some details and each of these modules us a call or call intercepted this is it makes an initial decision based on the caller i.d. whether it's a white list of color a color from your contact list now a white list could be made of fewer contact list it could be some public white listed number like maybe a public school or public hospital or whatever for whatever phone number you want in your white list. [00:20:55] So all the white lists of calls are passed all the blackness of calls are dropped now in our previous work we have shown how to dynamically build a blacklist from the from different data sources and how to update them dynamically every day so every day the blacklist would be updated and we would be able to catch new callers the main 2 functions of scepter is intercept the cost to acquire dog incoming audio stream and in direct recorded voice messages know these are the messages that the virtual assistant is going to be saying into the outgoing audio stream. [00:21:31] The name recognizer this is the part where we're trying to detect the name so if the caller knows the name it's a want to call it the caller does not know the name it's an On want to call and the user is allowed to said the correct name or names you can have multiple names as well 1st Name Last Name it could be a combination of both depends on how hard you want the challenge to be. [00:21:52] Now the backbone of this name recognize a model is a Q word spotting algorithm so there are multiple keyword spotting out algorithms even available commercially and. These are all. Barking algorithms and they work very well so. A keyboard spotting algorithm would be able to detect a correct name we have used snow boy of the reason as snow boy is a very lightweight model which could be used on an Android phone that has a high accuracy and another issue is well another advantage of using snowball is let's say you are using this app and you want the correct name to be your name let's say your name is Jason so you want to train the model with your name Jason now. [00:22:39] Since you are going to be using your voice train it it's not accepted that the virtual assistant would ask you to make let's say 100 examples of this name it's expected that there's going to be very limited training examples which no Boyett can train a model with just 3 examples so you say Ward 3 times and it would give you a model you could download that model and use it in your app so this is one of the reasons we use nobody because it works with limited examples and we use in the I like a black box we're not doing anything there we're using it like a black box to detect whether the correct name has been said. [00:23:19] Now all don't want to cause a pass to this more you'll recall detector and. In this form as I've said before the v.a. interrupts the caller at a predefined time stem at 28 2nd the v.a. is going to interrupt don want to color and it's going to use voice activated detection to see whether the caller became silent now what it does is at the 20th 2nd the v.a. is going to say something like this sorry I didn't get your call Sorry I didn't get the name can you please repeat the name now it takes about 5 seconds to say this and what the v.a. is going to be tracking is during this 5 seconds is the other during most of these 5 seconds is the other party silent or not so we see that if the other party is silent at least half the time during the virtual assistant just speaking. [00:24:09] We say that this is a human being because they have been silent for at least half the time but if they have not been silent of all we said the caller. Another transcriber is the part where we would since we want to provide this context with each call we have this module where we transcribe the conversation the caller is having with a virtual assistant now there are different transcribing A.P.R.'s or libraries we're using Google Cloud speech it has a very high accuracy and transcription can be performed for from a mobile device now here is the part when I'm going to talk about the last design goal of our virtual assistant so we talked about privacy we want to handle everything locally Now here is a little bit of pick up so we are we want everything to be handled locally so definitely we won the transcription to be handed locally as with now Android provides services where you could transcribe your ordeal but the issue is the incoming stream or the input channel always has to be the microphone so they didn't provide right now that you could use any income input channel for example here the input channel would be from the other side whatever the caller is saying it would be from the view IP channel or from the phone channel instead of the microphone. [00:25:32] But Android doesn't allow that hence we couldn't do it locally but. Google pix says called Screen featured there doing this transcription service as well so it is possible that you could. Code base to allow. Other input channels as well to transcribe your argues but since our school was not here or. [00:25:56] We're trying to build a prototype or proof of concept app for this so we want to show that this thing actually works and it wouldn't hamper usability So that's why we use Google Cloud speech where we. Applaud the recorded messages and transcript as returned but it is possible to do this locally as Google has shown by changing Android or if Android allows other input channels for the transcription service. [00:26:28] For implementation we envision our virtual assistant to be embedded with the phone app so when the phone app every phone call comes in a virtual assistant intercept your call but in the same way a current Android system does not allow interjecting outgoing voice messages. Without Android or smart fixations So that's why we implemented avoid app to conduct or use a study and to see how the virtual assistant performs in blocking unwanted cause but if I enjoyed changes or if we changed Android Oist modifications for this it is possible to embed it in a phone app just like Google pixel did. [00:27:09] We conducted the user study to see how the virtual assistant works or not exactly how it works or how the users feel and using our robocall guard. So for user study we recruited about $21.00 users. And all of them were from Georgia Tech so they were mostly tech savvy university students and. [00:27:34] All the users played the role of a color and a color so for each experiment we had a pair of users that say we we have user a and user b. when user is playing the role of a color user b. is the Cali vice versa user b. is acting as a color user b. user is the called me at the time. [00:27:53] Now we have 2 experiments here and within each experiment there are 2 scenarios so for experiment one the caller was provided with the correct name meaning that we have our view IP prototype app and we have trained it with the correct name for this use a study the correct name was Taylor so when the caller when the user came in for experiment one we said that the correct name is Taylor and for the 1st scenario the caller was given a script they were asked to read this line when the virtual assistant picked up and asked for a name Hello can you please forward my call to Taylor and since they said the correct name the call was accepted and it was forwarded to the colony on the other side the caller picked up before the caller picked up the call he was shown the transcript of the call and the call he was told that only if you think that it would be a wanted call for Taylor only then you should pick up so the caller was playing on a roll of tape so the call you see is that a call comes in which says Hello can I please talk to Taylor which is a want to call for Taylor to call it picked it up and then the caller and call you could have a natural conversation in the 2nd experiment the caller was not given a specific script they were asked to call their friend Taylor they were saying that when you know that this is a call you're making to your friend Taylor to make movie plans and everything else was the same. [00:29:22] For the 2nd experiment the caller was provided with an incorrect name or no name at all in the 1st scenario the caller was given a script Hello can you please for my call to Robert So you see that since Taylor was the correct name Robert was definitely an incorrect name so there called was not forwarded in the. [00:29:43] Last scenario the caller was instructed to make a call twin office trying to sell a computer so there was no name involved here didn't know what name to say they were just playing a role of a telemarketer. Let's look at the use of study results so these are the 4 questions related to the caller and experiences so the 1st 2 questions are for callers the 1st question was it was easy to interact with the view question one is the blue bars so you see that most of the callers agreed that it was easy to interact with the view the 2nd question was the delay you experienced before the other parson other person responded to the caller to the call is acceptable the 2nd question is a red bars so most of them were either neutral or agreed that it was acceptable. [00:30:35] The last 2 questions were for the Callie's So the 3rd question is the transcript was able to provide sufficient information to infer the topic of the incoming call and the yellow bar as you can see that most of them were either neutral or Agreed some of them disagreed and I'm going to. [00:30:52] Explain a bit more why some of them disagreed. One of the reason is when we say that make a call to your friend Kaylor to make movie plans so when the virtual assistant picked up and said Hello can you please say the name of the person you're trying to reach most of them only said Taylor nothing else and they're called was forwarded and when the call e phone rang and they saw that the transcript only said Taylor without the transcript was not sufficient enough. [00:31:22] And for the topic of the call now a different game shows that this is a want to call because it said the correct name it said Taylor but it didn't it didn't say what the call was about just say Taylor now since we don't have any control over what the caller is going to say there's really nothing much we could do to improve this. [00:31:41] The 4th question was the transcript was able to provide sufficient information about the blocked calls which is the green bar so you see that most of them agreed are strongly agreed that he has the sufficient transcript was sufficient enough and the reason is for the block cause either they were given a script like hello can you for my call to Robert or they were trying to sell a computer or we also made because it was an actual call recording so it was a longer message and the transcript was big enough to give some information of what the caller was about but these are some general questions about the virtual assistant a 3rd of the callers and the quality the 1st question is. [00:32:24] The founder of our benefit to them because it provides prior knowledge about the incoming calls and most of them agreed that it did it was indeed beneficial the 6 question I think I would like to use an applicant for the v.a. frequently most of them agreed and 7 Question Most of them felt that they were comfortable with the virtual assistant intervening in the phone calls. [00:32:53] Now let's look at the v.a. performance of how the video was a fission in blocking these unwanted calls now there could be 2 scenarios where the v.a. is interacting with the human caller or the v.a. is interacting with the caller. If it's interacting with a human caller there are 2 cases either it's alleged caller or an unwanted caller. [00:33:15] Alleged with caller we have these 4 experiments as I've talked about before with each of the $21.00 callers and the 1st 2 experiments they were given a correct name so all of these calls should be forwarded and all of these calls of the said The correct name they were detected as correct name and detected as legitimate callers and the virtual assistant was able to forward these calls. [00:33:35] For the unwanted callers in the 2nd experiment the callers were not given a name at all or de were given an incorrect name so in this case they were all playing a role of an unwanted caller and all of the these calls were blocked and they were stopped from reaching the user the user wrist provided with the notification of this blocked call and different script of the conversation. [00:34:00] Let's talk about a bit with interaction with the caller so we have a data set containing about 8000 more than a 1000 calls coming in to 100 part during April 23rd to me 6 in 2018. Now since these calls are coming to a honeypot it is possible I mean by nature these are unwanted calls but it's possible that there are Ms dial calls into or they're noisy cause into this honeypot So we perform topic modeling to filter out. [00:34:30] Interesting campaigns like Tech Support free cruise Google listing things like that so we extracted out these interesting campaigns or cause regarding 2 specific campaigns that are related to cause and then we perform d v scan clustering on this to filter out exactly what these campaigns are and which messages are or which recordings are related with each of these campaigns so we were able to create a cluster of $79.00 robo calls Now each of this cluster either they were identical calls or they were very similar causing the saying a similar message. [00:35:10] So we chose one and we chose randomly one call from each cluster and we made a call to our virtual assistant saying this message and all of these calls were detected as I want it. Now we are detecting desirable cause but we are also labeling whether it's a human color or color with interruption. [00:35:31] With trouble color labeling what we have seen is because that are shorter then 20 seconds are labeled sorry because that are longer than 20 seconds all of these are because our labeled accurately because we interrupt them and there is no stopping from the site so we are able to label them correctly however because we're mislabeled the reason is We are interrupting at the 20th 2nd now let's say the robo call is 15 seconds they talk about 15 seconds and then they're silent there's nothing else so after 28 2nd you would only find silence and you would think that this is a human call or you would mislabeled them. [00:36:08] But with further analyzes what we have seen is even though these are short because now the robo calls are trying to achieve something they're either trying to fool you are trying to get something from you so they have a purpose within those short 15 seconds they have to convert message so what we have seen as 86 percent of those horrible cause are trying to tell you to do something like press 9 or enter one press a digit or president to talk to a representative or things like that so most of them are trying to see the sports press or enter which alleged I'm a human caller would not say when you ask them who you're trying to call or why are you calling they wouldn't ask you to press the number so if a call contains these key words we label it as a rule because for the short term because in this way with this added step we are able to label the ninety's that more than 97 percent of all because correcting you know let's compare it to guard with other commercially apps available. [00:37:09] Like I said most of the apps right now hi are true color you mail they're relying on Blacklist there's only Robel color which claims to do call content and a license to detect and want to detect robo calls not unwanted calls now this is another issue with killer as well that killer is only to try mining robocall or versus human call or if it's from a telemarketer or a debt collector or an actual human being who's involved in a scam wouldn't be able to stop it because they're not a robocall are. [00:37:42] There to do some experiments with. This is a small scale experiment so we use are 20 verify to your phone number so this is not a bad phone number or a known militias of blackness a phone number so. We a picked a phone number like that and we randomly chose 10 robocall messages let's say free cruise tech support and so on and we use that number to make these called Trouble killer Now keep in mind that this number is a good number meaning that this number was never used for scam but the message that it is playing is a robocall message so it's kind of like caller id spoofing I'm spoofing another number but I'm playing a role because messages we saw their trouble killer was not able to block any of the real cause even though they contained campaign messages or cruise or something like that but our virtual assistant robo call Guard was able to block all of these 10 calls. [00:38:42] The 2nd thing we did is. We spoke to 10 phone numbers so what it is let's say we chose f.t.c. is data set so f.t.c. contains user reported comments about. Bad phone calls or a scam phone calls that you receive so we chose about I think it was for August something I forgot So August 28th or something so it's a month old all this 100-9000 Yes So we picked 10 most popular phone numbers from August 2019 from f.t.c. So these are the 10 most popular phone numbers that people have complained they were really annoyed with those numbers. [00:39:25] And we perform the experiment in September so these phone numbers are very popular in making phone calls and they are a month old so it is expected that these 4 numbers will end up in any blacklist that these commercial apps are using so we used schools card another app to school for each of these 10 popular phone numbers and we made calls for global killer and we saw that 9 out of 10 of these calls are real killer was able to block it we didn't perform I mean the same thing goes for our app as well since these are called messages all of them are blocked but the interesting thing to see here is these are known bad phone numbers Robel killer was able to block them but when we used a good phone number or was not able to block them even the killer in their path and they claim that they're doing called Content analyzers but in practice they're mostly relying on Blackest to furder. [00:40:21] Do to put forward the dense into this we did another small experiment which was we took the least popular phone numbers from the day before so we performed this on I think September 9th so from f.t.c. is reports we took the least popular phone number so the least popular is definitely one any phone number that has one complaint so these are very new phone numbers and it's expected that by the next day they won't be available in any of the blacklist we chose 10 phone numbers like that and we've made phone calls to prove a killer and real killer was able to block a new 2 out of can cause versus real color which was able to block all of them so that even though this is a small scale experiment it shows that what the real killer in practice there currently relying on Blacklist more than call content so with caller id spoofing it's not as good as a robo call guard performs right now. [00:41:22] So after the slide I had discussion and cons. So with discussion I'm going to talk about some of the limitations and some of the good sides of the guard one of the biggest limitation we have here is we are not defending against targeted attacks so the robocall or that knows your name would be able to pass this challenge now in a future work we are working on those that even if they see the name we're working on how we could stop the caller. [00:41:54] The other limitation is let's see there is a human caller who calls you and they couldn't say your name let's say this alleged callers a call that is wanted for you but since they don't know your name they wouldn't be able to pass this challenge this is one more limitation now the reasoning with this is even though they're called would be blocked you would be immediately notified of a blocked call and you will see the transcript what the other person said and also the name of the other person during this processing we also asked the caller's name so you wouldn't name or you would know who called so if this is a wanted call for you you could immediately call them back now the good sides of robo call guard our virtual assistant is that it works with caller id spoofing even the presence of caller id spoofing since it does call content analyzes it would know if it's on want to call or not and you would get some context before you pick up a call and it not only works for unwanted callers but unwanted live human callers like telemarketers or debt collectors it would be able to stop them so for conclusion. [00:43:05] We have. Made this call to type since we were not able to overcome the Android limitations I mean we could do this it would take more time but that was not in our school research so we wanted to make a prototype and perform the user studies just to see how the users react to this kind of experience experience and most of them were comfortable in using the virtual assistant and it performs pretty decently when blocking rule because I'm want to go because. [00:43:33] That's all thank you very much and I would be happy to take any questions or any feedback you have thank you. And sorry for the computer turning out. Yes please. So we chose this imperatively the reason is when the virtual assistant picks up and says hello you have reached the virtual assistant Can you please say the name that takes about 5 seconds and then we allow about 15 seconds. [00:44:06] To the caller to say the correct name so there's about 20 seconds time before interrupting we allow this time to the other party to say the correct name or let's say it's a human caller and they tend to say longer sentences like Hi I'm meant blah blah and I'm trying to reach them this is the purpose of my call so we allow of with error of margin we allowed this 20 seconds so after this 20 seconds even if they had haven't said the correct name that's when we decided to interrupt so that's why we have this predefined time stem that this is when we're going to interrupt and doing only during this time going to look for voice activated protection whether they're silent or not. [00:44:49] Yes from the call. The time starts from when the call has been picked up. Again. So what it does is when let's say you have called someone who is using Virtual the virtual assistant you're called would be picked up the virtuous woman say that it's a virtuous and can you please say the name and you know the name you immediately say the name for example with that hi I'm trying to reach Taylor immediately the virtual assistant would be able to detect the correct name with just Taylor and they would say that while I'm forwarding your call can you say your name so while they're forwarding your call you say that let's say your name is John you say Hi I'm John Snow and to forward your call and the caller the call you would see that John Snow has called and this is what they have said to you. [00:46:18] That's a very good question so let's say I guess it would I don't have an exact answer for this very good answer for this right now but I think that with more as more people tend to use it they would become more familiar with this app and even with the word or food delivery system they would definitely know who they're trying to deliver it to they would know your name so if they become more familiar to the app and the reach this type of called screening applications are virtual assistant there would be more interested in seeing the name right now there's a very less people using this so they don't care they don't care about going up to this challenge solving this problem and then reaching you so maybe that's why they don't care right now but as these apps become more familiar these types of calls screening things people would care more I guess. [00:47:13] If there are more questions. Thank you very much. Well it's kind of like a motivation of why we did this so our previous work was on blackness I've presented this in the cybersecurity lectures well before I think the year before this one so that work was mostly on blackness back then I think it was in 2018 so all the applications that were available out there these robo call defending applications all of these were depending on Blacklist so true color and all that I think killer came up in 2000 and something I forgot the year but most of these were using this blacklist and our current project project back then was evaluating how good blacklist worked and we thought blacklist were working pretty well but we were surprised that it's only effective like 55 percent of the time so then we came to thinking what else we could do now there is Google Voice what Google Voice did or does right now is if you're using Google Voice a call or a call comes in and Google assistant will ask the name of the caller so they would say who are like What is your name or why are you calling but they don't actually filter out these calls for you and for some people who I guess most of us receive multiple robo calls per day and if your number has been around like some numbers the tend to be more dirty than others so you receive more calls and you receive a multiple of these will because and they keep on coming to you when you are continuously and annoyed with the ringing of your foreign lead to any time during the day so with Google assistant or Google Voice this wouldn't stop your phone would still ring and you would still need to intervene what you want to do with your call versus if you had a virtual assistant which works like a less sophisticated human being let's say and they filter all these calls for you so only if the thing that this call is legit Amid you would want to know about this call you're made aware of this call Otherwise your phone doesn't ring the virtual assistant takes care of it and instead of bringing your phone let's say it 20 times a day you had just received a being notification that you've this call has been blocked then you know what the call was about. [00:49:45] So I hold that answer to your question. Is there any more questions you. Get in that that's a very yeah that's a very good question that's one of the I would say the key issues with the system we have right now is let's say most of the issue and then names tend to be harder for him for Americans or Europeans to pronounce so I would say that since Q were spotting is not the scope of this work right now so right now what we're trying we're trying to introduce this virtual assistant in the call blocking applications that are there right now and show that instead of just relying on caller id if we relied on call content that would give us much better results but you're right with with name there is an issue now the name is just an example of a challenge with our future work we're currently working on something else in addition to the name so I would say that since with a fast advance the keyword spotting algorithms would definitely work better and with snow boy I have tried with my name most of the time. [00:51:05] With other people who have never heard my name just read it somewhere and I've asked them to say say it to snow Boy boy I was able to detect it correctly even though it was not pronounced exactly like I trained to be so with these kinds of area advances I would thing that you were spotting algorithm would get better and the step would also get better but this is right now not the school for research so we're not concentrating on that. [00:51:31] Thank you for the question. Are there any more questions. This is one other thing and thank you for Question So this is one of the thing that we have tried is. So let's say my name is Sharman you put it and. What we have experimented it but let's say. [00:51:57] It and someone says Just when you are just going to a small boy is very intelligent in determining if even if they say half your name they're able to. Detected but there's also some issues with the let's say I was training at the trial Bunny and I only stayed by me and it was able to detect it as far as well so I would see all of these fall in the keyword spotting area so since we're not working on that we would kind of leave it leave that for the ai researchers to come up with something better than what is right now but right now it still works we tried with different combinations like Mr Shaw when you found it or maybe someday Dr Sharp when you point it so tried with different combinations and see what work and right now it works pretty well but you're right it could definitely improve there are definitely issues with the name spotting algorithm right now but we're hoping that it would get better and then the virtual assistant would definitely work better as would. [00:53:00] You. And I couldn't hear it can you say that again. So let's say because what you're saying is an on call is detected as a rule because it is a detected as an unwanted call and then or overcall. Ok so that would happen when if it's detected as an unwanted call but it's a want to call one of the possibilities is they don't know your name in for our system they don't know your name but this is a wanted call so their call has been blocked so like I said that you would get the user would get a notification and the transcript immediately and they would be able to call back if they wanted the other scenario is they know your name but. [00:53:52] Somehow the spotting algorithm was not able to detected in that case maybe they're called was forwarded I would say made the same argument is one of them is that you were spotting algorithms will get better so this thing will work better in detecting the correct name of they actually say the name and also that even for a false positive you would get a notification and a transcript immediately and you would be able to call them back if it's actually very important to you. [00:54:28] I'm sorry. Yes So if it's an unwanted call or so in total we allow about 35 seconds for them to say the name so after we have made the greeting there is 30 more seconds for them to say the name within this 30 seconds are actually within the total length of the called within the 35 seconds of the whole call during any point if they say the correct name the called would be forward even after the 28 seconds. [00:54:58] But even. In doing this 35 seconds they don't say the name they're labeled as an unwanted caller now in the unwanted color scenario we have 2 more deals so we have a color and honorable color or human color so in the 20th 2nd time step if there interrupted be labeled him as human color so they're on wanted since they didn't say the name but they're human because they were interrupted for over color didn't say the name So definitely they're on wanted but they were also not interrupted as well so they're labeled as a rable color so anyone who doesn't see your name within the 35 seconds is going to be an unwanted caller and their call would be blocked meaning that your phone wondering but you would definitely know that a call was blocked. [00:55:56] Yes so the virtual assistant would say that it has blocked a call and we have these in the app we have these different folders and human so they would say that which label it has given to the block called so if you go to the robo calls for her to see the history of your cause or it's a pop up notification actually so you would say that the virtual assistant just blocked a call for you and this is what this it. [00:56:24] It would block the number it would just block the call yes yes so you can definitely call them back it would not it's not abating its own blacklist so it will not block the caller id it will just blog that specific call. Yes. So there are these want to draw because like you said maybe a doctor's appointment or a public school announcement so one of the things we talk about in our paper is we can have these numbers in our white listed in our white list so these will because are able to pass through us but let it snotty in our white list so you have never seen this number before or you don't have it's not your regular doctor's office somewhere you went new and in that case if they don't know your name if they don't say your name generally one doctor's office or pharmacist call they typically say your name with those kinds of protocols so they would be able to pass this challenge but if they don't see your name in that case they would be blocked but you would get a transcript and if you think that this is important to you you could be able to call them back. [00:57:47] I'm sorry. Not to use a bill in the sense that you can choose it with your phone app because like I said for the Android or us restrictions but it is we developed it as a viewer absolute to use it the caller right now hold the caller on the call he has to have the app and they have to call it through the V.I.P.'s So this was just a prototype we didn't want to change whole Android or us for this but if Android ever moved this restoration or if they made this modifications we would be able to embed it with the phone app itself and then your phone calls would be interrupted and you could use the virtual assistant. [00:58:30] Is there any more questions but thank you all for coming and thank you very much for your very good questions and your important feedback thank you.