[00:00:05]
>> Great thanks Keith. Thanks for the invitation. I think the title the title says what this with this talk is about I don't ever I never actually imagine that I would be involved in this line of research I've got. A long career in cybersecurity. But I was kind of dragged into this issue of election security when I became dean and Georgia Tech as had had various involvements in the state of Georgia.

[00:00:37]
Because Georgia was one of the 1st states to adopt electronic voting machines and. As you will find out very quickly in this talk the no work they've never worked it's been an embarrassment to the to the state into the into the country and so Georgia Tech has had ample opportunity to weigh in on.

[00:01:02]
Boating machines and voting technology along the way. A kind of gut. An education about. Not only boating technology but about elections in in general and one of the things that happened was it kind of caused me to rethink. What I. Conceive of the computer scientists role in in things like election election technology and it really I hope you get a sense from my talk that this is really an opportunity to take with a fresh look.

[00:01:43]
At processes that we all kind of take for granted and apply things that we that we know how to do as information technologists as human centered computing professionals as usability professionals cybersecurity specialists and so you know I'll just begin my talk by by telling you that nothing that I'm going to say today is.

[00:02:10]
Is a. Is a is a revelation I mean the the idea that American elections are a mess as kind of been built into the fabric of what this country country is and those of you who didn't grow up in the in the U.S. maybe haven't heard all of the all the stories This picture is is from the election of 824 in which John Quincy Adams is purported to have bought off his presidency.

[00:02:44]
Defeating Andrew Jackson by making Henry Clay secretary of state or whatever whatever it was is called the corrupt bargain and so you see. In that period of time newspaper articles and cartoonists kind of drawing pictures of people in the street and upset about about the corrupt corrupt bargain.

[00:03:06]
It's always a long time ago today. People are drawing pictures for the newspaper and marching in the streets and upset about I'm a robot. Thank you. People are upset about about the way elections are run maybe for different reasons not not because not because we're trying to figure out what what a democracy is but but because the world is changed and the way that elections can get can get off track has changed dramatically and the impact it has not only in the country but the world has changed dramatically.

[00:03:58]
I don't normally begin talks by citing judge's orders but this one is particularly interesting this is this is an order that that Judge Amy Totenberg a federal judge in U.S. District Court northern district of of Georgia wrote. Just 3 weeks ago and the reason the reason I'm showing it to you why I'm showing 2 for 2 reasons 11 is.

[00:04:24]
I don't think I've ever said this in public before you should all go out and download a copy and read it it's like reading a novel and if you if you just if you just look at the at the title headings for the judge's order you get a sense of.

[00:04:40]
What she's what she's talking I don't think I've ever seen a judge's order with a subsection in title The past is prologue which is the history of of how we got to this state that we're that we're that we're in here. What's interesting about this is that this is the 1st time that I know of that that a court a federal court has intervened in how elections are conducted anywhere and this isn't just an intervention into how elections are conducted this judge said it's very likely that the way the Georgia conducts its elections is unconstitutional that was the point of this lawsuit.

[00:05:22]
The way we conduct elections today with technology is unconstitutional and so what I want to do is kind of walk you through why I things that you probably have in your mind about about elections. Are maybe kind of a over simplification of of how the world really really works why is it that the stuff that sounds pretty simple we have elections for student body presidents we have elections for for.

[00:05:53]
For for heads of of clubs. We have elections all the all the time it just you collect the votes and you count them up and whoever gets the most votes wins that sounds like a pretty easy thing thing to do why is it that we're making such a big deal of this and the reason is it's not as simple as it seems and so I've organized the talk around kind of pointing out some complications that you may not have thought of and how technology impacts that and more interesting Lee Maybe for this group.

[00:06:28]
What you should think about this as graduate students and researchers because there are some real unsolved. Problems in. In computing that should be addressed and they should be addressed by people at institutions like ours so the 1st complication is that there's no one in charge there is literally no one in charge of elections in the United States the only thing that the Constitution of the United States says about elections is in Article one Section 4 and.

[00:07:06]
It uses the language of of the of the 18th century but but but what it basically says is is we're not in charge in the United States the states are free to run elections as they choose. And for most states that means that municipalities townships counties are responsible for running for running elections so when you watch on on on the night of a presidential election all of those election returns come in.

[00:07:42]
You're seeing the result of somewhere between $10.12 separate elections that were held that day and somehow all of that has to come together in in give a result. You know people who wrote the 1st article of the of the Constitution I think didn't really intend that we would have things like voting machines and we would have we would have.

[00:08:13]
Hundreds of millions of people. In the in the country and that we would be spread across all these all these time zones so so when problems came up. They tried to solve them and in the case of of national elections they found lots of problems to be to be solved.

[00:08:32]
So we've learned later on much later on for example that not only are the states post to conduct elections but they have to be conducted fairly believe it or not no one thought to write that in the constitution and elections have to be fair was what is fair mean it means that everyone has has an equal right to vote.

[00:08:54]
Someone observed very early on in the in the 19th century that the people were taking their votes out in the alleyway and selling their votes to the highest bidder so they decided that that wasn't really a great thing to do and so and so they adopted the British idea of a secret secret ballot that wasn't in the Constitution either but now you if you're a public election in the United States you keep have to keep your ballot secret by law you can't even take a picture of it you can't take your i Phone into the polling place and take a picture of your of your ballot that turns out to be illegal because you can take that out and prove that you voted for for.

[00:09:35]
Whatever and someone might be willing to pay for that has to be open to all eligible voters eligible voters in the 1000 in the 18th century were male and they were they were female. One person one vote so so with regard to race or national origin if you're a citizen you have the equal vote to anyone else and it has to be secure from from tap where you can't just pick up ballot boxes and put them on the back of pickup trucks although that happens a lot and kind of carry them to some place and and tamper with them with the votes that's also illegal but these things happened over the course of decades couple 100 years and so and so when you think about the initial complications involved in in elections it's not a simple problem you can't just say well there's this nice abstraction of what a what an election is in that's the problem I'm going to solve it is literally this amalgamation of things that has to have to happen and they have to come together in such a way that and this is really the unbelievable part that that that when an election is held the person that loses the election the party that loses the election is convinced that they've lost we think about that for a 2nd that's a really difficult it's a really difficult hurdle to have if you if you if you engage in a in a close election and you want to win you now have to step back and say I see the results of the election are clear I have lost the election and so and so all of this kind of makes for a complicated.

[00:11:20]
Complicated setting and. The country sort of limped along with with with this complication for a long time until. The 2000 election. The election the George Bush. Gore election was decided by I don't even know how it was decide it was it was it was decided decided by a secretary of state in Florida it was decided by a Supreme Court that was decided by by a bunch of lawyers I got together.

[00:11:56]
In a room but it was kind of an unprecedented thing and it had to do with the fact that we had sort of exhausted the available technology to conduct elections given that complicated situation that I just that I just told you about so the 2nd complication is that people got fed up with what happened in 2000 in 2000 remember.

[00:12:18]
The. The what was so close in in Florida that they kept doing recounts after recounts and the recounts Woods flow from Bush to Gore you had to stop that at some point the Supreme Court said Stop it stop it now and this picture this this lawyer who's got a magnifying glass and is looking at that holes punched in cards.

[00:12:42]
So-called hanging chad is the thing that people have in their mind as the turning point in which the U.S. started to automate the election process of thinking in 2000 was that computers would be the way to get out of this kind of wind that instead of having all these clues the system scattered around the country we would have a single act of Congress covering the entire United States called the Help America Vote Act which would computerize elections the federal government provided $4000000000.00 to the states to do that and the states ran off to spend that $4000000000.00 by purchasing computers that would some of which would be things that you would use on Election Day some of which would be used to tally votes and report votes and record who's registered to vote.

[00:13:30]
And so on. So that was that was triggered by the problem with hand Mark paper ballots you know we we had we had up until that time because there were no computers or primitive electromechanical devices anyway we had the problem that people would. Would have to go up to it's a private booth and with a pen or or or by making a hole in a punch card indicate their choice.

[00:14:01]
In an election and and the the the HAVA act took a look at all the problems with that and said well we can fix those if we if we computerised So what are some of the things they tried to fix they tried to fix the fact that that the.

[00:14:19]
The polling place wasn't really accessible to voters with various kinds of comparisons so if you had any kind of disability at all you were kind of disadvantaged in that environment computerization was thought to be a way to make voting more accessible to a variety of voters. If you had a paper ballots in which people could write anything they would write crazy things.

[00:14:45]
So this is this is a famous ballot called called the lizard people ballot. In the 2016 Senate election in Minnesota. Franken you know Al Franken from the from the Saturday Night Live became U.S. senator his reelection campaign. Hinged on. Human beings looking at ballots like this one and saying what doesn't make any sense we're going to throw that ballot so this this is become the lizard people ballot and it's one of the examples of people choose to to to show how belts can be difficult to interpret and then in then.

[00:15:25]
Ballots are light you put them in a box the box is pretty light someone can come up and scoop that box up and carry it away so physically securing paper ballots was a difficulty believe it or not the guy in the blue shirt is not me. It is.

[00:15:41]
I don't know who is some actor someplace someplace. But but there were lots of lots of difficulties with with hand Mark paper ballots and the HAVA Act was intended to. To solve some of those problems which leads us to the 3rd complication and really the reason that we're there we're talking about this today.

[00:16:04]
All of the computerization you should notice the word computer in the word computerization any time you introduce a computer into any process at all we now know that that that that computer runs the risk of being misprogrammed mis configured misused or hacked by 3rd parties and that hacking part has become very visible since the 2016 election is probably the reason maybe the only reason some of you are aware that there's a controversy about elections at all is that is that you know we have all of these reports that that that foreign actors Russians whoever has infiltrated the U.S. election system and possibly change election election results so what do we do about this well you can you can do what what what was done basically through the 2000 which is to kind of ignore the problem thinking that well we would notice if if somehow one of these computers were hacked.

[00:17:14]
But but over the course of a decade or so. A few cybersecurity. Centers started looking at how difficult it would be to hack. An election and there are lots of different kinds of computers in the election system they focused on the machines that are in the polling place the touch screen machines that you touch in records your vote and that's what gets gets gets counted.

[00:17:43]
In particular they looked at the kinds of voting machines that were used in the state of Georgia things called direct recording equipment this is this is this is a Deore. Voting Machine is the thing with the with the i Pad looking screen on the on the left. Those those are unspeakably horrible computers.

[00:18:07]
They they run they run Windows C.E. operating system. Which which I used to sell and I can I can if you haven't looked at it I can guarantee you there is no security subsystem in a Windows C.E. device. And so and so it's not surprising that that people started looking at ways to get into the guts of a voting machine and in unbeknownst to the people that are running the elections flip votes.

[00:18:39]
That has been done over the last 20 years for machines exactly like the machines are you there we use in the state of Georgia. And and I keep track of this I don't know of a single attempt to hack one of the Georgia dairy machines that's failed. Now here's what I mean I don't I'm not saying that that there's not some person in the basement someplace that was playing around with voting machines in fail to do it I'm saying that this is like 8 ball so if you if you call green green ball in the corner pocket you've got to make that you've got to make that shot so so anyone who said I can do this too but voting machine like the ones you see on the screen here who has then tried to do it has succeeded at doing.

[00:19:24]
And and it's become successively easier to get into these machines remotely to change votes the most recent innovation was due to this guy here Alex Halderman is a faculty member at the University of Michigan who's showed how the magnetic cards the P.C. cards that are yes P.C. card still exist magnetic cards that are used to record the votes on the individual machines can be infected with malware you put that malware on to the voting machine because the card is literally inserted into the voting machine itself and the those cards go from machine to machine so malware spreads through through a voting installation a polling place very very very very quickly.

[00:20:14]
The other thing about these machines is that the people who wrote the HAVA act didn't anticipate that that manufactures would build machines with no paper backup so the only way that you have to determine whether or not the election was fairly decided was to go back and recover but what do you recounting your recounting electronic records that are stored in compromised devices that's not very helpful what you would like is some paper record of what the voter intended and that's not possible so that's a complication of.

[00:20:55]
The 4th complication. And it's essentially a complication that that's a result of realizing how bad the current environment is so people have looked at the 1st 3 complications and we need to have a different way of of of voting and so manufacturers started designing these things called ballot marking devices which are intended to replace D. or E. devices they're not so much computers for recording and holding storing the information they are simply $3000.00 pencils that are used to mark to mark the.

[00:21:35]
The the devices which sounds like a good idea. I'm not sure about the $3000.00 part sounds like it's maybe more cost effective to use a use a pen but but but people seem to be an amber and with with these bell marking devices they have introduced themselves a new complication so Judge Totenberg order says you've got to stop using direct recording equipment you have to move to something else the state of Georgia like a lot of other states to say well we're going to move to these kinds of computers.

[00:22:08]
To to conduct. Conduct elections. That gives rise to most of the complications I'm going to talk about next one thing that that may have occurred to you is that there's a complication that I haven't mentioned. Something that sounds like a complication. It's possible that. That ballot marking device can cheat or any any way of tallying votes can can cheat.

[00:22:40]
That is actually that is actually a portion of the voting problem that has been solved. Or at least we have we have workable solutions with workable engineering engineering solutions if I have a record of the intention of a voter. I can I can compare the result that was reported by the voting machines to the actual ballots that the voter marked.

[00:23:05]
And and conduct a statistical test called a risk limiting audit. That will tell you whether or not the result differs from what the voters actually. Actually did and. This is this is a. This is a technology that was invented by one of my one of my co-authors about 10 years ago it based it basically allows you allow you to compute a probability of error as you go through sampling randomized randomized ballots comparing the percentage that that that you observed during the sampling process to the the percentage that was.

[00:23:46]
That was reported. But it does require you to have some record of voter intent and most ballot marking devices in fact all ballot marking devices throw that away so like the D R E's there is no durable record of what the voter intended there's only a durable record of what the machine says the voter intended and that gives rise to a lot of interesting problems in computing and here's here's a way to think about these these problems is a 5th complication is what what we call the essential security flaw of ballot marking devices.

[00:24:32]
A ballot marking device because it's interpret user input can deliberately misinterpret that input you can put malware on it can be missed configured can be misaligned. There's no way to prevent an undue detected discrepancy between what the voters see on the screen when they mark a ballot and what's what's recorded.

[00:24:59]
That property is called software independent so so one of the security principles that underlies all election security is that any technology has to be software independent that is you can't make special reference to the software that's used to record the votes in your trust in it in order to have trust in the final the final result so an undetected cheat can't result in an undetectable error in the in the in the result there's a 2nd condition that that we came up with pretty recently and it's that.

[00:25:38]
Even if somehow you managed to to to detect the machine that was cheating you have to be able to correct it somehow there has to be something that you can do about it so this this principles called contestability. And the essential flaw of ballot marking devices is that most of them all of them that you can imagine.

[00:26:02]
Are not software independent there are there are some ideas for how to make them software and different but none of them are contestable that even if you managed to detect an error in the vote there is no action that you could that you could take that would correct that.

[00:26:21]
That are now why you ask. Can't you just say this with the voters responsibility to check check for errors I mean the bell marking device can print out and the one that words SEPTA by in Georgia prints out something that looks like a receipt that's the thing here with it has bar codes on it and just check it every voter has to check to see whether or not the machine cheated it's the it's the ballot that gets scanned and recorded it's not the it's not the electronic record why can't you just why can't you just do them and the answer is.

[00:27:05]
Kind of a deep answer. I'm going to kind of take you through the the levels here. First of all let me let me say that absolutely voters have to be responsible for checking their own work whether it whether your hand marking a ballot or pressing a button on it on a screen you have to you have to be responsible for your own work but if you're using a ballot marking device you're not only checking your own work you're checking the machine and that area detection step is not something that human beings are very good at so voters are responsible for checking their own actions they can't check the results of the machine so that leads us to the 6 complication the thing that I said saved checking the results the existence of audits become impossible if the voters can't verify ballots if they can't check the results of ballots to see whether or not their vote was correctly you correctly record and that's because the statistics behind audits I'm not going to drag you through everything but the statistics behind audits essentially is an interactive computation of this statistic T. So so you're sampling randomly sampling ballots and and what you're relying on is the fact that you're looking at a random variable to determining which way it goes winner or loser and then and then updating the value for for T. that has to be an observation that can't be something that has a probability distribution associate we can probably you can have a loser ballot you can't you can say well I'm going to maybe give it to the winner or the or the loser has to be it has to be an observation if you don't have.

[00:29:01]
The marking that you made as a voter on a piece of paper that's been securely stored at the time the audit is held you can't trust that audit trail and you can't conduct a risk limiting audit so so this this thing that I'm telling you about is literally the foundation on which all current election technology rests and if that crumbles your trust in the election has to has to crumble so you as you might expect there's a lot of controversy about others and a lot of research going on that kind of balances both sides of this this question I am coming down on the side that says that this is pretty devastating for voting machines in general that it's very hard to recover from this idea that voters have to be able to recall every choice they made in an 18 page ballot and that has to be that has to be locked in their head by the time that they leave the voting booth as opposed to writing those choices down on a piece of paper putting them in a secure vault so let me walk you through them.

[00:30:17]
So this is my cartoon of what what goes on in an election in which computers are used or used to vote you have a line of people outside a polling center you have someone like this person in the in the black shirt who forms an intention to go in and vote for someone that intention could have been formed weeks ago months ago or could have been formed just at the moment that they walk into the to the voting booth it doesn't matter you form an intention.

[00:30:51]
That intention is a psychological state so you don't have access to it no 3rd party has access to what that intention the only person in the world that knows what that intention is is the voter himself or herself. And the way that that gets known is for the voter to express that intention by marking about now you can go into the booth and make a mark on the ballot.

[00:31:23]
That's a direct observation of the intention that's what the voter said voter can check it if the voter made a mistake they can target the ballot and go ask for a new ballot but at the time they put it into the ballot box that's the best evidence you have with the voters intention was that's what a hand Mark paper ballot election is literally and then you go count the count the ballots and conduct risk Lamido audits and do all that stuff to make sure nothing bad bad happen here's what happens with the current generation of voting machines instead of marking that ballot you walk into an alleyway and you whisper your vote to a stranger that stranger takes what you said in either disregards it or properly translates it and marks the ballot for you that literally mark the belt they produce one of these one of these things and that's what gets recorded as your as your ballot so so what we're talking about here is this choice that you that that you have who who do what who do I trust.

[00:32:48]
If you had gone into the booth and had marked the bell you would have had an intention and you know it's written on piece of paper is that what your intention was Is this what your intention intention was and how would you know how would you how would you decide that leads to the 7th complication that's a really hard question.

[00:33:14]
that's a really hard question so i can give you i can give you the the observational part of this which is which is not quite so difficult maybe's surprising but but is it's not so the difficult if you give that task to to a group of voters actual actual voters in and and tell them used pressed the button on a belt marking device when it spits out that receipt checked that ballot to make sure that the machine didn't miss record your boat half of the people no matter what you tell them half of the people will be fused to do it half what we've used to do it this is been known for a long time this is been known since people 1st started talking about about boater border verification why don't they do it who knows this not important to the statistics if half of the ballots are in verified those of the bell as i call them poison bells the poison bellus are opportunities a targets of opportunity for hackers all you have to do is change those though it's because no one will notice it will the won't look at him but let's suppose you somehow managed to motivate a sizable fraction the vast majority of voters to toot to do this thing that we've never observed and and actually verify their their ballots half of the voters won't be able to tell if there's a change now this sounds kind of town or counterintuitive everyone's going to remember i voted for hillary clinton ari voted for for donald donald trump that may or may not be true But they certainly don't remember.

[00:35:03]
Whether or not a yes vote for Proposition 16 meant that they were for the thing that 16 was trying to prevent or against the thing that 16 was trying to trying to promote. And they probably don't know what a secretary of state is and so when they vote for the secretary of state they've made it may just be a name.

[00:35:24]
And judges particular those who won without party affiliations are just names you're not quite sure which judges you. You voted for but for whatever reason half of the people at least half of the people who decide to check their ballots can't tell if an error has been has been made and my assertion is that there's literally nothing you can do about that there's literally nothing you can do about that no encouragement no training is going to is going to help there's a group of there's a group of people who think that with with proper user interface design with proper ballot design you can you can encourage people to be much more effective at that.

[00:36:14]
Ballot verification number of things get in the way of that. 11 is that the rates of error detection that are required. Are well above 90 percent in order for the statistics to work you have to you have to 90 percent of the time detect an error if it's occurred in the in the bell you can't assume that the machine wasn't hacked you have to assume well this machine might have been hacked I need to verify this ballot and my verification is the thing that's going to stand as the basis for conducting risk limiting limiting audits.

[00:36:53]
Just for purposes of comparison a professionally trained proofreader operating under optimal conditions can maybe and take out misspelling so just just. Downright errors in text can maybe detect 80 percent of the errors in text of voter conducting a very similar task has to be essentially perfect in detecting detecting errors.

[00:37:30]
In this not only proofreading if you just look at sources of human error in any task. Detecting the error is much harder than preventing the error to begin with so people are pretty good at producing relatively error free text they're horrible at detecting errors in their own text they're pretty good at producing error free code it's pretty hard for for a peer reviewer working by him or herself to detect bugs in in code so you walk through all of the areas in which these kinds of activities take place in the numbers are somewhere between 50 and 80 in 80 percent so the world is not working in your favor if you think that you can detect detect errors there's also kind of this cognitive overlay to all of this.

[00:38:31]
18 ballot choices which is kind of typical for a ga ga ballot 18 ballot choices is a big ballot it's well beyond the George Miller 7 plus or minus 2 numbers so you're going to have to chunk this information somehow in order to in order to to remember it the fact of the matter is memory errors creep in at around the stage around around 7 or 8 a choices and people are simply going to misremember what they what they did they're going to shove what they did into an associate of memory network that is wrong for some reason and that happens over over and over again.

[00:39:15]
Even if they met remember correctly. When presented with an incorrect choice so the machine has now cheated it's presented you with with with an end an incorrect choice people are inclined through something called Choice blindness to believe what's in front of them rather than what they actually did in the numbers here are not encouraging So if you present people with choices very sad much simpler than than ballot choices.

[00:39:46]
Something like 13 percent of. The population will be able to detect an error that was made by just simple simply tell you here's a choice that you me I'll give you 2 pictures. And you choose one picture and then I'll surreptitiously switch that switch that picture and you say is this is this the picture that you that you chose 13 percent of the people will say yes that's not the picture I chose the vast majority of respondents are blind to the choices that they have that they've made because cognitively we construct the stories around stuff that we that we do and that that translates into.

[00:40:27]
Into making errors in things like ballot verification and on top of that there's all the stuff that you know about user interface design voters are not operating in perfect conditions. They're not bright 18 year olds they are tired stressed out middle aged people or older who are mildly cognitively impaired because people my age lose memory after a while and.

[00:40:57]
And they're stressed they want to get out of the out of the the voting booth. So none of this none of this is an optimal situation in which you want to be able to to check the results of a ballots so that's an issue. None of the things on this slide have been deeply researched.

[00:41:24]
You can you can bring analogies in from someplace with so we know because there's a long there's a long history and extensive literature on educational technology we know for example that that gooey increased cognitive load on anyone who uses them so if you're if you're a blind voter.

[00:41:43]
Or visually impaired voter and you have to all the sudden remember 6 screens in order to determine whether or not whether or not. You voted for John or Mary as secretary of state. That's done under a cognitive load and you're going to make errors you're simply going to make errors.

[00:42:10]
Brings me to the complications so the if complication is that is that the devices that we're talking about which have been developed to be more usable to more to be more it accessible are in my view absolutely not. And and I can say that with great confidence because there are almost no usability studies of these machines the only usability studies that I've been able to find are usability studies that are sponsored by vendors by sponsors of development sponsored by developers of the of the technology so so truly independent laboratory evaluation of the usability of voting machines simply does not exist as a discipline it's one of these weird.

[00:42:55]
You know black holes in the research literature that you fought you find usability studies of just about every other kind of information technology but this is one where nothing has has been has been done I don't want to underestimate the complications involved in doing this research. It's very hard to imagine how you would write an I.R.B. for this kind of research it's not clear at all where you would conduct the research you would have to if doing under realistic conditions you would have to have have to get access to actual polling places and because of things like secret ballot you're not going to be allowed to observe what's what what goes on Nevertheless it's.

[00:43:36]
It's an area that deserves to be studied I would I would say that the track record for. For touch screen voting voting machines is not very encouraging and. You know the the. The best results that you find these are results that are sponsored by developers many of the of the machines manufacturers of the machine are are that that on.

[00:44:08]
What is the Likert scale for usability those it there's a there's a there's a scale that that people use is it's a one to 11 to 10 scale the best the best that that direct recording machines ever score is is moderately usable moderately usable. Relatively unusable by visually impaired voters which is weird because that's who the machines are designed for.

[00:44:38]
Really unusable by cognitively impaired impaired voters. But but overall we know very little about about how to present ballot choices to human beings and. Be able to construct machines that are more and more usable. OK let me just finish with a couple couple observations one is. This is a pretty serious.

[00:45:07]
Problem for the upcoming election for the 20202020 election we just had an election last year in in Georgia. And you know the question is for people like me who are always you know ringing fire alarms and saying saying you know though it's been hacked it's been really hard to find evidence.

[00:45:28]
That the votes have been have been hacked will not true in 2018 so in 2018 we have I think for the 1st time statistical evidence that votes were hacked so this is. This is. A chart that shows. The what's called the under vote. For the governor's race Lieutenant Governor and go in governor's race in 2018 and what you're what you're looking at is is the number of.

[00:46:09]
Of votes helping with the scale so so it's the it's the it's the number of ballots in which a choice was not made so everyone who went to the polls. Because it was a governor's race voted for governor would be weird to think you would go in and not vote for governor in fact almost everyone everyone did and in a normal ballot.

[00:46:40]
Which you find out is a people get tired the ballots long and they don't know the candidates and so lieutenant governor's candidates were not as well known as the governor's candidate so some people left the black. And then more people left blank secretary of state and public service commissioner insurance commissioner So all these down ballot races that.

[00:47:02]
I did I can't keep all those in my mind. To vote for for for the Democrat for for governor and Republican for lieutenant governor and the home. So so what happens is that is that there's a natural drop off in every election in what's called the undervote. And it just it goes down the slope and that top line is kind of what it looks like.

[00:47:31]
Governor lieutenant of Secretary of State Attorney General. Agriculture commissioners that is that so so all of those races at the at the top as you go down the ballot people people lose interest. At this polling place in this in this in this precinct what happened was that everyone among the people who went to the polls to vote for Stacy Abrams and voted for Stacy Abrams.

[00:48:06]
Remarkable number of those voters decided to vote for the agriculture commissioner but not the lieutenant governor so this this thing that we call the shark's tooth here this this big this big drop off is something that was observed in the in the 2018 lieutenant governor's lieutenant governor's race the the interesting thing about this is that that black line on top.

[00:48:37]
Is the reporting of results from hand Mark paper ballots what do you suppose the orange line is voting machines so this was observed on the machines it was was it webs it was observed. To just poor proportion a degree in African-American precincts. It was observed on one machine out of 7.

[00:49:11]
In the precinct in which. In which the other 6 machines. All decided all statewide races for the Democrat except one machine decided for the for the for the Republican so so so Philip Stark they got the guy who is responsible for risk limiting audits. Took the task of conducting the analysis of of these machines of these results and presenting us with with the likelihood that this would happen by accident so how might this happen it might happen because people.

[00:49:54]
Were tired. Because I forget what some of the reasons are but but there are going to have been lots of reasons invented for this might occur fi kind of take all of those reasons in and normalize the results for for those reasons what you're left with is that the probability.

[00:50:15]
P. value of the of the. Of the of the test of the hypothesis that that this happened by by accident taken into account all the other ways it could have happened. Is is one in 10000. So this is a big deal this is a big deal it's that it's actually.

[00:50:35]
It's actually the 1st I think. Evidence that we have that that that things can happen in elections with voting machines that are unobservable to election officials that have an impact on the outcome. so what do you do what do you do i you know i'm i'm i'm in the camp that says that you you you follow the european model in the european model is is that is that you get rid of his many computers in the process is possible that that you due at the german high court says which is that if the if the average person can't understand how the votes were captured and tallied then the election shouldn't shouldn't be healed and and that's not where we are right now in in the united states although judge totenberg was order comes perilously close to that if you're a fan a voting machines because her order says that you can't use the old direct recording equipment machines and if if the bell it marking devices that georgists purchasing don't get installed or don't work properly you have to move to a hand mark paper bell system you have to move to a system system like this so i think i'll just leave you with that with that observation you know come back to the the club bar going to bait 124 it's always problems with elections these problems that are introduced by the use of computers in in the pulling place just seem to be very different then than the the normal chaos that surrounds us us elections and as i said a lot of the a lot of the problems are on an exam a people have looked pretty carefully at the cyber security problems we think we have an understanding of the of the threat models and and what's likely to happen with the various attack surfaces at these machines prevent the cognitive stuff that underlies it is completely virgin territory and i think some of these problems are really a really interesting to to look him so that's the end