Test The System

Many times we have a list of requirements that the system should do. Then we write software/build hardware that we hope fulfills those requirements.

Are we done? No. We must TestTheSystem.

All too often, we run a test and are surprised to find that the system fails some requirement. (Or someone adds a new requirement, and we already know without testing that the system doesn't do that). So we fix things up in ways that hopefully make it fulfill that requirement. Then we run a test and confirm that it meets that requirement.

Are we done? No. When we run a test for one requirement and it passes, that just confirms that it meets that one requirement. We must run the rest of the tests to make sure we didn't accidentally break something else.

TestDrivenDevelopment requires that we NeverWriteaLineOfCodeWithoutaFailingTest.

So what are the failing tests for a voting machine?

In particular, how do we measure the reliability of a voting machine system?

How can we decide which of 2 voting machine systems helps people make fewer mistakes? I'm assuming that humans make innocent mistakes when they vote and when they count votes. SecuringVotingMachines talks about ways to test if a machine has been maliciously tampered with. Here I'm talking about ways to test several voting systems to see which more reliably indicates voting intent.

With hand voting ("Everyone in favor of ___, raise their hands ... Everyone opposed, raise your hands ...") with up to a few dozen people, it's easy for everyone who cares to count hands and make sure the official count matches the true count. Voter intent is fairly easy to confirm -- "Bob, I thought you preferred #3" "Yes, if #1 were the only other option, but in this case I prefer #2" or perhaps "oops, you're right, I forgot about that -- let me pull my hand back down before they start counting."

Unfortunately, this makes the voters vulnerable to violence if they vote the "wrong" way (ElectoralEngineering). To work around this problem, the U.N. has mandated anonymous secret votes for elections (actually, "secret vote or by equivalent free voting procedures." - see http://www.un.org/Overview/rights.html ).

This additional requirement makes it much more difficult to confirm voter intent, and more difficult for interested observers to confirm ("test") that the official count matches the true count.

EditHint: refactor some of VotingMachineDiscussion here.

In particular, many people believe that the "butterfly ballots" did a bad job indicating voter intent. However, the only test I am aware of showed that all the other voting methods in Florida 2000 were even worse. So replacing butterfly ballots with something else is a CureWorseThanTheDisease.

I'm a little disappointed in the discussion of voting machines so far. We've had a list of up-front requirements ... and we're relying on someone's word that it actually meets those requirements.

I expected someone to jump in with an analogy to TestFirstProgramming.

Rather than have a big list of up-front requirements (WaterfallMethod), most of which are impossible to check during the vote (and some of which are practically impossible to check at any time), perhaps it would be better to make a list of tests that would make sure it actually is working in the desired fashion.

My understanding is that ExtremeProgrammers find it worthwhile to spend extra effort doing things that have no (direct) effect on how the program operates: (a) writing tests, and (b) changing the program ("refactoring") to make it easier to test.

Perhaps we can come up with similar ideas on how to (a) test a voting system, and (b) change/design a voting system to make it easier to test.

I find it hard to imagine that we can have a completely automated test; so we'll end up with tests like

Find someone unfamiliar with the system. Tell him we're testing the voting machine; ask him to sit in this wheelchair, go in through this door, vote for "None Of the Above", and get out as quickly as possible. With 20 people, we should get less than 1 error (at least 19 votes for "None of the Above"; a total of either exactly 19 votes (one was invalidated) or exactly 20 votes (at most 1 vote for someone else on the ballot), no more and no less. The median time to get in and out should be less than __300__ (?) seconds.

This tests targets innocent confusion on the part of voters. Other tests will target malicious tampering. Perhaps some tests will overlap both categories.

To make a system easier to test, perhaps we'll come up with ideas like "Always display a "partial vote count so far". But to keep the vote anonymous, don't update it on *every* vote. If there are only 2 ways to vote (yes/no), update it whenever both "yes" *and* "no" have incremented by at least 10, and then someone votes for the opposite of the previous vote.

You can probably come up with better tests.

-- DavidCary

Always display a "partial vote count so far".
- I'm not sure displaying partial vote is a good idea.

I'm glad to hear you have a better idea for testing voting machines. Mind sharing it with us ?

To preserve the rights of late voters, no indication of the current vote totals should be displayed. This is really the same issue as preventing the major TV networks from announcing preliminary totals prior to poll closures.

Odd, I always felt that it would be unfair to the early voters. In any case, which "rights" are you talking about ? It doesn't violate the right to an anonymous vote, because no one can find out how anyone else voted (if it's done right, which the above does *not* do. Ooops.). It doesn't make anyone's vote count more or less than anyone else. Is there some *other* right that has slipped my mind ? -- DavidCary

Consider the last voter. A quick review of the vote tally will indicate, except in the most unlikely scenario that his vote is of no value. The tabulation of ballots should not occur until all ballots have been cast. This is simple fairness.

We can keep the last voter's vote secret this way:

Always keep track of the partial vote count so far. But don't display it (except at the end of the day). Instead, have 2 other registers (that start at zero). Every 50 votes, the contents of the invisible "subtotal" register are moved to the "display" register, then the current "partial vote so far" is moved to the "subtotal" register. At the end of the day, the display is showing what the partial vote count was, not at the start of this batch of 50 votes (some 0 to 50 votes ago), but at the start of the previous batch, some 50 to 100 votes ago. It's impossible to detect what the last voter did, unless everyone in that last (50 to 100 votes) voted the same as the very last voter.

This still gives late voters "extra information" about the partial vote so far, that early voters don't get. Is there a better way to test voting machines for accuracy ? If not, I'd rather have easily-tested machines that leak small amounts of non-personally-identifiable information, rather than untestable machines that do not leak that information. -- DavidCary

(moved from VotingMachineDiscussion)

How to measure reliability?

According to the paper "Are Chads Democrats? An Analysis of the Florida Presidential Recount" by Matthew Spiegel in December 2000 (http://lakshmi.som.yale.edu/~spiegel/florida/FloridaChads.pdf), optical scanner ballots were less reliable than the notorious punch card ballots in the controversial U.S. 2000 presidential election: the number of changes per ballot is smaller for the punch card ballots (one change per 1614 votes) than the optical scanner ballots (one change per 1094 votes).

The evidence is indirect, but I don't know of a better way to measure reliability, at least with that system. Can we design a better system that would allow us to measure reliability more directly? (As opposed to just claim it has better reliability, without measuring it?)

We could all benefit from a measurement metric that would help determine reliability. For my part, the Diebold optical scanners worked pretty well here in Illinois for the 2002 general election: in what used to be a highly Republican state the Democrats swept every Republican statewide office holder out of their chairs except for one. Pretty good indication that the ballots weren't fixed -- at least, not by Republicans, who own the companies that make the voting machines, the phone lines, and the computers that did all the vote tallying.

There's some more indirect evidence, with a different conclusion, at "Cross-party voting in Florida seems to depend on the local voting technology" (http://ustogether.org/election04/mitteldorf/Liddle.htm).

(EditHint: "reliability" isn't exactly the word I want. What exactly is it that we want to measure? See MeasureTheRightThing, HowDoYouMeasureMaintenance, SoftwareMetrics, PerformanceIndicators, ModestElectronicVoteProposal)

More charts and graphs that (indirectly) test voting machines:

CategoryVoting