Comments of a Florida based professional software engineer regarding HBO's "Hacking Democracy"

[ddeclue]

Hello Friends:

My name is Douglas De Clue and I live in Orange County Florida where I work
as an automated test equipment (ATE) engineer writing software that performs
data acquisition, hardware control, communications/networking, and hardware
testing.

I have been doing this sort of work since approximately 1996 in a succession
of companies around the country as a contracting engineer.

In many aspects these tasks are very similar to the tasks performed by
automated voting equipment such as that produced by DieBold, ESS, and
others.

I have also worked in the past as a software test engineer where the purpose
was to test the behavior of software against specifications for FDA safety
testing of hemodialysis machines.

I am 40 years old, hold a Bachelor of Aerospace Engineering degree from
Georgia Tech since 1989 and have been programming computers since I was 12
years old in 1978 and have been writing software academically since the 1985
and professionally since 1992 when I began working as an engineer.

Since 2004 I have been heavily involved in voter database issues here in
Orange County where I have provided substantial volunteer I/T support to the
Orange County Democratic Party and to a variety of Democratic candidates for
office in the 2004 and 2006 elections including the Kerry campaign, the Rod
Smith campaign for Governor, and several other candidates so I also have a
substantial background in campaigns and voter database issues.

I am currently also a precinct committeeman for the Democratic Party here in
Orange County Florida.

During the 2000 elections, I was not active in the party but I DID reside in
Broward County Florida at the time and became painfully aware of the vote
counting issues of that election. (So called "butterfly ballot" or
Hollerith card voting has obvious verifiability issues as we saw in that
election and we should not return to such Hollerith card systems.)

Because I reside in Orange County, I have had the opportunity to actually
meet Clint Curtis on several occasions (who claims that the GOP once tried
to hire him to write software to hack the vote), who was a candidate who ran
for the United States Congress, 24th Florida Congressional District in this
2006 election almost entirely on the issue of vote verification.

I also met one of the women working with BlackBox who appeared in the
documentary that aired on HBO last night entitled "Hacking Democracy" from
Volusia County this year at a local DFA meeting.

Having said all of this let me now comment on what I observed in last
night's airing of the HBO doumentary "Hacking Democracy":

1) This documentary ironically aired after one of the largest political
landslide elections in recent decades when it was apparent for several
months that the Republican Party in power was headed for imminent and
overwhelming defeat. If votes were going to be stolen, this would have been
the election to do it. The Republicans knew they were going to lose badly
for at least 2 months.

What happened? Did all the vote stealers go on vacation this time?

Why didn't the GOP put the fix in? Even if they did not feel they could
outright steal the entire election, why didn't they pick a few of the closer
races, at least in the Senate, to steal? There were clearly several very
close races in Montana and Virginia after all where if they were going to
steal the election they could have tinkered with these races in very
marginal ways to win them.

Based on this election, I don't believe the vote was actually stolen in 2000
or 2004 using technical hacking means but rather through the Republican
party simply doing a better job of mechanically turning out their base
voters than the Democratic Party in combination with old fashioned voter
suppression through fear tactics (in some instances) like the ever popular
"if you vote you're going to jail" leaflet, campaign dirty tricks in some
instances like phone bank jamming, and through a faulty "felon" purge that
occurred here in 2000 that was heavily weighted to punish Democrats.

In reality, plain old fashioned gerrymandering of election districts has had
the most impact on elections by far here in Florida making most districts
uncompetitive simply by very careful selection of election district lines
using FREDS data and Census Bureau TIGER line data and other demographics
database info to push all minority voters into bizarrely shaped
supermajority districts such as Corrine Brown's here in Central Florida
where she is assured a win ever year because of overwhelming African
American support in her own district while surrounding districts have little
if any African American votes in them causing them to tip to the Republicans
since the African American vote is heavily (90%+) Democratic.

If you want to say "the fix is in", then in reality, it is much more about
the gerrymandering that occurred in the 2000 reapportionment - at least here
in Florida - than about technical vote hacking.

Instead of having 2 Democrats and 2 Republicans therefore in Central Florida
we have 1 Democrat (Corrine Brown) and 3 Republicans (Mica, Feeney, Keller)
in Central Florida through this gerrymandering. If only a small portion of
the African American vote in Corrine Brown's district were redistributed to
the other 3 districts in the area, there would be 2D's and 2R's because the
8th district would then be competitive enough for Democrats to win. The
districts for all levels of races - Congress, State Senate and State House
are all bizarrely gerrymandered in a manner which would make the
partitioning of Berlin and Germany among the four allied powers after World
War II look extremely reasonable by comparison.

2) The documentary "Hacking Democracy" was, in my own opinion, very unfair
in its treatment of the Supervisors of Election in Volusia and Brevard
Counties and treated them as though they were guilty until proven innocent.
I felt this betrayed the purpose in this effort by making the
blackboxvoting.org folks look very biased and not objective in their
conclusions.

3) The Diebold people nonetheless came across to me as doing a song and
dance to protect what appeared to be a product that had multiple technical
deficiencies - especially in the California legislative hearing - rather
than simply admit that there were defects, what those defects were, and how
they planned to correct them. This would have done much to inspire
confidence in their systems that the singing and dancing did not do.

4) I would not describe the simple manipulation of voter database tables
stored on SOE Sanchez' computer system as a "hack".

That really stretches the term "hack".

All that was done was the opening and manipulation of a standard database
file (it appeared to simply be a Microsoft Access .mdb file) using some
simple SQL queries which did the obvious and expected thing - it altered the
stored records. It wasn't really much of an impressive "magic trick" to me
as a programmer.

Not mentioned in the documentary is that every file stored on your computer
has a time and date stamp and a file size and that every NTFS formatted hard
drive has a master file table (MFT) and journal logs that track file changes
on the hard drive. A simple SQL record change as demonstrated in the video
would be fairly easy to detect using common forensic techniques so I would
not be willing to describe the vote change demonstrated on SOE Sanchez'
computer using the SQL query as "untraceable".

The real issue that was not addressed by BlackBoxVoting.org here was the
fact that they were storing voter information in aggregate form as the
official election record in the first place.

5) Regarding the change performed to the database file at the SOE office, I
would like to recommend the following safeguards be implemented by SOEs to
minimize the risk of hacking in the future:

a) After an election, individual voting machine datacards should be
immediately backed up to a write once media like CD-ROM or DVD-ROM as they
are uploaded so that a permanent record exists that cannot later be easily
altered or better yet replaced entirely by write once media like DVD-ROM or
CD-ROM so that they can not be altered after the fact. Such write once
media must be treated like evidence in a criminal case with a chain of
custody, seals, and secure storage.

b) Voter data should NOT be stored in relational databases as simple
aggregate totals. This makes it too easy to alter the results en masse. It
should only be presented that way for final viewing on the SOE website.

c) Instead, voter data should be stored in a journal or transaction log
format where each individual vote is individually recorded into a file along
with other corroborating information like:

i) Software Version ID and an MD5 (CRC32) checksum for that software.
ii) Voting machine serial#
iii) Voting data card serial#
iv) SOE Operator#
v) A globally uniqued voter sequence (GUID) for that particular machine
generated by a remote SOE machine that provides this to all voting machines.
It in turn would keep track of which machines it issued voter sequence
numbers to and when so as to make it hard to fake without access to both
machines.)
vi) Time and date stamp.
vii) Vote totals for each race for the machine as well as how the voter
voted in each race.
viii) A CRC32 (cyclic redundancy check) number computed for each
transaction. This is a complex checksum algorithm that would make it very
difficult if not impossible for a "hacker" to go back after the fact and
alter the records and is the basis for file verification when downloading
software and also in many common anti-virus programs.

You could also add the actual voter ID number from the SOE rolls to this
list of items to insure additional security but then it would be possible to
determine how people voted.

Tracking the operation of each machine like this in a transaction log format
rather than in simple relational database tables with aggregate counts would
make it much much harder to fake the results because the votes would then
have to be faked one by one rather than en masse as demonstrated in the
video. It would be very hard to fake the votes in a convincing manner that
could not be discovered by poll watchers who keep track of when people come
and go and how many show up at the polls at a given time.

6) This transaction log, any databases, the hard drives, the datacards and
the operating systems must be:

i) Password protected
ii) Encrypted
iii) Protected by physical security and chain of custody must be insured.
iv) Norton "Ghost" imaged or otherwise forensically copied to a permanent
write once media for preservation before and after the election.

It seems like all of these very obvious security techniques were being
ignored or by-passed in the Sanchez' demo and I hope that they are actually
being practiced in the real world.

7) All machines for voting must NOT have internet access and must be placed
on private networks with no outside access. This will do much to eliminate
the threat of external hacking - however, most hacking still occurs from the
inside in the real world.

8) Instead of storing the vote totals on each voting machine on the flash
memory cards that Diebold is using, I would say that the votes need to be
recorded on a write once media like CD-ROM or DVD-ROM so that they can not
be altered after the vote. This would also make it much easier to prevent
malicious code execution from the cards or changing the contents of the
cards by someone with a card reader as their security expert did.

9) The constant and ominous reference in the documentary to negative ballots
and counting backwards was not properly explained in the documentary and has
a simple more innocent explanation that I wish to comment upon here:

a) The counting backwards would most likely occur at 32767 votes because
numbers are probably being represented in the computer program as 16 bit
signed integers which means that because of how math is performed by
computers that after 32767, additional votes added would "roll over" the
count and cause it to start counting backwards. (For the non techie people
out there, think of this as being like your car odometer rolling over at
99,999 miles.)

b) Variables in voting machine software should never be declared as "short"
which means signed 16 bit integers. Instead they should be declared as
"unsigned long" which means unsigned 32 bit integers or in other words that
negative numbers would not be possible and that the count would go between 0
and 4,294,967,295 or slightly over 4 billion. This exceeds the entire
population of the United States by a factor of 14 (whereas 32767 is small
enough to actually occur in real races) and would prevent the roll over
scenario from ever occurring and it would also prevent malicious persons
from preloading the vote counts with negative numbers as demonstrated in the
video by BBV's security expert.

This problem is almost certainly not really malicious on the part of
Diebold, ESS, or other vendors but rather a simple coding oversight by the
programmers of the software who did not consider the unintended consequences
of choosing signed variables or short (16 bit variables) with respect to the
roll over issue and the possibility of pre-biasing the votes on the cards.

10) I definitely agree with BBV that the touchscreen devices are too easily
subject to either malicious code and simple miscalibration of the screens
because no permanent paper ballot record exists that can be hand counted and
such systems must be discontinued and removed from operation ASAP.

Paper ballots involving having the voter manually connect the arrow as used
here in Orange County FL or fill in the circle such as SOE Sanchez' office
uses however CAN be hand counted and thus are manually verifiable and
provide an important check on software programming errors and potential
malicious human behavior.

11) When the government buys custom software for DOD, FAA, NASA, FDA, or
other government agencies such as voting software obviously is, the software
is not "open source" to the general public - however, the government as the
customer is always given the source code for maintenance, inspection and
test purposes as well as compiling instructions to allow the gov't to verify
that the delivered compiled code is actually the same as the source code
provided, along with MD5/CRC32 checksums to verify the compiled code, design
documentation and specifications for testing.

12) Truly black box software without source code should never be allowed in
voting since it is almost impossible to tell what the software does by
testing to specifications alone.

Specifications testing hardly ever provide a complete check of the
functionality of any complex piece of code and should be viewed as a mere
starting point in software quality, not an end.

Because they apparently did not understand this simple fact about software
testing, I believe that the BBV people treated the CIBER people in
Huntsville Alabama rather unfairly in their "ambush" interview. The CIBER
people were simply given a check list of test specifications by the
purchasing gov'ts which likely as not wasn't complete due to a lack of
technical sophistication on the part of the purchasing SOEs and a lack of
insight into the development process of the vendors.

The CIBER people simply ran the check list they were given and had no way to
test the software beyond that without having design documentation and source
code which they were likely not given by the vendor. They did the job they
were contracted to do, nothing more, nothing less and it really wasn't fair
to bash them for not having access to information that the vendor did not
provide to them.

The software is, after all, tested "to specifications" and not "to
perfection" which is of course much harder to define. Those
specifications, like it or not, are almost always incomplete or faulty in
some manner in any event.

13) Beyond specification testing, an actual source code inspection and
analysis by a number of different experts is also a required (although not
necessarily sufficient) condition of software quality assurance.

In the end there is never any total assurance that software is entirely bug
free and we have seen a number of spectacular examples of software quality
failures in recent years including cases like the Mars probe that cratered
in a few years ago due to inconsistent engineering units in the software, a
lost earth orbiting satellite due to an incorrect constant for the
rotational velocity of the earth, a radiation device for treating cancer
that delivered lethal radiation doses and others where extensive testing
failed to reveal unintentional flaws in the software prior to operational
use.

Such mission-critical or safety of "life and limb" type software (sometimes
jokingly referred to as controlling "elevators", "nuclear power plants",
"airliners", and "iron lungs") is notoriously difficult to make bug free and
I have personally been involved in writing and testing such software for
wind tunnel systems where control of a 5,000 kW (6700 horsepower) electric
motor was involved. It is no joke if you lose control of a 5,000 kW motor -
people can be killed and millions of dollars of damage can occur - so it
takes a great deal of effort to get it right.

Yet, nevertheless, we continue to have spectacular failures of
mission-critical software in a variety of areas. (See:
http://awads.net/wp/2005/12/05/ten-worst-software-bugs/ or
http://www.wired.com/news/technology/bugs/0,69355-0.html?tw=wn_story_page_pr
ev2)

Ultimately, typically only about 1/3rd of bugs can be discovered through
specification testing, another third or perhaps somewhat more may be easily
found through cursory code inspection, but the remaining 1/3rd ends up being
discoverd through end user operation in the field. This is why it is
important to implement extensive beta testing by a variety of users before
software is generally released for normal operations.

14) Wherever software is used by the government for any purpose that will
affect our lives such as voting or as evidence in a court trial (say in the
case of DUI intoximeters or radar guns) such software must be made exempt at
least some of the protections of copyright and patent laws so that it can be
inspected by the government, recognized experts, and the public at large to
insure confidence in the system.

In criminal evidence cases, such black box devices as intoximeters and radar
guns deny the accused the right to confront the evidence against them -
essentially it becomes guilt by "magic 8-ball". In voting it then becomes
election by "magic 8-ball".

Such "open sourcing" of voter and other gov't owned systems like
intoximeters and radar guns would also make it much easier to identify and
fix flaws in the system because far more sets of eyes would be looking for
the problems and it would not be possible to keep such problems hidden for
long.

15) Voting systems based any of the Microsoft Windows operating systems are
particularly vulnerable as we have already seen from the many many monthly
security patches issued over the past 5 years+. All commercial operating
systems are vulnerable to some degree or another (see the NSA white paper:
http://www.jya.com/paperF1.htm) and for this reason it is necessary to
always hand count a 3% random sample of paper ballots in each election to
insure system integrity and if discrepancies are found to recount larger
samples up to and including a full manual recount if necessary.


My final conclusions regarding the HBO documentary "Hacking Democracy"

a) Technical vote hacking most likely did NOT occur in the 2000, 2002, 2004,
or 2006 elections.
b) Vote hacking is however theoretically possible to some degree depending
upon the software and hardware being used by the SOE and the technical
sophistication of the SOE personnel using it.
c) Voting software must be made open source.
d) Paper "opti-scan" ballots appear to be the best solution to providing
election traceability because they can also be hand counted.
e) There are numerous actions that I have outlined that can be taken to
significantly harden the voting software against hacking.
f) No software is "hack proof" or "bug free" however and a 3% sample hand
count should always be performed as a safety check.
g) The following voting issues which are more seriosu than vote hacking need
to be addressed expeditiously:

i) Gerrymandering. Blatant district gerrymandering using FREDS, US Census
Bureau TIGER line data, and other demographic resources must be stopped and
districts need to make organic sense following geographic boundaries like
rivers, highways, city limits, etc. so that communities of interest are
preserved and so that districts are not biased by racial or economic factors
to prevent competitive races.
ii) "Felon" and other types of "purges" whereby voters are removed from the
ballot and either denied a vote or forced to vote provisionally when these
purges are often subject to cases of mistaken identity given the issue of
common names.
iii) Intentional voter suppression "dirty tricks" like the "if you vote
you're going to jail" leaflet that appears in election after election in
African American and Hispanic communities.
iv) Dirty tricks like campaign phone bank jamming, false messages using
robodialer systems, and other such tactics.

h) Legislation at the state and Federal level needs to be passed to insure
that the fixes I have recommended here and other fixes that other experts
recommend are implemented promptly to minimize the risk of a hack in future
elections.

Respectfully,
Douglas J. De Clue
Orlando, FL
[e-mail]