Part 2 ** CA State Sen. Bowen Finally Grills 2 ITAs. (LONG transcript)

[nicknameless]

http://www.bbvforums.org/cgi-bin/forums/show.cgi?tpc=1954&post=19356#POST19356

(Commentary from the BBV.org website): "With this part of the transcript, you'll see that either Wyle lied or Diebold lied. We proved in Emery County Utah that Diebold made significant alterations in the Windows CE program."

Minutes 60-90 (there are 200 minutes total):

=====================

Senator Debra Bowen: Let me welcome Senator Poochigian first.

Senator Charles Poochigian: Thank you. You were just describing what you think ought to be sort of the standard with respect to testing by the particular testing companies and the QA role sort of in the ideal. Is it fair to infer from what you said that you don't think they're doing that or are you agnostic on that point?

Systest (Brian Phillips): I'm not sure I understand the...

Senator Charles Poochigian: Well the question is, you said that, in telecommunications there seems to be a different standard but that you have a higher expectation that these folks are actually doing some of this themselves, not simply relying on the services that you provide, as I understood you. If that's so are you suggesting that there's some failure on their part to do what you said they should be doing or are you…?

Systest (Brian Phillips): No, no I don't think so and fortunately I've had an opportunity to meet with the managers and directors of the QA departments of most of the voting system vendors that we work with.

Just as you might imagine there's a lot of interaction between our groups because we're sending things back through their organization. All very very competent and very diligent people, but they're also working under real world time pressures, budget constraints and everything else associated with that.

When I started my consulting practice in 1991 I went from the aerospace industry, Department of Defense industry to the commercial world. My goal was to take the best that we could do in the aerospace industry and apply it to the commercial industry but keeping in mind real world budgets, time constraints, and everything else -- which meant, where do you cut out?

And everyone has to deal with that, they've got time constraints, development has so much time to get software ready, quality assurance and testing groups have so much time to get it through all the tests, re-work the tests as they find defects, regression test everything, then get it to the independent testing authorities for our testing. And then there are schedule constraints there, because they're anxious to be able to offer this product to the state of California, for instance for certification testing to meet their particular timelines.

So there's never a question of do they want to do this or do they have the right processes to do it. But there are real world pressures, I guess you'd say.

Senator Debra Bowen: Have you ever failed a voting system?

Systest (Brian Phillips): No. The reason for that is we will never issue a qualification report unless all tests pass. I mean we won't even submit a report to NASED saying "this company failed."

Senator Debra Bowen: So then the answer is no but only because you just wouldn't…

Systest (Brian Phillips): We're not going to, we wouldn't take it that far

Senator Debra Bowen: It's pass or what? Pass or don't fail?

Systest (Brian Phillips): There isn't an official report saying "this company has applied and they failed." I have had two companies who have applied, gone through testing and we didn't complete it because they weren't going to pass. There was no way. And they were sent back to the drawing board essentially.

Senator Debra Bowen: Okay.

Wyle (Jim Neu): The same parallel applies in our testing of DoD components or anything else. We typically don't quote "fail" them because we're working with the developer If a failure occurs which could be a crack or anything else it goes back for redesign it comes back under change of scope to be tested again and eventually it goes forward. We typically don't issue a failing report because if they're not able to fix it, they just don't come back and the product just never goes to the marketplace.

Systest (Brian Phillips): I will say in our IV&V world in the public sectors we have failed companies, we were doing IV&V for the state in Colorado in their unemployment insurance system development effort with Accenture. Essentially we failed Accenture in Colorado.

Senator Debra Bowen: We've had IV&V vendors who failed systems in California, too. The IV&V process has been extremely valuable to keep taxpayers from continuing to pour money into a project that just wasn’t on the right track.

Systest (Brian Phillips): An ITA, our process is very much, in essence, an IV&V effort.

Wyle (Joe Hazeltine): We've had systems as well at Wyle that it became clear early on that they were not going to meet the standards and it wasn't really a matter of rejecting it, it was a matter that the vendor realizing that he wasn't going to get there and withdrawing the effort.

You'd mentioned earlier about is it possible to have error –free software and the answer to that is really "no." It is possible however to have software which meets standards and standard requirements. And say the standard requirement for accuracy in a voting system is one error in 10 million votes cast, I mean, that's the standard which is being set, and we're testing 1,549,703 consecutive ballots without an error to try and demonstrate that. So that's a measurable thing which we can do. To say "no errors" is not possible.

Senator Debra Bowen: So then how do you account for Sequoia last week in Chicago, Hart Intercivic and ES&S in Texas, I don't know who the vendor was in North Carolina where the server was set at 3200 and over 7,000 people voted, how do we account for a testing process where each and every one of those systems had a NASED number, passed ITA testing and yet, at least from my perspective, failed miserably in the real world.

Wyle (Joe Hazeltine): Well I don't know all the details on the cases but from looking at it looks like there were some human factors issues as well as they were set up incorrectly it wasn't the machine it's the way that it was set up at the polling place.

Senator Debra Bowen: That's certainly true with the server error in North Carolina. That certainly was a factor in Chicago where I understand that 4,000 poll workers had never seen the machine that they were setting up.

Wyle (Joe Hazeltine): And the one in North Carolina where the screen said that the memory was full and it was ignored. I mean the machine was identifying that it was a condition outside its bounds but it wasn't being properly monitored.

So the answer to the question, error free is not possible but I do think you can set performance standards that can be met and testing can demonstrate that. Another standard which is in the VSS is a mean time between failure of 163 hours for a machine that going to be used for eight.

Senator Debra Bowen: I can't remember the source but I looked at the mean time failure charts recently, somebody did a graphic of that and it looked to me like light bulbs were about 200 times more reliable than voting machines, but the consequence of losing one light bulb in here…

Wyle (Joe Hazeltine): I can believe that, a light bulb has about a 5,000 hour life but the light bulbs aren't being exercised quite the same as a voting machine, they're not being carted back and forth. There are some issues I think in the environment around a voting machine that requires additional testing to demonstrate its reliability, which is being done.

Wyle (Jim Neu): Typically the more complex a system is, the lower the mean time between failure. There's a considerable industry involved in analyzing the bad actors or the items that are causing the most failures but the standard has been set, 163 and we do test it at standard.

Senator Debra Bowen: But we've got from Mr. Phillips already that they basically bench test or test two to three machines and that you've got thousands in the polling places, and that's part of the problem. Is Wyle's experience similar in terms of the number of machines they're testing?

Basically I think the bottom line here to take away is you're not charged with testing under real world conditions. You look at the instructions, that's part of the documentation package but somehow reviewing those instruction sets, I think it's called the Technical Data Package, is what you call it, reviewing that to make sure it's complete doesn't translate into once the equipment goes to the polls it gets set up and operated properly.

Systest (Brian Phillips): In other words have we tested to ensure sort of, that they're foolproof?

Senator Debra Bowen: Well I think it goes beyond that because if a lot of the problems are Operator error and I certainly am inclined to concur that a lot of the problems are, you're asking people-- and I haven't even factored in the average age of poll workers. I don't know what it is around the country but I do know what it is in California and I do know that a few years ago in one large county when poll workers were given cell phones so that they'd have a way to call in with questions, in some polling places nobody in the polling place could figure out how to turn the cell phone on. So, that's a cell phone. Not a voting machine.

So the question really is, what do we get by a review of the technical documentation, the manual if the result in the field, you know, in Chicago, is a major meltdown?

Wyle (Jim Neu): I think you get a lot. But I have to say that we have to test to the standard. If we took it upon ourselves to test to something other than the standard we could be accused of being inconsistent, unfair, biased or whatever. So the standard has to be developed, we have to work with the standards agencies to develop the checklist that we spoke about and then we have to rigorously test to the standard and not to more and not to less.

Senator Debra Bowen: So who should be responsible for dealing with the actual use of voting equipment once it gets into the hands of, here it's the county registrars of voters it might be the township in other states, and what happens there? Because that's not part of your routine.

Wyle (Jim Neu) Well, outside of my role as a Wyle person I can tell you that having reviewed some of the background material there is some, what appears to be fairly good ideas about how things need to be implemented in the field. They're clearly not our responsibility and nothing to do with how we test. But having read some of the background material there are clearly people who are developing concepts that would help to reduce the other kinds of errors.

Senator Debra Bowen: Okay. A few more questions about the testing. So the volume test is not something that you're responsible for, it's just not a part of the real world test.

Systest (Brian Phillips): That's correct. There are no requirements that we have to show that x number of systems hooked together could work properly. There are volume tests within there such as the number of ballot positions, accuracy and things like that which we do test.

Wyle (Joe Hazeltine): That's no different than military or other systems where you take what they call the first article and the first article goes through a rigorous set of tests, as these machines do here. What you're trying to do with that first article is show that the design meets the requirements. Manufacturing is a different question from design.

Senator Debra Bowen: Well it actually leads to another set of concerns that have been raised. Have you ever had instances in which vendors gave you deliberately bad information or withheld information?

Systest (Brian Phillips): I can't think of any time when they've ever withheld anything intentionally, I mean there have been omissions that we've caught when they've sent it in. I can't think of where anybody's ever sent anything to us that was maliciously wrong.

Senator Debra Bowen: Wyle?

Wyle (Joe Hazeltine): You're addressing the vendors?

Wyle (Jim Neu): In terms of the voting machine vendors.

Wyle (Joe Hazeltine): We've never had a case where a vendor has never been more than cooperative. We've had instances where a failure has given us bad data, but not the vendor, the vendor corrected the problem. Whatever the anomaly was, we got something that was really unanticipated.

Senator Debra Bowen: I ask that question in part because of the, going back to a Pennsylvania certification in which Diebold wrote () 'We do not certify operating systems with Wyle (this is a Diebold employee). Therefore we do not need to get Win CE (the Windows CE 3.0) certified by Wyle. We do not want to get Wyle certifying and reviewing the operating systems. Therefore we can keep to a minimum the references to a Win CE 3.0 operating system."

Wyle (Joe Hazeltine): I'm not familiar with the document that you're looking at.

((Whisper: That's back when they were...))

Senator Debra Bowen: Oh, it's back when they were Global, GES, Global Election Systems.

Wyle (Joe Hazeltine): Is this some type of a hearing or testimony that they had done?

Senator Debra Bowen: I'll get you a copy of the e-mail. It's not something that I uncovered. But I think it goes to the fact that Windows CE requires significant customization to work on a voting machine, such as a Diebold touch-screen. It's not Commercial Off the Shelf Software, it won't work without being customized. Diebold has the source code for Windows CE and can modify core features and yet here they are basically writing saying we don't want Wyle to be looking at the Windows CE 3.0 system even though it could only run on a Diebold touchscreen if it were customized.

Wyle (Joe Hazeltine): Well that's, you can read it that way, another way that you can read this is that Windows as a commercially off the shelf software does not require ITA certification.

Senator Debra Bowen: That would be true if it ran as a COTS software operation on the machines.

Wyle (Joe Hazeltine): I'm saying the program resident in the operating system would need to be looked at but the operating system itself would not.

Senator Debra Bowen: It's my understanding that Windows CE, the operating system, requires customization that it cannot run on a Diebold touch-screen as Commercial Off the Shelf Software.

Wyle (Joe Hazeltine): I'm not an expert in that I'd have to defer.

Senator Debra Bowen: It's software, I understand.

Systest (Brian Phillips): My take on that is if they've modified any of the source code to Windows CE 3.0 it is now subject to ITA qualification.

Wyle (Joe Hazeltine): It's no longer COTS.

Systest (Brian Phillips): Any COTS software modified is no longer COTS and therefore it would have to be tested. Now that information would have conveyed to the ITA and if they -- in this case, if Diebold or Global at the time didn't tell the ITA that they've had to -- because we assume that if we're working with an operating system like Windows CE or Windows XP or 2000 or anything like that it's Windows out of the box.

Senator Debra Bowen: Let me just back up, because we all understand what we are talking about here, but I dare say that if you go outside of this room and ask people "What are they talking about?" they'll say "I have no clue." So let's start with "What is Commercial Off the Shelf Software" and how do you test, what are the test protocols about it and what is the justification for it, because I actually think it's very logical and people will understand it.

Systest (Brian Phillips): I mean in the simplest terms, Commercial Off the Shelf software, COTS is something that anyone would be able to go and purchase, called shrink-wrapped software be able to purchase either over the Internet, or in a store, something like that. It's available to anyone, it's not designed and developed specifically for, in this case, for the voting world. It can be used, and we'll use an operating system as a good example of that, it can be used to run any type of voting system. So that in its simplest form is what COTS is all about. That implies that the manufacturer of the COTS software has done enough thorough testing by both themselves and by the users out in the community to have a good solid application that doesn't have any bugs and defects.

Senator Debra Bowen: So the theory is that whatever issues there are with Commercial Off the Shelf Software that one could go buy in a shrinkwrap – well I think Egghead doesn't exist anymore, but when they did -- that whatever flaws there are in that are likely to have been uncovered in a larger user universe so that it would be a waste of resources to test Commercial Off the Shelf software that's being used exactly as it came out of the shrink wrap.

Systest (Brian Phillips): Yeah. There's also the question of whether or not we'd have access to the software, to the source code itself, I mean, Microsoft is going to protect their software. And so what they're going to say to the vendors is "If you have to have our source code and have it reviewed, use a different operating system." So those types of things. Because they may not open it up. So there's questions of that.

Senator Debra Bowen: So the operating assumption then for us looking at testing is that software on which the ITA didn't do code review that is running on the voting machine wasn't reviewed because it was Commercial Off the Shelf software.

Systest (Brian Phillips): It was COTS, right.

Senator Debra Bowen: So how do you deal with something like Windows CE where, unless I'm wrong you cannot just -- there is no version of Windows CE that you can just load that will run on a touch-screen without being customized. It's actually a change in Microsoft's policy, Microsoft released Windows CE out to the world in order to encourage device manufacturers to use the Windows CE platform.

Systest (Brian Phillips): There's a number of ways to handle a situation where it's customized COTS software. If the vendor in this case, Diebold/Global, has permission to modify the software, then they have access to the source code. Differences can be run on the source code to show where the modifications were made and those changes can be then reviewed to ensure that they were changes that were necessary in order to run on that particular type of device, that there were no security issues, there was no malicious code etc. and that the changes themselves meet whatever coding standard practices are required. You should be able to do that and that is not that extensive of an effort. If it turns out that they've modified everything within Windows CE that's become an extensive effort. But regardless you would still know what's changed and what hasn't changed. If they somehow don't have permission to show that, then that question has to go in front of the technical committee to find out "How do you want us to handle this?" We can't enforce them to do something they may have contractually worked out that they can't release the source code. So we would not, Systest Labs would not qualify that product in that particular case.

Wyle (Joe Hazeltine): To this e-mail, if I may, you know, the first statement "We do not certify operating systems with Wyle, in fact is a correct statement."

Senator Debra Bowen: Okay, that was going to be my next question. Is the Windows CE operating system viewed as software or firmware?

Wyle (Joe Hazeltine): I'm not sure I know.

Systest (Brian Phillips): It's software and there really, what needs to be understood is there's really no different between software and firmware, it's all software it's just how it's stored on the device.

Senator Debra Bowen: Right. I'm only asking that question because of the -- you test end to end, Systest -- but Wyle and Ciber have decided that they're not going to test – so -- Ciber's not here, I would love to ask them if they looked at the Windows CE operating system.

Wyle (Jim Neu): You know I don't know for certain the answer to your question but the answer really revolves around the fact that if it was represented to us -- if it were firmware -- and it was represented to us as COTS unmodified, then we would not review it. If it was represented to us as COTS that had been modified and it was firmware, then we would review it. So this person has made the assumption here that his Win CE apparently is not modified. Because he says therefore we don't need to get Win CE 3.0 certified.

(("this person" is Diebold Election Systems top programmer, Talbot Iredale, who wrote the e-mail they are discussing and also personally made the modifications to Windows CE, rendering it non-COTS. He also wrote some of the rob-georgia patches.))

Senator Debra Bowen: That's exactly why I –

Wyle (Jim Neu): If he knew that it was modified then he's probably incorrect in his statement. But I can't comment on whether he knew that or not.

Senator Debra Bowen: I can't comment on whether this person knew or not, I do believe that there is no such thing as a Commercial Off the Shelf Windows CE system that will run without modification system on any device, much less a touch-screen voting system.

Wyle (Joe Hazeltine): Oh I agree with that, it's an operating platform but every platform is different.

Wyle (Jim Neu): So if it is identified to us as non-COTS software and if it's in the firmware, in the chip, then we will review that software.

[Wilms]

1. I hadn't realised Bowen was able to get some of these ITA's on record...

...if not under oath.

Techies, note these transcripts.

[Amaryllis]

2. Kick

Senator Debra Bowen: So then how do you account for Sequoia last week in Chicago, Hart Intercivic and ES&S in Texas, I don't know who the vendor was in North Carolina where the server was set at 3200 and over 7,000 people voted, how do we account for a testing process where each and every one of those systems had a NASED number, passed ITA testing and yet, at least from my perspective, failed miserably in the real world.

Wyle (Joe Hazeltine): Well I don't know all the details on the cases but from looking at it looks like there were some human factors issues as well as they were set up incorrectly it wasn't the machine it's the way that it was set up at the polling place.

Wyle's reply brings to mind Land Shark's thread on glitches...doesn't matter if intentional or not; end effect on democracy is the same.

[kster]

3. K&R...nt

[kster]

4. Kick.nt

[nicknameless]

5. A kick for part 2

:kick:

[Kurovski]

6. Kick to the top. (nt)

[Steve A Play]

7. Kick!

:kick: