Thou shalt have no other gods before the ANSI C standard 1623

Trevor L. Jackson, III

Naah, actual end is neither necessary nor sufficient.

It is not necessary: Hoare logic, weakest preconditions, program verification. It is not sufficient: It can only prove the presence of bugs, not their absence.

Testing can only disprove an invariant; it cannot prove that an invariant always holds, as the good man (Dijkstra) told us oh so long ago.

True. Testing can prove code incorrect. It cannot prove it correct.

Naah. Hoare logic. Weakest precondition. Program verification. They can prove things about programs. And even if you aren't going to that level of work, usually you can get more butturance by analyzing the source code than you can by running the program on some test inputs.

It can be applied to large systems. It is true that it is not effective at verifying that large- (or medium-) sized systems are secure with any degree of certainty whatsoever. But then, no methodology is effective at verifying the security of large- (or medium-) sized systems, so what can you do? In particular, testing is significantly less effective at buttessing the security of large systems than the methodology I described.

Right. That's why source code alone is not enough. You need an understanding of the architecture of the system, of the relevant components, of how it is supposed to work and how it actually works. Specifications, design documents, access to the system developers, ability to execute the program and play with it -- those are all very helpful, and almost necessary, to this task.

You don't start a security review by examining source code. You start by understanding and evaluating the architecture, threat model, security goals, etc.

I think you have misunderstood my point. I'm not trying to argue in favor of code inspection (for instance, I'm not saying that code inspection is all you need to do, or even the most important thing you can do, in a security review).

Rather, I'm arguing against reliance on testing. I'm arguing that testing is not the most important thing you do during a security review. I'm arguing that its effectiveness at buttessing the security of a system is limited. I'm arguing that it is only a small part of a security review.