Thou shalt have no other gods before the ANSI C standard 1619

Trevor L. Jackson, III

Actually, no, that's not what I had in mind. What I had in mind is that testing is not a very effective way of buttessing the security of a piece of software. Testing is fine for finding average-case bugs, but tends to be pretty lousy at finding worst-case bugs -- and security is all about the worst case, not the average case.

Yes, you can do a lot better if you combine testing (running the program on inputs proposed by the tester) with access to the source code, analysis of the software architecture and its security requirements, deep thought about possible attacks, adversarial methods, and so on. But, if you've got access to all those resources (source code, design documents, specifications, security architecture, threat model) and if you're doing that kind of detailed security analysis of the system (e.g., code inspections, adversarial analysis of potential attacks, etc.), then the testing itself probably isn't adding much value. The most common case is that you can do most of your security analysis without needing to run the code.

Perhaps your mental model of how security analysis works is a little wrong. Were you thinking that the security reviewer would do all this security analysis, come up with a bunch of inputs which might or might not break the program, and then test the program on those inputs? That's not usually how it goes, in my experience. Once you've done the analysis, you often know pretty well whether or not those inputs are going to break the program, before you run it. Sure, often you build an exploit and check that your attack works just to double-check your work, but the testing phase itself often adds relatively minor value -- most of the value tends to come in the security analysis phase.

There are ways in which testing can help improve the security of a system while you are building it, to be sure. Unit testing and black-box testing is probably a good idea on general quality grounds. Once you've found a security hole, adding regression tests to make sure it doesn't come back is probably not a bad idea. These are a useful minimum baseline. However, their benefit is somewhat limited. And, when it comes time to evaluate the security of a system you've already built, testing tends to be of very limited value. Testing is rarely the #1 most useful thing you can do for security.

That's my experience-opinion, anyway.