Thou shalt have no other gods before the ANSI C standard 1616

Hank Oredson

Agreed! This is a much better formulation of the problem. I like this much better. It makes clear what the goal is, and leaves it up to the security evaluator to use whatever tools are most effective at the tool (rather than artificially constraining them to use testing, or any other one method).

Other typical goals include "evaluate its security and buttess the likely risks", or "tell me what are the most important things I could be doing to improve its security". These are more common than "break it, please", in my experience, but all three are important.

Now you can ask, what is typically going to be the most effective method? How much of the time is the reviewer going to spend on tasks like: reading design specs? questioning developers? forming hypotheses about possible failure modes? analyzing the threat model and security goals? examining the test plan, quality processes, and prior security defenses that have already been put in place? inspecting code? running automated code-scanning tools? developing test cases and executing the program on these inputs?

My guess is that, in a limited-time review done by a competent security professional, the latter (writing and running test cases) is likely to account for only a small fraction of the overall effort and overall value. That's because often testing doesn't rank as high in terms of metrics like "bugs (or surprises) found per hour spent" or "progress towards evaluating the security of the system per hour spent".

To give a concrete example, here is an example of one security review I have participated in: Two of us spent about a week (ok, 4 days, but those were long days...) evaluating the security of a system called E. We spent maybe 10-20% of our time on writing and executing test cases. I think we were pretty productive. If we had spent 80-90% of our time focused solely on testing, I don't think we could have made nearly as much progress in that time.

To give another example, here is another security review I participated in recently: We spent about a week on evaluating the security of a voting system. I doubt that we spent more than a couple of hours on testing or running the system. We didn't need to.

Part of the point I'm making is that generally in a security review, you should be starting with a more holistic, systems viewpoint. You can make a lot more progress that way than by just diving into the code at some random point. And testing is so low-level that it is almost never where you should start.

Another part of the point I'm making is that the power of testing at uncovering new security holes (that you didn't already know about) is very limited. I have found it more common that I identify potential security defects in some other way, and then I use testing to confirm my hypothesis. Testing just has very limited power at inducing worst-case behavior from the system; you need to figure out what the worst cases are some other way, and that is exactly the hard part.