Thou shalt have no other gods before the ANSI C standard 1621
Trevor L. Jackson, III
Adversarial Testing, was Thou shalt have no 1626
Trevor L. Jackson, III" No, it is that both will apply the same process of reasoning. And since that is decidedly *not* explicitly part of the rules of the game...
Occasionally, black-box end is indeed quite useful during a security review. But most of the time, I find that the majority of the time, work, and value of a security review comes from activities other than testing. Your experience may vary.
The key point is that running the software usually gives you only a very limited insight into what the system is doing, how it is structured, and whether it is secure. Sometimes it does indeed give useful insights, but usually you can do better by starting with a review of the architecture, design documents, threat model, security goals, and then progressing on to code reviews and other detailed analysis of selected parts of the system.
Naah. I'm tilted towards systems issues. What is the threat model? What are the security goals? For each security property we hope the system we will have, what is the TCB for that property? How large is that TCB? How much confidence can we have in it? Have basic principles (least privilege, fail-closed rather than fail-open, etc.) been followed?
Performance optimization is different from security review.
Adversarial Testing, was Thou shalt have no 1625
Douglas A. Gwyn I disagree. The value of the model is precisely the fact that a...
I'm not sure what you mean by that. Are you thinking of race conditions and TOCTTOU vulnerabilities? I've usually been able to do a better job of diagnosing such conditions without a heavy focus on running the software. Indeed, these vulnerabilities often only occur under rare environmental conditions. The hard part is identifying those rare environmental conditions, and it is not clear whether running the code will help a lot. In any case, race conditions and TOCTTOU vulnerabilities are one clbutt of security defect, but by no means the only or largest clbutt.
You can use it that way, but that is not the accepted meaning of the word. Inherent to testing is that you are *running* the code on some inputs. There are lots of ways to evaluate software without running it, but those ways cannot be described as "testing".
Testing cannot certify that desired invariants hold. It can find bugs, but cannot certify their absence.
That's an example of a question that testing alone cannot answer. Establishing this will require looking at the source code, and once you are examining the source code, testing is usually of secondary importance. (It is still a good idea, but it tends not to be the primary source of value during a security review.)
I don't know why you think it cannot be established by examining source code. (Obviously, you need to have source code to the memory manager if you want to determine whether the memory manager scrubs its memory.)
There are a few examples of things that cannot be established by examining source code: for instance, compiler "bugs" (or unexpected compiler behavior). However, these are typically only a very small part of any security review.
Thou shalt have no other gods before the ANSI C standard 1622
David Wagner snip IME there is a different. Checking invariants requires a run-time evaluation of an expression, typically a predicate, and that evaluation typically requires some non-shippable...
Alt Folklore Computers Newsgroups