Thou shalt have no other gods before the ANSI C standard 1620

David Wagner

I suspect that is false. Detecting exploitable conditions often involves simulating the atual operation or exercising the real software in a lab setting. Once the actual behavior of the system has been established then throughtful analysis can be used to evaluate the hazard posed by the expoitable condition.

Testing is not a bunch of monkey pounding keys. It is at least as strenuous as the initial design process. But it cannot focus on the intent, requirements, or specs for the software. It has to focus on the what was actually built rather than what was intended to be built. That often requires considerable hands-on operation.

No.

Your mental model appears to be tilted toward defects such as fundamental math or bad coding, which are subject to effective hands-off analysis. That model does apply to security protocols and other sci-crypt-related issues.

But it does not apply to software. Consider that one cannot even guess where most performance is lost in an application that has recently reached the point as which the full feature set is available. That is why no sane developer attempts to improve the performance of a software package without extensive performance profiling.

Since many exploitable sofware defects are related to timing issues, the need for extensive investigation of what the software actually does is a critical input into the analysis process by which we determine whether or not it does something we don't want it to do.

I think this illustrates a terminology issue in this thread. I suspect you are using the term testing to mean exercising the software. I am using the term testing to mean measuring or evaluating the software.

Many times those measurements are simply cerifying that the invariants that were supposed to hold do so in practice. Does scrub() always clear the buffer? It turns out that the answer depeds on whether the memory manager has made a consolidation pbutt over the storage arena between the allocation and initialization of the buffer and the point at which scrub() was invoked. The failure of the scrub() invariant is an exploitable hole. But in practice it cannot be deduced from the application level source code because the flaw is caused by a level-crossing behavior that emerges from artifacts of the implementation rather than the intentions, specs, and design of the system.

As I understand your usage of the term I agree. But there is more to the process of testing as I use the term than simply exercising the system.

Let's say that testing the feature set and testing the security of the system are two distinct processes.

tj3