Thou shalt have no other gods before the ANSI C standard 1613

David Wagner

We do have such an industry, but it is tiny precisely because it is not well rewarded. Perhaps we are facing the some kind of challenge Detriot faced in the auto industry. Bad development organizations will only improve when threatened by better ones.

The answer is complex. It depends a great deal on the nature of the project. I can only make (arm waving) suggestions based on overall project size. If the project was a 3-man effort, the increase in cost is negligible -- it might be cheaper over the lifecycle. At 30 people you might be looking at 2x. At 300 you might be looking at 50x.

Well first of all the term articulate is actually a hiring criteria aimed at something else. You want really good listerners. Good listeners are better communicators. That manifests itself in the (testable) form of articulateness.

The reason good communication is so important in the qualification phase is that it the the last phase you have before systems get fielded. Fielding defective systems is simple unacceptable.

The cost of finding and fixing defects is extremely high in the qualification phase. Literallty thousands of times higher than in the requirements phase, and hundreds of times higher than in the design phase. So you have to buttume that you are basically spending money like water and good communication among the members of the qualification team makes the process faster. This is not a time-to-market issue. This is

Just look at the cancellation and failure rates for development projects. Do you think those were canceled during the requirements, specification, or implmentation phases? Except for projects that fail to conform to the "stay out of the way" rule, I believe pre-qualification cancellatio to be rare. Cancellation due to an extended qualification phase is endemic.

You need a team of qualifiers rather than testers (your definition of tester). They have to be competent to evaluate the reqs, the spec, the code, and the doc. That skill set is rare and thus expensive. Many organizations limp along doin the best they can, which produces the syndrome you observe -- the qualification staff, schedule, and budget all shrink as the other phases grow. This is the prevalent recipe for bad software.

"Look at" in terms of comprehension of it, of course. But IME that comprehension either leads to investigation of specific targets (i.e. components) or it is essentailly wheel spinning.

I may not have described the issue in #6 adequately. It is not the purpose of that rule to direct someone such as yourself or any of the other serious participants in this thread away from the properties and potential vulnerabilites of the overall system. I meant to suggest that if the qualifiers don't get down to specifics and spend a lot of time on refuting the possible attacks that might be mounted against individual modules or components then the qualifiers are just going through the motions. It is a point worth mentioning because that is a kind of process defect that I see all the time. "Oh yes we have a great QA plan, it is ### pages thick". Good qualification, whether security related or not, simply cannot be performed against a checklist. You can't visit each module. You have to pick the key ones and grok them in the original sense of grok.

Of course. Item #6 was poorly presented.

But your comment suggests another reinforcement of the requirement for top-notch communication during the qualification phase. You mentioned "... understand which components are trusted for which purposes ...". IME that is rarely adequately documented. The indequacy is aften caused by the evoution of the understanding of the system that occurs as it is built. This is why an investment in multiple design-implementation cycles pays off so well -- you aren;t done until a cycle occurs in which you don;t learn anything critical.

The qualifiers have to ferret that kind of information out of the developers themselves often by (gently) interrogating them about what they actually built rather than what the design or implementaiton plan said they were supposed to built or what they intended to build. The skew between the developers intentions and the code they produce is the most fruitful area for defects. (And I suspect exploitable "features").

tj3