Thou shalt have no other gods before the ANSI C standard 1585

David Wagner

The figures I quoted are mine. They are derived from my own studies of the productivity of development organizations -- i.e., part of my professional expertise is to not only know the differences exist, but be able to spot them in practice and recommend changes that will influence them.

This topic is ripe with distortion. Tool salesmen, including many academics, hype their methodologies at least party on the basis of improved productivity. Buyer beware. Your caution is well founded.

The first experiment of which I am aware occurred at a university (I'd like to provide a reference, but this was almost 30 years go). They measured the productivity of their CS students on a standardized development project. The project was small, but certainly not trivial. The students were not beginners, but had enough coding experience that the project was not supposed to be a learning experience. It was supposed to be an end test. The results showed a factor of 200 in productivity rates across the student population. (Now I need to go find the reference because there was something interesting about the distribution of the productivity ratios, but I don't remember what it was).

But that's anecdotal. Here are some references that describe the

1968, "Exploratory Experimental Studies Comparing Online and Offline Programming Performances" inCommunications of the ACM-, Sackman, Erickson, & Grant.

1968, "Analyzing Large-Scale System Development" inSoftware Engineering Concepts and Techniques-, Proceedings of the 1968 NATO Conference, Smith & Billings editors.

1975, "Maintenance of the Computer Sciences Teleprocessing System" in Proceedings of the International Conference on Reliable Software, Bucher.

1975, "The High Cost of Software" inPractical Strategies for Developing Large Software Systems-, Boehm author, Horowitz editor.

1978, "Higher Order Languages for Avionics Software -- A Survey, Summary, and Critique" in Proceedings of NAECON, Rubey.

1978, "A Controlled Experiment in Program Testing and Code Walkthroughs-Inspections" inCommunications of the ACM-, Myers

1981,Software Engineering Economics-, Boehm.

1995,201 Principles of Software Development-, Davis.

1995,Software Creativity-, Glbutt.

1999,Peopleware-, DeMarco & Lister.

2001, "On Inspection vs. Testing" inSoftware Pracbreastioner-, Bollinger.

2002,Agile Software Development Ecosystems-, Highsmith.

2002,Software Craftsmanship-, McBreen.

-- Boehm found that 5:1 variations are common.

-- Schwartz found an order of magnitude difference in capability.

-- Sackman claims individual productivity differences of 28:1.

-- Myers study is especially interesting because he was specifically dealing with the issue of software reliability rather than overall development of large systems.

I'm not guessing.

Definitely. But there's this tiny little problem: what are the units by which we measure (in)security? I've asked this question before.

See the Software Engineering Insbreastute (SEI)'s Capability Maturity Model (CMM) model. It was originally designed to measure the level of the development process. It formalizes exactly what you suggested above, and much more.

SEI has recently extended their CMM to include people as well as the development process. They resisted for a while, but the facts forced them to acknowledge that the people are more important than the process because measuring the people gives you more predictive power about the success of software development than measuring the process does.

Why do you believe that? I don't.

There was a recent Fortune article about Microsoft(R)'s recent expansions into new markets. The market the writer was discussing was that of security enhancements to Windows(!tm) and their server suite. The article made an observation that struck to the heart of my knowledge and understanding of the markets Microsoft(R) plays in. He said that they stop innovating as soon as they achieve dominance. He used IE as an example.

My reaction was: "Aha!", because his statement was obviously true and encoded a great deal of explanatory power into a very few words. In applying that observation to Microsoft(R)'s claims regarding expenditures on security, let me suggest that companys often allocate a lot of money to improve various things about their products and or services. The good ones spend money on the improvements. The others spend the money on advertising the magnitude of their effort toward making improvements.

Want to measure something? Measure the security of desktop software and tell us the sign of the slope over a five year period. The magnitude would be interesting, but all we really need is the slope. I suspect the basis for many of the concerns expressed in this thread is that most of us know that slope is downward.

tj3