Thou shalt have no other gods before the ANSI C standard 1441

when we were working on the original payment gateway and this thing called SSL with this little small startup in menlo park (that later moved to mountain view)

after they had gone thru all there extensive testing ... we produced a fault-failure matrix which listed all possible faults-failures (some amount of this is traditional enumeration of edge conditions) we could come up with in conjunction with all possible states ... and had them demonstrate that all possible conditions in all possible states were handled.

this is somewhat like chip logic checkers (before chips became too complex to do complex coverage) ... one of the earliest was the los gatos state machine (LSM, later renamed logic simulation machine for public consumption) ... which was followed by EVE (endicott verification engine ... and somewhere inbetween was yorktown simulation macine).

as chips became more and more complex ... you started to see by at least the mid-90s situations where there might only be a couple percent test coverage ... and the evoluation of testing methodologies using things like genetic (adaptive) algorithms.

misc. past posts about lsm, ysm, eve, etc:

by comparison in the mid-70s ... when i released the (mainframe) resource manager .... we developed an automated benchmarking methodology and defined something like 1000 different benchmarks to validate and calibrate the resource manager. the science center

had done a lot of work on performance monitoring, workload profiling, performance management, performance simulation, and the early inception of capacity planning. part of this was an APL performance model that took in real live data from running systems and was later used as a product called "performance predictor" on HONE

where marketing and sales people could ask what-if questions about customer configurations (i.e. what benefit was there to more disks, faster disks, faster cpu, more real storage, etc).

however, for the resource manager validation ... the APL model was feed the results of the first 1000 or so benchmarks and then was allowed to choose configuration, workload, and system parameters for another 1000 benchmarks (examining the result from each benchmark before defining the next one). This set of 2000 benchmarks took something like 3 months elapsed time to run:

--