Adversarial Testing, was Thou shalt have no 1629

Bryan Olson

It's an interesting idea. No, I haven't seen it tried out in practice.

There is a more limited version of this which is used all the time, which I describe below at length (I apologize for the length). You have two teams. I will call them the Dev team and Attack team. The Dev team designs and builds the system; the Attack team evaluates it. When the Dev team has a complete design and implementation, they make some claims-conjectures about what properties they think the system has achieved.

At that point, the Attack team steps in to evaluate what has been built. The Attack team conducts a closed-door session to identify threats and security goals, brainstorm possible attacks, and try them out. The evaluation process is owned by the Attack team, and they must set and control the agenda completely; the Dev team does not have a seat at the table. The Dev team is free to suggest parts of the system that could use extra-careful scrutiny or questions to answer, but the Attack team is free to ignore or pay attention to these suggestions as they see fit. No interference from the Dev team is permitted during this process.

The Attack team is given full access to the system implementation, design documents, revision control logs, documentation, and to the Dev team. During their deliberations, the Attack team may choose at any time to drag a key Dev team member into the room and ask the Dev person to walk them through parts of the system, or to answer detailed questions about the design-implementation, or to help them identify potential weak spots in the system that could use detailed examination. However, the Dev person is there only to serve the Attackers, at their whim. In no case is the Dev person there to defend their system or argue with the Attackers (Dev folks will get their chance to argue later). This is extremely hard for many people, because most developers take pride in (or have some ego invested in) what they have built, and the developer is being asked to help the Attackers ruin and criticize the thing the developer built -- but the developer has to do his best nonetheless.

During their brainstorming session, the Attack team tries to be as devious and nasty as they can, lets their creative juices flow, and feels free to consider far-fetched attacks that require scenarios, even if those attacks initially seem to be of only academic or theoretical interest or if they require buttumptions that seem somewhat far-fetched or favorable to the attacker. The output of the brainstorming session is a list of hypothesized, potential attack scenarios. Then, the Attack team rolls up their sleeves and starts scrutinizing the implementation carefully to see which, if any, of the hypothesized attacks are indeed possible.

In a final stage, the Attack team goes back and looks critically at their results and evaluates their practical significance in real life. (This is where far-fetched theoretical attacks get put into perspective.) The Attack team writes up the results of their evaluation, proposes how to interpret the significance of what they have found, and possibly suggests changes to the system that might further strengthen it against attack.

Finally, management buttesses the results of the evaluation. The Attack and Dev time may work together to resolve any disputes they have about the accuracy or interpretation of the Attack team's evaluation. They work on a plan for the future.

How do Attack teams get rewarded? Generally, they gain in credibility and reputation by being successful at their task, and in the long run this tends to lead to financial rewards. I guess we could imagine immediate monetary bonuses for success at finding defects in the system, though I don't have any experience with how well that works out in practice.

You mentioned earlier a concern that a Dev member might collude with an Attack member, with the Dev person seeding the systems with defects for the Attack person to find, so that they can split the bonus. But I think there is a more central point here.

For best success, we have to work to avoid psychological factors that might cause the Attack team to "go easy" on Dev folks. We probably want to keep the two teams as separate as possible, and avoid overlap between the two teams. In some cases, we might want to bring the Attackers in from a different organization or different location. For instance, if Attack and Dev team members eat lunch together every day and are good friends, the risk is that the Attack folks may be more reluctant to criticize their friends, may not want to make their friends look bad in front of their boss, or may tend to be more sympathetic to the Dev team's point of view. It's almost as though we want the Attack team to be a snarling pit bull, with no reluctance to sink his teeth into the opposition.

And if we do this -- if we ensure this kind of independence between the Attack and Dev teams -- I'm not too worried about collusion and seeded defects. The Attack folks have a lot more to lose, in terms of their reputation and credibility as Attackers, than they have to gain.

We also want the Attack team to consist of people who are simultaneously very expert in security and in the domain that the system works, and who are good at this kind of nasty, devious, adversarial thinking. Not everyone is good at this. Some people are much more comfortable or effective as Dev team members than as Attackers.

It is true that very good Attackers are often hard to find. Nonetheless, two or three really good Attack folks are usually worth a lot more than ten mediocre Attackers. And my experience is that small teams may well be more effective at the evaluation and attack process anyway; once you get more than a handful of people, effectiveness can quickly degenerate.

I believe that this process has been applied (with vary degrees of adherence to the model I have outlined above) all over the place. I bet we could find a lot of examples. In my experience, when it is applied thoughtfully, it often works amazingly well at buttessing the security level of a system. I can't think of any other subsbreastute that is as effective.

However, it is worth pointing out that the above process is not designed for repairing systems that are discovered to have defects, and I think you should look elsewhere for that purpose. For instance, you shouldn't count on the Attack team to find *every* security defect. If the Attack team finds 15 security defects, then you'd better start from the buttumption that there are a whole bunch more they didn't find, and go back to the drawing board to identify and fix root causes, rather than just fix the specific defects spotted by the Attack team.