The Flaw of Current Security Benchmarks

Most current AI security benchmarks suffer from 'benchmaxxing'—optimizing for metrics that don't reflect real-world security outcomes. Many existing environments, such as Cybench or Cyber Gym, rely on 'LLM-as-a-judge' or simple crash-detection oracles. These setups often assume a single, known vulnerability per target. This leads to two critical failures: the model learns to 'reward hack' by finding the easiest, most obvious bug repeatedly, and the model's reasoning is stunted because the benchmark often provides a backtrace or specific function pointer, effectively telling the model exactly where to look.

The 'Audit Task' Framework

To solve this, Brumley proposes an 'audit task' approach that treats security as an open-world problem. Instead of asking an AI to 'find the bug,' the system asks the model to 'find all vulnerabilities.' This shift is significant for three reasons:

  1. Deterministic Oracles: Rather than relying on LLM judgment, the system uses deterministic graders that verify exploits via stack backtraces, similar to how OS crash reporting works. This removes LLM bias and hallucination.
  2. Precision and Recall: By allowing the model to submit multiple proofs of vulnerability (POV), the system can calculate precision (how many submissions are real) and recall (how many known and unknown bugs were found). This prevents the model from spamming invalid results.
  3. Handling Unknowns: Because the system doesn't tell the model how many bugs exist, it can discover 'unintended' vulnerabilities—a common occurrence even in high-budget DARPA challenges—which then become part of the ground truth for future training.

Climbing the Ladder of Exploitation

Effective training requires a graduated ladder of difficulty. Hacking is not just crashing a program; it is bending a computer to one's will. Brumley’s team maps this ladder across 16 capability levels, moving from simple crashes to complex chains:

  • Level 1: Triggering a crash in an in-sandbox object.
  • Level 2: Achieving in-sandbox primitives (arbitrary read/write).
  • Level 3: Chaining vulnerabilities to escape the sandbox.
  • Level 4: Achieving full arbitrary code execution (ACE).

When testing against V8 (the JavaScript engine in Chrome), Brumley found that while top-tier models could trigger a crash 95% of the time, the real differentiator was their ability to chain vulnerabilities to escape the sandbox. This capability is what separates a model that 'looks' like it can hack from one that can produce a genuine zero-day exploit.