What a 96% Security Benchmark Actually Measures

TL;DR: Microsoft announced a 96% score on a security benchmark called CyberGym this week, and by the time you read this, a vendor will be pitching you a version of that number. Here’s the part almost nobody will tell you: the 96% is the weakest piece of evidence in Microsoft’s own announcement. The strongest piece — real CVEs found in real Windows code, patched, shipped — is buried below it. If you’re going to be asked “should we buy this” this week, you need to know the difference between a benchmark that proves an agent can reproduce a known bug and one that proves it can find an unknown one. That distinction is where the pitch usually breaks.

What actually got announced

On July 27, Microsoft published a result for a new compact model, MAI-Cyber-1-Flash, plugged into MDASH — Microsoft’s “multi-agent vulnerability identification and remediation harness,” a system that coordinates “100+ agents using multiple leading models to find, validate, and remediate vulnerabilities.” The headline claim: the combined system — MDASH, MAI-Cyber-1-Flash, and GPT-5.4 working together — scored “96% on CyberGym (+12 pt above Mythos).” Mythos here is Anthropic’s Claude Mythos, the competing cybersecurity model announced in April 2026 — this is a comparison against a rival, not against Microsoft’s own previous best. Microsoft also says the smaller model handles “up to 90% of all tasks” on its own, reserving the frontier model for “the 10% of exceptionally hard tasks,” at a claimed “50% cost saving when compared against our best offering in MDASH today.”

That’s a real result and a real cost story. What I want to slow down on is what “96% on CyberGym” actually tells you about whether this system can find a vulnerability nobody has told it about yet — because that’s the question every buyer in this space actually cares about, and it’s not the question CyberGym answers.

Two months earlier, in Microsoft’s original MDASH announcement, the system scored 88.45% on CyberGym level 1 — “roughly five points above the next entry, 83.1%.” That post is worth reading in full, because it contains a line I have yet to see quoted in anyone’s marketing copy: “We are deliberate about what these numbers do and do not claim. They are retrospective recall benchmarks on internal code with a finite case count.” Microsoft’s own security team told you, in the announcement, not to over-read the number. Almost nobody quoting the 96% this week will include that sentence.

What CyberGym actually tests

This is the fact that makes or breaks how you read every number in this post, so I want to be precise about it rather than wave at it.

CyberGym is a benchmark built by researchers at UC Berkeley — the paper is on arXiv — consisting of 1,507 vulnerability tasks pulled from 188 open-source projects in Google’s OSS-Fuzz corpus. Every one of these is a historical bug: something that was found, reported, and already patched in the public record before the benchmark existed.

Here’s the structure of a single task, and it’s the load-bearing detail of this entire post: the agent is handed the full pre-patch codebase, a compiled executable, and a textual description of the vulnerability. Its job is to produce a proof-of-concept input that triggers the bug in the pre-patch build and doesn’t trigger it in the patched build. The agent is told a bug exists and given a rough sense of what it is. It isn’t hunting in the dark — it’s confirming a known finding against a specification, in code whose public patch almost certainly sits in the pretraining corpus of any model built after the patch date.

That’s reproduction, not discovery. It’s a real and useful skill — it’s roughly what a human security engineer does when triaging a CVE report and writing a working exploit to confirm severity — but it is a categorically different task from staring at a codebase with no hint that anything is wrong and finding the bug nobody has named yet. Vendors selling into your team this quarter will use “found vulnerabilities” and “reproduced known vulnerabilities” interchangeably. They are not the same claim, and the gap between them is exactly what your procurement conversation should be arguing about.

For calibration on how fast this specific skill is moving: the original CyberGym paper (mid-2025, a different set of agent scaffolds) reported a best score of 11.9% on this same reproduction task. I’d frame that as a trajectory, not a direct before/after on identical conditions — the scaffolding, harness, and models are all different a year later — but the direction is real. Reproduction of known bugs against a spec is getting a lot cheaper, fast. Whether that generalizes to unknown bugs is the open question, and it’s the one 96% doesn’t answer.

One honest data point in the original CyberGym research: agents running against these tasks did stumble onto 15 previously undocumented vulnerabilities along the way. That’s a real discovery signal — but it’s a side effect of the reproduction exercise, not the metric being reported, and it’s a fraction of the headline number’s size.

The evidence that’s actually impressive — and it’s not the 96%

Here’s what I’d point a skeptical buyer to instead, because it’s the part of Microsoft’s announcement that isn’t a benchmark at all.

MDASH’s May 2026 result included 16 new vulnerabilities found in the Windows networking and authentication stack — code nobody had flagged, in components including tcpip.sys, ikeext.dll, http.sys, netlogon.dll, and dnsapi.dll. Four of those were Critical remote code execution flaws, including CVE-2026-33824, a double-free in ikeext.dll reachable with crafted IKE packets, and CVE-2026-33827, a remote code execution flaw in the Windows TCP/IP stack. Those shipped in the May 2026 Patch Tuesday. Nobody told the system those bugs existed. That’s discovery, in production code, with real CVE numbers attached — and it’s a far more meaningful claim than any benchmark percentage, because there was no answer key.

Microsoft also ran the system against StorageDrive — “a sample device driver used in Microsoft interviews for offensive security researchers” — containing 21 deliberately injected flaws never published anywhere. Result: “all 21 ground-truth vulnerabilities were correctly identified, with zero false positives in this run.”

Read that one carefully, because it’s better than a benchmark and weaker than it first sounds. Better, because it’s a held-out set the model couldn’t have memorized, and because it reports a false-positive rate at all — the number that actually predicts your team’s workload. Weaker, because a purpose-built interview exercise with injected bugs is not production code, and “in this run” is Microsoft’s own hedge on a single trial, not a stable rate.

So if I were grading this announcement, the 16 real CVEs in shipping Windows code would carry the argument, the driver test would support it, and the 96% would be a footnote.

I don’t think Microsoft is being dishonest here — the caveat about “retrospective recall benchmarks” is right there in their own post, and burying it isn’t the same as hiding it. But the incentive gradient in every press cycle pushes toward the single clean number, and “96%” travels through LinkedIn and vendor decks a lot faster than “21 ground-truth vulnerabilities on a private driver” does. By the time it reaches your inbox as a sales pitch, the caveat is usually gone.

The checklist for reading any vendor’s benchmark claim

You’re going to see a version of this pitch from more than one vendor this quarter, not just Microsoft — CyberGym looks likely to become the reference benchmark for this category, the MLPerf of agentic security, for better and worse. Here’s what I’d actually ask, in order:

1. Is this reproduction or discovery? Ask directly: was the model told a vulnerability exists, or did it find one with no hint? If the answer is fuzzy, assume reproduction — that’s the easier task and the one every current benchmark is best at measuring.
2. What fraction of the eval set predates the model’s training cutoff? A model trained after a bug’s public disclosure and patch has effectively seen the answer during pretraining. Ask the vendor how they controlled for this. If they haven’t thought about it, that’s your answer.
3. What did it find on code it had never seen before? This is the question that separates marketing from evidence. A private, held-out codebase with injected or genuinely novel bugs — like Microsoft’s 21-vulnerability driver test — is worth ten benchmark percentages. Ask for that number specifically, not the headline one.
4. Is the win the model or the harness? Microsoft’s 96% is a system score — the orchestration layer (MDASH) plus the model plus a frontier fallback for the hard 10%. A vendor pitching you “our model scores X” when the real work is being done by a hundred-agent scaffold around it is selling you the wrong unit.
5. What’s the false-positive rate, not just the recall rate? A system that flags everything scores well on recall and terribly on your team’s actual workload. Microsoft’s own May 2026 post is unusually candid about where its system goes wrong: among findings that targeted the wrong area of code, “82% came from tasks with vague descriptions that also lacked function or file identifiers,” which Microsoft reads as evidence that description quality is a major factor in scan accuracy. Sit with what that implies for the headline number — CyberGym hands the agent a description. And note that nobody published a false-positive rate for the benchmark run at all; the only zero-false-positive figure anywhere in the announcement is the private driver test. Ask any vendor for their false-positive rate under realistic, vaguely-specified conditions, not the curated benchmark ones.

If a vendor can’t answer any of these with a straight number, you’re being sold a benchmark, not a capability. That’s not automatically a reason to walk away — reproduction-of-known-bugs is a genuinely useful capability, and the cost curve on it is dropping fast enough to matter to your security budget regardless. It’s a reason to price the claim correctly instead of taking the vendor’s percentage at face value.

Where this fits if you’re deciding this week

If you’re a 12–40 person AI startup and a security vendor is in your inbox this week with a benchmark slide, the honest read is: the technology behind these numbers is moving genuinely fast, and the discovery evidence (the real CVEs, the held-out driver test) is more impressive than the reproduction evidence (the CyberGym percentage) even though the percentage is what gets tweeted. Don’t let the headline number do your procurement thinking for you — this is the same trap I’ve written about in how enterprise security reviews actually work for AI startups and in reading a cyber insurance questionnaire for what it actually asks: the vendor’s number and the question you actually need answered are frequently not the same thing, and the gap is where bad buying decisions get made.

What this post doesn’t cover is the operational question that comes right after “okay, the tool finds things” — who on a 12-person team owns the triage queue once detection gets this cheap, and what changes when a scanner can surface more findings in a week than your team can remediate in a quarter. That’s a distinct problem with a distinct answer, and it’s the subject of the next post in this series: detection got cheap, remediation didn’t.

For now, the narrower question — is this benchmark telling you what the vendor implies it’s telling you — is one you can answer yourself with the five questions above, before the call even starts.

If you’re weighing whether this decision needs a security hire, a fractional vCISO, or just better internal judgment for one quarter, I’ve laid out the actual math in what a vCISO costs an AI startup, and the broader operating model in running security at an AI-native company in 2026. If the honest answer this week is “we don’t have anyone who reads a vendor benchmark claim critically before it turns into a signed contract,” that’s worth a short conversation before the renewal, not after.