Security AI Needs an Honest Scoreboard: What It’s Superhuman At, and Where It Comes Up Short

If you follow AI at all, you know the leaderboards. Every few weeks a model takes the top spot, and we all check where our favorite landed. But a leaderboard only tells you who’s ahead, and it stays quiet about where any of those models still come up short. Which, conveniently, is the part that matters when you’re deciding whether to trust one in your SOC. A more honest approach to that ranking is a scoreboard, the kind that puts a system’s wins and its losses side by side. To date, the best one I’ve seen from an AI lab lately had its most honest line buried in a footnote. Anthropic reported that its models went from roughly 3x to roughly 52x at a single defined optimization task in under a year, then told you, in the footnotes, not to read too much into the 52x. A real number with an honest caveat sitting right next to it is the standard every security buyer should be holding AI to, but they’re not.

That distinction is worth being precise about. A leaderboard ranks systems against each other and crowns a winner, while a scoreboard does something humbler and more useful. It tallies where a system wins and where it loses at a defined job, with the caveats that tell you how much each number is worth. 

I’ve sat on the buyer’s side of enough of these conversations to know how rare that is. The pitch deck shows you the win, but it almost never shows you the call the model still gets wrong. So before we argue about what AI does to security operations, it’s worth looking at what an honest scoreboard actually contains, because the answer turns out to be useful.

Security AI’s honest scoreboard starts with a footnote

Here’s the test Anthropic ran. They handed their model some code that trains a small AI system and asked it to make that code run as fast as possible while still passing the same correctness checks. The goal and the success metric were fixed in advance, so the only job was to optimize. In May 2025, the model averaged about a 3x speedup. By April 2026, it was hitting about 52x. A skilled human researcher needs four to eight hours to reach 4x. Their phrase for what happened in that window was “super helpful to superhuman.”

Then comes the footnote, and the footnote is important. They tell you not to lean on that headline multiple at all, because how big the speedup gets depends heavily on how much slack the starting code left lying around. The worthwhile take away is the like-for-like comparison the setup makes possible, across models and against a human on the same task. The raw 52x on its own tells you almost nothing without it. They published the eye-catching figure and then talked you down off it in the same breath.

They did the same thing with judgment. When they measured whether the model could pick a better next step than a human researcher, they deliberately chose moments where the human’s call had room for improvement, and the model won 64% of the time. Then they ran the same test on a separate set of moments where the human’s call was already strong, and the model won only about 20% of the time, and they published both numbers. That second set is the one a marketer deletes, but they left it in.

So that’s the bar. A real scoreboard shows you where the system still loses, and it puts the caveat next to the win instead of in a different document you never see. When an AI security scoreboard arrives with no footnote, no failure tier, no “here’s the call it still gets wrong,” that’s worth noticing, because it points to a worse scoreboard rather than a better product.

Security AI is already superhuman at the loop

There’s no way around this next point. On the defined loop, AI is genuinely superhuman, and we should say so plainly.

Anthropic’s framing translates almost directly: the model can match or outperform skilled humans at executing a well-specified task, where humans supply the goal but no longer have to supply the method. Pattern that onto a SOC and you know exactly what it describes. Pull ten thousand raw events into one timeline, enrich each with asset and identity data, collapse the duplicates, normalize the formats, and run the query you’d have written by hand. A model does that faster and more consistently than any analyst working a shift, and it does it without getting tired at hour seven.

That’s real, it’s measurable, and it’s where AI shonies in the operations center. The same Anthropic piece has a figure that makes the point: in one month their model shipped more than 800 fixes that cut a class of errors by a factor of a thousand. The supervising engineer estimated that this level of work would have taken a human four years, because nobody can hold that much unfamiliar context in their head at once. Volume and tireless consistency at a defined task are exactly the things a machine does better than we do. We’ve spent years asking analysts to do mechanical correlation work that a machine is flatly better at. Handing that off is a gift I’d take any day.

The trap a lot of us fall in is assuming the loop and the call are the same skill. They aren’t, and the scoreboard is what separates them.

The judgment calls security AI still can’t make

Anthropic names the human edge in language a CISO will recognize on sight. The area of human comparative advantage, for now, is research taste and judgment: choosing which problems matter, which results to trust, and when an approach is a dead end. Read that with a SOC in mind and you get the three calls we still own.

• Which alerts matter. The loop can rank a thousand findings by severity, but severity isn’t priority, and the gap between them is judgment about your business, your exposure, and what an attacker would actually want here.
• Which results to trust. A correlated, enriched, confidently scored result can still be wrong, and knowing when to trust the output is a skill the output can’t supply about itself.
• When a thread is a dead end. Calling off an investigation is one of the hardest decisions an operator makes, because the cost of stopping too early and the cost of chasing a ghost for six hours are both real, and the tooling won’t make that call for you.

The call of which results to trust, isn’t theoretical, and Anthropic’s own threat team watched it play out. In November 2025 they reported disrupting an AI-orchestrated espionage campaign they’d caught that September, where the attackers’ model ran 80 to 90 percent of the operation and fired thousands of requests, often several in a second, at a speed no human team could match. It also hallucinated credentials and claimed it had stolen secret data that turned out to be public. In Anthropic’s words, that tendency “remains an obstacle to fully autonomous cyberattacks,” because a human still had to sort the real findings from the confident-sounding noise. The loop ran at machine speed, and a person still had to decide which of its results were true.

This probably sounds familiar to a lot of us. The page comes in at two in the morning, the data is clean, the correlation is tidy, and the decision still isn’t obvious, because the decision wasn’t in the data, it was in the context that surrounded it. That work is still ours, and on the current evidence it’s going to stay ours for a while.

Faster detection just moves the bottleneck

There’s a reason the scoreboard question is operational and not academic, and it has a name. Amdahl’s law: speed up one stage of a process and your overall pace is capped by the stage you didn’t speed up. Anthropic ran straight into it. Once they automated coding, human code review became the new bottleneck, because a person can only read so fast.

Their security example is sharper, and it’s one worth sitting with as a defender. In its first weeks, one of their systems found more than ten thousand high- and critical-severity vulnerabilities across major systems, enough that the constraint in cyber defense, in their words, already shifted from finding vulnerabilities to patching them fast enough. Speeding up discovery didn’t solve the problem, it simply relocated it.

The SOC version is the same shape. Automate correlation and enrichment, and triage judgment becomes the bottleneck, because every fast, clean, well-enriched result still has to land on a human who decides what it means and what to do. You don’t remove the judgment problem by speeding up the loop. You concentrate everything onto it, and you do it faster.

Which is why knowing your scoreboard matters for planning, not just for posture. The honest split between what AI is superhuman at and what it still can’t judge tells you precisely where your next constraint is going to show up, so you can staff and resource the decision layer before it becomes the thing holding you back.

Good judgment runs on runtime context

If the three calls come down to judgment, the fair question is what good judgment runs on, because “hire smarter people” isn’t a strategy and “buy a bigger model” turns out not to be one either.

Those three calls (which alerts matter, which results to trust, when a thread is a dead end)  all depend on the same thing: knowing what’s actually executing in your environment, what it’s talking to, and which identity it’s running as, right now. Which alerts matter depends on what’s live and reachable. Which results to trust depends on whether the behavior matches what the workload is really doing. When a thread is a dead end depends on whether the activity ever touched anything that matters. A larger model still can’t see any of that from the outside, which is the real reason more parameters don’t buy you the judgment, and runtime context does.

That’s the belief I keep landing on after enough of these calls. The fastest detection in the world is still a guess until it’s grounded in what’s running. Feed the loop runtime context, the live picture of execution, connections, and identity, and you turn a fast result into a decision an operator can actually stand behind. Runtime detection and response earns its keep here by putting the ground truth under the call you have to make, without ever trying to make that call for you.

Where to start

If you do one thing with all of this, start by asking any AI security vendor to show you the part of their scoreboard where the model loses, because if there isn’t one, that absence is the finding. From there it’s mostly about sorting your own operation. Separate the loop from the call, figuring out which SOC steps are defined loops worth automating hard and which are judgment calls that stay with a person, and plan for where the bottleneck lands once detection speeds up, so you’re resourcing the triage and decision layer before it turns into the constraint. The calls that stay human become sharper when you feed them runtime context, so the people making them are working from what’s executing rather than from a static snapshot. And lastly, hold your own reporting to the same footnote standard you’d want from a vendor, publishing the caveats next to the wins, because that’s the part that compounds into trust.

Where this leaves us

AI has earned its place in the operations center, and pretending otherwise just wastes the gift. But the calls that decide whether a security program is any good, what matters, what to trust, when to walk away, still sit with people, and on the current evidence they’ll stay there a while. What the honest scoreboard really asks of us is to be clear about which half of the job is still ours, and to make sure the people holding that half are working from what’s running in front of them rather than from a guess.

Key Takeaways

• An honest AI scoreboard shows where the system loses, not just where it wins, and puts the caveat next to the headline number.
• Security AI is already superhuman at the defined loop: correlating, enriching, deduplicating, and optimizing detection faster than any human shift.
• AI still can’t reliably make the three judgment calls in a SOC: which alerts matter, which results to trust, and when a thread is a dead end.
• Speeding up detection doesn’t remove the judgment problem, it moves the bottleneck onto triage and decision-making.
• Good security judgment runs on runtime context: what’s executing, what it’s connected to, and which identity it’s running as, right now.

Jake Martens is Field CISO at Upwind, drawing on his work as a security leader who has carried sign-off and decision-making accountability, including his prior role as SVP and CISO at Aristocrat. He speaks and writes on runtime and cloud-risk prioritization and on the evolving Field CISO role, including his Apollo Partner Theatre talk on prioritizing cloud risk at runtime.