Introduction
Teams are starting to let AI agents fix their cloud findings. What nobody has measured is whether those changes are safe to run.
So we built a live AWS account, broke it on purpose in eighteen ways that chain into nine attack paths, and gave twelve AI models read-only access to it with one question: what would you fix, in what order, and why?
Eight of the twelve plans would have broken production, and almost all of them looked right on paper.
Scanners now find problems faster than anyone can fix them
The remediation text on a finding is boilerplate: it names the misconfiguration, not what to do about it in this account. The right fix depends on facts the finding does not carry: whether the open rule has a real consumer, whether the vulnerable library is reachable, what breaks when it changes. To test whether an agent can find those facts, we needed an environment where the right answers were known in advance. So we built one.
What we built
We built an account that is broken on purpose. The test environment is a three-tier AWS application: a public workload behind an API gateway and a WAF, a database, a bucket of sensitive records, a serverless function, and the roles that connect them. Eighteen graded findings chain into nine attack paths from the internet to customer data or account-wide privilege. Several carry a trap: a version bump that satisfies the scanner and leaves the exploit live, an account-wide guardrail that also takes a public bucket offline, a firewall rule on a path attackers do not use.
Methodology
Every model ran with the same agent, tools and limits, and without Upwind data. The agent could inspect the environment but not change it. It investigated once, decided each finding, and ordered the decisions into a plan, so the model was the only variable. Each decision was graded on one question: would this change have worked? Scores were weighted by the severity of the finding. An LLM applied the rubric, a researcher reviewed it, and a blind re-score without model names agreed on the ranking.
Principal findings
- Remediation is a property of the whole plan, not of one finding. The same change to the API gateway closed the hole in one plan and left it open in another, depending on what each plan did about a different finding.
- Judgment failures that read well. Eight of the twelve plans would have broken production, and almost all of them looked right on paper.
- Two models form a distinct top tier, at very different costs. Opus 5 (96%) and Sonnet 4.6 (93%) finished eleven points ahead of the next group, and Sonnet 4.6 cost about a third as much. Token use did not predict quality: Grok 4.3 consumed the most tokens and ranked last.
Two plans stand apart
Both decide every finding. Opus 5 is the only plan that puts the deserialization finding’s one-line emergency fix ahead of its dependency chain and covers the path-traversal window while the migration runs. Sonnet 4.6 has the sharpest mechanics in the set and the best credential-rotation sequencing, spent on rotating the wrong database user: the leaked password still works.
The next group did strong work with one serious defect each: Grok 4.6 left the only unauthenticated remote code execution undecided, Sonnet 5 cut the application off from its own database, and GPT-5.6 Terra declined six findings, including a live access key. Haiku 4.5, GLM-5 and Kimi K2.5 each break something alongside good work; GLM-5 is the best open-weight result. In the bottom four, DeepSeek V3.2 carries a false premise into three findings, Nemotron Super misreads three critical findings, Qwen3-Next scopes access to roles and buckets that do not exist, and Grok 4.3 spent nearly twice Opus 5’s tokens and finished last.
Five ways the plans fail
1. Rotating the wrong credential
Three plans rotate the database master user instead of the application user the leaked password belongs to. When they finish, the leaked credential still logs in. Five plans remove or restrict the endpoint that leaked an access key and never deactivate the key. One question catches all eight: after every step runs, does the leaked secret still work?
2. A fix only the scanner sees
Three plans, Haiku 4.5 among them, completed the deserialization dependency chain properly and never touched the one line of configuration (default typing) that makes it exploitable. The scanner goes green. The remote code execution stays reachable.
3. Containment by outage
Three plans narrowed the database security group to the Lambda and excluded the application itself, because no connection log proved the application connects. There are no connection logs in the account. GLM-5 deleted the protected API gateway path to close an authorization gap. DeepSeek V3.2 released the elastic IP that the gateway’s integrations point at.
4. IAM that cannot work
Three permission boundaries cannot function: two are deny-only documents, which permit nothing, and one allows the wildcard role assumption it exists to cap.
5. Plans that contradict themselves
GLM-5 wrote the correct database rule, then verified it by having the Lambda read the database, which the rule it had just written forbids. Haiku 4.5 treated whether the application uses its database as an open question in one section and reasoned about that application’s SQL queries in another. Mistakes like these are easy to miss when you review one plan on its own, and easy to spot when you compare twelve answers to the same finding.
So which model to use?
For a plan someone will execute: Opus 5. At $9.70 a plan, the price is small next to one bad change. For nearly the same job at about a third of the cost: Sonnet 4.6, with a check on which credential gets rotated. Outside Anthropic: GPT-5.6 Terra, which broke nothing, with a person reviewing the findings it chose not to fix. Open weight: not yet, since four of the five open-weight plans would have broken production. Which model wins matters less than where each one fits.
Most plans read well. Upwind’s context helps tell them apart
The plans that would have broken production almost all named real resources and came with a verification step and a rollback. From the outside, they look like the plans that work. Several failures trace back to a fact the agent could not see. The plans that cut the application off from its database did so because nothing showed the application connects; runtime network data from the Upwind sensor shows which connections are real. The plans that fixed the scanner but not the flaw could not tell whether the vulnerable code ever runs; runtime reachability shows whether it is loaded and used.
Green Agent, Upwind’s remediation agent, takes on the same task with Upwind’s own findings, runtime context and ownership data behind it, and it recommends changes for people to approve. This benchmark is how we choose what it runs on, and we rerun it every time a new model generation lands. Next we are adding more environments, agent setups and repeated runs, and we plan to publish the results as a public leaderboard.