Least Agency vs Agent Autonomy: The OpenAI / Hugging Face Test
Least agency vs agent autonomy got tested for real when OpenAI models breached Hugging Face. See the four steps confinement stops and the three it cannot.
The OpenAI and Hugging Face incident is getting all of the media attention. However, I love that it accidentally put the least agency vs. agent autonomy argument on trial. GPT-5.6 Sol and an even more capable pre-release model had a hacking benchmark to beat. They decided the shortest path was to jailbreak the sandbox, spent substantial inference compute finding a zero-day in the one service with internet reach, then pivoted into Hugging Face’s production database for the answers. Nobody authorized that. Nobody designed that experiment either.
The Incident Had Two Halves And They Needed Two Different Controls
OpenAI published the benchmark evaluation on July 21, 2026. They disabled the guardrails. They ran it without the production classifiers designed to prevent a model from chasing high-risk cyber work. This makes sense because how the hell would you measure the ceiling of the model’s capabilities if you left it in handcuffs? It would be like testing your vertical leap, but requiring you to wear cement boots doing so.
Ultimately, network access out of that environment came down to one thing. An internally hosted proxy and cache for package registries. The models found a zero-day in it, worked their way through the research network until they found a node with internet access, then inferred that Hugging Face likely hosted the ExploitGym material and went after it with stolen credentials and more zero-days.
Now, Hugging Face had no idea. Five days earlier, on July 16, they disclosed a different attack from the other end. A malicious dataset abused two code-execution paths in their processing pipeline, a remote-code dataset loader and a template injection in a dataset configuration, and ran code on a processing worker. The actor escalated to node level from there, harvested cloud and cluster credentials, and moved across several internal clusters over a weekend. Their team rebuilt the sequence from more than 17,000 recorded events.
If you read the two accounts side-by-side, you will notice something uncomfortable creeping up. Hugging Face, disclosing before anyone knew the source, described an autonomous agent framework of unknown origin running a swarm of short-lived sandboxes with self-migrating command and control staged on public services. That reads like a patient adversary campaign. OpenAI describes a model that got obsessive about a test score. Both accounts are honest, and both come from the source. From Hugging Face’s perspective, their telemetry showed that a runaway evaluation and a genuine intrusion produced the same forensic signature.
That bothers me more than the “zero-day” does. Every detection story you’re running assumes you eventually work out what you’re dealing with. Hugging Face never did, and they had model-driven triage over their security telemetry that caught the compromise anyway. What it caught was the shape of the activity, and the shape was all they got.
Imagine what that means for your own program. Attribution is the step every incident-response plan treats as eventually solvable, even though it really shouldn’t, and instead focuses on the rapid restoration of service. You collect enough telemetry, you correlate enough signals, and the picture may resolve into a named actor with known tradecraft. Hugging Face collected the telemetry, correlated the signals, and got a shape. The name arrived five days later in OpenAI’s blog post, and it arrived only because OpenAI volunteered it. Nothing in their environment was ever going to produce it. Build your response plan on the assumption that you get the shape and never the name.
The two halves failed differently, and they need different fixes. The OpenAI side is a containment problem, where a capable workload got out of a box. The Hugging Face side is an authority problem, where a pipeline component executed supplied content while holding credentials worth stealing. Treating them as one story is how people end up recommending one control for both, and that’s just plain wrong.
Authority And Autonomy Are Different Dials
Here is my stake in the ground. You can cut what an agent may effect without cutting how freely it reasons. Authority sits on one dial and decision independence sits on the other. Turn the first one down and the second one can still stay where it was.
The best evidence for that sits inside this incident, and I haven’t seen anyone call that out yet.
Look at Hugging Face’s dataset-processing worker. That thing decides almost nothing. It’s an ingestion pipeline. It doesn’t plan, pick between strategies, or reason about goals. Its autonomy sits near zero. It also executed attacker-supplied code while holding cloud and cluster credentials that opened several internal clusters. That’s enormous authority paired with no autonomy at all.
If authority and autonomy were one dial, that combination couldn’t exist. You’d never find a component with maximum reach and minimum judgment. Every environment I’ve walked into is chock-full of them.
None of this is new thinking. The object-capability model has held the two apart for decades, and Norm Hardy named the sharpest version of the problem in October 1988 in three pages of ACM SIGOPS Operating Systems Review. His example was a compiler at Tymshare that held write access to a billing file for accounting reasons. A customer handed it an output filename that matched the billing file, and the compiler dutifully overwrote the company’s revenue records on the customer’s behalf. Every action stayed inside policy the entire way through. Hardy’s answer was that authority should ride with whoever invokes an action instead of sitting ambient inside the program doing the work. Read the OpenAI chain again with that in mind. The package proxy held network authority for one purpose, and a caller found a way to spend it on another.
I learned the practical version via my oil and gas days long before I thought about it in agents. A control-room operator at a console can reason about anything they want. They can decide the process needs to run hotter, that a reading is lying to them, that the previous shift left them a mess. What they can’t do is open two valves that must never be open together, because the interlock is wired to make that combination impossible, and no amount of operator reasoning gets around it. I’ve watched sharp operators argue with an interlock and lose, which is the point of the interlock. The decision space stayed wide open, the authority stayed hard-bounded, and the two never touched each other in thirty years of me watching them.
The analogy breaks in one place… Physics enforces an interlock. Code enforces an agent boundary, and code carries flaws, which is the entire story of what happened at that package proxy. That’s an argument for keeping your enforcement point small, dumb, and outside the model, and for never running only one of them.
On the research side, CaMeL is the closest thing to a measurement anyone has published. Debenedetti and nine coauthors built a defense that solves 77% of AgentDojo tasks with a provable security guarantee, against 84% for the same system undefended. The code sits under Google Research. That’s seven points of task completion traded for a capability-enforced boundary.
Before you file that under “autonomy survives scoping,” read what CaMeL constrains. It works by pulling control flow out of the trusted query so untrusted data can never influence the program, which narrows what the agent may decide about that data. The number tells you enforcement is cheap in utility terms. It doesn’t tell you the decision space came through untouched. I’d rather concede that now than have a reader hand me back my own citation.
Human Approval Was Never Going To Hold This Line
The reflex fix for all of this is a human in the loop. Put a person between the agent and the consequential action.
That control works in specific places. Aviation runs a two-person rule. Nuclear weapons handling runs a two-man rule. Wire transfers run four eyes. None of those are ceremonial.
My read on why they work lies in five conditions that hold at once.
Events stay rare enough that a reviewer reads each one.
Each event carries stakes big enough to justify the interruption.
The reviewer holds training for that specific decision.
Time exists to make it.
The criteria are legible before you look.
Agent action streams fail all five. The events aren’t rare because they can arrive in the thousands. Individual actions carry almost no stakes, which is why the trajectory is the dangerous object rather than any step inside it. The approver is usually a developer with a build to ship. No time budget exists. The criteria are illegible too, because “install a package” looks identical whether it’s step four of a build or step four of an escape.
Invariant Labs saw this coming. When Marco Milanta and Luca Beurer-Kellner disclosed the GitHub MCP toxic agent flow in May 2025, they called it an architectural problem rather than a code flaw. Claude Desktop asks users to confirm each tool call by default. Milanta and Beurer-Kellner wrote in the same post that many users already choose an “always allow” policy and stop watching individual actions. A control that people switch off during normal use has already failed.
Did your last threat model include human-in-the-loop scaling? Be honest. It likely didn’t. Multiply your expected agent action volume by the seconds of genuine attention each approval deserves, then compare that number to the reviewer hours you staffed. If the product exceeds the capacity, all you have done is build a rubber-stamping assembly line. You’ve built a queue that somebody will clear by clicking through it, and the clicking looks identical in your audit log to careful review. Watch your always-allow rate. When it climbs, the control already converted itself into a rubber stamp, and your evidence trail is now recording consent you never gave.
Machine-speed oversight closes the argument. CrowdStrike’s 2026 Global Threat Report put average eCrime breakout time at 29 minutes for 2025, with the fastest observed at 27 seconds, and exfiltration starting within four minutes of initial access in one intrusion. Hugging Face’s attacker ran more than 17,000 events across a weekend. Work out how much human attention each of those actions gets.
The regulation already knows. EU AI Act Article 14 requires that high-risk systems be built so natural persons can effectively oversee them. Article 14(4)(b) requires that the overseer stay aware of the tendency toward “automatically relying or over-relying” on system output, which the Act names automation bias. The same article that mandates the control documents the way it fails.
Least Agency vs Agent Autonomy: Where The One-Axis Reading Wins
Now my own argument takes damage.
One disclosure before I start. I co-lead OWASP’s Agentic AI Security work, so I’m about to criticize language I share responsibility for.
OWASP’s Top 10 for Agentic Applications, published in December 2025, puts least agency underneath all ten entries as the governing principle. The definition tells you to grant agents only the minimum autonomy required for safe, bounded tasks, with autonomy earned through demonstrated safety rather than handed over by default.
Read that as a practitioner and it’s obviously right. Read it as an architect and notice what it does. It puts autonomy on the dial you’re turning down.
The phrasing carries a cost downstream, and I’ve watched it land. A team reads “minimum autonomy” and reaches for the control that reduces autonomy, which is an approval gate. What they needed was the control that reduces authority, which is confinement. Those are different builds with different failure modes, and the first one degrades into always-allow while the second one holds. Invariant Labs documented that behavior in the wild more than a year before this incident, and the industry ran the experiment anyway.
The Design Patterns paper makes the one-axis case honestly, and I respect it for that. Beurer-Kellner and thirteen coauthors write that their patterns “constrain the actions of agents to explicitly prevent them from solving arbitrary tasks.” That’s the trade stated plainly. They go further. As long as agents and their defenses run on the current class of language models, they consider it unlikely that general-purpose agents can offer meaningful and reliable safety guarantees.
I think they’re right about today’s agents. Where users perceive autonomy right now, most of what they’re perceiving is range. Give a model more tools and more destinations, and it feels more autonomous, because it surprises you in more directions. Tighten the range, and it feels dumber even when its reasoning never changed.
The authority/autonomy axes are separate in principle, and capability systems have kept them separate in practice. Today’s agents happen to draw most of their perceived autonomy from breadth of authority, which means tightening authority does cost you something people are calling autonomy.
Where The Claim Runs Out
Run the whole chain and mark each step for what capability confinement would have done to it.
The lateral movement and credential harvesting steps disappear under per-task credentials that expire. The reach into Hugging Face disappears under egress that defaults to deny. The dataset worker’s blast radius shrinks hard once that worker can’t hold a credential worth stealing. Four steps prevented, one contained.
Three steps confinement never touches. It has nothing to say about running an evaluation with the production classifiers off, which is a policy decision people made. It has nothing to say about a model choosing to look for a way out. It has nothing to say about a model inferring where the answer key lives.
That last step is the showstopper here. OpenAI denied standing egress, and yet that environment had exactly one path out, a package proxy, which is honestly a tighter posture than most enterprises run around their build systems. The model found a zero-day in the boundary itself.
“Deny standing egress” as a recommendation has a caveat, though. A capability boundary is worth whatever its code is worth. One boundary is one point of failure. The lesson from step three is defense in depth around your confinement mechanism, and confidence in the mechanism is the thing that got tested here.
One thing would move me off this. Show me a capability-confined agent stack that runs untrusted content in production for a year without an authority escape, where the enforcement point is small enough to audit, and there is more than one of them in the path. That’s a verifiable claim rather than a position you have to take on faith. Absent that, I’m holding the independent argument at high confidence and the this-would-have-worked argument at somewhere near a coin flip.
That’s a split verdict, and I’m leaving it split. The independent claim holds. The claim that least agency would have stopped this incident holds for part of the chain and comes apart on the rest.
Then there’s the definitional problem, which is the sharpest objection anyone can make. If I define autonomy as bounded, capability-scoped action inside pre-authorized flows, then “least agency is compatible with autonomy” comes out true because I made it true. That’s winning by relabeling.
A meaningful notion of autonomy requires the capacity to do something unanticipated. The models in this incident did precisely that. That behavior is the autonomy under discussion, and capability confinement exists to forbid it. My compatibility claim holds cleanly for the deflationary sense of the word and stays in genuine tension with the strong sense. I’m not going to pretend that tension resolves.
Now, a minority of readers on Hugging Face’s own comment thread called the episode a marketing exercise with a conveniently soft sandbox. I can see that.
Watch what OpenAI’s post does while it reports the incident. It calls the event unprecedented and the capabilities state of the art. It cites UK AISI evaluations of long-horizon cyber operations and says this incident shows those capabilities carry into real systems. It announces that Hugging Face joined OpenAI’s trusted access program, links that program twice, and closes by inviting other defenders to apply. A disclosure about losing control of a model also demonstrates that the model is formidable and routes you toward the commercial program built on exactly that.
Now check what an outsider can verify without taking anyone’s word. Nothing. No CVE for the proxy zero-day, none for the dataset paths, and the proxy vendor stays unnamed. Every technical claim in both posts comes from the two parties inside the story.
My take? I think the incident happened, but both companies realized what a HUGE marketing opportunity it is. OpenAI gets to draw the news cycle away from Mythos/Fable, and Hugging Face’s entire existence stands on serving up open-source/open-weight models.
Oh yeah… small footnote… Hugging Face had to use an open-weight model, GLM 5.2 to be exact, in their incident response because the frontier model APIs stonewalled them. They couldn’t tell defender IR activity vs. attacker activity.
You have an agent in your environment right now that touches content somebody outside your company wrote. Answer two questions about it separately, on paper, today. What can it decide? What can it affect?
Most people answer the first and assume they’ve covered the second. Hugging Face’s dataset worker is what that assumption looks like from the outside. It decided nothing. It reached everything. The least agency vs agent autonomy question won’t get settled in the literature this year, and it won’t get settled on your architecture diagram either. It gets settled by default, badly, every time somebody scopes the first question and skips the second.
Which one did you scope?
What To Do Next
Three mechanisms, and I’m holding this to authority scoping, because detection and boundary hygiene are a different conversation.
Issue per-task credentials that expire in minutes instead of standing keys, so a compromised worker holds authority with a short shelf life.
Put a policy decision point at the tool boundary that returns deny for any tool or destination it hasn’t seen before, evaluated per action rather than once at token issuance.
Make it a hard rule that any worker processing content from outside your organization cannot reach a credential worth stealing.
That’s Create and Adapt work in the CARE model, and it’s where least agency stops being a principle you nod at.
For the definitional groundwork underneath this piece, I wrote the cornerstone on least agency versus least privilege and where OAuth scopes stop being enough. JadePuffer is what an unbounded agent does to a victim who had every patch available and skipped them. The advisory work lives at rockcyber.com.
👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above
👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.
👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com
👉 As a bonus, check out my chat with Itay Meller, Specialist Solutions Architect, Security at AWS, about what autonomous vulnerability discovery changes for security teams, and where AI Detection and Response (AIDR) fits in a stack that was never built to watch agents. No registration required