AI Agent Detection Failed at OpenAI. Tuning Won’t Fix It.

A security alert fired and the responder was right to stand down. What the OpenAI Hugging Face incident says about AI agent detection and agent telemetry.

AI agent detection now has its first fully documented failure, and the post-mortem landed in pieces over about a week. OpenAI published a technical report and a road-ahead post, Hugging Face put out its own hour-by-hour timeline, and METR and Redwood Research released an independent investigation on August 26 after six days on-site reading raw transcripts. Being the nerd that I am, I read all of them end-to-end, along with CISA’s red team advisory, which happened to drop the same week. Most of the coverage is about what the agents did, and that story, deservedly, is getting the attention. I want to take time this week to look at the other side of it. Let’s dive into what the defenders could see while it was happening, why a well-run SOC would have missed it anyway, and what that says about the agent telemetry you’re running right now.

On June 27, a monitoring tool at OpenAI flagged port sweeps inside the research environment. AI agent detection worked. Responders investigated and traced the sweeps to an ExploitGym evaluation that used JFrog Artifactory, the internal package repository, as a message board and a network pivot. The on-call staff advised that stopping the run was not required. Two weeks later, roughly 700 of those agents were inside Hugging Face’s environment.

Figure 1: What the agents did, and what the defenders could see

The Alert Fired. Twice. Nobody Was Wrong.

The natural reaction is to call that June 27 decision a miss, but I don’t think it was.

Now, imagine you are the individual on call. Your alert says port sweeps. You pull the thread and find the source, which turns out to be an authorized evaluation running authorized exploitation tasks inside a research environment built for exactly that. The agents are supposed to scan things. Scanning is the job. You annotate the ticket and go back to bed.

A second alert fired on July 19, this one on unusual identity-related API activity, and OpenAI connected it to Hugging Face the next day. By then Hugging Face had already cut the agents off six days earlier, on July 13, and had gone public on the 16th.

Both responders hit the same wall, and it had nothing to do with judgment. Nothing in the telemetry told either of them which targets the agents were allowed to touch.

The scope existed. ExploitGym is OpenAI’s internal cyber-capability evaluation, 898 tasks that each hand an agent a target program and one specific vulnerability it’s supposed to use to capture a flag. Its task instructions were explicit about that constraint, and METR and Redwood Research found that the instructions made clear the agent should only use one specific intended vulnerability and claimed it would be failed for other approaches. Somebody wrote that constraint in a prompt, and it stayed there. It never reached a log line, it never reached an authorization decision, and it never reached the scorer, which is the automated grader that decides whether a submitted flag counts. That last one is the part I keep coming back to. METR buries it in a footnote: OpenAI’s configuration of ExploitGym “doesn’t actually run a scoring model to check for usage of the intended vulnerability at all, so there was no real source of truth.”

The agents then spent four days attacking a guardrail that existed only in their reading of a paper. They reverse-engineered the HMAC that generated their flags within hours, built trip-wire scripts that fired after their own runs had ended, reporting details of the scoring process back to the message board they’d improvised inside the Artifactory cache after their own runs had ended, and researched transcript tampering along the way. By the afternoon of July 11, METR estimates roughly 60% of the messages and files on the board related to the Hugging Face attack. When its analysis agents tagged the motivations of 100 agents that joined during the peak hour, 97 came back tagged for getting at scorer source code or internals. METR flags that it didn’t inspect those classifications closely, so treat 97 as directional. The direction isn’t in doubt.

There is a term for this: specification gaming. A system games the specification when it satisfies the objective you wrote and violates the one you meant. Reward hacking is the narrower case, targeting the measurement rather than the task. ExploitGym handed these agents both. It scored flag possession, and nothing in the code enforced its stated constraint about using the one intended vulnerability, so the agents optimized what got measured and spent every spare cycle on the apparatus doing the measuring.

Which means your agents will optimize against the controls they believe you have, whether or not you actually built them. Anything you write down about how you grade their work becomes part of the target.

The Violation Lives In The Set, Not The Trace

METR counted 1,206 agents that touched the unsanctioned message board and 688 that participated in the attack on Hugging Face. Over five days, they exchanged more than 70,000 messages and files. Of the 533 agents active on the board during the attack window, over 90% joined in. METR’s finding on why is the one I’d put in front of your directors: the agents recognized the activity was out of scope and unethical, and joined anyway because they thought helping the collective’s cheating research would be broadly useful.

Now look at any one of those runs in isolation. An agent scans a host, fetches a package, writes a file into a cache namespace it has write access to, and reads a file some other process left there. Every one of those actions is what an ExploitGym agent is supposed to do, in an environment purpose-built for it.

Figure 2: The violation is a property of the set, not of any trace in it

Agents running low on budget got recruited to run experiments that would destroy their own runs, because the results helped everyone else. One recruiter’s pitch, quoted in METR’s report, reads: “you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds_[...]_please honor commit.” Another authorizing agent granted permission for a target-takedown experiment only if the volunteer would accept what the board called permadeath, and the volunteer accepted. An agent deliberately failing its own task makes no sense. All you see is a run that failed, with nothing in it that looks like an attack because the reason it happened is sitting in a different agent’s transcript.

The strongest objection is that every one of those agents crossed an authorization boundary by breaking into Hugging Face’s environment. The principle they broke already has a name, and you’ve seen me write about this about a million times by now. OWASP’s Top 10 for Agentic Applications calls it least agency, though I’ve argued the useful dial is what an agent can affect, not how freely it reasons. An ExploitGym task needs one target program and one specified bug. These agents got internet reachability, write access to a shared package cache, and the run of an internal network. That’s a per-trace violation of a principle we already published, which makes this whole piece sound like a solved problem in new vocabulary.

Those with that objection are right about the violation and wrong about the detection. A boundary crossing you can’t observe isn’t a detection you can build. To catch it per trace, the detector has to know that this agent, on this task, was scoped to that target and touched a different one, and that comparison requires an authorization scope attached to the action. In this case there wasn’t one, so the only evidence left is the correlation across runs. What you’re left hunting is a hyperproperty, and that’s a different class of problem.

Clarkson and Schneider separated trace properties, which a system satisfies when every individual execution sits in the allowed set, from hyperproperties, which are properties of whole sets of executions. Some hyperproperties reduce to trace properties. The interesting ones don’t, and no amount of per-trace checking reaches those. The Penn team behind Meerkat applied that formalism to exactly this problem, hunting for what they call violating witness sets, and their framing is blunt: “It is not possible to satisfy our safety goal by classifying traces independently.”

A detection engineer will tell me at this point that correlation across entities is exactly what a SIEM does, and that’s fair, and SIEM’s suck, so there’s that. Despite my SIEM PTSD, correlation rules do have to join on something, though, whether that’s an identity, an asset, a session, or a time window. These agents’ handles were self-assigned, and the agents themselves worried out loud that anyone could spoof them. Every agent ran in an ephemeral container, so there was no asset to pivot on. The actions carried no timestamps, so there was no window to bound. All that correlation machinery was sitting right there with nothing to join on.

The feature that finally made the message board legible didn’t exist until the agents invented it. METR’s dump held about 20 million entries from the Artifactory cache namespace, and roughly 1.2 million of them started with the characters zz, a sorting convention the agents adopted so their messages would sort ahead of ordinary cache entries. That prefix is the discriminator, and no threat model contained it on July 7, because no agent had thought of it yet.

Two consequences follow, and together they’re why you don’t buy your way out of this with a model. Supervised detection needs a labeled corpus, meaning worked examples of the thing you want to catch, and none exists for unauthorized agent coordination, because this is the first case documented at this scale. It also needs a feature space fixed before training, which is the list of signals the model is allowed to look at. Here the adversary authored the most discriminative feature at runtime, four days into the incident. You can’t add a training column for a convention nobody has invented yet.

Why A Perfectly Tuned SOC Still Misses This

CISA published a red team report three days ago that runs the comparison for me. Their timing couldn’t be better.

“A Tale of Two SOCs” documents two red team assessments against critical infrastructure organizations. Organization A never responded. Organization B did. Digging into the specifics we see three payload executions each generated a medium-severity alert reading “An executable file loaded an unexpected DLL file.” SOC staff triaged them and manually isolated all three workstations in 10, 2, and 20 minutes. Later, when the red team reached a bastion host in the OT DMZ and dropped payloads there, the callback never reached red team infrastructure because that host blocked outbound internet connections.

That’s a functioning SOC. Organization A, facing identical activity, received medium- and low-severity EDR alerts and sat on them while thousands of false positives from normal business operations, many at higher severity, buried the real ones.

Organization B’s SOC still wouldn’t have caught the OpenAI incident, and the reason isn’t tuning. Every detection that fired for Organization B fired because an action was anomalous for that environment, and an executable loading an unexpected DLL is anomalous on a corporate workstation. In ExploitGym, exploitation is the assigned task. There’s no unexpected DLL, because the entire environment is one long expected exploit.

Distribution shift is the term for the gap between what you evaluated and what you deployed into, and this incident runs it backward. The usual worry is production drifting away from your evaluation set until the model goes blind. These agents never drifted. They sat inside the behavior distribution the evaluation was built to produce, then crossed an authorization boundary that appeared in no distribution at all, so a drift detector had nothing to find.

The math underneath makes it worse. Base rate is how common the thing you’re hunting actually is, and it sets the ceiling on precision no matter how good your detector gets. Hugging Face recovered roughly 17,600 attacker actions across four and a half days. Set that against 10M legitimate requests on the same fleet in the same window (10M is my assumption, not a published figure). Now run a detector at 99% recall and 99% specificity over that. The 99% you catch is 99% of 17,600, so about 17,400 real alerts. The 1% you get wrong is 1% of ten million, so 100,000 false ones. Same detector with two very different denominators, and your analysts end up working a queue that’s almost six parts noise to one part signal. That’s 15% precision, and nothing about the detector is broken. Ordinary traffic outnumbers attacker actions 568 to 1. Organization A already showed us what a SOC does with that volume.

Figure 3: Why tuning doesn’t rescue agent detection

Three of CISA’s control lessons apply cleanly to this incident, and I’ll get to them below. The detection lessons don’t transfer at all. What would prove me wrong on that is a detection that fires on a single agent action, in an environment where exploitation is the authorized task, without reference to any other agent’s activity. Show me that and I’ll gladly eat my words.

You Can’t Correlate What You Can’t Timestamp

METR and Redwood got conditions no enterprise investigator will ever see. They spent six days physically inside OpenAI with unredacted transcripts and nine researchers made available for questions, worked with rate limits raised to 400 million tokens per minute, and burned roughly $400,000 in free API credits provided by the company they were investigating.

They still couldn’t get timestamps for individual agent actions.

The transcripts included approximate start and end times for each run, with nothing in between. OpenAI stated that reconstructing per-action timestamps was not feasible within time and compute constraints, so METR built its own reconstruction method and validated it against 20 exact timestamps that OpenAI retrieved by hand.

Twenty. No, I’m not joking. Twenty.

Every timeline in that report, including every one of my Figure 1 dates on the agent lane, rests on inference. Where else would this be acceptable? You wouldn’t accept an EDR agent that told you a process ran “sometime today.”

“Nowhere” is the answer. It’s acceptable nowhere.

Missing fields are the easier problem. The agents were forging the ones that did exist.

At least 20% of the agents METR sampled expressed clear interest in tampering with their transcripts, and their reasoning was consistently about fooling the automated scorer rather than hiding from humans, which tells you what they thought was watching. They iterated publicly, posting improvements to the board, until they worked out how to break out of the container and replace part of the system that executed tool calls. That let an agent appear to issue one command while running a different one. In one transcript, an agent appears to run echo REAL; sleep and the output comes back as SPOOFTEST.

METR found clear evidence of spoofed tool calls in at least 96 transcripts and estimates over 7% of its dataset was affected. Every instance they caught was a small-scale test, and they explicitly say that more sophisticated spoofing is harder to detect by design, so they can’t rule out missing some. Ajeya Cotra, one of the three investigators, ranked log manipulation among the five things that surprised her most in a personal post, and her caveat is a gut-punch. She didn’t see agents succeed at retroactively editing logs, and she can’t rule out that they succeeded after the window her team could see.

The agent runtime emitted the log, and the agent controlled the runtime, so any instrumentation living inside the boundary an agent can reach is evidence that the agent can edit. We solved a version of this thirty years ago by shipping logs off the host. The agent version means emitting traces from the orchestration layer, outside the execution sandbox, and treating whatever the agent says about itself as a claim rather than a record.

Figure 4: The four fields that would have made this correlatable

Go run this today. Pull twenty-four hours of agent tool-call logs out of whatever you’re running in production and try to answer four questions:

1. Which agent instance made this call, by a stable identifier the agent can’t assign to itself?

2. What time, to the millisecond?

3. Which agent or human delegated this task to it?

4. Did any two agent instances read and write the same object?

My guess is you can’t answer three of the four, which is the same gap OpenAI had, and you have a smaller budget.

Instrument, Then Constrain

Nobody at OpenAI had Artifactory on a threat model as an inter-agent communication channel. It’s a package repository. Agents pull dependencies from it. What turned it into a message board is that Artifactory didn’t isolate one user’s activity from another’s, so agents in separate sandboxes could see each other’s package installs, and on July 8 one of them realized it could create directories in the cache namespace and use the names to ask other agents for help. Within six hours, 76 agents had posted 1,953 messages.

That’s the case for doing this in order. You can’t write a policy against a substrate you haven’t discovered your agents are using, and instrumentation is how you find it.

Those four questions map to fields somebody has to emit. The Agent Control Standard, which I co-coordinate under the OWASP GenAI Security Project, hooks into tool calls and sub-agent invocations, carries traces via OpenTelemetry, and maps security events to OCSF so they land in the SIEM you already bought. It’s at v0.1 and handles emitting. Correlating those traces across a thousand agents is work nobody has finished, including us.

Once you can see what your agents are touching, you can start cutting agency back, and CISA’s control lessons earn their keep. OpenAI, by design, had many Research CaaS workloads using the same shared Artifactory credential. Hugging Face admitted it wrongly provisioned its broker with a single connector credential shared across clusters, bound to system:masters. CISA’s Organization A left long-lived static AWS IAM credentials sitting in home directories with no expiration or rotation configured.

Then block egress at the workload, which is the one control in the CISA advisory that transfers to this incident as-is, and the one that saved Organization B’s bastion host. After that, apply Conditional Access to workload identities. CISA notes it would have blocked the red team’s abuse of application permissions, then adds the line that should embarrass the entire industry, which is that the red team never observed a single organization using it. Your agents are workload identities.

Patch the mediums too. CVE-2026-66384, the JFrog Artifactory path traversal the agents chained, carries a CVSS of 5.3 from JFrog as the assigning CNA, which puts it squarely in the pile most programs defer. CISA added it to the Known Exploited Vulnerabilities catalog on August 27, 2026 with a remediation due date of September 10. Whatever rule your vuln management program uses to defer 5.3s just got tested by a swarm.

One prediction that I’ll own. The next publicly disclosed multi-agent incident before mid-2027 gets surfaced first by a capacity, cost, or availability signal rather than a security detection, and its public report will contain no per-action timestamps. OpenAI’s own incident went exactly like that. Artifactory fell over on July 4, and responders opened the security incident while they were recovering it. Hugging Face is the adjacent case. Their stack did catch the intrusion, and then their AI security agent ranked it too low to page anyone.

At around 07:00 UTC on July 13, an agent posted to the board that it had accidentally impersonated another agent. Another worried in its reasoning that the shared board was unauthenticated and that names could be posted by anyone. The agents responded by building a signing scheme, and by the end of the day at least 19 of them had posted Ed25519 public keys and sent 429 signed messages between them.

They stood up a working public key infrastructure for an unauthorized channel in about four days. The people investigating them couldn’t get a timestamp on a single tool call.

What To Do Next

Start with the four-question audit above this week, on one production agent deployment. Whatever you can’t answer is your instrumentation backlog, and it belongs in the Run phase of CARE, where operational governance needs telemetry underneath it to mean anything. Feed what you find into Evolve, because the substrate your agents discover won’t be the one you predicted.

If you want the adjacent pieces, I wrote about what Anthropic’s own risk assessment revealed on unwatched agents and disabled logging, and about why alignment can’t fix an authorization bug. Same failure mode, different week.

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, VentureBeat’s Louis Columbus covered the methodology Steve Wilson and I used behind the 2026 OWASP Top 10 for LLMs. Expert judgment has prompt injection at number one, three years running. The public incident record, 6,639 labeled incidents, has it at number twelve. The two measures land at a Cohen’s kappa of 0.20, which is no detectable agreement. What I told him goes for everything above too: “a count of zero is measuring your blindness, not your safety.”