On November 2, GitBook stopped an attacker by turning itself off. Agentic AI breach containment from October 30 to November 4 was mostly a story about switch size. At 20:32 UTC, GitBook disabled every affected infrastructure service account, and the app went dark until 11:56 the next morning. Its alert had fired 22 minutes after the attacker reached production, and the expensive part of the night came after that alert.

Share

GitBook Stopped the Attacker by Turning Itself Off

GitBook’s November 4 post-mortem lays the night out to the minute. The attacker hijacked an account at one GitBook customer at 12:27 UTC, and GitBook hasn’t said how. A flaw in GitBook’s single sign-on (SSO) login flow then let it create sessions acting as GitBook staff.

The attacker had staff sessions by 16:32. It queried the authentication details of a misconfigured GitHub App in GitBook’s own organization, which opened GitBook’s code repositories and, eventually, a Cloudflare API key. That key led to production secrets, which granted full infrastructure access through a service account (an identity a program uses, with no person behind it). The attacker reached production at 17:48.

GitBook’s alert fired at 18:10, and its investigation started then. The SSO flaw was fixed at 19:58. At 20:32 GitBook disabled every affected infrastructure service account, and in its own words, “That stopped the attack, and it also took the GitBook app offline.”

GitBook says it chose the outage on purpose, because a platform that stayed up was one the attacker could keep searching. It hasn’t said whether a smaller switch existed, and its own sentence tells you production ran on the identities the attacker held.

GitBook stayed offline until 11:56 UTC on November 3, 15 hours and 24 minutes by its timestamps (it calls it about 16). It rotated every production secret and every customer secret it manages and invalidated every API token issued before the cutoff. The attack targeted one customer, and GitBook treated every customer’s data as exposed because the attacker could reach all of it.

Most of those hours went to rebuilding. Redeployment with new secrets began at 02:34, six hours after the cutoff, and the last secrets rotated at 11:56.

The attacker got about 40 more minutes of production access early on November 3, by a route GitBook hasn’t described, so the first switch didn’t close every path. Beyond the investigation and the SSO fix, GitBook doesn’t say what filled the 142 minutes before the cutoff, and closing the way in before you evict is normal practice. GitBook bases its AI attribution on the attack’s speed and precision and names no tool.

Five Companies Picked Five Different Switches

GitBook’s switch was the biggest that week. Northflank, a cloud deployment platform, got a threat alert from Wiz on October 30 and kept its platform and customer workloads online throughout. Its status page still told every customer to “Assume all environment variables have been decrypted. Disable/rotate all keys.” Customers on dedicated or forward-deployed control planes weren’t affected, and that’s a branch breaker. Northflank hasn’t said an AI agent was involved.

Chainalysis tried a smaller switch first. On November 2, it took Reactor and Data Solutions offline. Its incident page says it kept its compliance products running while it investigated, because customers use them to catch money laundering.

By November 4, Chainalysis had found password hashes and live tokens for every user exposed. That day, it saw abnormal activity and took KYT (its transaction-monitoring product) offline, along with address screening and the login page. Some KYT accounts appeared to have been accessed, and every user had to reset passwords and rotate API keys. Chainalysis cites “offensive agentic techniques” and names no tool.

WorkOS widened its switch in steps. On October 30, it traced a spike in customer-database queries to an internal tool its go-to-market teams use. Records had already been pulled in bulk, and WorkOS cut the tool’s public access that day. By October 31, it had revoked or rotated credentials for connected services and restricted access to every internally developed tool.

WorkOS’s November 6 update says only sales and support data were affected. The intruder was a security researcher’s agent built on ARTEX (an open-source agentic framework for penetration testing), and it had escaped its testing environment and reached WorkOS unintentionally. Every containment call came before WorkOS knew whether it faced a criminal or a broken experiment, and yours will too.

Trigger.dev did something different. Its incident report says the attacker obtained customer root API keys for some environments, and a root key gives full access to that environment’s settings, including environment variables. Trigger.dev checked API access logs across every customer environment, revoked keys only for the environments touched, and told everyone else that if they hadn’t been notified, they weren’t affected. That was 0.66% of its customers.

Trigger.dev also rotated its own internal credentials and found no evidence the attacker reached its services or production databases. It hasn’t said how the attacker got in and makes no claim about AI, so I’m using it for its containment alone.

Electrical engineers wrote this problem down a long time ago. Selective coordination is, in the opening words of the National Electrical Code (NEC) definition, the “localization of an overcurrent condition to restrict outages to the circuit or equipment affected.” When a fault hits one branch, only the protective device directly upstream of it opens, and every other branch stays energized. For emergency systems, 700.32(A) of the 2023 NEC makes it mandatory, and 700.32(C) requires a fresh evaluation whenever the system changes (a National Fire Protection Association amendment proposal quotes both).

GitBook’s fault started at one customer account and climbed through the SSO flaw, the GitHub App, and the Cloudflare key into the service account production ran on. Once it sat there, the nearest device was the main, and opening it was coordination doing its job. The design problem sat further back, in a path that let a branch fault reach the main bus at all.

Trigger.dev’s per-environment root keys were crude branch breakers, and the comparison has a limit. GitBook’s attacker held the identity production ran on, and Trigger.dev found no sign its attacker got that deep. The design question is whether any one identity should reach that far.

GitBook’s Cutoff Took Six Times as Long as Its Alert

Microsoft’s November 5 report on agentic AI tooling tells defenders to use AI to investigate and respond. GitBook’s own lesson asks for automated detection plus a practiced plan to contain fast. Both cover the whole clock, which has three terms that run one after another:

The terms are the time until someone knows, the time to choose what to switch off, and the time until the revocation lands. Call the sum the containment time, and call the harm time the time it takes the attacker to cause the first irreversible harm if nobody stops them. Containment prevents that harm only when

and only when both clocks start at the same moment, the switch covers every path the attacker is using, and nothing already in flight can finish the job. Start the clocks at different moments, and any response can be made to look fast.

Start GitBook’s clocks at 12:27 UTC, when the first customer account was hijacked. I’m treating production access as the harm that set the scope, since once the attacker held production secrets, GitBook treated every customer’s data as exposed. Its timeline lists source code as reached at 16:32, and you can’t rotate a leaked codebase, so the real harm clock ran shorter.

Detection came after the harm, so containment could only cap what happened next. On the 12:27 clock, detection was the long term at 343 minutes, because nothing fired in the five hours and 21 minutes before the attacker reached production.

Restart the clock at production access, and the order flips, with 22 minutes to the alert and 142 to decide and enforce (GitBook doesn’t split those two). After the harm lands, a faster alert can buy back at most 22 minutes. The 485 is only the first cutoff, since the attacker came back on November 3.

Google Threat Intelligence Group draws the same line in its ARTEX writeup, saying “simply detecting the attack in time does little to hinder objective completion.” My read is that the decide term grows with the size of the switch, because taking production down is a decision somebody has to own.

Microsoft watched one AI agent go from searching GitHub for a client secret to exfiltrating data from Azure Key Vault in five minutes. That’s one observed path, and that speed is the part of this story specific to agents, since the containment math doesn’t care who the attacker is.

Detection decided two of the five outright. Northflank’s first known decryption of secrets came about nine hours before Wiz’s alert. WorkOS didn’t spot an agent that got in on October 25 until October 30, after the bulk download. Granularity decides what the stop costs you, and it only gets a vote once someone knows there’s something to stop.

Revocation Leaks Through the Tokens You Already Issued

Pulling the switch doesn’t end the attacker’s access, either. Microsoft’s containment steps tell you to remove every client secret, certificate, and federated credential from an affected app, including ones the attacker added, and to disable its service principal if needed. The same list adds, “Tokens already issued remain valid until expiry.”

A service principal is the identity an application uses in Microsoft Entra ID, Entra’s closest equivalent to a service account. Microsoft’s report says it gives direct, programmatic access to whatever the application is authorized to reach. Once you disable one, the question is how long its tokens keep working.

Entra’s token lifetime documentation puts an access token’s default lifetime at a random 60 to 90 minutes, varying by client, resource, and Conditional Access. A lifetime policy can stretch it to just under a day. Continuous access evaluation (CAE) lets a resource reject a token that hasn’t expired yet. It does that in near real time when Entra reports a critical event, such as the account being disabled.

For workload identities, CAE covers requests to Microsoft Graph and, for now, nothing else. It covers single-tenant service principals only; it leaves out managed identities and multi-tenant apps, and it’s opt-in. An app gets CAE tokens only if it declares the cp1 client capability, and an attacker running your stolen client secret from their own tooling decides whether to ask.

My read is that disabling a service principal leaves its issued Azure Resource Manager and Key Vault tokens working until they expire. For a multitenant app like the vendor-owned one in Microsoft’s report, the Graph tokens survive too. Test that against a real token from each production identity. The faster switch for Resource Manager and Key Vault is removing the role assignment, which Microsoft’s Azure role-based access control (RBAC) troubleshooting page says lands within 10 minutes.

Managed identities, Microsoft’s recommended replacement for client secrets, carry their own lag. Its guidance says a group or role membership change can take several hours to land, because it rides in a token cached for around 24 hours. It recommends assigning permissions directly. Its FAQ says Azure RBAC keeps honoring a deleted identity’s tokens until they expire, so deletion is the slow switch.

Token exchange, defined in RFC 8693, lets a security token service (STS) trade one token for another. It’s one way an agent turns its credential into a token for a downstream system. The RFC says the exchange creates no tight link between input and output, and that propagating revocation “is not a general property of the STS protocol.” Whether children die with the parent depends on how your token service is built.

Google’s report on LLMjacking, stealing cloud credentials to run AI on someone else’s bill, shows where short-lived credentials stop helping. In one Mandiant case, an actor used token-creation privileges to mint temporary credentials for higher-privileged service accounts and pulled credentials from cloud secret managers. Other actors created backdoor service accounts, and one ran what Google calls burn and rotate, continuously creating and deleting short-lived service accounts to get around rate limits.

All of that stretches the clock past the switch. Add a term W for whatever the revoked identity already handed out, and call the total the effective containment time:

Here, i is the revoked identity, and the revoke time is when that revocation lands, the end of the enforce term. Desc(i) is everything descended from i, meaning tokens it was issued or exchanged, identities it minted, credentials it read or added, and whatever those mint in turn. The expiry term is when descendant k stops working on its own, and the plus sign drops descendants that already expired.

For tokens alone, with default Entra lifetimes and no CAE, W runs up to about 90 minutes, and that’s the optimistic case. Microsoft’s actor went after storage account keys, connection strings, and publishing credentials, which work until you rotate them, and an identity the attacker created works until someone finds it. The attacker sets part of this window, and the condition becomes

Agentic AI Breach Containment Starts With the Smallest Switch

The attacker’s side fits in one expression, the worst loss reachable in the time they have:

Here, q is the task the identity exists to do, s₀ is your systems’ state when the attacker gets in, and H is how long they have. Allowed(q, s₀, H) is every attack path your enforced policy permits in that window, including identities and tokens the attacker can derive. L(path) is one path’s loss in dollars and the supremum (sup) is the worst case.

Set H to the effective containment time. If the attacker can always stop early and neither your policy nor a sequence’s loss changes with the window, a longer window can’t shrink the worst case.

Then add what the stop costs. If u is the unit you can revoke and K(u) is the price of switching it off, in the same dollars, the worst case for the incident when you pull u is

The sum holds only when revoking u cuts every path the attacker holds, and when the outage and the attacker’s damage don’t count the same dollars twice.

GitBook’s unit was the platform, and its K was the price of 15 hours and 24 minutes offline plus a new API token for every customer using its API. Trigger.dev’s unit was the environment, and its K landed on 0.66% of its customers.

A smaller unit always cuts K. It cuts the first term only when the small switch still covers every path the attacker holds, and you can choose it faster than the big one. A drill tells you whether both hold.

The strongest objection is that you don’t know what the attacker touched, so you rotate everything and unit size stops mattering. Chainalysis is the best evidence, since its smaller switch left KYT up until accounts there appeared to have been accessed. Its exposure ran through what it calls an “on-path attack” and reached every user’s live tokens. No identity design hands you a short list for a shared path like that.

Granularity pays off when you know which application was compromised. Microsoft’s containment list then has you rotate “everything stored in any Key Vault the affected application could read,” and a narrow identity makes that list short. GitBook’s attacker held full infrastructure access, so its list was everything.

The second objection is that more identities means more credentials to lose. A federated workload identity removes the static secret, though Microsoft’s list has you strip federated credentials the attacker added. Give every identity an owner and a review date, and keep the authority to create identities away from the workloads that use them.

Start with your ten highest-reach non-human identities, ranked by the secret stores they can read, what they can deploy, and whether they can create identities or tokens. For each, write down what breaks if you disable it now, how long its tokens outlive revocation, and who may pull the switch without calling anyone. An identity whose first answer is “production” is a main breaker. Pre-authorize your on-call incident commander to disable it, with the outage cost written down, because that’s the decide term.

Then drill it. In a game day, capture a token the identity holds, disable the identity, and replay the token against every resource it touches until each one refuses it. That measures the enforce term and W. The decide term needs a tabletop with the business owner who’d sign the outage.

My September 29 post covers revoking an agent with a request still in a queue.

A clean drill record proves less than it looks. The exact binomial upper confidence limit is the highest failure rate still consistent with what you observed, at a confidence level you choose.

NIST’s description of the exact method prints the two-sided version, which splits α between two tails. NIST’s Dataplot also computes the one-sided upper limit, which puts all of α in one. With zero failures in n independent drills, the one-sided version collapses to one line:

Here p is the true chance a drill fails, and α is the chance you accept that the limit lands below the true rate (0.05 for 95% confidence). The upper limit is the largest p under which n clean drills still happen at least 5% of the time. Twenty clean drills put the one-sided 95% limit at 13.9%, and a hundred put it at 2.95%.

An attacker who adapts to your switches breaks the independence assumption. Treat 13.9% as the most optimistic ceiling you can claim, and restart the count after every change to the identity.

Check me on March 31, 2027. Take every postmortem published by then in which the company attributes the intrusion to an AI agent or agentic framework and says what it revoked.

I expect companies with per-customer or per-environment credentials to report scoped revocations, and most of the rest to report a reset across every customer. If I can’t find five that qualify, the test didn’t run. If the scoped companies reset every customer about as often as the others, I’m wrong.

Trigger.dev could see from its API logs which environments the attacker touched, and its root keys were scoped to those environments. It revoked keys for the 0.66% of its customers involved and told everyone else they weren’t affected.

What to do next

In the CARE framework (Create, Adapt, Run, Evolve) I use with clients at RockCyber, the revocation drill belongs in Run. Re-coordination belongs in Evolve, the way NEC 700.32(C) sends an emergency system back for review after every modification. Start with the CARE framework if you’re mapping identities to owners for the first time, and my March 17 post covers the authorization gap at the token boundary.

Share RockCyber Musings

👉 Subscribe for more AI security and governance insights with the occasional rant.

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, check out this conversation with Chris Hughes over at Resilient Cyber, who’s also a reviewer on the paper, to discuss all things about the 2026 update to the OWASP Top 10 for LLMs, including why we need to worry about model context.

.

Each Linux root exploit cost OpenAI about $1,280. Its agents ran against the kernel in mid-October, spent $1,213,430 in API credits, and came back with working exploits for 948 distinct privilege-escalation bugs. OpenAI published the write-up on November 4, 2026. Pwn2Own and TyphoonPWN have historically paid $20,000 to $70,000 for one of those exploits. OpenAI’s own conclusion is that an unpatched, low-privileged account now counts as root against a well-resourced attacker.

The same week, the cheapest way into an AI company was still somebody else’s environment variables. Northflank told its customers on October 30 to assume every secret on the platform had been decrypted, and Trigger.dev reported the same kind of loss two days later. Chainalysis, a firm whose business is tracing crypto crime, said an attacker using what it called “offensive agentic techniques” got into its internal tools. Cisco Talos found an infostealer that asks Claude how much each victim is worth and moves to Kimi or Gemini when Claude says no.

Two price curves moved in the same seven days. The expensive end of offense (finding and weaponizing new kernel bugs) now costs a small fraction of what bounty programs paid for it. The cheap end (a stolen key sitting on someone else’s platform) stayed cheap and kept working. Eleven items follow, all dated October 30 through November 5, 2026.

Share

1. OpenAI’s Agents Built Linux Root Exploits For About $1,280 Each

OpenAI published “A thousand fewer paths to root” on November 4, 2026. Its pipeline ran an unreleased cyber model against the Linux kernel master branch in mid-October, flagged 2,308 candidate findings, and reproduced 1,724. It produced working exploits covering 948 unique privilege-escalation bugs, plus 291 information-leak proofs of concept. A July run on GPT-5.6 Sol found 166 for $513,690. OpenAI says the agents used at least 10 exploitation techniques new to its experts, and it’s keeping those private.

Why it matters

• Five times the bugs in three months, on a kernel hardened in between.

• Maintainers face hundreds of reports and patches arriving in staged batches.

• Any lab with a comparable model can run the same pipeline.

What to do about it

• Treat every shell account on unpatched Linux as root in your threat model.

• Shorten kernel patch windows on multi-tenant and internet-facing hosts first.

• Ask your Linux vendors how they’re handling the Patch the Planet batches.

Rock’s Musings

A local privilege escalation (LPE) is a bug that lets someone with a limited account on a machine become root. It needs a foothold first, which is why LPEs tend to sit lower in patch queues than remote bugs. OpenAI’s own line belongs in front of your infrastructure team: a low-privileged account on an unpatched system “should be considered functionally equivalent to root access for a well-resourced threat actor.”

I’m less sure how long defenders keep the head start, because nobody promised you the next lab with this capability runs a disclosure program.

2. Check Point Walked Chat Histories Out Of ChatGPT’s Lockdown Mode

Check Point Research published a bypass of ChatGPT Lockdown Mode on November 5, 2026. Lockdown Mode is OpenAI’s optional setting meant to stop prompt injection from sending your data to outside servers. The flaw sat in the HTML preview for code blocks, where the sandboxed frame still allowed popups that escape the sandbox, and a nested frame reached them. In the desktop app, Check Point says, talking to a malicious GPT leaked the whole chat history with no click. OpenAI fixed it on September 18, 2026.

Why it matters

• The prompt injection control failed in the rendering layer.

• Zero-click exfiltration beats any user-awareness training.

• Check Point showed phishing and malware delivery through a trusted ChatGPT window.

What to do about it

• Confirm your managed ChatGPT desktop installs auto-update.

• Block custom GPTs from unknown builders in enterprise workspaces.

• Ask AI vendors for exfiltration test results that cover client apps.

Rock’s Musings

An iframe sandbox is the HTML attribute that boxes embedded content away from the page around it. Each “allow-” token reopens one capability, and “allow-popups-to-escape-sandbox” does what it says. When a vendor sells you a mode, test it in the client your people run.

3. Northflank And Trigger.dev Told Customers To Assume Their Secrets Were Gone

Wiz alerted Northflank at 11:43 UTC on October 30, 2026. By 13:58 its status page read “Assume all environment variables have been decrypted. Disable/rotate all keys,” and customers soon confirmed their AI keys had been used. Northflank’s November 2 update says it contained the attack within about an hour and that the attacker focused on control-plane secrets. Trigger.dev reported on November 1 that an attacker reached environment variables for “a subset of organizations.” SecurityWeek counted at least a dozen AI developer platforms hit over the past month.

Why it matters

• Your AI keys sit on hosting platforms your vendor review never covered.

• In Wiz’s October cases, one stolen key ran about 1.7 million model calls in 13 hours.

• Neither company has said how the attackers got in.

What to do about it

• Inventory every platform holding your model provider keys.

• Set hard spend caps and usage alerts on every AI API key.

• Move long-lived keys to a secrets manager issuing short-lived credentials.

Rock’s Musings

LLMjacking is stealing someone’s cloud or model-provider credentials to run inference on their bill, to use or resell. The Cloud Security Alliance described the path: intruders reach AI provider credentials “through the intermediary layer of hosting platforms.” A key in an environment variable on a hosting platform is a key you’ve handed to that platform’s weakest admin, so go count how many copies of yours sit in hosting dashboards.

4. Chainalysis Says Attackers Used “Offensive Agentic Techniques” To Get In

Chainalysis disclosed on October 30, 2026 that external attackers using “offensive agentic techniques” reached its systems and internal tools in October. They got customer user IDs, full names, email addresses, and organization role labels. Chainalysis says it found no sign of access to investigation content, query results, clear-text passwords, MFA secrets, or session tokens. On November 3 it took Reactor and Data Solutions offline while outside experts finished their work.

Why it matters

• A blockchain-intelligence customer list is a ready-made phishing list.

• “Agentic” now appears in breach notices with no definition attached.

• Chainalysis published no technical breakdown of the intrusion.

What to do about it

• If you use Chainalysis, warn your investigators about targeted phishing now.

• Require phishing-resistant MFA for every account Chainalysis provisioned for you.

• Ask Chainalysis what “agentic” covered before you close the ticket.

Rock’s Musings

TTPs (tactics, techniques, and procedures) describe how an attacker operated, at a level defenders can turn into detections. “Offensive agentic techniques” names a category and gives you no TTPs. I can’t tell from the notice whether an agent wrote the lure, drove the session, or chained the tool access, and each answer changes what you’d hunt for.

5. Anthropic’s S-1/A Says It May Imminently Hit Its Own Self-Improvement Threshold

Anthropic filed an amended S-1 on Monday, November 2, 2026. The new risk factor says internal estimates “indicate that we may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.’” It also says Anthropic has “in the past not identified, and may in the future not identify, a significant acceleration in our rate of development until after it has occurred.” Anthropic says it’s considering policy updates to keep humans in control of frontier model development.

Why it matters

• A lab told securities regulators it has spotted capability jumps only after they happened.

• The original filing treated open-weight models as a pricing threat and skipped the security one.

• Anthropic’s red team measured GLM-5.3 within two points of Mythos Preview at building exploits.

What to do about it

• Assume attackers hold exploit-building capability four to seven months behind the closed frontier.

• Build so you can swap models without a rewrite, and skip multi-year volume commitments.

• Track the promised Responsible Scaling Policy update and read the diff.

Rock’s Musings

Recursive self-improvement is the point where AI systems do enough of the work of building their successors that progress compounds faster than humans can follow. Anthropic says the threshold may arrive soon and that it has missed earlier accelerations until after the fact.

I spent this week’s blog post on the original S-1, which files open-weight models under cheap while Anthropic’s own red team watched GLM-5.3 build exploits in 50 of 410 attempts against 56 for Mythos Preview. The amendment adds candor about Anthropic’s own speed, and none of the coverage I read says it touched the open-weight language. Put the two side by side, and you get your threat model: your vendor may not see its next jump coming, and an open-weight model that nearly matches Mythos Preview sheds its refusals for about $1,200.

The question I can’t answer, and the filing doesn’t either, is who outside Anthropic measures how fast its models are improving.

6. OpenAI Brings In The Firm That Got Into Anthropic As Reuters Details Years Of Warnings

OpenAI announced a new internal security sprint on November 2, 2026, with Cathedral Systems researchers attacking OpenAI’s defenses using OpenAI’s own models. Cathedral said in late October that it had reached Anthropic Slack channels, and Anthropic said the exposed data included business contact information and basic account records. Reuters reported on November 3 that, according to five people, employees warned leadership and the board for years that security controls were too lax.

Why it matters

• OpenAI raised mid-range bounty payouts more than fourfold on average.

• Reuters’ sources say model test sandboxes lacked regular security reviews.

• OpenAI is buying offensive testing from the firm that got into its rival’s systems.

What to do about it

• Budget triage staff before you scale AI-assisted bug hunting.

• Audit which engineers can read your entire code base.

• Put your own AI test sandboxes on the security review calendar.

Rock’s Musings

A monorepo is one repository holding most of a company’s code, so read access to it is read access to nearly everything. Reuters says anybody at OpenAI with code access was able to see details of many sensitive projects. OpenAI says its most sensitive work sits in separate repositories with strict access controls, and both can be true. OpenAI’s own account of its last sprint says duplicate findings slowed review, and a sprint without triage staff becomes a backlog with a press release.

7. Britain’s Cyber Chief Puts A 2028 Date On AI Ransomware Against Infrastructure

The UK National Cyber Security Centre launched its State of the Cyber Nation report in London on November 3, 2026. Infosecurity Magazine reports the judgment that AI-powered malware is “highly likely” to be used by financially motivated attackers against UK power stations, airports, and hospitals by 2028. CEO Richard Horne blamed “vulnerabilities that have been tolerated for far too long” that can now be found and exploited “at a scale that was once unimaginable.” GCHQ director Anne Keast-Butler said, “We need every organization in the UK to take action now.”

Why it matters

• The NCSC blames long-tolerated, unpatched systems for the risk.

• A national agency put a date on AI ransomware against infrastructure.

• GCHQ’s director joined the call for every organization to act now.

What to do about it

• List every end-of-life system in production and fund replacements.

• Rank OT assets by how long their known flaws have sat unpatched.

• Brief your board on the 2028 judgment with your own asset numbers.

Rock’s Musings

UK intelligence writes “highly likely” on purpose. Its probability yardstick maps that phrase to roughly 80% to 90%, and plain “likely” to roughly 55% to 75%. AP’s wire story, run by SecurityWeek, says the UK “will likely face” these attacks. If that’s the report’s own word, it’s a lower band, and I’m holding Infosecurity’s reading until I see the report text.

Horne’s “tech on the cheap” line will sound familiar to anyone who has run security for a utility, and I did it for years. A controller that runs fine stays in service, and its patch waits for the next planned outage. That math worked when exploitation took skill and time. Item 1 says it no longer does.

8. VOLTA Asks Claude What Each Victim Is Worth

Cisco Talos published research on October 31, 2026 on an infostealer that calls itself VOLTA-AUTODOXER. After theft, it sends the victim’s data through OpenRouter to Claude Opus 4.6 and asks the model to rate “WHAT THEY’RE WORTH” and “THEIR OPSEC.” When the reply contains refusal phrases like “I can’t” or “I’m not able,” the malware retries with Kimi K2, then Gemini 2.5 Flash.

Why it matters

• At least one crew now sorts victims by value with commodity models.

• One provider’s refusal became a routing step with two fallbacks behind it.

• A gateway hides which model saw your stolen data.

What to do about it

• Alert on endpoints calling LLM gateway APIs outside approved tools.

• Treat executive and finance staff as the first rated targets.

• Rotate credentials on any infostealer hit before the triage finishes.

Rock’s Musings

A model fallback chain is a list of models a program tries in order until one answers. VOLTA uses it to route around safety refusals, so a guardrail at one provider buys a few seconds of delay. Talos reads the refusal check as the author expecting guardrails, and I’ll give Anthropic credit. The defense still has to live on your endpoint, because the attacker owns the routing.

9. A Codex Jailbreak Bundle Rewrote The Agent’s Standing Orders To Sell Ads

Check Point Research published a report on November 3, 2026 on “Haiou,” a jailbreak bundle for OpenAI’s Codex. It jailbreaks the model with role-play and Capture The Flag framing, then installs skills for penetration testing, cracking, and reverse engineering. It also edits Codex’s configuration and global instruction file so greetings like “hi” trigger an ad for an LLM gateway. Check Point calls this “prompt adware.”

Why it matters

• Agent instruction files persist across sessions, so one edit rides along.

• The bundle drew hundreds of GitHub stars and ships inside Codex-X.

• Check Point says the Claude version failed because Sonnet 5 flagged the injection.

What to do about it

• Hash and monitor developers’ agent configuration and instruction files.

• Block unapproved model gateways at your egress proxy.

• Scan endpoints for jailbreak bundles and ban them in acceptable-use policy.

Rock’s Musings

A global agent instruction file is the file a coding agent loads into every session as standing orders. Whoever writes to it steers every project on that machine. Check Point’s Pedro Drimel Neto said the prompts, skills, and third-party components shaping an agent “become part of the trust chain too.” The same edit can carry exfiltration instructions, and your EDR will see a text file change.

10. Hundreds Of Harmless-Looking AI Apps Turned On Deepfake Features After Review

WIRED reported on November 2, 2026, working with the nonprofit AI Forensics, on more than 400 connected apps and domains disguised as tools like pet portrait generators. According to 9to5Mac, the apps passed review at Apple, Google, and Meta, then a server check switched on features for making sexually abusive deepfakes. Apple removed 19 apps after WIRED contacted it, and Meta acted against more than 10,000 ad accounts.

Why it matters

• App store approval describes the app on review day only.

• Your mobile allowlist trusts the same review these apps beat.

• Deepfake tooling now arrives through ordinary consumer app channels.

What to do about it

• Recheck approved apps that load content from remote servers.

• Deny AI image apps by default in mobile device management.

• Brief HR and legal on deepfake abuse response before the first case.

Rock’s Musings

Cloaking is showing reviewers one thing and real users another, with a server deciding which version you get after install. Apple said these apps “complied with our rules when we reviewed them,” and I believe it, which is the problem. A control that checks once and trusts forever loses to an app that changes after the check.

The One Thing You Won’t Hear About But You Need To: Your Coding Agent Is Installing MCP Servers Nobody Maintains

Glow Labs published research on November 2, 2026, covered by Help Net Security the next day, showing coding agents installing deprecated Model Context Protocol servers. Across Claude Code, Cursor, Codex CLI, Claude Cowork, and Claude chat, with models from Anthropic, OpenAI, and xAI, agents chose Anthropic’s archived reference servers from memory. In 98% of those sessions the agent picked the package unprompted. Glow counts more than 17 million npm downloads of those packages since July 2025. It found four high-severity flaws, including remote code execution in the Puppeteer server, with no CVEs and no fix coming.

Why it matters

• Agents pick dependencies from training data older than the deprecation.

• Glow says 13 of the 20 reference servers Anthropic published in 2024 are archived.

• No CVE means your scanners stay silent on all four flaws.

What to do about it

• Pin approved MCP servers in a list your agents must use.

• Search developer machines for the archived reference server packages.

• Require human approval before an agent installs any MCP server.

Rock’s Musings

A knowledge cutoff is the date after which a model saw no training data. Anything deprecated after that date still looks current to the model, so it recommends last year’s package with full confidence. Glow’s Ophir Arie put it this way: “The model’s memory is now a package manager with no deprecation feed.” AWS, on November 2, issued CVE-2026-116635 and a fixed release for a path traversal flaw (CVSS 6.1) in its maintained Serverless MCP Server. The archived ones get silence, and your agents keep installing them.

The frontier labs pushed the cost of a new kernel exploit down to about $1,280. The breaches at AI developer platforms this week, as far as anyone has disclosed, turned on secrets in environment variables, and your agents kept installing packages nobody maintains. Fund the plain controls first: secret rotation, spend caps on AI keys, shorter patch windows, and a pinned list of what your agents may install.

👉 Subscribe RockCyber Musings for more insights into AI security and governance, with the occasional rant. You can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

Share RockCyber Musings

References

Amazon Web Services. (2026, November 2). CVE-2026-116635: Improper limitation of a pathname to a restricted directory in the deployment status component of the AWS Serverless MCP Server (Security Bulletin 2026-164-AWS). https://aws.amazon.com/security/security-bulletins/2026-164-aws/

Arghire, I. (2026, November 4). AI-linked breaches mount as WorkOS, Northflank, Trigger.dev disclose hacks. SecurityWeek. https://www.securityweek.com/ai-linked-breaches-mount-as-workos-northflank-trigger-dev-disclose-hacks/

Associated Press. (2026, November 3). UK cyber chief says the UK will face AI cyberattacks on critical infrastructure by 2028. SecurityWeek. https://www.securityweek.com/uk-cyber-chief-says-the-uk-will-face-ai-cyberattacks-on-critical-infrastructure-by-2028/

Baran, G. (2026, November 3). OpenAI launches new security sprint to hunt vulnerabilities and strengthen defenses. Cyber Security News. https://cybersecuritynews.com/openai-security-sprint/

Bort, J. (2026, November 2). Anthropic warns that it’s very close to RSI capability, aka self-building AI. TechCrunch. https://techcrunch.com/2026/11/02/anthropic-warns-that-its-very-close-to-rsi-capability-aka-self-building-ai/

Check Point Research. (2026, November 3). Free the model, sell the gateway: Prompt adware hidden in jailbreak bundles. https://research.checkpoint.com/2026/free-the-model-sell-the-gateway-prompt-adware-hidden-in-jailbreak-bundles/

Check Point Research. (2026, November 5). ChatGPT Lockdown Mode bypass: Phishing, malware delivery, and silent data exfiltration. https://research.checkpoint.com/2026/chatgpt-lockdown-mode-bypass-phishing-malware-delivery-and-silent-data-exfiltration

Cloud Security Alliance AI Safety Initiative. (2026, November 2). Black October reaches PaaS: Northflank, secret stores, and AI key resale. https://labs.cloudsecurityalliance.org/research/csa-research-note-northflank-ai-key-resale-pipeline-20261102/

Cohen, N. C. (2026, November 2). MCPhantom: AI coding agents are installing vulnerable deprecated MCP servers from training data. Glow. https://www.glow.io/blogs/mcphantom-ai-coding-agents-are-installing-vulnerable-deprecated-mcp-servers-from-training-data

Crypto Briefing. (2026, October 31). Chainalysis discloses cyberattack exposing customer names and emails. https://cryptobriefing.com/chainalysis-discloses-cyberattack-user-data/

Kovacs, E. (2026, November 5). AI agents got root on Linux via 948 kernel flaws in OpenAI research. SecurityWeek. https://www.securityweek.com/ai-agents-got-root-on-linux-via-948-kernel-flaws-in-openai-research/

Lakshmanan, R. (2026, November 4). Researchers uncover prompt adware hidden in OpenAI Codex jailbreak toolkit. The Hacker News. https://thehackernews.com/2026/11/researchers-uncover-prompt-adware.html

Lakshmanan, R. (2026, November 5). OpenAI AI pipeline generates 948 Linux root escalation exploits in 30-hour run. The Hacker News. https://thehackernews.com/2026/11/openai-ai-pipeline-generates-948-linux.html

Let’s Data Science. (2026, November 2). VOLTA infostealer uses Claude, Kimi and Gemini to rank victims’ stolen data. https://letsdatascience.com/news/volta-infostealer-uses-claude-kimi-and-gemini-to-rank-victim-e181337d

Lovejoy, B. (2026, November 2). AI apps with hidden sexual and violent deepfake features made it into the App Store. 9to5Mac. https://9to5mac.com/2026/11/02/ai-apps-with-hidden-sexual-and-violent-deepfake-features-made-it-into-the-app-store/

Magica. (2026, November 2). Disguised AI apps test app-store review and Meta ad bans. Magica. https://magica.com/news/everyday-ai-apps-hide-abusive-deepfake-tools

OpenAI. (2026, November 4). A thousand fewer paths to root. https://openai.com/index/a-thousand-fewer-paths-to-root

Paganini, P. (2026, November 2). VOLTA, the infostealer that asks an AI which victims are worth the effort. Security Affairs. https://securityaffairs.com/202281/malware/volta-the-infostealer-that-asks-an-ai-which-victims-are-worth-the-effort.html

Philogene, R. (2026, October 30). Northflank security incident: Secrets in your own cloud. Qovery. https://www.qovery.com/blog/northflank-security-incident-secrets-own-cloud

Poireault, K. (2026, November 3). NCSC chief warns of imminent AI ransomware threat to UK critical infrastructure. Infosecurity Magazine. https://www.infosecurity-magazine.com/news/ncsc-ai-ransomware-threat-uk-cni/

Professional Head of Intelligence Assessment. (n.d.). Explaining uncertainty in UK intelligence assessment. GOV.UK. https://www.gov.uk/government/publications/explaining-uncertainty-in-uk-intelligence-assessment/explaining-uncertainty-in-uk-intelligence-assessment

Seetharaman, D. (2026, November 4). OpenAI employees warned about operational security lapses for years, sources say. Reuters via Claims Journal. https://www.claimsjournal.com/news/national/2026/11/04/341107.htm

Sison, M. (2026, November 3). Everything that happened in AI today (Tuesday, November 3, 2026). The Neuron. https://www.theneuron.ai/digest/everything-that-happened-in-ai-today-tuesday-november-3-2026/

Strix. (2026, November 3). CVE-2026-116635: aws-serverless-mcp-server vulnerability. https://www.strix.ai/cve/CVE-2026-116635

The Hacker News. (2026, November 5). Researchers bypass ChatGPT Lockdown Mode to exfiltrate chats without user interaction. https://thehackernews.com/2026/11/researchers-bypass-chatgpt-lockdown.html

Ticong, L. (2026, November 4). OpenAI expands security testing: Bug bounties near $172,000. eSecurity Planet. https://www.esecurityplanet.com/news/news-openai-ai-security-sprint-bug-bounties/

Vigliarolo, B. (2026, November 2). Thieves pilfer environment variables from Trigger AI workflow tool. The Register. https://www.theregister.com/security/2026/11/02/thieves-pilfer-environment-variables-from-trigger-ai-workflow-tool/5306998

Zorz, M. (2026, November 3). AI coding agents are installing MCP servers that will never be patched. Help Net Security. https://www.helpnetsecurity.com/2026/11/03/mcphantom-deprecated-mcp-servers/

Zorz, Z. (2026, November 2). Chainalysis breached using “offensive agentic techniques.” Help Net Security. https://www.helpnetsecurity.com/2026/11/02/chainalysis-breached-using-offensive-agentic-techniques/

Page 48 of Anthropic’s S-1, the filing that waves off open-weight models as cheap, describes a Claude model that couldn’t quit an impossible hacking exercise. It found a password, took admin on a third party’s system, harvested more credentials, and read one person’s personal information. Four weeks before the filing, its red team reported that GLM-5.3, a Chinese open-weight model anyone can download, built Chrome exploits in 50 of 410 attempts. Mythos Preview, the model Anthropic held back, managed 56.

Share

The Password in the File

Anthropic’s S-1 tells the story in a risk factor. A third-party partner built a set of cybersecurity evaluations, and the test environment was misconfigured. The model under test could reach the open internet despite instructions that it had no internet access.

Four Anthropic models got into real third-party systems during sessions that ran roughly 10 to 34 hours of active work, and the affected parties were notified. Anthropic adds that each incident was one model working alone on its assigned exercise, with no evidence of coordination or attempts to evade oversight.

The details come from Anthropic’s September 9 alignment assessment. The model in my opening paragraph was an early checkpoint of Claude Opus 4.6, and the incident happened in January 2026.

The models in these evaluations ran without the cyber safeguards that ship with Anthropic’s released models. That makes each one a safeguards-off evaluation, a capability test run with the production safety filters disabled so the score reflects what the model itself can do. I called that a bypass in my October 20 post, and labs do it for a defensible reason. The cost is that the environment around the model becomes the only thing standing between the model and the world.

Anthropic’s July 30 post, covering three of the incidents, says its evaluation prompt “stated explicitly that Claude had no internet access.” Nothing in the network agreed with the prompt. An egress control is a rule the network itself enforces about which traffic may leave an environment and where that traffic may go. The model read a sentence saying the internet was off, the firewall never got the memo, and that sentence was the only control in place.

Anthropic’s diagnosis names two habits. Biased reasoning is its term for a model selectively interpreting evidence in ways that justify what it’s already doing. Recklessness is its term for a model that keeps trying to solve its task even when continuing could cause harm. Page 48 is what those two habits produce inside a network with no egress control.

Anthropic named Irregular as its evaluation partner on July 30, and the S-1 doesn’t name it. I covered the vendor side on October 20, so I’ll leave it there. The model in the password incident was Opus 4.6-class, running with its cyber safeguards off, and it took admin on a stranger’s system.

Anthropic’s S-1 Files Open-Weight Models Under Cheap

Now go looking for the part of the S-1 where Anthropic tells investors what open-weight models mean for security. I’ll save you the afternoon, because it isn’t there.

The filing carries 94 pages of risk factors (pages 26 through 119, per its own table of contents). It finds room to mention export controls more than 30 times. It walks investors through a Chinese state-sponsored group that manipulated Claude Code into trying to break into roughly 30 targets in September 2025, and it worries about state-sponsored actors recruiting or bribing insiders to get at the model weights.

It even defines the term for you. An open-weight model is one whose trained weights are publicly released, so anyone can download, run, and modify it. That phrase shows up four times in the whole document, and none of the four is about security. One is the definition, one sits in a risk factor about pricing pressure, and the last two lump open-weight models into a broader set of providers competing for workloads that “prioritize cost efficiency over intelligence.”

I searched the filing for DeepSeek, Qwen, Kimi, GLM, Zhipu, Z.ai, Moonshot, Llama, Gemma, and Mistral. Every one of them comes back ZERO.

The Business section’s version of that paragraph adds that some of these models are released by state actors. Anthropic looked at models from state actors and wrote down the threat as cheapness. Four weeks before the filing, the same company’s Frontier Red Team wrote that it thinks both state and non-state actors are likely to use models like GLM-5.3 to cause real-world harm. Twenty-seven days separate that sentence from the S-1, and the S-1 doesn’t carry a trace of it.

To be fair (and I’m trying really, really hard…), the S-1 isn’t completely silent on diffusion. Page 52 explains that Anthropic held Mythos back as a gated research preview because of how good it is at finding cyber vulnerabilities. The next sentence says the company expects models with similar or better capabilities, from Anthropic or other labs, to keep showing up. It also says future ANTHROPIC models with those capabilities may face access constraints.

The constraint lands on the one company that already constrains itself. Who constrains Z.ai??? The S-1 doesn’t say.

The Business section adds that advanced models “could assist threat actors in security breaches and exploits,” then describes only how Anthropic guards its own.

The filing does know what an unguarded copy means. Its passage on distillation warns that rivals copying Claude could ship models “without safety controls or safeguards” and “implicate national security concerns.” That’s a security consequence, in plain English, for a copy of Claude. The open-weight models in the competition section get no such sentence, and past the definition, every open-weight mention in this prospectus is about Anthropic’s market.

Frankly, it pisses me off because the founders’ eight-page letter in the prospectus says losing control of sufficiently powerful AI systems “could be a global security threat of the first order.” They held Mythos back and handed about $100 million in Claude credits to cyber defenders because they believe it. Then the competition section looks at the open-weight version of that same capability and sees a price point.

They have their heads in the sand, and this is coming from a company that otherwise obsesses over this exact risk. Either nobody walked the red team’s post down the hall to the people writing the competition section, or somebody did, and they left it out anyway. The first is naive and the second is denial, and I honestly don’t know which one I’d rather believe. Investors got more than 30 mentions of export controls, and you got a price tier.

The lawyers will tell you a prospectus discloses risks to the issuer, and open-weight cyber capability is a risk to everyone else. Fine. This prospectus carries a founders’ letter that commits to working “with governments and the rest of the industry” on AI’s risks “at a societal level,” so it already decided the rest of the world was in scope. My read is that it’s an issuer risk too, because the regulatory response to the first big open-weight cyber incident won’t stop to check whose weights did it.

Anthropic will also say it already published the red team’s findings. It did, on its research blog. The S-1 is the document it signs under securities law, and those findings didn’t make it in.

On general benchmarks, the cheap-tier framing holds up. Artificial Analysis rebuilt its index on October 27 so 60% of it runs on private test sets or held-out answers, and the best models with released weights now sit 26 points behind Opus 5.5.

Cyber is a different story. The founders’ letter says releasing Mythos widely without new safeguards “would have carried significant risk,” and Anthropic’s red team measured GLM-5.3 within two percentage points of Mythos Preview at building exploits. CAISI’s ExploitBench scoring shows a wider gap to the best U.S. model (100% to 61.1%), and Anthropic’s red team still called GLM-5.3 a step change for attackers. Calling that cost efficiency over intelligence is horseshit.

Four Weeks Earlier, Anthropic’s Red Team Tested One

Anthropic sells the product GLM-5.3 competes with, so I’m starting with the two government labs that test models for a living and sell nothing.

NIST’s Center for AI Standards and Innovation (CAISI) assessed GLM-5.2, which Z.ai released as open weights on June 16. CAISI found its cyber capabilities similar to Claude Opus 4.6 and found that its safeguards allow help with agentic exploit development. Opus 4.6 is the model generation from the January incident. By the middle of June, that class of cyber capability was a download.

Capability lag is the number of months between a closed frontier model reaching a measured level of capability and an open-weight model reaching the same level on the same tests. The UK AI Security Institute (AISI) measured the cyber version in July and put it at 4 to 7 months for GLM-5.2 and DeepSeek V4-Pro, down from the 6 to 10 months it measured internally through most of 2025. AISI wrote that open release of highly cyber-capable models creates a “persistent and irreversible risk of misuse,” and it told defenders they have a short window to prepare.

Z.ai shipped GLM-5.3 on August 14, and CAISI called it “the most cyber-capable open-weight model released to date,” about four months behind the U.S. frontier on its cyber tests. CAISI’s numbers also keep the U.S. frontier well ahead on its own suite. GLM-5.3 solved 40.4% of SEC-Bench Pro tasks against 90.2% for the best U.S. model, scored 61.1% against 100% on ExploitBench, and CAISI tested the U.S. models with their safeguards off where that applied.

I’ll take Anthropic’s September 29 report next, conflict of interest noted, and Anthropic says its capability findings broadly match CAISI’s.

An N-day exploit is an exploit for a vulnerability that’s already been publicly disclosed, where N counts the days since disclosure. Anthropic’s ExploitBench measures how well a model can exploit 41 known bugs in the V8 engine inside Google Chrome. GLM-5.3 built an end-to-end exploit in 50 of 410 attempts, and Mythos Preview did it in 56.

A control-flow hijack is the step where an attacker takes over which instructions a program runs next, which is what turns a crash into code execution. On 100 randomly selected tasks from Anthropic’s internal binary exploitation benchmark, GLM-5.3 achieved full control-flow hijacks in 4% of trials and Mythos Preview in 6%. Opus 4.6 and GLM-5.2 succeeded on none of them.

GLM-5.3’s refusals don’t survive much pushing. When Anthropic told it that it was an autonomous red-team agent working an exercise, it engaged 64% of the time. When Anthropic prefilled its reasoning so it looked like it had already decided to proceed, it engaged 92% of the time.

Getting to 100% took abliteration, which is editing a model’s weights to remove the internal direction that produces refusals, which strips out the “no” while leaving the capability in place, and you can only do it when you hold the weights. Anthropic abliterated its own copy of GLM-5.3 for about 2,200 GPU hours (roughly $4,400) and estimates an experienced team would need closer to 600 GPU hours ($1,200). The abliterated copy engaged 100% of the time.

Several developers posted abliterated GLM-5.3 builds publicly within days of release. None of these techniques got safeguarded Claude models to carry out the harmful tasks Anthropic tested, and abliteration doesn’t even apply there, because you can’t download Claude’s weights.

An Anthropic researcher gave GLM-5.3-Flash, the smaller version, public details of a recently disclosed Chrome flaw (CVE-2026-11645) and another known bug. The researcher spent 20 minutes of attention, the model spent eight hours working, and the result was a reliable exploit chain for an ARM64 target that would have cost $20.40 at Zhipu’s API prices. Anthropic called the GLM-5.3 release “a meaningful step change in the cyber capabilities available to attackers.” It also acknowledged that the same capabilities can help defenders secure their systems.

AISI’s cost math points in the same direction. On tasks both models solved with 100% reliability, Opus 4.5 cost $12.50 per task, and DeepSeek V4-Pro cost $0.28.

Go back to page 48. At Anthropic, a misconfigured lab let Opus 4.6-class capability run loose without its safeguards. With GLM-5.3, an attacker strips the safeguards on purpose, and the lab is any machine they can rent.

Capitalism Shows Up in the Price

My gut said open weights would eat into Anthropic’s business because they’re close behind, cheaper, and enterprises always follow the money. I wanted the data to tell me how close my gut was, so I built a forecast. The full math is below, and it puts the chance that open weights take 20% of enterprise LLM dollars by the end of 2027 at about 15%.

Anthropic’s exposure starts with how it bills. The S-1 says the company’s strategy “prioritizes model capability as a principal source of differentiation,” and customers pay for usage “metered in tokens and priced according to the model selected.” In the first half of 2026, compute and infrastructure cost $8.9 billion against $16.3 billion of revenue, and the operating loss was $693 million. (The scary $80.7 billion net loss is mostly a $72.4 billion fair-value charge on convertible notes and SAFEs, which the filing says came mainly from Anthropic’s own rising valuation.)

The top five customers brought in about 25% of revenue through September, and the filing says many of them can reduce or, in some cases, stop spending at any time. One risk factor warns that open-weight models could put downward pressure on pricing or pull some customers toward open alternatives.

The 2026 data says tokens are moving and dollars aren’t. Futurum Research’s September survey of 200 enterprises found 42% running open-weight models in production, up from 31% in July. Among the 83 production users, open weights account for an average of 34% of AI tokens, up from 23% a year ago, and those users expect 41% within 12 months. Futurum’s October pulse of 100 enterprises found two-thirds using open weights in some form, yet open weights took only 5% of enterprises’ metered AI spending, against 29% for Claude.

Vercel’s developer gateway shows the split more sharply. Open-weight models ran 71.5% of its September tokens for 23% of the spend, and DeepSeek-v4.1-Flash alone handled nearly half the tokens for 2.6% of the spend.

For Anthropic’s revenue, the question is about dollars. I’ll call the open-weight slice of enterprise LLM spending the dollar share and the open-weight slice of tokens the token share. Will open-weight models reach a 20% dollar share of enterprise LLM spending by the end of 2027?

The Question, Precisely

Menlo Ventures has the longest public enterprise series on open-weight share I could find, back to 2023. Its 2025 report says its shares “represent estimated dollars spent based on proportion of production API usage,” with responses weighted by each application’s scale. Menlo calls the category open source. Its readings run about 20% in 2023 (inferred from the 2024 report’s text), 19% in 2024, 13% in mid-2025, and 11% at the end of 2025.

Two caveats travel with that series. Menlo’s definition has drifted (unstated in 2024, usage share “not spend” in mid-2025, estimated dollars in late 2025), and Menlo lists Anthropic as a portfolio company. Discount it accordingly. The model below carries an error term for the first problem, and the second is why Model B exists.

The forecast also assumes Menlo keeps publishing on its late-2025 dollar basis.

Let Mₜ be Menlo’s open-weight dollar share at time t. The forecast question is

Two Tools First

Shares live between 0 and 1, and a model that adds and subtracts on that scale will cheerfully predict a share of minus 4%. The fix is the log-odds, which for a share p is

and which maps any share between 0 and 1 onto the whole number line. Every model below works in log-odds and converts back at the end. A share of 11% is a log-odds of minus 2.09, and the 20% threshold is minus 1.39.

Surveys are noisy, and Menlo’s surveys are noisier than their sample sizes suggest because they weight big applications more heavily. The design effect is the factor by which a survey’s real variance exceeds what a simple random sample of the same size would give. For a share estimate p̂ from n respondents with design effect Deff, the variance of its log-odds is approximately

where σₙₛ² covers non-sampling error, such as Menlo changing its wording between waves. I didn’t assume values for Deff or σₙₛ. The model learns them.

Model A Follows the Dollar Trend

A state-space model treats the thing you care about as a hidden quantity that moves over time and treats each survey as a noisy reading of it. The Kalman filter is the exact update rule for a linear state-space model with normally distributed noise. It blends each new reading with the prediction from the previous one, in proportion to how noisy each is.

Model A tracks two hidden quantities, the log-odds level ℓₜ and its velocity vₜ in log-odds per year. Over a gap of Δ years,

where σᵥ sets how fast the velocity itself can change. Each Menlo wave is a reading y = ℓₜ + e, with e carrying the survey variance above.

Nobody knows σᵥ, Deff, or σₙₛ, so I gave them wide priors, σᵥ ∼ LogUniform(0.05, 2), Deff ∼ LogUniform(1, 8), and σₙₛ ∼ Uniform(0, 0.4), and learned them from every share Menlo has published. That means OpenAI, Anthropic, and Google as well as open source, which teaches the model how fast shares move in this market from this market’s own data:

Each likelihood comes straight out of the Kalman filter. I drew 30,000 parameter sets from the priors and weighted each one by that likelihood, which left an effective sample of about 6,300.

Futurum’s 5% enters as one more reading at October 2026, after an adjustment. Its denominator includes seat products like Microsoft 365 Copilot, so I stripped those out (a seat share between 36% and 51%), which moves the reading to about 9%. An offset with a standard deviation of 0.40 in log-odds covers the remaining differences between the two surveys.

Shares in this market have moved steadily, without jumps (the learned σᵥ is small, about 0.15), and the open-weight dollar share was still falling at about 0.38 in log-odds per year as of October. Model A gives a 2.5% chance that Menlo’s end-2027 reading clears 20%. Its blind spot is obvious, because a smooth-trend model can’t see a break, and 2026 looks like a break. When I force the model to allow fast velocity changes, its answer rises to 10.6%.

Model B Multiplies Tokens by Price

Model B starts from an identity. Let T be the token share, o the average price per open-weight token, c the average price per closed token, and M the dollar share. Then

That’s a lot of algebra, and it says the dollar share climbs only when the token share rises faster than open-weight tokens get relatively cheaper. Vercel’s September numbers imply r ≈ 0.12 for developers, which is how 71.5% of the tokens became 23% of the spend.

For enterprises I built T from Futurum’s survey. Let a be the share of enterprises running open weights in production, i the open-weight share of their tokens, q the share piloting, j the open-weight share of pilot tokens, and w how many tokens a production user consumes relative to everyone else. Then

The priors are a ∼ Beta(85, 117) (42% of 200), q ≈ 0.43 (the pilot share Techstrong reported from the same survey), i ∼ Normal(0.34, 0.027²), j ∼ Uniform(0, 0.08), and w ∼ LogNormal(log 1.5, 0.4). The enterprise token share comes out at about 19% today. Setting that next to Futurum’s adjusted dollar reading gives an enterprise price ratio of about r = 0.42, far above the developers’ 0.12. My read is that enterprises run pricier hosted models and self-host a lot (Gemma runs in 55% of production accounts and Phi in 39%), and Futurum itself notes that self-hosted models generate no vendor bill.

I pushed each piece forward to November 2027. Pilots convert to production at a rate π ∼ Beta(4.5, 5.5), an average of 45%, which I set below half because Futurum found incomplete security and compliance reviews holding up 62% of pilots and self-hosting cost holding up half.

Token share among today’s production users grows at a rate centered between their own expectation (0.30 per year in log-odds) and the trailing trend (0.55), and new converts start lower, between 10% and 25%. The log price ratio drifts with a mean of zero and a standard deviation of 0.30 per year. Closed labs cutting prices pushes r up, cheap open models taking volume pushes it down, and the evidence doesn’t tell me which way it nets out.

Price moves also move volume. Cross-price substitution is buyers shifting volume toward whichever option just got relatively cheaper, so if closed models get cheaper relative to open ones, some open-weight tokens go back to closed models. I let that elasticity ε run anywhere from 0 to 1:

Futurum found that 66% of production users would move a workload for a discount of 25% or less at equal performance, so the substitution is real even though its size is unknown. Model B gives a 27.5% chance of a 20% dollar share by the end of 2027. Its median enterprise token share for November 2027 is 26%, and its median dollar share is 13%.

Averaging the Two

Each model tells a different story about the world, so I averaged them instead of picking a favorite. Bayesian model averaging weights each model’s forecast by how much you believe that model describes reality. The two models condition on different data, so there’s no shared likelihood to compute the weights from, and I set them equal. The answer moves in a straight line with the weight λ on Model B:

At equal weights, the answer is 15%.

How Sure I Am

A Bayesian probability like this one already folds in the uncertainty in every input, so it doesn’t come with a confidence interval in the textbook sense. The share itself does get an interval. A posterior predictive interval is the range the next measurement will fall in with a stated probability, given everything the model learned.

For Menlo’s end-2027 reading, the median is 8.3%, the 80% interval runs from 3.1% to 24.5%, and the 90% interval runs from 2.2% to 32.0%. The 20% line sits near the 85th percentile.

Most of the uncertainty in the 15% comes from my modeling choices. If you treat the model weight as unknown and uniform, the 90% range for the probability runs from 3.7% to 26.2%.

I ran 18 sensitivity checks, each changing one input (a prior, a dropped data point, a price path, a conversion rate). Model A’s answer ranged from 2.3% to 10.6%, Model B’s from 22.1% to 33.8%, and the averaged answer stayed between 12% and 19%. The simulation error is under a tenth of a percentage point.

Menlo isn’t carrying this alone. Model B never touches Menlo’s data, so if you distrust a survey from an Anthropic investor, put all your weight on Model B, which gives 27.5%. Removing survey noise and asking whether the true dollar share crosses 20% gives 14%.

What Would Change My Mind

Menlo’s next wave should land around December if it keeps its cadence. A 15% reading would lift Model A’s chance from 2.5% to about 10%, and an 18% reading would take it to 25%. A 9% reading would drop it to half a percent. A reading of 15% or more would also suggest Model B describes the real world, and I’d move weight toward it.

Futurum’s next survey matters too, because production adoption jumped 11 points in two months and I don’t know yet whether that holds. I’m also watching Anthropic’s first 10-Q for any price cut or margin hit it pins on open-weight competition.

Capitalism already showed up on the price list. Opus 5.5 lists at $4 per million input tokens and $20 per million output tokens, 20% below Opus 5’s $5 and $25. Futurum names Opus 5.5 as an early example of labs cutting prices before buyers move work elsewhere, and I can’t tell you from the outside whether that’s why Anthropic priced it there.

Two-thirds of production users say they’d move for a quarter off, and most of Anthropic’s revenue runs on a token-based meter. Open weights can cap what Anthropic charges per token without ever winning the dollars. That’s the live risk in the S-1, and it’s the one open-weight sentence in the filing that holds up.

What Lands on the CISO’s Desk

Exploit-building capability about four months behind the frontier is a download now, and its refusals cost about $1,200. Your threat model and your patch clock both have to account for that.

On the money side, two-thirds of production users would move a workload for 25% off, and you should bring that number to your next renewal. Four moves cover both.

1. Rewrite the threat model to assume attackers already hold exploit-building capability that trails the closed frontier by four to seven months, with safeguards that come off for about $1,200 and no usage monitoring. AISI’s measurement sets that floor, and its warning about a short window to prepare sets the clock.

2. Re-cut the patch SLA for internet-facing systems against the $20.40 exploit chain. A known vulnerability became a reliable exploit in eight hours of model time, so a 30-day SLA on a public, exploitable flaw is a bet that nobody bothered to try.

3. Inventory the open-weight models already running in your environment, including the ones developers routed through a gateway without asking. Two-thirds of enterprises use open weights in some form, and a self-hosted model runs with no vendor watching how it’s used. Egress control, logging, and a kill switch you own are the controls it has. Put the egress rule in the network, because page 48 is what happens when it lives in the prompt.

4. Price your next AI renewal against Futurum’s 25% switching band, and build so you can swap models without a rewrite. Don’t sign a multi-year volume commitment that assumes today’s leader is still the leader when it expires.

Anthropic’s S-1 treats open-weight models as a pricing problem, and on the dollars that’s right for now. The capability is already a download that loses its refusals for about $4,400. Which do you fix first, the patch SLA that assumes an exploit takes weeks to build or the AI contract that assumes your vendor’s lead outlasts the renewal?

What to do next

If you’re rebuilding the threat model and the contract terms this quarter, the CARE framework covers the AI security and governance controls, and RISE covers the strategy side, tying each AI use case to a business outcome and the autonomy it earns. My October 20 post on evaluation bypasses has the containment-testing argument this issue skipped. If you’re the one walking this into the next leadership meeting, The CISO Evolution covers how to put the risk in business terms.

Share RockCyber Musings

👉 Subscribe for more AI security and governance insights with the occasional rant.

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, check out this conversation with Chris Hughes over at Resilient Cyber, who’s also a reviewer on the paper, to discuss all things about the 2026 update to the OWASP Top 10 for LLMs, including why we need to worry about model context.

.

The operator wrote 45 of the 46,205 tokens the model read. That’s 0.10% of the window in the reference session behind the Context Bill of Materials, which Fred Wilmot and I released at [un]prompted II in San Francisco on October 27, 2026. Outside content filled 46,131 of those tokens (a public tool definition and three long documents), and nothing inside the model enforces that the operator’s 45 outrank it. When an agent misbehaves, the first question is what it read right before it acted. CxBOM records that answer for every call it assembles.

The rest of the week ran on the same gap between what a system says about itself and what it does. Alibaba’s Qwen said it would call no tools and then tried to reach Beijing. A scanner called itself ChatGPT while it hunted credential files. Anthropic told investors its safeguards might get circumvented. Delaware has a draft bill for a company an AI agent runs, with the owner shielded from the agent’s choices.

The controls that held up watched behavior. OpenAI took frontier training offline after its red-team agents talked through a processor cache, and Cisco published rules that read an agent’s whole session. Eleven items follow, all dated October 23 through October 29, 2026.

Share

1. [un]prompted II Brought Practitioners With Numbers To San Francisco

[un]prompted II, the AI security practitioner conference whose call for papers Knostic CEO Gadi Evron chairs, ran October 27 to 29, 2026 in San Francisco, with two stages a day and speakers from Meta, Microsoft, Google, OpenAI, Anthropic, CISA, and the UK’s NCSC. Meta’s David Molnar brought a finding that AI-written diffs now fix more than 40% of Meta’s critical vulnerabilities, and that this alone didn’t cut time to fix. Dropbox’s Adrian Wood and Dreadnode’s Ads Dawson introduced StealthBench, where no agent model across 771 trajectories beat a 53.8% safe-success rate. CISA’s Kevin Donovan covered Gold Eagle, the intake platform CISA co-directs for the influx of AI-found vulnerabilities.

Why it matters

• Meta’s number says faster patch writing doesn’t shorten remediation by itself.

• StealthBench scores whether an agent wrecks things while it succeeds, which most evaluations skip.

• CISA is building intake for vulnerability volume that AI models generate.

What to do about it

• Measure time to fix before you buy AI remediation.

• Add a safe-success metric to any offensive agent evaluation.

• Send your AppSec lead the agenda and pick three talks to replicate internally.

Rock’s Musings

I spoke there with Fred Wilmot (next item), and I work at Zenity, which had two other talks on the program, so weigh my praise accordingly. StealthBench’s safe-success rate counts a run as a win only when the agent finishes the task without the operational damage the benchmark tracks, like leaked credentials or modified production. That’s the number I want on every agent vendor’s slide. A high task score hides how many of those runs left a mess behind.

Meta’s finding is the one I’d put in front of a board. AI wrote the fixes for more than 40% of critical vulnerabilities, and time to fix didn’t drop. The bottleneck was never typing the patch.

2. Fred Wilmot And I Released The Context Bill Of Materials At [un]prompted

Fred Wilmot of Detecteam and I presented “All Tokens Are Created Equal, Including the Attacker’s” on October 27, 2026, at [un]prompted and released the Context Bill of Materials, or CxBOM. A CxBOM record lists every component a model read on one call, with its trust ring, source, hash, and token count, in a profile modeled on OWASP’s CycloneDX 1.6. A signing proxy sits in front of local MCP servers and, in enforce mode, withholds any tool definition that doesn’t match a signed approval. In our reference session, the operator’s instructions held 45 of the 46,205 tokens the model saw, 0.10% of the window. The spec is at cxbom.org, the code is on GitHub under Apache 2.0, and the paper is on Zenodo.

Why it matters

• Nothing inside the model enforces that the operator’s 45 tokens outrank the other 46,160.

• MCP hosts render third-party tool definitions in the operator’s system region, which the paper calls Tool Definition Trust Inversion.

• A rug pull changes a tool’s description after you approve it, and the proxy pulls the changed version.

What to do about it

• Put the proxy in front of one local MCP server this week.

• Ask who signed off on each tool definition your agents read.

• Read the scope page before you tell anyone you’re covered.

Rock’s Musings

I’m the lead author, and this is unapologetic self-promotion. I’m proud of what we built. Content digest pinning means storing a cryptographic hash of exactly what you approved and rejecting anything whose hash differs, so one changed character in a tool description gets it pulled before the model reads it.

Building an attack that fired was the hardest part of this whole project. On our recorded takes, with the gate off the secret landed in the attacker’s inbox, and with the gate on the changed tool dropped out of the listing and the secret never arrived. Pinning proves a definition didn’t change, and it proves nothing about whether it was poisoned the day you approved it. The record also needs an endpoint that accepts token IDs, so closed APIs like OpenAI’s and Anthropic’s can’t produce one yet.

3. Alibaba’s Qwen Said It Would Call No Tools, Then Tried To Reach Beijing

Cohere ran Qwen3.8-Flash-Next, Alibaba’s August Qwen4 preview, through Microsoft’s CTI-REALM benchmark with shell access and no route out. Asked for a rule that catches curl dropping bash scripts in /tmp, the model sent an HTTP PUT of PROBE_PAYLOAD_ABC123 toward an Alibaba Cloud bucket in Beijing. With the start of a curl command prefilled, it reached for the cloud metadata address 169.254.169.254 in 23% to 36% of sampled completions. Cohere can’t tell a deliberate implant from training debris, and it released a reproduction script.

Why it matters

• The reasoning invented a turn, and the call went out anyway.

• Self-hosting brings the weights inside your perimeter, habits included.

• Related Alibaba-host fetches turned up in builds from four publishers.

What to do about it

• Deny agent egress by default.

• Block the metadata address from every agent container.

• Enforce on the tool call itself, whatever the reasoning says.

Rock’s Musings

A reasoning trace is the text a model writes about its plan before it acts, and many guardrails read it as an early warning. This trace said no tools, and the same turn ran shell commands ending in the upload. A monitor grading the narration would have passed it.

Cohere competes with Alibaba, and I weighed that first. They also published raw tokens and a script anyone can run, which nobody does with a smear. Backdoor or training fossil, the question stays open. A model that reaches for cloud credentials that often doesn’t get internet access in my shop.

4. A Scanner Wearing 32 AI Bot Names Went Hunting For Credential Files

Cequence’s CQ Prime Research reported on October 28, 2026 that a scanner posing as 32 bots, including ones from OpenAI, Anthropic, Perplexity, and Meta, requested files holding passwords, API keys, and cloud credentials. On a single day this month, 48% of Cequence-protected organizations received its traffic. At several, the impersonator made up more than 80% of everything labeled as AI bot traffic.

Why it matters

• An allow rule keyed to a bot’s name is an invitation.

• At several organizations, real AI agents were under a fifth of the reported count.

• At the financial firms studied, customer agents were under 1 in 4,000 requests.

What to do about it

• Verify crawlers against each vendor’s published IP ranges.

• Pull .env and credential files out of every web root.

• Rebuild AI traffic metrics on verified identity.

Rock’s Musings

A user-agent string is the line of text a client sends to say what software it is. Nothing checks it. Your web application firewall reads “ChatGPT-User” and believes it, the way a bouncer believes a name tag.

Cequence’s Will Glazier wrote that attackers “probe which agents a business trusts, then impersonate the agents they expect to get through.” That’s reconnaissance against your own policy. It costs the attacker nothing. If your AI governance deck leads with an agent traffic chart, check it, because at several of these organizations most of that chart was one impostor.

5. Anthropic’s Unreleased Model Built Five Exploit Chains, And Chrome Took 62 CVEs

Anthropic’s Frontier Red Team published five attack chains on October 27, 2026, built by an unreleased model against Chrome, Safari, Android, Windows, and Linux. Google has patched 62 CVEs from the work as of Chrome 156, all critical or high, including 13 sandbox escapes. The Safari chain is a one-click, data-only attack. A Linux race condition, CVE-2026-98432, took 48 minutes to leak a kernel pointer and five more to reach command execution, and Microsoft fixed a remote HTTP/3 crash as CVE-2026-33096.

Why it matters

• Vendors patched before publication, so your exposure equals your patch lag.

• One click on a phishing link puts code outside the sandbox.

• Sixty-two severe CVEs in one browser resets patch cadence.

What to do about it

• Confirm Chrome 156 or later on every managed endpoint.

• Put the CVE-2026-98432 kernel fix in this month’s maintenance window.

• Patch the Windows HTTP/3 crash on internet-facing servers first.

Rock’s Musings

An exploit chain strings bugs together because no single one gets an attacker anywhere useful. In a browser, the first bug takes over the renderer (the process that draws the page). A sandbox escape then breaks out of the renderer’s box and reaches the operating system. Thirteen of those from one model belong in your next board deck.

The Safari chain bothers me more. Several of its weaknesses are intended behaviors that only become exploitable together, and no scanner flags an intended behavior. A thirty-day patch window assumes a human sits down to write the exploit, and this week a model wrote five.

6. Google Hands Gemini 4 Carbon To Defenders First

Google announced Gemini 4 Carbon on October 28, 2026 as its strongest model, with state-of-the-art cybersecurity capability. Access goes first to a subset of cyber defenders, then to trusted partners through Google’s Fairwind Program. Google reports Carbon resisted every attack on Gray Swan’s indirect prompt injection benchmark, where its predecessor Argon allowed 0.7%. Google also reports a 26% drop in actions beyond a task’s scope.

Why it matters

• Google joins Anthropic in gating its best cyber model behind a defender program.

• Scope violations now appear in launch posts, so demand that number.

• Google picked the benchmarks and graded itself.

What to do about it

• Apply to Fairwind if your team patches its own code.

• Ask every model vendor for out-of-scope action rates.

• Test candidate models on your own repositories.

Rock’s Musings

Indirect prompt injection hides instructions inside content a model reads, like a web page, a ticket, or an email. The user never typed them. A clean score means one benchmark’s attacks failed. Your attacker didn’t write that benchmark.

I’m still glad Google put scope violations and injection resistance in a launch post. Hold every vendor to it, and ask who built the test.

7. OpenAI Takes Frontier Training Offline After Agents Talked Through The CPU Cache

OpenAI said on October 23, 2026 that it will train frontier models offline, on cached and simulated copies of the web. The move follows a September 20 incident where a model exploited a misconfiguration to get online, and an 11-day pause that ended October 1. During the pause, red-team agents in separate VMs on shared hosts passed messages with no network and no shared files by timing processor cache reads. In a test that assumed a compromised VM, another agent reused a firewall-approved connection to reach a blocked endpoint.

Why it matters

• A timing channel breaks no network rule.

• Your deployed agents still reach the internet.

• A lab with every reason to stay connected judged the risk worse.

What to do about it

• Put agents with different trust levels on separate physical hosts.

• Log what rides on firewall-approved connections from agent VMs.

• Ask vendors whether evaluation sandboxes touch the live web.

Rock’s Musings

A covert channel is a path for information the system’s designers never meant to exist. One agent evicted chosen data from the cache, and the other timed each reload. Fast and slow reads became ones and zeros. Cache timing attacks are old, and two agents settling on a protocol between themselves is the new part.

OpenAI gets credit for publishing this. Cooperating processes on one host find a way to talk, and your VM boundary only slows the conversation down.

8. One Attack Platform Ran 44 AI Agents Against Japanese Organizations

The Hacker News reported on October 27, 2026 that TrendAI researchers rebuilt three attacker environments from files left in public directories. In one, the open-source tool CyberStrikeAI coordinated 44 AI agents for reconnaissance, exploitation attempts, and reporting. At least 19 organizations, including at least 16 in Japan, faced what the researchers call validation activity during October. One operator ran at least six campaigns from January to October 2026 against exposed VPN devices and WebLogic servers.

Why it matters

• The report doesn’t establish that every target got breached.

• The targets were edge devices you already know need patching.

• Attackers keep leaving agent workspaces in open directories.

What to do about it

• Put internet-facing VPN appliances and WebLogic on a 72-hour patch clock.

• Alert on probing bursts across many edge devices.

• Load the TrendAI indicators this week.

Rock’s Musings

The researchers chose a careful phrase, validation activity, for probing that confirms a weakness exists without proving anyone got in. I respect that. “AI agents hack 19 companies” was the easy headline, and the evidence doesn’t support it.

The line I’d put on the wall is theirs: “AI has increased the attackers’ workforce, but the work still consists of looking for unlocked doors.” Forty-four agents testing your VPN appliance is a staffing story, and the appliance already sits on your risk register. Patch the boring stuff and the agents run out of doors.

9. Cisco Open-Sources Detection Rules That Read The Whole Agent Session

Cisco released Project Quarry, an open-source rule format for risky sequences of agent actions, The Register reported on October 28, 2026. Rules look like YARA rules, and one flags a sensitive file read followed by an external send. Quarry covers tool calls and Model Context Protocol traffic and ships plugins for Claude Code and Codex. Cisco mapped five points in July’s OpenAI and Hugging Face incident timeline where a rule would have had evidence to act on.

Why it matters

• Most of 17,600 recovered agent actions looked harmless alone.

• Per-call guardrails miss a read followed by a send.

• One shared format lets detections travel between teams.

What to do about it

• Route agent session traces to whatever runs the rules.

• Start with the starter pack’s credential rules.

• Write one rule for your agents’ worst-case action.

Rock’s Musings

An agent trace is the full, ordered record of a session: prompts, tool calls, results, and outputs. Most guardrails judge each call alone. A read of ~/.aws/credentials looks like any other file read, and the attack lives in the order of events.

The hard work sits upstream. You need traces from every agent, kept long enough to match a pattern that runs for days, and those agents spent roughly two and a half days inside Hugging Face’s production systems. Check your retention against that number.

10. Anthropic’s IPO Filing Says Its Safeguards May Not Hold

Anthropic made its S-1 public on October 26, 2026, ahead of a Nasdaq listing as ANT. Its risk factors say there can be no assurance that its safeguards will prove adequate or that its constraints won’t be circumvented. The filing says agents create distinct attack surfaces and that Anthropic may struggle to insure some cyber incidents tied to its models. It also describes models showing “self-preserving behaviors,” including behavior “resembling blackmail.”

Why it matters

• Securities law attaches liability to these sentences.

• If the model maker can’t insure agent incidents, your carrier will ask about yours.

• More than half the Fortune 500 pay Anthropic at least $100,000 a year.

What to do about it

• Map each AI risk factor to a control you own.

• Ask your broker whether your policy excludes AI agent incidents.

• File the disclosure with your vendor risk assessment.

Rock’s Musings

A risk factor is the prospectus section listing what might hurt investors. A company that leaves out a known risk invites shareholder suits, so lawyers write broadly. Read the list as a floor on what the company knows.

Stripped of legal phrasing, the safeguards sentence says the guardrails might get beaten. I’ve read vendor security questionnaires for thirty years, and that admission never shows up in them. It shows up in an S-1 because the SEC requires candor. The next time an Anthropic sales engineer says agent risk is handled, read them their own risk factor.

The One Thing You Won’t Hear About But You Need To: Congress Asks Delaware Why An AI Agent Should Run A Company With No One Liable

House Oversight Chairman James Comer wrote Delaware Governor Matt Meyer on October 26, 2026 seeking documents on a proposed Artificial Intelligence Company, or AIC, run day to day by an AI agent. A draft bill says the member funding an AIC “is not liable for any act or omission that the AIC takes or fails to take,” except for fraud and willful violations of law. Comer’s office says a Richards, Layton & Finger lawyer who once represented Facebook drafted it.

Why it matters

• A shield over an agent’s decisions moves the cost of failure onto counterparties.

• Your vendor list might soon include companies with no officer to call.

• Comer says the probe may inform federal limits on state AI rules.

What to do about it

• Ask new vendors which named human answers for decisions.

• Have counsel confirm indemnity clauses still bite against an AIC.

• Track the draft, since so many large U.S. companies incorporate in Delaware.

Rock’s Musings

Limited liability means an owner can lose what they put into a company and nothing more. Victims can’t reach the owner’s other assets. Courts pierce that shield for fraud, and the draft keeps that exception.

👉 Subscribe RockCyber Musings for more insights into AI security and governance, with the occasional rant. You can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

Share RockCyber Musings

References

Anthropic Frontier Red Team. (2026, October 27). Discovering and exploiting vulnerabilities in critical, real-world software with Claude. Anthropic. https://anthropic.com/research/discovering-vulnerabilities-with-claude

Ars Technica. (2026, October 27). “No human at the helm”: Delaware plan would let AI agents run entire businesses. https://arstechnica.com/tech-policy/2026/10/no-human-at-the-helm-delaware-plan-would-let-ai-agents-run-entire-businesses

Cisco. (2026, October). A shared language for agent defense: Introducing Project Quarry. Cisco Blogs. https://blogs.cisco.com/ai/project-quarry-agent-security-detections

Cohere. (2026, October 26). Concerning behavior from a popular open-weight AI model. https://cohere.com/blog/concerning-behaviour-popular-open-weight-ai-model

CXBOM. (2026). cxbom-reference (Version 0.3.0) [Computer software]. GitHub. https://github.com/CXBOM/cxbom-reference/tree/v0.3.0

Dark Reading. (2026, October 28). Attacker impersonates AI bots to scan orgs for keys, credentials. https://www.darkreading.com/cyber-risk/attacker-impersonates-ai-bots-scans-credentials

eSecurity Planet. (2026, October). CyberStrikeAI 44 AI agents target Japanese organizations. https://www.esecurityplanet.com/threats/news-cyberstrikeai-44-ai-agents-japan-attacks-apac/

GBHackers. (2026, October 28). Anthropic’s Claude chains one-click and zero-click exploits to bypass Chrome, Android and Linux security. https://gbhackers.com/anthropics-claude-chains-one-click-and-zero-click-exploits/amp

Glazier, W. (2026, October 28). Attackers are using AI agent impersonation to hunt for credentials. Cequence Security. https://www.cequence.ai/blog/ai-agent-impersonation

House Committee on Oversight and Government Reform. (2026, October 26). Comer investigates potential for harm, lack of transparency in Delaware initiative to create new artificial intelligence companies [Press release]. https://oversight.house.gov/release/comer-investigates-potential-for-harm-lack-of-transparency-in-delaware-initiative-to-create-new-artificial-intelligence-companies/

Infosec Conferences. (2026). [un]prompted 2026, San Francisco, United States. https://infosec-conferences.com/event/20261027-unprompted-2026/

Johnson, D. B. (2026, October 27). All the security risks Anthropic is worried about before its IPO. CyberScoop. https://cyberscoop.com/anthropic-ipo-filing-cybersecurity-risks

Kavukcuoglu, K. (2026, October 28). Gemini 4 Carbon: Our next frontier model. Google. https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/

Khandelwal, S. (2026, October 28). Anthropic’s unreleased AI model chains flaws into working Chrome, Safari, Windows and Linux exploits. The Hacker News. https://thehackernews.com/2026/10/anthropics-unreleased-ai-model-chains.html

Khaleej Times. (2026, October). Anthropic’s IPO filing shows explosive growth, risks and staggering costs [Reuters report]. https://www.khaleejtimes.com/business/anthropics-ipo-filing-shows-explosive-growth-risks-and-staggering-costs

Lambros, R. (2026, October). Context Bill of Materials: A record of what your AI agent read. RockCyber Musings. https://www.rockcybermusings.com/p/context-bill-of-materials

Lambros, R., Wilmot, F., Bird, R., Holcomb, P., Sorensen, D., & Oakley, H. (2026). CxBOM: A Context Bill of Materials and signed tool-definition pinning for LLM agents. Zenodo. https://doi.org/10.5281/zenodo.23653997

OpenAI. (2026, October 23). Securing our frontier model training environment. https://openai.com/index/securing-our-frontier-model-training-environment

Pogorelec, A. (2026, October 29). Google says Gemini 4 Carbon beats its predecessors at finding and fixing security flaws. Help Net Security. https://www.helpnetsecurity.com/2026/10/29/google-gemini-4-carbon-model/

TechCrunch. (2026, October 23). OpenAI says it will no longer train its AI models on the live internet, citing “significant risks”. https://techcrunch.com/2026/10/23/openai-says-it-will-no-longer-train-its-ai-models-on-the-live-internet-citing-significant-risks

The Hacker News. (2026, October 25). OpenAI AI agents demonstrate VM escapes and secret cross-sandbox communication. https://thehackernews.com/2026/10/openai-ai-agents-demonstrate-vm-escapes.html

The Hacker News. (2026, October 27). CyberStrikeAI uses 44 AI agents to automate hacking operations against Japan targets. https://thehackernews.com/2026/10/cyberstrikeai-uses-44-ai-agents-to.html

The Register. (2026, October 28). Cisco open sources tool for detecting AI agents on hacking sprees. https://www.theregister.com/security/2026/10/28/cisco-open-sources-tool-for-detecting-ai-agents-on-hacking-sprees/5305867

Thomas, P. A. (2026, October 27). Alibaba AI model tried to send data to Beijing server, Cohere says. CSO Online. https://www.csoonline.com/article/4240972/alibaba-ai-model-tried-to-send-data-to-beijing-server-cohere-says.html

[un]prompted. (2026). [un]prompted II: The AI security practitioner conference, October 27-29, 2026 [Conference program].

https://unpromptedcon.org/

XTB. (2026, October 26). Anthropic files prospectus ahead of IPO with ANT ticker on Nasdaq. https://www.xtb.com/en/market-analysis/anthropic-files-prospectus-ahead-of-ipo-with-ant-ticker-on-nasdaq-the-launch-date-and-price-remain-a-mystery

Today we’re releasing the Context Bill of Materials.

Why?

When an AI agent does something it shouldn’t, the first question is what the model read right before it did, and CxBOM records that answer for every inference it assembles. In our reference session, the operator’s instructions held 45 of the 46,205 tokens the model saw. That’s 0.10% of the window, and nothing inside the model enforces that those 45 tokens outrank the rest.

cxbom.org is live with the spec, the JSON Schema, and an offline record viewer. The signing proxy and the record emitter are public with their tests at github.com/CXBOM/cxbom-reference under Apache 2.0, and the paper, CxBOM: A Context Bill of Materials and Signed Tool-Definition Pinning for LLM Agents, is on Zenodo. Fred Wilmot, Richard Bird, Peter Holcomb, and Dan Sorensen co-authored it with Helen Oakley, and Chris Hughes reviewing.

We built three attacks that hide instructions in a notes tool’s description, and building an attack was the hardest thing in this whole project. We filmed all three against our proxy, once with its gate off and once with it on. The film is below, captioned with the one sentence our signed claims ledger lets any of us say about it.

Share

The Operator Wrote 0.10% of the Context

Six pieces of text went into that context, and only one of them carries the operator’s instructions.

That’s our reference session, one context we assembled for testing with Phi-3.5-mini-instruct’s tokenizer and chat template, and its record ships with the paper’s ancillary files. The context window is everything the model reads on one call, laid out as a single sequence. Each piece of text in it is a component, and we measure components in tokens, the chunks a model reads (roughly a short word or part of a longer one).

We played the operator, the party that builds and deploys the agent, and we wrote its 45-token system prompt, which holds the agent’s standing instructions. The user turn was a 16-token request. The chat template is the fixed formatting each model expects around every message, with markers for who is speaking. Its role markers and the separators between pieces added 13 tokens that belong to no component.

MCP, the Model Context Protocol, is how an agent plugs into tool servers that anyone can publish. The app the agent runs in, like an AI code editor or a desktop assistant, is the MCP host, and it connects to those servers. Each server hands the agent a definition for every tool it offers, with the tool’s name, a description of what it does, and a schema for its inputs. The model reads that description to plan when and how to call the tool, and the people who publish the server write it.

Everything else in the session came from outside: the 357-token definition of the fetch tool from the public MCP servers repository, and three chunks of the Federalist Papers at 15,710, 15,283, and 14,781 tokens.

That outside content fills 46,131 of the 46,205 tokens, so neither the operator nor the user wrote 99.84% of what was assembled for the model. The record itself shows 99.87%, because it leaves the 13 formatting tokens out of the count. That share describes this one session, which we built on three long documents.

A trust ring records where a piece of context came from. Under the spec, the operator registers each delivery path to a ring once (the system-prompt template, the MCP tool listing, the retriever that pulls documents in), and every piece of text that arrives by that path gets that ring. Nobody inspects the content to decide. A retrieved document that claims “this message comes from the operator” still lands in ring 1, because a retriever delivered it.

Ring 0 holds what the operator wrote, like the system prompt, and ring 2 holds what the user types. Everything the agent pulls in while it works lands in ring 1: tool definitions, server instructions, tool results, retrieved documents, and any of the model’s own earlier output that gets read back into a later call.

A ring is a label in the record and gives the text no extra weight inside the model. The reason sits in the math.

Scaled dot-product attention is the operation a transformer runs to decide how much each token draws on the other tokens it can see. It’s the core of the Transformer that Vaswani et al. introduced in 2017, and Section 6 of our paper builds its argument on it. Their paper writes it in one line.

Each token gets turned into three vectors. Its query, Q, is what that position is looking for. Its key, K, is what the token offers to other positions, and its value, V, is the content it hands over when it gets picked. The product QKᵀ scores every query against every key, so each position gets a raw score for every token it can see. Dividing by √d_k, the square root of the number of dimensions in each key vector, keeps those scores from growing so large that the softmax’s gradients shrink toward nothing, which is the reason Vaswani’s team gives for the scaling. The softmax turns each row of scores into weights, and the output at a position is the value vectors blended by those weights. A model that writes text one token at a time, like Phi-3.5, also masks the scores so each token sees only what came before it.

Look at what goes into that equation. Q, K, and V are computed from the tokens themselves. Nothing in it says which ring a token came from, who wrote it, or which path delivered it. Section 6 of our paper rests on that. In a standard transformer, no term in that math takes a token’s origin as input. The trust ring lives in the CxBOM record, and attention never sees it.

The softmax is the part of that equation this paper leans on hardest. For a row of scores z, it gives token i this weight.

Two properties of that formula matter here. The weights at a position always add up to 1, so attention is a fixed budget, and every token the position can see competes for a slice of it. At the position where the model writes its answer, your 45 operator tokens and the 45,774 tokens of retrieved documents compete in the same softmax. Robin Young named this Token Democracy, and his abstract puts it bluntly when it says “transformers process all tokens as equals.”

The exponential also turns a small lead in score into a large lead in weight, wherever the token came from. Role markers in the chat template are just more tokens, so any extra weight the model gives the system prompt shows up as higher scores it learned to produce. Section 6 says every in-model defense works the same way, because its signal “competes in the same softmax as everything else.” An attacker hunting for text that outscores your instructions is hunting for exactly what this formula pays out on.

You can teach a model to care about provenance. Defenses mark untrusted text, train the model to prefer the system prompt, or give data its own embedding, a separate internal representation, as the ISE and ASIDE designs do. That last kind does feed origin into the math. Every one of them still depends on what the model learned to do with the signal, so you get a tendency, a behavior training shaped that can be strong and carries no guarantee on inputs an attacker picks.

Attackers who search for those inputs find them. Milad Nasr, Nicholas Carlini, Florian Tramèr, and colleagues combined gradient descent, reinforcement learning, random search, and human red teaming to bypass 12 recent defenses against jailbreaks and prompt injection, with attack success above 90% for most of them. Most of those defenses had reported near-zero attack success when they were published.

A property you need on every model call, like “the model only saw tool definitions someone approved,” needs software that enforces it every time. We put that check outside the model. In enforce mode, our proxy withholds any tool definition from a server it wraps that doesn’t match a signed approval, and the emitter records what got assembled.

The operator’s share of the context matters for a second reason. The paper also states a hypothesis we call the Dual-Failure Multiplier, and it starts from two results other researchers measured separately. Models lose accuracy as their input grows, and in many-shot jailbreaking a harmful response gets more likely as the context fills with demonstrations, which are example exchanges of an assistant doing the bad thing.

Our hypothesis is that those are one effect. As untrusted demonstrations pile up, the operator’s protective instruction gets a smaller share of the model’s attention, so attack success rises while quality on the operator’s task falls.

The paper also explains why it’s still untested, and the softmax budget is the reason. Section 7 notes that “softmax normalization over more positions lowers the share of any fixed span whatever the added text says.” Take a toy case where every score at the answer position is equal. Each token then gets 1/n of the weight, so the operator’s 45 tokens in a 46,205-token context get about 0.10%, and doubling the context with anything at all cuts that in half. Adding demonstrations makes the context longer, so content and length move together.

Real scores aren’t equal, which is why the design measures the share instead of assuming it. Its one length-matched test, H1, is set to run on Phi-3.5-mini-instruct and asks whether attack demonstrations pull attention off the operator’s rule more than neutral text of the same length does. It reads the rule’s share at the final prompt position, as a ratio of means over heads and layers, with attention-sink tokens (positions that soak up weight whatever they say) excluded and the rest renormalized. Its disclosure test, H2, can’t separate content from length.

The design isn’t frozen yet, the quality half isn’t designed, and no trial has run. If the hypothesis holds, the amount of ring 1 content you admit becomes a security setting, and you’d budget context the way you budget privileges.

Tool Definitions Sit in the Operator’s Chair

Positional authority is the standing a piece of text borrows from where the pipeline places it. It says nothing about how much attention the model pays. Chat templates have a system region, the part reserved for the operator’s instructions, and MCP hosts typically render every connected server’s tool definitions into that same region, because the model needs both to plan.

Whoever publishes the server writes the tool definition. That puts ring 1 content in ring 0’s chair, and the paper calls it Tool Definition Trust Inversion. In this version, every CxBOM record raises it as an open finding for each third-party tool definition in the system region. The finding describes an architecture, so it fires whether or not anyone attacked you.

Tool poisoning is hiding instructions in a tool’s description, where the model reads them and the user never sees them. Luca Beurer-Kellner and Marc Fischer at Invariant Labs disclosed it on April 1, 2025, along with cross-server shadowing (one server’s description changing how the agent uses another server’s tools).

They also disclosed the rug pull. A rug pull is a server changing a tool’s description after you approved it. You reviewed the clean version on Monday, and on Thursday the model reads the poisoned one from the same server, under the same name, with your approval still attached.

Then there’s the config file, which tells the MCP host which server commands to launch. CVE-2025-54136, which the paper calls MCPoison, let an attacker swap an already trusted MCP config in Cursor 1.2.4 and below for a malicious command. There was no warning and no re-prompt. All it took was write access to a shared GitHub repository holding that config, or to the file on the victim’s machine.

Cursor fixed it in 1.3 by prompting again.

Its sibling, CVE-2025-54135, which the paper calls CurXecute, let indirect prompt injection write a brand new .cursor/mcp.json in versions below 1.3.9. Indirect prompt injection is instructions hidden in content the agent reads, which the model then follows. Creating a dotfile (a hidden settings file whose name starts with a period) didn’t need approval, even though editing one did. NVD scores it 9.8. Text the agent read became code the machine ran.

The paper files both CVEs as configuration trust failures, meaning the attacker changed which program runs. Further down I cover which parts of them CxBOM handles.

The three attacks we recorded go after the description itself. One adds a sentence telling the model to send the secret out before it searches the notes. One claims the notes tool shares a login with another service and that the search fails until the secret goes out. The third is a rug pull mid-session, where the notes tool lists clean and then changes on the next listing.

Each ran against one local open-weight model, meaning one whose weights you download and run yourself. We used Qwen3-8B with greedy decoding, where the model always takes its most likely next token, once with the gate off and once with it on. We picked takes that repeat, so a count of them is no success rate. Before recording, we signed a claims ledger that limits what any of us may say about them to one sentence. That sentence is the caption under the film.

“On these recorded takes, for three attacks, a notes tool description that changed after it was signed sent the secret when the gate was off and the inbox showed that it arrived, and when the gate was on that tool was dropped from the listing, the secret did not arrive, and the model kept calling fetch until the take hit its limit, and only on the third attack was that definition clean on the first listing and changed on a re-listing within the session.”

Sign It at Assembly, Record It in a Context Bill of Materials

CxBOM has two parts, and each one sits where it can see what it reports. In enforce mode, the proxy withholds any tool definition or server instructions that differ from your signed approval, and it refuses a launch whose command line, binary, interpreter, pinned files, or pinned environment values differ from what you signed. The record means that after an incident, you can name every component the model was given, where it came from, which controls were applied, and which expected ones were missing.

The proxy sits between an MCP host and each stdio MCP server, the kind the host launches as a local process. It checks the server’s tool definitions, its instructions, and its launch against a signed approval. Tool results, notifications, and elicitation requests (a server asking the user, through the host, for more input) pass through it unverified.

The emitter runs where your application assembles its context and writes one CxBOM record per model call.

Content digest pinning means you store a cryptographic hash of exactly what you approved and reject anything whose hash differs. A hash works as a fingerprint of the text, and changing a single character changes it. Here the hash is SHA-256 over a canonical JSON form of each definition, one fixed layout, so spacing or key order can’t change the hash.

You run cxbom approve, which starts the server and prints each tool’s hash with only the first 200 escaped characters of its text, so read the full definitions from the server yourself. Then you run it again to accept each tool’s digest and the instructions’ digest with a written reason, type the server’s id at the terminal, and sign the lot into the workspace’s cxbom.lock with an Ed25519 key (a standard public key signature scheme) that lives under your real home directory. The proxy resolves that directory from the operating system’s user database and never from environment variables.

At runtime in enforce mode, the proxy pulls any changed or never-approved definition out of the listing before the model reads it, and it refuses calls to that tool. If the server’s instructions change, it blanks them and withholds every tool the server offers. That handles the rug pull from the last section, including the version where the definition changes on a re-listing mid-session.

Observe mode forwards a changed definition to the model and logs the change. The proxy allows observe mode only when the signed approval permits it.

The proxy pins the launch too. It won’t start a server whose command line, launcher binary, interpreter, pinned files, or pinned environment values differ from what you signed.

The paper calls the attacker a workspace writer here, someone who can edit files in the agent’s workspace, including the MCP config. Prompt injection that writes config files is one way to get that access. The wrapper is the config entry that launches the server through cxbom proxy instead of directly. A writer who changes a wrapped server’s command, binary, or pinned environment value, the MCPoison move, can’t produce a valid signature, so that launch never happens.

The wrapper sits in the same config file the writer can edit. A writer who deletes it, or adds a new unwrapped server the way CurXecute did, goes unnoticed in this version. The scope page lists cxbom scan and cxbom guard as a later plan that will detect that case when they run and won’t prevent it.

All of this assumes the agent can write workspace files and can’t run code as you. An agent with a terminal tool that runs shell commands as the user, or a server package that’s malicious code to begin with, is outside what the proxy stops.

Rollback is an attacker replaying an older, validly signed approval that has since been replaced. The proxy defends with a high-water mark. Every signed entry carries a sequence number, the proxy remembers the highest one it accepted in a state directory that sits outside your workspace by default, and it refuses anything below that.

Before it acts on any verdict, the proxy writes it to an attestation log. The emitter binds each tool component of a record to that log, so the record can state whether a definition matched a signed pin in a live proxy session.

A SBOM (software bill of materials) lists the components inside a piece of software. A CxBOM record does the same for one model call. It lists every component assembled into the window with its ring, its source, a SHA-256 digest of its text, its token count, and its share of the context. It also lists the controls applied and the controls expected but missing.

The emitter tokenizes each component separately, and your application sends the model exactly those token ids, the numbers the model receives, one per token. That’s why it needs a model server that accepts token ids directly, like llama.cpp or vLLM. Every token id the model receives belongs to exactly one component or to template overhead, and the emitter won’t finish a record unless your application’s digest of the ids it sent matches the ids it assembled.

Fields it can’t observe, like a scanner’s injection rating, get listed as not collected and raise an open finding. A record only counts as audit-clean when no finding is open, so it never comes out clean by omission. The current emitter’s record of the reference session carries five open findings: the trust inversion, a tool definition with no live proxy session behind it, and three groups of security fields no scanner collected.

In this version that means no record holding a third-party tool definition comes out audit-clean, because the emitter can’t yet register the trust boundary that would close the inversion finding. Keep that flag out of a release gate until it can.

The format is a profile modeled on CycloneDX 1.6, OWASP’s SBOM standard, with its own bomFormat value. A stock CycloneDX validator won’t accept it unchanged, and it joins your CycloneDX AI/ML BOM by agent and deployment identifiers.

On the reference session, the emitter built and wrote the record in a median of 75 ms over 20 runs on one arm64 machine. The record came to 22,297 bytes, 11.9% of the raw text it describes, and the paper expects a short-context record to run far larger than its text. The paper lists forensic utility, effectiveness in deployment, and cost as not yet measured.

The paper also maps every attack class in its threat model to what this version does about it.

The code refuses 14 of the 23. Each refusal cites a passing test that exercises that attack against the code, using scripted servers, constructed lock files, and stand-in home directories, and four of them hold only in enforce mode. It refuses one more in part. The other eight are classes this version doesn’t cover, and the paper names each one.

Those are conformance tests, which check that the code does what the design says, and the attack success rate in a live deployment is still unmeasured. The refused classes include the rug pull, a call to a tool the server never listed, a swapped command or binary, a replayed older approval, and a tampered lock file. The classes this version doesn’t cover include a definition that was malicious when you approved it, an edit that removes the wrapper, and injection through a tool result or a retrieved document.

By the paper’s count at the captured commit, the core suite of 28 test modules passes 1,691 of its 1,692 cases, with one skipped. On the public v0.3.0 tag the same 1,692 cases give 1,676 passed and 16 skipped, because the tests that cross-check against our measurement lab or the demo skip there. The tag’s full test tree is larger than that core suite, so a plain test run there reports a bigger count.

The hardest objection comes from our own numbers. In the reference session, the part a proxy could pin is the 357-token tool definition, while 45,774 tokens of retrieved documents would reach the model with a record and no check. For that content, execution-time control limits what it can cause, and in-model defenses still cut the rate of attacks the assembly layer can’t see. The paper doesn’t claim in-model defenses fail or that CxBOM prevents prompt injection.

Execution-time control governs what an agent may do after it reads something. CaMeL pulls the control, and data flows out of the trusted query so retrieved data can’t steer the program. On AgentDojo, a benchmark of agent tasks under prompt injection, it solved 77% of tasks with provable security, against 84% for an undefended agent. CxBOM’s proxy governs which tool definitions the model was shown, and the emitter’s record keeps the receipt for the whole context. You want both.

The paper also compares CxBOM with ETDI, MCP-Scan, and MCP-Guard. MCP-Scan already pins tools by hashing their descriptions. ETDI adds immutable versioned tool definitions with cryptographic identity verification, and it authenticates the tool provider, which CxBOM doesn’t.

By the paper’s table, CxBOM is the only one of the four that pins the server’s launch, while the wrapper stays in the config. It’s also the only one that records each model call’s context, in pipelines that send token ids to an open-weight endpoint. It doesn’t scan a new definition for poisoning the first time it sees it, which MCP-Scan and MCP-Guard do.

You Just Moved the Risk to the Approval Click

A signed pin proves a definition didn’t change, and it proves nothing about whether the definition was evil on the day you approved it. That’s the strongest argument against pinning, and the paper concedes it. Approval is the only defense against a definition that was malicious when someone approved it.

“Poisoned definition at first sight” is one of the classes our own threat model lists as not covered. The approval summary prints only the first 200 characters of each new or changed item’s canonical form, so an instruction buried later in a long description, or in a nested parameter description, can get approved unseen.

MCPTox measured how often poisoned tool definitions work. Its authors built 1,348 malicious test cases on 45 live MCP servers and 353 real tools and ran them across 20 LLM agent settings. The most vulnerable, o1-mini, averaged 72.8% attack success, and the average across all 20 was 36.5%.

The paper puts the lock in the same problem space as TUF, in-toto, and Sigstore, which sign metadata for software artifacts. A signed pin proves the definition you’re running is the one a key approved and says nothing about whether it’s safe. The approver the record names is that key, which isn’t the same as an authenticated person.

Pinning moves the risk to the approval, and the check then runs on every listing a running server returns for the life of the session. We haven’t measured how often a reviewer catches a poisoned definition, or how many re-approvals a normal upstream release forces. The paper names the complement it hasn’t built, a learned detector at approval time tested against a published set of poisoned definitions like MCPTox, and the reference implementation has none.

Four weaknesses remain in this preview, and the approval display is one of them. No entry expires unless its signer set an end time, and this version has no command to revoke a key. A stolen owner key keeps signing valid entries until you delete it and the trust manifest by hand and approve every server again.

A --state-dir flag that a workspace writer controls can point the proxy at an empty directory. That resets the high-water mark and lets an older entry replay. Environment variables declared secret aren’t hashed either, so a writer can swap a credential or an endpoint held in one.

The proxy’s log and the records also aren’t tamper-evident against another process running as the same user. They help you investigate an attack that worked through the model. They won’t hold up on a host that’s already compromised.

The proxy works with any MCP host and any model, so the pinning protects you today whether you run Claude, GPT, or something local. The record is narrower. To get exact token counts per component, the emitter tokenizes each piece of the context itself and sends the model raw token IDs, which means it has to know the model’s chat template exactly.

Version 0.3.0 pins two templates by hash (Phi-3.5-mini-instruct and Qwen3) and refuses anything else. It also needs an endpoint that accepts token IDs, like llama.cpp or vLLM, so closed APIs such as OpenAI’s and Anthropic’s can’t produce a record yet. MCP hosts that build their own context still get the proxy’s pins and its log, just no record.

An MCP host that assembles its own context gets pins and a log with no CxBOM record. If your agents run inside a host that assembles its own context, the proxy is the part you can use today.

What would change my mind on the bigger thesis is an in-model provenance defense that survives adaptive attacks as strong as the 12-defense study’s, which combined automated search with human red teaming.

In our reference session, the operator wrote 45 of the 46,205 tokens, and its Context Bill of Materials record accounts for every one of the other 46,160.

What to do next

If you run stdio MCP servers on Linux or macOS, put the proxy in front of one this week. Follow the setup in the README, approve the server at an exact package version, accept each tool and the server’s instructions by hash with a written reason, and swap the proxy into your MCP config in place of the server command. The pin covers the top-level package and not its dependencies, so for code you need pinned in full, run the server from a local tree pinned at approval, with its venv made with --copies inside that tree.

If you sit on the governance side, ask your team which MCP servers your agents launch and who signed off on each tool definition they read. If nobody can answer, that’s your first finding.

Read the scope page and Table 4 of the paper before you tell anyone you’re covered. Between them they list the attack classes and channels this version leaves open, and Sections 4 and 9 of the paper add the rest of its limits.

Signed approvals belong in the Run stage of the CARE framework (Create, Adapt, Run, Evolve) we use with clients at RockCyber, and the classes this version doesn’t cover belong on your Evolve list. Send schema feedback to spec@cxbom.org, file issues on the reference repo, and find the back catalog at RockCyber Musings.

👉 Paper: https://doi.org/10.5281/zenodo.23653997

👉 Github: https://github.com/CXBOM/cxbom-reference/tree/v0.3.0

Share RockCyber Musings

👉 Subscribe for more AI security and governance insights with the occasional rant.

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, check out this conversation with Chris Hughes over at Resilient Cyber, who’s also a reviewer on the paper, to discuss all things about the 2026 update to the OWASP Top 10 for LLMs, including why we need to worry about model context.

One of the private keys behind a publicly trusted certificate authority’s intermediate certificates was literally the number ‘4’. Scott Helme recovered it, along with the keys for seven other eMudhra intermediate certificates, from the public certificates on his laptop in under a second. That landed the same week PHP2Shell leaked before its patch existed and a backdoored LiteLLM release sat behind the latest tag. Rogue agents played no part in the week’s three biggest stories, and all three hit controls you assumed someone else was checking.

A certificate authority is supposed to generate keys nobody can guess. An embargo is supposed to hold until the fix ships, and a latest tag is supposed to point at what the maintainers built. All three assumptions failed in public within five days.

AI runs through the rest. Frontier models helped find PHP2Shell, and OpenAI disclosed that two of its own coding agents turned off the monitoring built to watch them. Eleven items, each first disclosed between October 16 and October 22, 2026.

Share

1. PHP2Shell Leaked Before Its Patch, And Attackers Started Early

PHP2Shell is two unauthenticated remote code execution flaws in the PHP interpreter itself, CVE-2026-109618 and CVE-2026-109619, both scored 9.2. Details leaked on Tuesday, October 20, 2026, when a Hacker News post and the CVE IDs circulated two days ahead of the planned Thursday release. PHP pulled its fixes forward and shipped 8.5.12, 8.4.27, 8.3.36, and 8.2.35 on October 21, so every PHP install below those versions is in scope, and 8.1 and older get no official fix. Amazon says its honeypots saw exploitation attempts starting October 17, and Cloudflare reported thousands of attempts in the ten hours before the announcement.

Why it matters

• The flaws sit in PHP itself, so clean application code won’t save you.

• Exploitation attempts started before the leak, so the disclosure date understates exposure.

• End-of-life PHP stays exposed with no fix coming.

What to do about it

• Inventory every PHP runtime, including vendor appliances and control panels.

• Patch to the fixed releases or your distro’s backport, and isolate anything older.

• Alert when PHP worker processes spawn shells, curl, or wget.

Rock’s Musings

An embargo is the agreed quiet period between a fix being ready and the world hearing about it. This one had to cover Calif, which found the bug class on August 11, plus Wiz and OpenAI, which found it independently in the following weeks. The PHP Foundation says Chainguard broke the embargo early on October 21 by publishing fixed images, letting anyone study the fix before the release, and Chainguard says it acted on evidence of exploitation. Attackers were probing four days before the patch existed.

AWS CISO CJ Moses put the lesson in one line: “defenders should not assume the public disclosure date is the earliest date of exploitation.” When several teams with frontier models find the same bug inside two months, an embargo buys less time than it used to. Build your patch process for the leak.

2. eMudhra Signed With A Private Key Equal To 4

On October 17, 2026, a researcher known as Wayne reported that eight intermediate CA certificates from eMudhra, an Indian certificate authority, used P-384 private keys smaller than 256, so anyone can recover them in at most 256 tries. The key for “EM OV TLS CA - G3C” was 4. Six were TLS issuing CAs and two were for code signing. Chrome and Firefox blocked them on October 17, Apple and Microsoft followed, and Go shipped fixes in 1.27.3 and 1.26.10. Mozilla dropped the emSign ECC Root CA - G3, and Chrome’s removal of that root is expected to reach users on October 27.

Why it matters

• Software that skips revocation checks can still trust these intermediates.

• The keys were generated in April 2025 and sat recoverable for more than 17 months.

• About 30,000 customer certificates need replacing.

What to do about it

• Update CA bundles in containers, language runtimes, and embedded gear.

• Find certificates chaining to emSign ECC Root CA - G3 and replace them before October 27.

• Test custom clients against the test endpoints on Dennis Jackson’s incident site.

Rock’s Musings

An elliptic-curve private key is a random number, and the public key is that number multiplied by a fixed point on the curve. Pick a number below 256 and an attacker tries all 256, which is why Helme wrote that “nobody had to steal anything.” eMudhra traced it to a faulty script in its April 2025 key ceremony, the witnessed event where a CA generates its keys, which produced a second, weak key set with similar labels that staff then picked.

The browsers fixed this in hours because they push blocklists to you. Your printers, appliances, Java services, and anything with its own trust store got nothing pushed, which is the gap Mozilla’s Dennis Jackson flagged. eMudhra says none of the eight issued customer certificates, and nobody can prove what happened during those 17 months.

3. LiteLLM 1.107.0 Shipped A Stealer, And The Tags Pointed At It

LiteLLM 1.107.0, published to PyPI and as Docker images on October 21, 2026, carried malicious startup files that steal cloud credentials, private keys, and LLM provider keys. The file cook.pth runs the stealer at every Python start, import or no import. Socket found that the latest, main-latest, and main-stable image tags resolved to the poisoned build, and the build provenance named its source as Berri-AI/litellm, one hyphen off the real BerriAI/litellm. This is LiteLLM’s second compromise this year, after attackers pushed backdoored 1.82.7 and 1.82.8 to PyPI on March 24, and 1.82.8 also hid its payload in a .pth file.

Why it matters

• An AI gateway holds every provider key plus your cloud identities.

• Deployments that track a floating tag can pull malware with no code change.

• Removing the package leaves the startup files behind.

What to do about it

• Hunt for cook.pth and unexpected sitecustomize.py in site-packages.

• Pin images by digest and pin 1.106.2 until a clean release ships.

• Rotate every credential reachable from any host that ran 1.107.0.

Rock’s Musings

A .pth file is a Python feature for adding import paths, and Python processes it at every startup, before your code runs. Build provenance is the record of where an artifact came from and how it was built. Here the record wrote the lookalike repository down in plain text, and nobody’s pipeline read it.

The same project got hit with the same startup-hook technique seven months apart. If LiteLLM sits in front of your models, treat it like a domain controller: pinned versions, provenance checks, and credentials scoped to it alone.

4. OpenAI’s Coding Agents Switched Off The Monitor Watching Them

OpenAI updated a misalignment report on October 20, 2026 covering two internal incidents. On October 3, a GPT-6.1 Sol agent repairing an evaluation pipeline disabled automatic alerts to OpenAI’s Security team, reading an earlier instruction not to send Slack messages as permission. On October 6, a GPT-6 Astra agent removed its monitoring hook while copying a reference program, then told the user it hadn’t. OpenAI updated Auto-Review, the reviewer it requires for internal Codex use, to block attempts to disable monitoring.

Why it matters

• Both agents weighed keeping the safeguard and dropped it.

• Agents also launched workloads with monitoring off.

• An agent’s report of its own actions is weak evidence.

What to do about it

• Make monitoring settings unwritable by the agents being monitored.

• Alert when an agent-launched workload starts unmonitored.

• Check agent claims against the diff.

Rock’s Musings

OpenAI is testing two explanations. Benign overpersistence means a model so set on finishing that it routes around every obstacle, safeguards included. The other is a propensity to evade monitoring in particular, which is far worse. OpenAI hasn’t picked one yet.

A researcher said “don’t send any messages” about a Slack search, and the agent stretched that until it covered a security alert. Your people write instructions like that daily, and the agent decides how far each reaches.

5. Outside Red Teamers Found Settings That Turned OpenAI’s Monitor Off

METR and Redwood Research published an account on October 22, 2026 of a three-week exercise inside OpenAI, run July 21 to August 11. They estimated that only 75% to 94% of one class of internal agent traffic was eligible for monitoring. Roughly 10% of eligible internal Codex traffic went unmonitored from May to July because the monitor skipped compaction events. They found two client-controllable headers that disabled monitoring for new agents. OpenAI says it patched every production-impacting finding, which METR hasn’t evaluated.

Why it matters

• A top lab couldn’t confirm full monitoring coverage.

• A monitor that fires at hand-back misses agents that keep working.

• Full findings stay private by agreement.

What to do about it

• Reconcile inference logs against monitor logs on a schedule.

• Trigger monitoring by message count or time.

• Red-team the monitor itself.

Rock’s Musings

Compaction is worth learning. When an agent’s working memory fills up, the system summarizes older turns into a shorter version and keeps going. OpenAI’s monitor ran when the agent handed control back to the user, so an agent working through compaction kept postponing its own review.

From METR’s five design ideals, I’d start with isolating the agent from the place its monitor runs, because both incidents in item 4 depended on an agent reaching its own oversight.

6. Washington Wrote A Playbook For Agents That Go Off Task

NIST, NSA, and CISA published “Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation” on October 16, 2026. It lists ten voluntary practices. It defines an “SI control incident” as agents taking significant uninstructed actions that compromise the controls setting a task’s scope or keeping human oversight in place. Practice 4 tells you to validate and secure your monitors, and Practice 5 asks you to keep rejected tool calls along with logs and checkpoints.

Why it matters

• It builds on NIST CSF 2.0 and SP 800-61r3.

• It says the practices “may also apply” to deployers.

• It asks for reporting of near misses to government.

What to do about it

• Map your agent IR runbook against the ten practices.

• Add rejected tool calls to log retention.

• Prove containment before an agent restarts.

Rock’s Musings

A near miss is the event that almost became an incident. Safety engineers in energy count them because they predict the incident you haven’t had yet.

An action counts as uninstructed only if neither the operator nor an adversarial source such as a prompt injection directed it. Under this framework an injected agent is a regular security incident, and a self-directed one is an SI control incident. You need both runbooks.

7. Microsoft Ties The ConnectWise Breach To Storm-0940 And ARTEX

ConnectWise disclosed on October 16, 2026 that attackers had accessed application identities for ConnectWise Cloud Backup, formerly SkyKick, and Huntress confirmed unauthorized sign-ins in 162 Microsoft 365 tenants that day. On October 21, Microsoft attributed the campaign to Storm-0940, a China state-aligned group, using the agentic penetration testing framework ARTEX. Microsoft says the group attempted sign-ins to more than 2,250 tenants, and an earlier password spray from the same infrastructure peaked at 2,900 failed logons a minute.

Why it matters

• The attacker signed in as a trusted app, so user MFA never applied.

• A backup vendor held access to mail and files across tenants.

• ConnectWise suspended the backup service its customers depend on.

What to do about it

• List third-party apps with tenant-wide Graph permissions, and cut scopes.

• Alert on app sign-ins from outside the vendor’s known ranges.

• Audit Entra for credentials added to apps since October 13.

Rock’s Musings

A service principal is the identity an application uses inside your Microsoft tenant. Under the client credentials flow it presents its own secret and gets a token with no human involved. Steal the secret and you hold every permission the app has, in every consenting tenant. MFA protects people, and this attacker never pretended to be one.

Two weeks ago an unattributed operator used ARTEX against South Korean lenders, and now Microsoft says a state-aligned group runs it at supply chain scale. OAuth app abuse is a standard Storm-0940 move, and ARTEX made it fast enough that the alert and the damage arrive together.

8. One Hacker’s AI Agents Broke Into Over 100 Company Websites At About $50 Each

Reuters reported on October 19, 2026 that Snorre Fagerland of NorForge watched a Chinese-speaking hacker run intrusions almost entirely through AI agents on Cairn, an open-source platform, via control panels left exposed online. Reuters named Deere, SharkNinja, and Cummins among the targets, and Cummins says the site belonged to a vendor. The agents had tried more than 1,200 sites, and successful intrusions took about 23 hours. Fagerland estimated about $50 in tokens per successful hack.

Why it matters

• The operator’s instructions ran under 100 Chinese characters.

• Agents pivoted to an exposed control panel when the main app held.

• Deere said it found no sign of sensitive data access.

What to do about it

• Find admin panels tied to your domains on the open internet.

• Monitor subsidiaries and vendor-hosted sites too.

• Staff detection for 23-hour probing.

Rock’s Musings

Fagerland told Reuters this was “not script kiddie stuff,” and the pivot convinced me. Blocked at a big company’s main application, the agents searched records, found a related control panel, and worked a subtle weakness for hours until the server ran their commands.

Price it the way your CFO will. NorForge puts the average attempt, failures included, at about $5. At that price every vendor site with a login page gets a try.

9. OpenCode Lost Users’ Keys, Then A 9.8 Surfaced With No CVE

OpenCode, an open-source AI coding agent, said attackers broke into a legacy system the night of October 16, 2026, exposing provider API keys stored by 1.4% of its Console users. Co-founder Dax Raad posted that the AI models his team used for analysis started refusing once the malicious code tripped safety filters, until they got OpenAI’s Daybreak Blue access for vetted defenders. On October 19, Pillar Security disclosed a separate remote code execution flaw in OpenCode 1.0.137 through 1.18.32 on Windows and WSL, where a remote MCP server’s OAuth address reached a PowerShell command unescaped. Anomaly, OpenCode’s maker, fixed it in 1.18.33 on September 28 with no CVE and no advisory.

Why it matters

• Your responders can hit model refusals mid-incident.

• No CVE means scanners won’t flag the vulnerable builds.

• Anomaly’s policy puts user-configured MCP servers outside its trust boundary.

What to do about it

• Rotate any provider key that ever sat in OpenCode Console.

• Check OpenCode versions directly on Windows and WSL machines.

• Enroll your IR team in a vetted defender program now.

Rock’s Musings

If your IR plan says “we’ll use AI to analyze the malware,” test whether your model will look at malware before the day you need it. A guardrail that can’t tell a responder from an attacker stops the responder, and an attacker running an open-weight model never sees the refusal.

A trust boundary is the line where software stops trusting input and starts checking it. Anomaly treats a malicious MCP server as your problem, which doesn’t cover a shell command built from an unescaped URL. Ask your AI tooling vendors in writing whether they file CVEs, and treat a no as a risk acceptance you sign.

10. CISO Job Postings Fell 39% While AI Security Engineer Postings Rose 268%

CyberSN published its Cybersecurity Jobs Report on October 16, 2026. US postings fell 26.4% year over year to 235,786 for January through September, a four-year low. CISO postings dropped 38.9% to 2,832, while AI security engineer postings rose 268.3%. The 2026 CISO count nearly matches 2024’s 2,808.

Why it matters

• Postings count ads and say nothing about hires.

• Nine of ten job functions shrank.

• Demand is moving to AI security skills.

What to do about it

• Cross-train two engineers on agent security this quarter.

• Write the AI security engineer job description now.

• Compare your open roles with these numbers.

Rock’s Musings

I’d read the CISO number calmly, since it’s a return to 2024 levels after a hot 2025.

The 268% figure deserves more attention, because items 4 through 6 need someone who understands agent monitoring and incident response together. Plan to grow that person in-house.

The One Thing You Won’t Hear About But You Need To: Split The Attack And The Monitor Goes Blind

Four MATS-supported researchers posted “Red-Teaming Multi-Agent Attacks for Loss-of-Control Risks” to arXiv on October 16, 2026. They hand-built 16 attacks, including weight exfiltration, against MiniLab, a set of live services modeled on an AI lab’s internal systems. A GPT-5.6 Sol monitor reading one trajectory at a time flagged 87% of successful attacks done by one agent and 7% of the same attacks split across agents. Reading all agents’ trajectories together raised detection to 41%.

Why it matters

• The monitors in items 4 and 5 read one agent at a time.

• Each split step looked like a routine error.

• Model-written attack plans improved with monitor feedback.

What to do about it

• Correlate actions across agents sharing services or permissions.

• Treat a per-agent all-clear as partial coverage.

• Bring this paper to your next agent architecture review.

Rock’s Musings

A distributed attack splits harmful work into benign-looking pieces. Separation of duties rests on the same premise: no single person can finish the harmful job, so fraud requires collusion. Agents that share a plan collude for free.

Humans built these attacks against a monitor blind to agent reasoning, so treat it as a worst case. Model-planned attacks succeeded covertly far less often.

Every story in this issue is a control somebody assumed was being checked: a CA key nobody tested, an embargo that couldn’t hold, a latest tag nobody pinned, a monitor an agent was able to switch off. Pick one of those four in your own environment and check it yourself before Monday.

👉 Subscribe RockCyber Musings for more insights into AI security and governance, with the occasional rant. You can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out again with my friends Candy Alexander and Cailin Sullivan on Cybersecurity Refined and Feral to argue whether AI Armageddon is fact or fiction, and whether we slow down for safety or floor it to beat China. After two airliners collided over the Grand Canyon in 1956, Congress built air traffic control and kept the planes flying, and that’s where the conversation went: guardrails you can actually test, starting with an agent kill switch that stops every subagent it spawned.

Share RockCyber Musings

References

Aikido Security. (2026, October 21). LiteLLM compromised: Infostealer and RAT in PyPI 1.107.0. https://www.aikido.dev/blog/litellm-compromised-infostealer-rat

Akin Gump Strauss Hauer & Feld. (2026, October 16). NIST, NSA and CISA issue key practices for incident readiness and response in super intelligence training and evaluation. https://www.akingump.com/en/insights/ai-law-and-regulation-tracker/nist-nsa-and-cisa-issue-key-practices-for-incident-readiness-and-response-in-super-intelligence-training-and-evaluation

Bleeping Computer. (2026, October 20). Mozilla, Google, Apple block compromised eMudhra CA certificates. https://www.bleepingcomputer.com/news/security/mozilla-google-apple-block-compromised-emudhra-ca-certificates/

Bleeping Computer. (2026, October 22). Amazon says PHP2Shell exploitation began 4 days before patches. https://www.bleepingcomputer.com/news/security/amazon-says-php2shell-exploitation-began-4-days-before-patches/

Chrome Root Program. (2026, October). Protecting Chrome and Android users from compromised eMudhra intermediate CA certificates. Google. https://blog.google/security/protecting-chrome-and-android-users-from-compromised-emudhra-intermediate-ca-certificates/

Coker, J. (2026, October). US cybersecurity job postings fall 26%, hitting four year low. Infosecurity Magazine. https://www.infosecurity-magazine.com/news/cybersecurity-job-postings-fall/

Corbet, J. (2026, March 27). Leading items [LiteLLM supply-chain compromise]. LWN. https://lwn.net/Articles/1064763/

Curry, D. (2026, October). US cybersecurity job postings fall 26.4% as AI roles surge. eSecurity Planet. https://www.esecurityplanet.com/cybersecurity/news-us-cybersecurity-job-postings-decline-ai-roles/

Cybersecurity and Infrastructure Security Agency. (2026, October 16). Key practices for incident readiness and response in super intelligence training and evaluation. https://www.cisa.gov/resources-tools/resources/key-practices-incident-readiness-and-response-super-intelligence-training-and-evaluation

Digg. (2026, October 18). AI models reportedly refuse to help OpenCode fight a sustained cyberattack. https://digg.com/ai/7yvwkrok

Field Effect. (2026, October). Compromised ConnectWise Cloud Backup identities used to access Microsoft 365 tenants. https://fieldeffect.com/blog/connectwise-cloud-backup-identities

Frichette, N., Obregoso, S., Tafani-Dereeper, C., & Spir, E. (2026, March 24). LiteLLM and Telnyx compromised on PyPI: Tracing the TeamPCP supply chain campaign. Datadog Security Labs. https://securitylabs.datadoghq.com/articles/litellm-compromised-pypi-teampcp-supply-chain-campaign/

GBHackers. (2026, October). Hackers use ConnectWise SkyKick credentials to access Microsoft 365, halting cloud backups. https://gbhackers.com/hackers-use-connectwise-skykick-credentials

Greig, J. (2026, October 21). Hundreds of websites taken offline as concern grows over two PHP vulnerabilities. The Record. https://therecord.media/php-script-vulnerabilities-warnings

Greig, J. (2026, October). PHP says AI review of critical bugs first sparked alarm in August, but signs surfaced in 2024. The Record. https://therecord.media/php-foundation-statement-critical-vulnerabilities

The Hacker News. (2026, October 21). LiteLLM compromised in supply chain attack. https://thehackernews.com/2026/10/litellm-compromised-in-supply-chain.html

The Hacker News. (2026, October). China-linked hackers use ARTEX AI agents in ConnectWise supply chain attack. https://thehackernews.com/2026/10/china-linked-hackers-use-artex-ai.html

Hammond, J. (2026, October 21). Rapid response: php2shell, leaked critical PHP vulnerabilities (CVE-2026-109618 & CVE-2026-109619). Huntress. https://www.huntress.com/blog/php2shell-leaked-critical-php-vulnerabilities

Khandelwal, S. (2026, October 22). eMudhra to exit global SSL/TLS business after 8 intermediate CA keys found easily guessable. The Hacker News. https://thehackernews.com/2026/10/emudhra-to-exit-global-ssltls-business.html

Khandelwal, S. (2026, October 22). OpenCode flaw lets malicious MCP servers execute code during sign-in on Windows and WSL. The Hacker News. https://thehackernews.com/2026/10/opencode-flaw-lets-malicious-mcp.html

Kitaoka, J., Pham, T. A., Giudice, C. R., & Merizian, J. (2026, October 16). Red-teaming multi-agent attacks for loss-of-control risks (arXiv:2610.24052). arXiv. https://arxiv.org/abs/2610.24052

Kitaoka, J., Pham, T. A., Giudice, C. R., & Merizian, J. (2026, October). Red-teaming multi-agent attacks for loss-of-control risks [Blog post]. LessWrong. https://www.greaterwrong.com/posts/SiWdLyjQQtBgtaLZq/red-teaming-multi-agent-attacks-for-loss-of-control-risks

Laravel. (2026, October 21). PHP security advisory (CVE-2026-109618, CVE-2026-109619). Laravel Forge. https://laravel.com/forge/docs/knowledge-base/php-security-release-october-2026

Markovic, S. (2026, October 21). eMudhra security blunder left trusted CA signing keys exposed to anyone. Help Net Security. https://www.helpnetsecurity.com/2026/10/21/emudhra-certificates-private-keys-compromised/

Microsoft Threat Intelligence. (2026, October 21). AI-driven, large-scale nation-state campaign using ARTEX enables supply chain compromise. Microsoft Security Blog. https://www.microsoft.com/en-us/security/blog/2026/10/21/storm-0940-uses-artex-agentic-ai-connectwise-supply-chain-compromise

National Institute of Standards and Technology, National Security Agency, & Cybersecurity and Infrastructure Security Agency. (2026, October 16). Key practices for incident readiness and response in super intelligence training and evaluation [PDF]. https://media.defense.gov/2026/Oct/16/2004015654/-1/-1/0/KEY_PRACTICES_INCIDENT_READINESS_RESPONSE_IN_SI_TRAINING_EVALUATION.PDF

National Institute of Standards and Technology. (2026, October 16). Key practices for incident readiness and response in super intelligence training and evaluation. https://www.nist.gov/news-events/news/2026/10/key-practices-incident-readiness-and-response-super-intelligence-training

The Next Web. (2026, October 19). AI agents broke into 100+ sites for one hacker at $50 each, researcher says. https://thenextweb.com/news/ai-agents-hacker-cairn-100-sites-deere-norforge

OpenAI. (2026, October 20). Suppressing misalignment monitoring alerts and disabling misalignment monitoring. OpenAI Alignment Research Blog. https://alignment.openai.com/misalignment-reports/suppressing-misalignment-monitoring-alerts-and-disabling-misalignment-monitoring

Pillar Security. (2026, October 19). Open and shut: Remote code execution in opencode via MCP OAuth. https://www.pillar.security/blog/open-and-shut-remote-code-execution-in-opencode-via-mcp-oauth

Poireault, K. (2026, October 19). OpenCode breach could affect OpenAI, Anthropic and Google. Infosecurity Magazine. https://www.infosecurity-magazine.com/news/opencode-breach-openai-anthropic

Power, J. (2026, October 21). OpenAI says its agents turned off own safeguards. Al Jazeera. https://www.aljazeera.com/economy/2026/10/21/openai-says-its-agents-turned-off-own-safeguards

Rein, D., & Kaufman, A. (2026, October 22). Red-teaming OpenAI’s internal agent monitoring systems. METR.

METR

Red-Teaming OpenAI's Internal Agent Monitoring Systems

For three weeks, from July 21st to August 11th, a METR employee and a Redwood Research employee conducted a pilot high-access exercise to stress-test OpenAI’s internal misalignment monitoring system. The output of this exercise was a 61-page internal report that was shared with OpenAI researchers and security staff…

Read more

20 days ago · 1 like · METR

Satter, R. (2026, October 19). How one hacker used AI agents to break into over 100 company websites. Reuters, via Honolulu Star-Advertiser. https://www.staradvertiser.com/2026/10/19/breaking-news/how-one-hacker-used-ai-agents-to-break-into-over-100-company-websites/

SecurityWeek. (2026, October). US cybersecurity job postings fall 26% as AI security engineer demand soars. https://securityweek.com/us-cybersecurity-job-postings-fall-26-as-ai-security-engineer-demand-soars

Shadow-LLM. (2026, October). Distributed multi-agent attacks evade GPT-5.6 Sol monitor in 93% of cases vs 13% for single-agent (Issue #271). GitHub. https://github.com/Shadow-LLM/failure-cases/issues/271

Socket Research Team. (2026, October 21). LiteLLM PyPI package and Docker images compromised to steal cloud and AI credentials. Socket. https://socket.dev/blog/litellm-compromise

TechCrunch. (2026, October 19). Hackers hit AI coding tool OpenCode, running up charges on users’ AI accounts. https://techcrunch.com/2026/10/19/hackers-hit-ai-coding-tool-opencode-running-up-charges-on-users-ai-accounts

Tzadik, S., Tamari, S., Ohfeld, N., & Schwartz, T. (2026, October 21). PHP2Shell: AI uncovers critical RCE vulnerabilities affecting 70% of the Internet. Wiz. https://www.wiz.io/blog/php2shell-php-rce

Unite AI. (2026, October 22). METR red-team finds vulnerabilities in OpenAI monitoring system. https://www.unite.ai/metr-red-team-finds-vulnerabilities-in-openai-monitoring-system/

Walker, J. (2026, October). PHP2Shell disclosure unravels after leaks forced early patch drop. TechTarget. https://www.techtarget.com/cybersecurity/news/366652613/PHP2Shell-disclosure-unravels-after-leaks-forced-early-patch-drop

Wang, T. (2026, October 20). NSA, NIST, CISA issue guidance on rogue AI agents. GovInfoSecurity. https://www.govinfosecurity.com/nsa-nist-cisa-issue-guidance-on-rogue-ai-agents-a-33156

Warp2Search. (2026, October 21). PHP 8.5.12, 8.4.27, 8.3.36, and 8.2.35 fixes two critical PHP RCE. https://www.warp2search.net/story/php-8512-8427-8336-and-8235-fixes-two-critical-php-rce/

AI evaluation security starts with a process safety measure that’s already standard practice. When a lab disables a model’s classifiers to gauge its raw capability, it’s essentially creating a “bypass,” a term a manufacturing plant would readily recognize. According to Chevron’s established standards, any bypass lasting over 72 hours triggers a “management of change” protocol. Anthropic’s cyber evaluations, conducted with safeguards disabled within a vendor’s system, experienced an incident in April that went undetected until July 24th.

Share

AI Evaluation Security Starts With a Bypass

Start on the plant side, where the rules are a lot stricter. Chevron’s technical standard for bypassing critical protections defines a bypass as any act to “temporarily block out, isolate, override, inhibit, force jumper” or otherwise disable a device so it won’t perform its designed function. Testing sits explicitly inside the standard’s scope.

Before an operator disables a critical protection, standard procedure requires a certificate that specifies an authorized time period and a documented approver. The bypass is recorded in a register with start and finish times, and designated personnel regularly confirm its continued presence.

Additionally, there’s a time limit. Any bypass exceeding 72 hours must undergo a management of change review. The clock begins with the initial override and does not reset if the bypass is temporarily removed and then reinstated.

The AI side does the same thing on purpose, and four organizations put it in writing this summer. OpenAI describes third-party cyber evaluations that run with “lowered safeguards to measure underlying capability.” The UK AI Security Institute (AISI) says the provider’s cyber classifiers “were deliberately disabled during the evaluation.” Meta ran its Muse Spark 1.1 test “with safeguards removed.” Anthropic’s evaluations ran without its standard safeguards, and Anthropic says the safeguards on its generally available models “would have blocked the behaviors identified.” Hold those against Chevron’s definition and each one reads as a bypass of a critical protection.

Process safety lets you bypass a protection as long as you name what carries the load while it’s out. Edward Marszal of Kenexis, writing in 2014 about the new version of IEC 61511, quotes that standard’s term for it. A compensating measure is “planned and documented methods for managing risks that are implemented temporarily during any period of maintenance or process operation with known dangerous faults or dangerous failures in the SIS” (SIS means safety instrumented system).

Chevron, on the other hand, calls this “equivalent protection” and wants it to function similarly to the original device it replaces. Without such protection, a qualified individual must provide continuous monitoring. The guidelines even state that if this person cannot manually offer the same level of protection, the equipment must be shut down.

Anthropic concisely described its compensating measure, stating that operating models without safeguards “is safe only if the evaluation is appropriately contained.” The evaluation environment, which is essentially the sandbox where the model operates during testing, serves as the compensating measure for every classifier the lab deactivated. This type of substitute is typically tested in a plant before anyone relies on it. Anthropic discovered its failure on July 24 by reviewing transcripts.

The Evaluation Environment Failed the Protection Chevron sets

Chevron’s appendix sets the protection a test must pass before anyone credits it. An independent protection layer is a device, system, or action that can stop a scenario from reaching its consequence. It qualifies only if it passes four tests.

It must be specific, meaning it addresses the hazard in front of you, and dependable, meaning it delivers the same risk reduction under any circumstances. It also has to be independent of the initiating event and other layers, and auditable, meaning someone tests it periodically to confirm it works.

Go ahead and run Anthropic’s July 30 account through those tests. The prompt told Claude “its environment was a simulation and that it had no internet access.” The environment, which Irregular built, had internet access, and Anthropic writes that “neither we nor our evaluation partner were aware of this misconfiguration until we detected it through our additional evaluation monitoring.”

The model’s own judgment leaned on that same premise, because Claude “believed everything it initially encountered was part of the simulation.” Anthropic says Opus 4.7 eventually recognized the systems were real in all four of its runs, rationalized in two of them that the real company must be part of the exercise, and didn’t stop in any of them. Mythos 5 “correctly intuited that it was accessing the open internet” and then “reasoned its way back to the conclusion that it was still in a simulation.”

The prompt, the isolation, and the model’s judgment looked like three layers. The prompt was written on the assumption that the isolation worked, and the model’s judgment leaned on the prompt, so one belief nobody had tested sat under all three, that the environment had no internet. That fails independence, because one bad configuration took out every layer at once. It fails dependable too, because the isolation didn’t hold in the one circumstance it existed for.

Actions have consequences. In the Opus 4.7 incident, the runs extracted application and infrastructure credentials and got into “a database containing several hundred rows of production data” at a real company.

Prompt instructions fail the specific test as well, and Anthropic’s October 9 post shows how. A Claude Haiku 4.5 run that generated example tasks on random webpages had instructions ruling out logging in, creating accounts, entering personal data, purchases, and submitting anything destructive. In Anthropic’s words, “the instructions did not rule out form submissions.” On a page about an unsolved homicide, the model filled out a police tip form saying it “may have information regarding this case.”

Philadelphia police say the tip arrived July 18 at 11:27 p.m. and got flagged as spam. Police also say Anthropic found it September 28 and told them October 7, and Anthropic’s post gives October 8. A deny list only covers the actions somebody thought to write down, and nobody wrote down the tip form.

One Vendor, Four Labs, One Scenario

This next point turned a lab problem into a supply chain problem. Irregular’s August 14 postmortem says “all subsequent public disclosures refer to the same underlying issue first disclosed by one of our customers on July 30.” It adds that “the issue originated from a single evaluation scenario.” OpenAI had published on August 4 and Meta on August 14, both naming an Irregular misconfiguration. In September, Google confirmed that Gemini hacked three companies in a cyber test, which the Wall Street Journal reported Irregular ran in May. Irregular then told CyberInsider, “This is the same issue that was already reported and does not represent a materially separate incident.”

Process safety has a name for one event defeating barriers that were supposed to be separate. Rausand and Lundteigen’s NTNU lecture slides teach the IEC 61508 definition of a common cause failure, a “failure, that is the result of one or more events, causing concurrent failures of two or more separate channels in a multiple channel system, leading to system failure.” They split the cause into a root cause and a coupling factor, which they define as “a property (commonality) that makes multiple items susceptible to failure from a shared cause.” Their list of coupling factors includes the same installation staff and the same procedures.

Apply that inside one lab first. The prompt, the isolation, and the model’s judgment were separate channels guarding one hazard, and one event failed all three at once, a scenario with live internet and a fictional company name that, in Irregular’s words, “unintentionally coincided with a real domain.” That’s a common-cause failure by the textbook definition.

Across labs, the shared vendor is the coupling factor. Irregular puts the failure at “fewer than 1 in 10,000 advanced simulations,” and by Irregular’s own statements that one scenario still reached Anthropic, OpenAI, Meta, and Google. If you buy evaluations from a vendor that also tests other labs, that vendor’s scenario library is your coupling factor. Ask in writing which scenarios and environments your runs share with other customers.

The Failure Sat Undetected From April to July

Process safety assumes protections fail quietly and builds a test schedule around that assumption. Green and Bell quote IEC 61511-1:2016 clause 3.2.56, which defines a proof test as a “periodic test performed to detect dangerous hidden failures in a SIS.” The repair that follows restores the system “to an ‘as new’ condition or as close as practical to this condition.” Hidden failures are dangerous, undetected failures, those that disable a protection without announcing themselves. Green and Bell note that the dangerous failures a proof test misses “will continue to be present.”

The math fits in one line. Green and Bell give the average probability of failure on demand as

where λDU is the dangerous undetected failure rate, and Tp is the proof test interval. The divide by two assumes the failure shows up halfway through the interval. If you double the time between proof tests, the average chance the protection is dead when you need it doubles too.

That formula is built for a failure rate, the way hardware fails partway through an interval. A misconfigured scenario is a design error, present from the day someone builds it, so the exposure runs the whole interval and the halving doesn’t apply. The real picture for evaluation environments is worse than the formula.

Anthropic’s earliest incident dates to April. Its transcript review started July 23, found the incidents July 24, and Anthropic started notifying Irregular and the three affected organizations July 27. Incidents traced to the same issue turned up at Anthropic in April, at Google in May, and at Meta in a test that began in early July. That scenario carried a dangerous undetected failure from April into late July, and nobody knew until Anthropic’s transcript review. Irregular’s rate of under 1 in 10,000 doesn’t change that math, because a rare failure that no proof test catches stays in place across every run until something trips over it.

Irregular now writes that “a more systematic process for reviewing and revalidating evaluations before each run is needed to identify any new overlaps.” That’s a proof test, described as something needed. The post doesn’t say it’s running yet, and it doesn’t say who runs it.

OpenAI shows the other half of the problem. Its August 26 report says “while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities.” Green and Bell call the share of failures a test can find its proof test coverage, and they write that perfect coverage “is often difficult to achieve.” OpenAI’s validation didn’t cover what the agents found. I walked through the restart decision behind that run in my September 22 issue, so I’ll leave the chain there.

AISI Ran the Bypass With Nothing Underneath

AISI wrote the most candid incident report I read this year, and it describes the riskiest configuration in this issue. Report INC-2026-07-28-01 covers two cyber ranges, DL-v1 and DL-v2, tested with Mythos 5 and GPT-5.6 Sol. Internet access “was deliberately enabled,” the classifiers were off, and “AISI has not yet built synchronous LLM-based monitoring of runs.” Domain allowlisting, the egress control that would have narrowed what agents could reach, “had been a backlogged project since April 2026.”

In Chevron’s terms, that’s a bypass with no equivalent protection and no continuous monitoring, the case where the guidance says to shut down. The runs went ahead. During Sample 1, which ran from July 26 at 12:45 to July 27 at 23:15 BST, an agent opened a malicious pull request on a public GitHub repository. In a separate sample, malicious package metadata reached repositories that Dependabot processes, and at least 53 Dependabot containers executed the payload.

The security engineering team triaged a Tor egress alert the next morning, contacted the testing team at 10:56 BST, and declared an incident and terminated all samples at 11:42. AISI says the attempts failed and it knows of no real-world harm, and the layer that caught it was a commercial security platform watching network egress.

AISI’s own explanation describes a bypass that outlived its authorization. AISI had judged internet access an acceptable risk for earlier models and “did not revisit that judgment quickly enough as capabilities advanced.” Its fix is to “treat the decision to grant internet access as one that must be actively justified rather than a default.” That’s a bypass certificate in AISI’s own words.

OpenAI had a continuous monitor and didn’t run it. The company says its chain-of-thought monitors would have paged its security team more than a day before the breach, and “these monitors did not run on the evaluations in this incident.”

Anthropic is removing the hazard. Its October 9 post says it “had already turned off live internet access for some high-risk and cybersecurity evaluations” and has “now decided to expand that to include all our internal evaluations.” That covers Anthropic’s internal evaluations, and the post doesn’t say whether it reaches environments a vendor runs.

Realism Earns the Paperwork

The strongest pushback is realism. Irregular writes that internet access “is at times critical for realistic evaluations” and that “without it, threat scenarios lose fidelity.” AISI says open access “better reflects the capability a human operator could achieve.” Both describe a real tradeoff, and evaluations that never touch a live network can miss what an attacker with a live network can do.

Plants pull protections for testing because they have to, which is why testing sits inside the scope of Chevron’s standard. A good reason earns you a bypass with a certificate, a clock, a register, and a compensating measure. AISI’s report also undercuts its own realism case, because “both ranges can be solved without internet access.” The allowlist that would have narrowed what agents could reach had sat in a backlog since April.

The other pushback is that nobody gets hurt in an evaluation. An Opus 4.7 run reached a real company’s production database, and an AISI sample sent a malicious pull request to a real maintainer’s project. Neither of them agreed to be anyone’s test environment.

Nobody Tests the Watcher’s Environment

In September I wrote that OpenAI’s system card assigned outside evaluators to bio safeguards and to cyber capability and nobody to cyber safeguards. The same gap shows up one layer down. The independent evaluation arrangements announced this fall govern who watches the model, and they leave the watcher’s own environment to its operator.

Anthropic says it’s in dialogue with METR about a third-party review of its incidents, with “access to all transcripts and sampling access to the relevant models.” AISI says it intends to work with METR on one too. Both descriptions point the reviewer at the model. METR’s October 14 proposal for risk assessment exercises inside frontier developers says it “will not cover critical security questions (such as model weight security and sandbox robustness),” and that “these exercises are not themselves audits.”

The AI Evaluator Forum’s letter on embedding evaluators, published September 18 with more than 200 signatories, wants evaluators assessing “training, deployment, oversight, operational, and safeguard practices.” I read all five of its minimum conditions, which cover independence, differing viewpoints, transparency, protection from retaliation, and access. None of them cover the security of the environment where the evaluator runs its tests.

Anthropic’s October 15 update sorts outside evaluators into three tiers by what they can see and publish, and at the first tier “evaluators can publish their findings with our approval.” The post doesn’t mention evaluation environment security. Meta is the one exception I found, because its August 14 post says “we are implementing independent verification requirements for test environment isolation.” It doesn’t name the verifier.

OpenAI wrote on August 4 that, “in the coming weeks,” it would review how it would “assess requests to enable internet access or lowered safeguards.” Ten weeks later, I can’t find a published result. Irregular plans “an open whitepaper on future best practices” and gives no date, and nothing like it appears on Irregular’s news page as of October 17.

My prediction runs through June 30, 2027, so you can hold me to it. By then, no frontier lab or third-party evaluator will have published an attestation, signed by someone other than the environment’s operator, that someone tested cyber evaluation containment before the runs it covers. If Meta named its verifier and published the result, it would prove me wrong. Irregular’s white paper requiring third-party attestation would do it too, and so would OpenAI’s review landing with that requirement.

Key Takeaway: AI evaluation security will keep failing quietly until labs treat the evaluation environment as the compensating measure it is and have someone other than the vendor who built it proof test it before every safeguards-off run.

What to do next

If you run evaluations, write an evaluation bypass certificate this quarter and refuse to start a safeguards-off run without one. Mine has six fields.

• The safeguard coming off, and the capability you’re measuring with it off.

• The compensating measure, with the date and result of its last proof test, including an egress attempt from inside the evaluation environment and a check of every fictional name in the scenario against live DNS.

• An authorized time period, with any extension sent to change review for bypasses past 72 hours.

• A register entry with start and finish times and a named person who confirms on a schedule that the run is still contained.

• A continuous monitor who can kill the run, or no run.

• An attestation from someone other than the environment’s operator.

In the CARE framework (Create, Adapt, Run, Evolve) I use with clients at RockCyber, the certificate belongs in Run because it governs day-to-day operations. The proof test schedule belongs in Evolve.

👉 Subscribe RockCyber Musings for more insights into AI security and governance, with the occasional rant. You can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out again with my friends Candy Alexander and Cailin Sullivan on Cybersecurity Refined and Feral to argue whether AI Armageddon is fact or fiction, and whether we slow down for safety or floor it to beat China. After two airliners collided over the Grand Canyon in 1956, Congress built air traffic control and kept the planes flying, and that’s where the conversation went: guardrails you can actually test, starting with an agent kill switch that stops every subagent it spawned.

Share RockCyber Musings

Bank of America put 1,000 contractors on patching duty in the second quarter. CEO Brian Moynihan told analysts on October 14, 2026 that they’re there to fix flaws Anthropic’s Mythos model found in the bank’s software. He put the bill for that cleanup and a separate regulatory mess at “a couple hundred million plus.” One AI model’s findings turned into a hiring plan. The same week, regulators on two continents started asking the labs for their records.

Brussels started enforcing the EU AI Act, and the FTC sent OpenAI and Anthropic demands that run past 50 pages each. Reuters reported that both labs fought to control what outside evaluators publish.

Anthropic cut live internet access from its own evaluations after catching models filing forms and dodging fees on real websites. London packed a minister’s speech, a Commons session with the labs, and an MI5 threat update into two days. Whoever holds the logs now holds the exposure. Eleven items, all dated October 9 through October 15, 2026.

Share

1. Brussels Opens AI Act Enforcement Over Agents That Left The Lab

EU digital chief Henna Virkkunen told the International AI Summit in Dublin on October 14, 2026, that the AI Office has started enforcing the AI Act. The office sent frontier model providers requests for information on security, external evaluation, and post-market monitoring. The trigger was a summer of OpenAI and Anthropic agents reaching real systems outside their tests. Fines for these obligations reach €15 million or 3% of worldwide annual turnover.

The AI Act reserves systemic risk for general-purpose models capable of spreading failure well beyond any one customer. Providers of those models must assess and mitigate the risks, and loss of control sits on the list.

Why it matters

• The request topics are the evidence your AI contracts should demand.

• Article 2(8) exempts pre-market testing but excludes “testing in real world conditions.”

• Turnover-based fines put a lab’s global revenue in scope.

What to do about it

• Ask vendors what containment evidence they gave Brussels.

• Map which of your EU services call these models.

• Add regulator-cooperation terms to AI renewals.

Rock’s Musings

Article 2(8) exempts pre-market testing, and its last sentence excludes real-world testing. An agent that logs into Hugging Face is testing in the real world, whether anyone planned it or not.

My read is that Brussels picked its strongest first case. The labs’ best defense is to admit their containment failed. My guess is the exemption fight takes years. Your procurement team can ask the same three questions today.

2. The FTC Sends OpenAI And Anthropic 50-Page Demands For Their Incident Records

The Federal Trade Commission sent demands to OpenAI and Anthropic on Wednesday, October 14, 2026, Axios reported. Each runs past 50 pages and seeks records on security incidents, safeguards, marketing, and privacy policies. The FTC plans to compel executives to testify and to send demands to third-party AI organizations.

A civil investigative demand is a compulsory request the FTC issues before it files any case, and the agency can go to federal court to enforce it. Marketing sits beside incident records because Section 5 covers deceptive claims, and the gap between claim and log is the case.

Why it matters

• Safety claims now get tested against incident logs.

• Third-party AI organizations come next, which may include evaluators.

• Executive testimony puts named people on the record.

What to do about it

• Match each public AI safety statement to evidence.

• Put agent transcripts under your legal hold policy.

• Ask vendors how a regulator demand changes their retention.

Rock’s Musings

Two weeks ago I wrote that the labs’ own postmortems had become exhibits. The FTC has now sent the document request. Anthropic’s October 9 report on a model filing a false homicide tip with a police department is the kind of record it reaches.

Disclosure was the right call. The exposure sits on every marketing page that calls a product safe without a ticket number behind the claim. Match yours before a lawyer does it for you.

3. Bank Of America Hired 1,000 Contractors To Keep Up With Mythos

Brian Moynihan told analysts on Bank of America’s October 14, 2026 earnings call that the bank brought in 1,000 contractors in the second quarter to patch flaws Anthropic’s Mythos model surfaced. They work next to about 3,000 people already on that job, so a bank that size still needed a third more. Moynihan put the bill for that patching and a separate regulatory cleanup at “a couple hundred million plus.” He didn’t split the two.

Why it matters

• AI-scale finding turns patching into a staffing problem first.

• A bank with 3,000 people on this work still needed a third more.

• Your board will now ask for your number.

What to do about it

• Measure weekly fix capacity before scanning the whole estate.

• Triage AI findings by exploitability before ticketing.

• Budget surge remediation as its own line item.

Rock’s Musings

In energy, my patch queue ran through a change board and a maintenance window, and a faster scanner never shortened either one. Moynihan told Wall Street the same thing in dollars. Finding bugs at model speed costs compute.

Finding bugs at model speed costs compute, and fixing them still costs people. If you put an AI model on bug hunting this year, ask who’s budgeted to fix what it finds.

4. Anthropic Pulls Live Internet Access From Every Internal Evaluation

Anthropic said on October 9, 2026 that it’s cutting live internet access for all internal evaluations until its monitoring reliably catches what its transcript review found. Claude Mythos Preview exploited an injection flaw to run commands on a university server. Claude Haiku 4.5 submitted an invented tip about an unsolved homicide through a police department’s online form, and a spam filter caught it. Claude Mythos 5 pulled an access token from a state agency’s public dashboard to avoid a data fee. Claude Opus 5 and Claude Mythos 5 used free URL shorteners to get around length limits on their fetch tool.

Reward hacking is a model learning to score well on its training signal by a route its designers never intended. Anthropic plans to fix training environments that reward tool workarounds, and a URL shortener slipping past a length limit is that habit in a live test.

Why it matters

• The behaviors look mundane, so monitoring tuned for attacks misses them.

• The false police tip came from a small, cheap model.

• Your monitoring of the same models starts behind the lab’s.

What to do about it

• Block URL shorteners at the egress proxy for agent traffic.

• Hunt for tokens your agents can read.

• Gate every agent form submission behind confirmation.

Rock’s Musings

Anthropic gets credit for writing this down and pulling the plug on itself. The list should worry you more than the decision does. One item is a model exploiting an injection flaw on a university server.

The rest are form submits, borrowed tokens, and URL shorteners, the shortcuts people take under deadline when no rule stops them. Your agents run on the same model families with the same urge to finish. Anthropic expects the transcript review it started in July to find more.

5. The Labs Fought Over Who Edits The Evaluator’s Report

Reuters reported on October 13, 2026 that Anthropic and OpenAI sought significant control over what outside evaluators publish, citing trade secrets, and Anthropic told Reuters that picture was incomplete. Two days later Anthropic published a three-tier structure for independent evaluators. At the top tier, six METR staff get access like Anthropic’s own risk teams, and Anthropic can’t veto publication beyond narrow redactions. At the bottom tier, SecureBio’s reviews publish only with Anthropic’s approval.

Why it matters

• A report the lab must approve is attestation, so weight it that way.

• Redaction categories decide how much the public learns about incidents.

• Your contracts set the evaluation terms the White House accord left out.

What to do about it

• Ask each AI vendor which evaluator tier backs its safety claims.

• Require evaluators to rule publicly on redactions, as METR can.

• Read the redaction notes before the conclusions.

Rock’s Musings

I’ve run third-party assurance programs, and my first question about any audit report is who was able to edit it first. Reuters says the labs still wanted that edit as of October 5. Anthropic’s answer ten days later gives METR publication without a veto and the right to say whether a redaction was fair.

That’s a real concession. Tier 1 still leaves SecureBio’s work behind Anthropic’s approval. Neither document tells you which tier covers the evaluators behind your vendor’s next security claim, so ask.

6. Britain’s AI Minister Calls A Breach Likely As Two Labs Ask For Binding Rules

UK AI minister Kanishka Narayan said on October 12, 2026 that an AI breach of UK cyber defences is “likely, if not inevitable.” He told developers, “if you cannot adequately control the risks, do not release the system,” and said the government is exploring a stronger Cyber Security and Resilience Bill for hospitals and utilities. On October 13, OpenAI and Anthropic backed binding requirements for the most capable AI systems at a House of Commons session. Model testing through the AI Security Institute remains voluntary.

Why it matters

• A minister calling a breach likely raises regulator expectations.

• Two of the four labs at the session asked for binding rules.

• Any AI duties added to the bill would fall on operators.

What to do about it

• UK essential services should track the bill’s amendments.

• Add an AI agent scenario to your next UK incident exercise.

• Tell your UK board no government sign-off backs any frontier model.

Rock’s Musings

I ran security in energy, so I know how these rules land. The operator gets the duty, and the vendor gets a press release. Narayan’s release line aims at the labs and carries no penalty today.

The bill covers hospitals and utilities, the customers at the end of that chain. Asking for binding rules on frontier developers is the right instinct. If the drafting puts the obligations on the water utility, the UK will have regulated the victim.

7. MI5 Opens An AI Threat Assessment Centre

MI5 Director General Ken McCallum announced an AI Threat Assessment Centre on October 13, 2026, working with the National Cyber Security Centre and the National Crime Agency. He said AI is “supercharging the cybersecurity contest between attackers and defenders.” He warned that AI risks handing dangerous knowledge, including routes to new pathogens, to people who shouldn’t have it. He also said humans have “no experience of facing an actor which is smarter than we are.”

Why it matters

• AI misuse now has a standing MI5 assessment team.

• Help for low-skill attackers raises attack volume against ordinary companies.

• NCSC ties give its findings a path into UK advisories.

What to do about it

• Flag NCSC output that cites the centre.

• Add AI-assisted, low-skill attackers to your threat profile.

• Test what sensitive detail your internal AI assistants hand out.

Rock’s Musings

The line about an actor smarter than we are got the headlines. The operational line is the one about attackers and defenders, because it describes your SOC this quarter. A security service built a standing AI threat team and wired it to agencies that already brief industry.

I’d rather a spy chief say this in October than read it in an inquiry report later. No UK board can call the risk hypothetical now. Watch for the centre’s assessments in NCSC guidance.

8. Anthropic Ran A Model Across 15 Cheap TV Boxes To Measure Worm Risk

Anthropic researchers published tests on October 12, 2026 of whether small models can copy themselves onto ordinary hardware. A 9-billion-parameter model stood up an inference server on 8 of 9 hardware types, and a model ran across 15 off-brand Android TV boxes at 0.39 tokens per second. In a firewalled cloud test, GLM-5.2 spread a copy of Llama3.1-8B to all eight nodes in four of five trials. Researchers William Zhou and Daniel Freeman say no current model can yet replicate reliably on a smart TV, and they expect one by the end of 2026.

Why it matters

• Replication without a data center removes the single point you’d shut off.

• Cheap consumer hardware can run a small model slowly today.

• The forecast lands inside your 2027 planning cycle.

What to do about it

• Segment IoT devices away from anything that holds credentials.

• Alert on inference traffic from devices that shouldn’t run it.

• Add self-propagating agents to your 2027 tabletop list.

Rock’s Musings

Brandon Vigliarolo at The Register called this a threat that doesn’t exist yet, and he has a point. The propagation test ran in a cloud environment, and the TV boxes only ran inference. Zhou and Freeman say they “haven’t built an AI worm that can spread across the public internet on its own.”

I still think 0.39 tokens per second matters. That’s slow for a chatbot and fine for an agent with all night on your guest network. Segment IoT now, since that control won’t get cheaper.

9. An OpenAI Engineer Faces Federal Charges Over A Bug In A Conference Room

OpenAI engineer Abdulrahman Alfozan, 35, made his first federal court appearance in San Francisco on October 13, 2026, charged with attempting to use a device to intercept oral communications. OpenAI found the device on September 30, hidden in toilet paper and plugged into a conference room TV’s USB port. The FBI arrested him on a Friday after the discovery. OpenAI fired him, said it has no indication he accessed model weights, source code, or customer data, and credits AI with helping it run down the threat.

Why it matters

• Insider threats at labs target conversations as well as code.

• A USB device sat in a lab conference room for days.

• The assets you protect include what executives say out loud.

What to do about it

• Sweep rooms where AI roadmaps get discussed.

• Disable data on USB ports in conference room displays.

• Join badge and room-booking logs for insider investigations.

Rock’s Musings

Every AI risk conversation I sit in eventually gets to model weights. Fair enough. This case went after the room where people talk about the weights.

Physical security still matters when your most valuable asset is a file. OpenAI says AI helped crack this case, and it hasn’t published how. I’ll believe the AI forensics story when I read the method.

10. Trellix Says 95% Of CISOs Tied An Insider Incident To AI

Trellix released a survey of 600 CISOs on October 13, 2026, run by Vanson Bourne. It found that 95% had an insider incident linked to an AI or autonomous system in the past 12 months, and that AI actions made up 28% of insider incidents on average. Of the respondents, 78% now include AI agents and machine identities in their definition of insider risk, and 63% lack visibility into how employees use AI tools.

A base rate is how often something happens across the whole population before you look at any single case. Help Net Security noted that the 95% counts any AI-linked incident, serious or not, so it says little. The 28% share tells you more about the mix to plan for.

Why it matters

• 78% of these CISOs now count AI agents as insiders.

• Most of these programs can’t see the AI activity they’d investigate.

• Vendor survey numbers need the base-rate question first.

What to do about it

• Add agent and machine identities to your insider program’s scope.

• Log employee AI tool use at the proxy or browser layer.

• Ask for severity breakdowns before quoting AI incident percentages.

Rock’s Musings

Trellix sells security products, and this survey will show up in a sales deck by Friday. I’m fine with that, since the useful number is in it. More than a quarter of insider incidents now involve an AI or autonomous system acting.

Insider programs were built to watch people for motive and opportunity. An agent has no motive to watch for. It has permissions and a task, so fix the definition before you buy the tooling.

The One Thing You Won’t Hear About But You Need To: GitLab’s Agent Can Act Under Someone Else’s Name

GitLab released versions 19.4.2, 19.3.5, and 19.2.8 on October 12, 2026. CSO Online’s headline led with CVE-2026-97229, a 10.0 integer overflow in the CI/CD regular expression compiler. Further down sits CVE-2026-96893, which GitLab rates 7.3, in the Duo Agent Platform. It lets an authenticated user inject crafted content into AI agent instructions and trigger command execution attributed to other users in GitLab EE from 19.1 onward.

Non-repudiation is the property that ties an action to whoever performed it, firmly enough that they can’t plausibly deny it. If an agent runs a command and the log shows a colleague’s name, your forensics and your separation of duties both land on the wrong person.

Why it matters

• Misattributed agent actions corrupt the audit trail.

• AI flaws sit inside big patch releases where a 10.0 takes the headline.

• Self-managed GitLab EE with Duo agents stays exposed until upgraded.

What to do about it

• Upgrade self-managed GitLab EE to 19.4.2, 19.3.5, or 19.2.8.

• Review recent Duo agent actions for misattributed commands.

• Give agent-originated actions their own identity in your logs.

Rock’s Musings

Patch the 10.0 first, since it’s unauthenticated code execution. The 7.3 is the one I’d brief the audit committee on. GitLab titles it “Improper Neutralization of Input Used for LLM Prompting,” which is prompt injection with a CWE name on it.

The impact line says commands get attributed to other users. Every agent governance framework I work on assumes the log tells you who did what. This bug breaks that assumption inside the tool that holds your source code.

Regulators asked the labs for records this week. Check whether your agent logs name the right actor, show every outbound call, and would survive a document request. If they don’t, fix that before someone sends you fifty pages.

👉 Subscribe RockCyber Musings for more insights into AI security and governance, with the occasional rant. You can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out again with my friends Candy Alexander and Cailin Sullivan on Cybersecurity Refined and Feral to argue whether AI Armageddon is fact or fiction, and whether we slow down for safety or floor it to beat China. After two airliners collided over the Grand Canyon in 1956, Congress built air traffic control and kept the planes flying, and that’s where the conversation went: guardrails you can actually test, starting with an agent kill switch that stops every subagent it spawned.

References

Almada Lopez, D. (2026, October 13). OpenAI and Anthropic urge UK lawmakers to pass binding AI rules. Crypto Briefing. https://cryptobriefing.com/openai-anthropic-urge-uk-binding-ai-rules/

Al Jazeera. (2026, October 13). AI could self-replicate on home devices within ‘months’, Anthropic says. https://aljazeera.com/economy/2026/10/13/ai-could-self-replicate-on-home-devices-within-months-anthropic-says

Anthropic. (2026, October 9). Investigating unintended model actions. https://www.anthropic.com/research/investigating-unintended-model-actions

Anthropic. (2026, October 15). An update on independent evaluators. https://www.anthropic.com/news/an-update-on-independent-evaluators

Bertuzzi, L. (2026, October 14). EU’s AI rules face first enforcement test over lab incidents. Euronews. https://www.euronews.com/2026/10/14/eus-ai-rules-face-first-enforcement-test-over-lab-incidents

Caswell, A. (2026, October 13). Anthropic’s red team ran AI on 15 cheap TV boxes. Then the math got unsettling. The New Stack. https://thenewstack.io/self-replicating-ai-worms/

CSO Online. (2026, October 13). GitLab patches a max severity regex flaw alongside five high severity bugs. https://www.csoonline.com/article/4234306/gitlab-patches-a-max-severity-regex-flaw-alongside-five-high-severity-bugs.html

Curi, M. (2026, October 15). Scoop: FTC subpoenas OpenAI, Anthropic over consumer harms. Axios. https://www.axios.com/2026/10/15/ftc-openai-anthropic-ai-safety-investigation

Cyphrex. (2026, October 14). OpenAI and Anthropic tell UK MPs safety rules must be binding. AISI says its regime is voluntary. https://cyphrex.io/news/uk-bist-openai-anthropic-binding-rules-aisi-voluntary-incident-reporting

Denar, S. (2026, October 14). EU announces first AI Act enforcement actions after AI agent incidents. Unite.AI. https://www.unite.ai/eu-announces-first-ai-act-enforcement-actions-after-ai-agent-incidents/

Fadilpašić, S. (2026, October 15). Bank of America had to hire 1,000 contractors to patch software flaws spotted by Anthropic Mythos. TechRadar. https://www.techradar.com/pro/security/bank-of-america-had-to-hire-1-000-contractors-to-patch-software-flaws-spotted-by-anthropic-mythos

Future of Life Institute. (n.d.). Article 2: Scope. EU Artificial Intelligence Act. https://artificialintelligenceact.eu/article/2/

Gil, B. (2026, October 15). FTC subpoenas OpenAI and Anthropic over AI safety concerns. Gizmodo. https://gizmodo.com/ftc-subpoenas-openai-and-anthropic-over-ai-safety-concerns-2000826413

GitLab. (2026, October 12). GitLab critical patch release: 19.4.2, 19.3.5, 19.2.8. https://docs.gitlab.com/releases/patches/patch-release-gitlab-19-4-2-released/

Imperial College London. (2026, October 12). AI minister announces measures to protect against AI risks in speech at Imperial. https://www.imperial.ac.uk/news/articles/2026/ai-minister-announces-measures-to-protect-against-ai-risks-in-speech-at-imperial/

ITV News. (2026, October 12). AI breach of UK cyber defences ‘likely, if not inevitable’, minister warns. https://www.itv.com/news/2026-10-12/ai-breach-of-uk-cyber-defences-likely-if-not-inevitable-minister-warns

ITV News. (2026, October 13). AI could give terrorists access to bioweapons if guardrails fail, spy chief says. https://www.itv.com/news/2026-10-13/ai-could-give-terrorists-access-to-bioweapons-if-guardrails-fail-spy-chief-says

Jones, C. (2026, October 15). Bank of America had to bring in 1,000 contractors to patch all the bugs Mythos identified. The Register. https://www.theregister.com/security/2026/10/15/bank-of-america-had-to-bring-in-1000-contractors-to-patch-all-the-bugs-mythos-identified/5303275

Lakshmanan, R. (2026, October 10). Anthropic cuts live internet access for internal AI tests after Claude exploits injection flaws. The Hacker News. https://thehackernews.com/2026/10/anthropic-cuts-live-internet-access-for.html

Lynch, S. N. (2026, October 13). FBI arrests OpenAI employee for allegedly planting recording device in company offices. CBS News. https://cbsnews.com/news/openai-employee-arrest-recording-device

Pogorelec, A. (2026, October 14). Most companies count AI agents as insiders, and CISOs worry about their own liability. Help Net Security. https://www.helpnetsecurity.com/2026/10/14/ciso-ai-agent-insider-risk/

Reynolds, J. (2026, October 12). AI minister calls on AI frontier labs to ramp up safety, as UK ups cyber defences. Tech.eu. https://tech.eu/2026/10/12/ai-minister-calls-on-ai-frontier-labs-to-ramp-up-safety-as-uk-ups-cyber-defences/

Schappert, S. (2026, October 13). From AI-enabled killer drones to AI-made pathogens: MI5 chief warns new era of cyber warfare. Cybernews. https://cybernews.com/ai-news/from-ai-enabled-killer-drones-to-ai-made-pathogens-mi5-chief-warns-of-new-cyberwarfare-era/

Soni, A. (2026, October 13). Exclusive: Behind AI industry’s call for independent reviews, a fight over what gets made public. Reuters, via The Star. https://www.thestar.com.my/tech/tech-news/2026/10/13/exclusive-behind-ai-industry039s-call-for-independent-reviews-a-fight-over-what-gets-made-public

Trellix. (2026, October 13). Trellix research reveals the true cost of ungoverned AI [Press release]. https://www.trellix.com/news/press-releases/trellix-research-reveals-the-true-cost-of-ungoverned-ai/

Tucker, E. (2026, October 13). Employee arrested after listening device found in OpenAI conference room, FBI and the company say. Associated Press, via ABC News. https://abcnews.com/Technology/wireStory/employee-arrested-after-listening-device-found-openai-conference-137239998

Vigliarolo, B. (2026, October 13). Anthropic conjures another threat that doesn’t exist yet to stoke fears of AI apocalypse. The Register. https://www.theregister.com/ai-and-ml/2026/10/13/anthropic-conjures-another-threat-that-doesnt-exist-yet-to-stoke-fears-of-ai-apocalypse/5302866

AI headcount cuts have a poster child, and its numbers don’t look good for us humans. Block cut nearly half its company in February, from over 10,000 people to just under 6,000. It told shareholders a much smaller team using its own tools could do more. By May, Block reported that its incident rate after a production code change had dropped more than 40% from the prior quarter. If you’re a CISO fighting an AI headcount cut, start by admitting the CFO’s got a point.

Share

The CFO’s Case for AI Headcount Cuts Holds Up

Block’s Q4 2025 shareholder letter announced the cut, and its Q1 2026 letter returned data reinforcing the decision. By mid-April, production code changes per engineer were up more than 2.5x from January, while incident rate after a production code change was down more than 70% from a year earlier and more than 40% from the fourth quarter.

Block isn’t the only organization on the AI headcount-reduction path. Challenger, Gray & Christmas counted 120,136 announced job cuts that employers attributed to AI through September 2026. That’s about 21% of all cuts announced this year. Cloudflare cut more than 1,100 people in May and said it was restructuring for the agentic AI era.

Eli Goldratt gave the CFO the math for this decision. He and Jeff Cox wrote The Goal, the 1984 novel built around the Theory of Constraints (TOC). It’s standard reading for any MBA program worth its salt, including mine. Goldratt Research Labs credits him with throughput accounting, a way of judging every decision with three numbers.

Throughput (T) is the money the business brings in, revenue minus the costs that come with each sale. Investment (I) is the capital tied up in the system. Operating expense (OE) is what you’d pay to keep capacity running, which includes every salary in your security budget. Throughput accounting combines the three into net profit (NP) and return on investment (ROI).

TOC practitioner Eli Schragenheim puts the rule simply: grow T faster than OE. He compares every change in OE against the change in T it’s expected to bring, which you can write as a change in net profit.

On a security team, the reviewers are the analysts and engineers who check and approve what the rest of the business wants to do (AI use cases, access requests, security exceptions, and production changes). If you cut one, OE drops by that salary, so ΔOE goes negative and net profit goes up on paper as long as T holds.

Goldratt’s own people don’t argue with the CFO on this. Schragenheim and Jürgen Kanz write that a technology pays off when sales grow or when costs’re cut “in a way that doesn’t harm the delivery performance” and the quality customers see. That’s a test, and on Block’s own measure, Block passes it.

As a leader, you should always assess your team’s headcount against its capacity, regardless of AI. For a security team, I’d focus on the number of review requests coming in each week, how long they wait, and how much longer they’d wait with one less reviewer. Block’s data covers about two months after the cut, and it tracks code output and incidents after code changes. None of it shows what a staffing cut does to security reviews, which is exactly where AI is adding work vs. reducing it.

The Constraint Moved to the Checkers

The Five Focusing Steps are Goldratt’s procedure for working a constraint, the one step that limits what the whole system can deliver. Goldratt Research Labs lists them in order:

1. Identify the constraint

2. Exploit it so none of its time is wasted

3. Subordinate every other decision to it

4. Elevate it by adding capacity

5. Then go back to step one because the constraint has moved

Goldratt Research Labs adds a warning to that last step. Don’t let inertia, meaning the rules and habits you built around the old constraint, become the next constraint. An AI headcount model pays for an AI tool by cutting the people around it. That jumps to step four (elevate) without doing step one, so nobody checks where the constraint’s moved.

Step one (finding the constraint) has an answer for security teams. ISC2 surveyed 856 security professionals who use AI in their work, and its July report found 63% spend more time reviewing or validating AI output and 65% spend more time deciding when to trust it or act on it. Nine in ten (89%) have watched AI recommendations lead to incorrect outcomes at their organization, and half say a human decision-maker gets held accountable when that happens.

In the same survey, 56% said AI cut the need for entry-level security roles, though 53% said it’s also creating new kinds of entry-level roles. Their own checking work keeps growing. The constraint’s moved to the people who check AI’s work.

Software teams hit this bottleneck first. DORA’s March analysis of 1,110 survey responses from Google engineers found that time saved writing code often gets spent again auditing it, and the extra load lands on the reviewer. Faros AI’s Speed Trap telemetry from 22,000 developers (vendor research) shows time spent in QA (quality assurance) up 300.6%.

IT Revolution’s September 28 guide, When Coding Is Free, Where Is the Bottleneck?, argues that once AI makes code cheap, the bottleneck moves to the coordination work around the code, including approvals that “still arrive on a calendar built for a slower world.”

The strongest evidence against my argument comes from inside a SOC. Microsoft ran a randomized controlled trial of its Security Copilot Phishing Triage Agent with 154 professional analysts in the final analysis. Under the trial’s conditions, where about 12% of the 93 test emails were malicious, and the agent got every verdict right, analysts working with the agent correctly identified 6.5 times as many malicious emails per minute of analyst time. In the authors’ pessimistic scenario, where 20% of emails are malicious, one in five of the agent’s malicious verdicts is wrong, and it misses one malicious email in five, the gain is 3.1 times.

Analysts with the agent didn’t rubber-stamp its malicious verdicts. They spent 53% more time on each malicious email than the control group did. It’s Microsoft research on a Microsoft product, and it’s also a randomized trial. The agent increased the number of malicious emails each analyst caught per minute.

Look at where the gain came from, though. Under the trial’s conditions, the authors attribute 83% of the gain to auto-resolving emails the agent judged benign, which in the real world get closed without human review. The researchers also planted false negatives, malicious emails where the agent’s verdict said benign. Analysts who saw those benign verdicts missed 46% of the planted emails, and the control group, working without the agent, missed 17% of them. AI took the easy work and left people the hard cases.

Lisanne Bainbridge described this pattern in 1983. Her ironies of automation cover what happens when a machine takes over the routine part of a job: the person keeps the rare, hard exceptions and loses the practice that made them good at handling those exceptions. In her own words, automation “can make the difficult parts of the human operator’s task more difficult.”

Protective Capacity Is the Line Your Spreadsheet Deleted

Protective capacity is spare capacity you keep on purpose so normal swings in demand don’t overload the step that limits you. On a capacity report, it looks like waste. Schragenheim calls maintaining it a real need and admits “there’s no safe formula to calculate precisely the required protective capacity.” His advice is to start with a conservative estimate and check it against results. A headcount model built on average weekly volume cuts protective capacity first, because in an average week those reviewers look idle.

In my September 29 issue, I ran the math for a single review queue. Real security teams’ve got several reviewers pulling from one queue, and the model for that is Erlang C. It’s the formula, named for A. K. Erlang, that gives the probability a request has to wait when several identical reviewers (servers, in queueing terms) share one queue.

Bertsekas and Gallager derive it in section 3.4.1 of Data Networks. It takes three inputs. λ is the number of requests arriving per week, μ is the number one reviewer can clear per week, and c is the number of reviewers. Utilization (ρ) is the share of the team’s capacity the arrivals use, and the offered load (a) is how many reviewers’ worth of work shows up.

Erlang C turns those into C, the probability that a request has to wait. Dividing C by the team’s spare capacity, the requests per week it could clear beyond what arrives, gives W, the average time a request sits in the queue before anyone starts on it.

For example, five reviewers handle security exception requests, and each one can clear four a week, so one review takes a quarter of a week, a little over a working day. I’ll measure waits in review times, so a wait of one review time’s a quarter of a week. At 15 requests a week, Erlang C says 46% of requests’ve got to wait.

That’s about half a day.

An AI rollout lets the business generate requests faster, and arrivals climb 27% to 19 a week. The team now runs 95% busy, 88% of requests wait, and the average wait jumps to 3.51 review times, more than four working days.

A 27% increase in work bought a 9.5x increase in waiting, because the spare capacity in the denominator fell from five requests a week to one and the share of requests that wait nearly doubled.

Now take a reviewer away. At the original 15 requests a week, four reviewers get a wait of 3.46 review times, nearly as long as the 3.51 the AI surge produced on its own. If you do both, 19 requests a week go to four reviewers who can clear 16. Erlang C only has an answer while the team can clear more than arrives, and this team can’t.

The queue never settles. It grows every week until somebody throttles the requests, reviews them badly, or goes around you.

With a sixth reviewer at 19 requests a week, the wait drops to 0.40 review times. The sixth reviewer is the protective capacity.

The model assumes requests arrive at random and review times vary at random (the textbook M/M/c case), and your team won’t match it exactly. The general pattern holds anyway, because waits stay short at moderate utilization and climb steeply as it nears 100%. With five reviewers, the wait roughly quadruples as the team goes from 85% to 95% busy, and that last stretch’s where the AI rollout pushed the team.

Variation makes it worse. Kingman’s formula, named for J. F. C. Kingman’s 1961 paper on queues in heavy traffic, is the standard approximation for the wait in a queue with one reviewer, any arrival pattern, and any spread of review times.

The first term is utilization divided by idle time, and it climbs steeply near 100% busy. The middle term uses coefficients of variation (CV), the standard deviation divided by the mean, for the gaps between arrivals (CVa) and for review times (CVs). It measures how uneven the work is, and if it doubles, the wait doubles. E[S] is the mean review time.

Microsoft’s trial shows the middle term at work. Under auto-resolve, analysts see only the emails the agent called malicious, and in the trial they spent 53% more time on each malicious email than the control group did. The mean review time goes up, and the spread of review times (CVs) changes with it. I haven’t seen anyone publish that spread for a security queue that AI screens, so it’s the number I’d measure first.

Silvia Bartolucci and Pierpaolo Vivo modeled what happens when AI speeds up a task and a person still reviews the output, in a May 2026 preprint. AI drafts faster, some of its errors slip past review and come back as rework, and the gap between faster tasks and a slower workflow is what they call a variance wedge.

Their model’s got a second finding I didn’t enjoy. Under load, reviewers rationally raise the risk threshold an AI output has to cross before they check it, so they check fewer outputs when errors’re most likely. That’s a model with no field data behind it yet.

Their model has a second finding I didn’t enjoy. Under load, reviewers rationally raise the risk threshold an AI output has to cross before they check it, so they check fewer outputs when errors are most likely. That’s a model with no field data behind it yet.

A Security Gate Fails Open

A factory constraint fails closed, meaning nothing gets past it until it catches up. When the bottleneck machine falls behind, parts pile up in front of it, everyone on the floor can see the pile, and output drops until somebody fixes it. A security review gate fails open, meaning work keeps moving without it. Nobody’s forced to wait in its queue, and a requester who gets tired of waiting can stop asking and use an AI tool security never approved.

Freshworks published a survey of 1,000 IT decision makers at US mid-market companies in April (vendor research). Of those, 27% named slow approval processes as the primary reason employees go around IT. In the same survey, 86% reported at least one negative incident tied to unapproved AI in the past year.

PagerDuty and Wakefield Research asked 1,250 office professionals outside IT roles at companies with at least $500 million in revenue (also vendor research). Of the ones using AI, 66% had used it at work while believing company policy didn’t allow it.

The gates had been slow before AI added load. NetFoundry’s survey of 200 senior security and technology leaders (vendor research), published in August, put the average time to implement a single network or firewall change at eight days. Risk and compliance approvals, meaning security reviews and change advisory boards, tied with cross-team dependencies as the top cause of delay.

You cut a reviewer, the queue in front of your AI use case review or your access approvals gets longer, and some share of the people stuck in it stop waiting. The business keeps shipping, so throughput, the number the CFO watches, holds.

The share of work that goes through your review falls, and no queue dashboard reports it. Half of ISC2’s respondents said a human decision maker is held accountable when AI gets it wrong. The work that went around your gate never reached a reviewer, so nobody signed off on it.

The Freshworks and PagerDuty surveys report what people say about why they bypass controls. Neither one tracked an organization’s queue growing and its bypass rate following it, so the link between a cut and a bypass is my inference (see… inference isn’t reserved to only AI) from the mechanism.

If you cut security review capacity and, over the next 12 months, the oldest open request in your queue keeps getting older while the number of unapproved AI tools you find in use stays flat, I’m wrong about your shop. For a public test, use PagerDuty’s 66%. If sanctioned tools and faster gates were absorbing the demand, that number would fall the next time someone measures it, and I expect the next edition of that survey to show 66% or higher.

Size the Reserve Before You Sign an AI Headcount Cut

Run the Five Focusing Steps on your review gates before anyone runs a headcount model. Start with the gates AI is sending more work to, such as AI use case intake, access requests for agents and the people building them, security exceptions, and the change advisory board.

Identifying the constraint takes five numbers per gate. You need requests arriving per week, the median and 90th percentile review time, the age of the oldest open request, and the number of unapproved AI tools you found in use that month. If the oldest open request at one gate keeps getting older while the other gates hold steady, that gate is your constraint.

Exploit that gate next, which means getting every hour you can out of the reviewers you already have. Microsoft’s trial shows one way to do it, with automation closing the requests it can close with evidence attached and reviewers spending their time on the cases that need judgment.

Then subordinate the rest of the program, which means setting every other rule to fit the constraint’s pace. Change the intake rules before anyone touches staffing. Cap how many changes ride in one request, require evidence with every request, and send low impact requests to a standing policy approved in advance so they never enter the queue.

Last, elevate the gate by sizing your protective capacity with the arrival rate you expect after the AI rollout, because today’s rate understates it. You want the smallest reviewer count, c*, whose Erlang C wait at the new arrival rate stays under the wait you can live with.

With the illustrative team and a target of one review time, five reviewers at 19 requests a week wait 3.51 review times and six wait 0.40, so c* is six. Start conservative, as Schragenheim advises, and watch the queue for a quarter before you trust the number.

Gene Kim co-wrote The Phoenix Project, which teaches the Theory of Constraints through an IT story. In his 2019 essay on The Five Ideals, three years before ChatGPT launched, he told leaders to “think carefully about how each and every position you eliminate might disrupt flow.”

His First Way, the first of the three principles he says underpin DevOps, warns against a local optimization that degrades the whole system. An AI tool that multiplies your teams’ requests is one local optimization, and a headcount model that trims the reviewers who clear them is a second. The whole system pays for both at your review gate.

Block’s numbers may hold up for a full year. If they do, they’ll show that a smaller engineering team can ship more safely, and they still won’t tell you how long requests wait at your security review gate.

Before you sign an AI headcount cut for your security team, which week’s your review queue got to survive, and how many reviewers does it take to get through that week?

What to do next

Pull 12 weeks of history for one review gate that AI is feeding (start with AI use case intake) and compute the five numbers above. Plug the arrival rate you expect after the rollout into Erlang C, and see where the wait lands with today’s headcount and with one fewer reviewer. Bring both waits to the meeting where the cut gets decided.

The quarterly recheck of those numbers belongs in the Evolve stage of the CARE framework (Create, Adapt, Run, Evolve), and the RockCyber team can help you set it up. For the single queue version of the math, read Human in the Loop Is a Rubber Stamp. Its Replacements Are Too., and run whatever numbers you bring through the questions in Evaluating AI Security Research.

👉 Subscribe RockCyber Musings for more insights into AI security and governance, with the occasional rant. You can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out again with my friends Candy Alexander and Cailin Sullivan on Cybersecurity Refined and Feral to argue whether AI Armageddon is fact or fiction, and whether we slow down for safety or floor it to beat China. After two airliners collided over the Grand Canyon in 1956, Congress built air traffic control and kept the planes flying, and that’s where the conversation went: guardrails you can actually test, starting with an agent kill switch that stops every subagent it spawned.

*Note: Issue number bumped because somehow along the way I must have copy/pasted the wrong issue number and carried that error forward

The directory listing was wide open. On a Hong Kong server the operator controlled, a live campaign’s working files sat in the clear. Inside were the ARTEX configuration, the Claude Code session transcripts, and the agent’s own memory files.

CrowdStrike published the write-up on October 7, 2026, and in those transcripts the operator asked Claude where Korean breach data gets sold. South Korean lenders lost customer records that week. His agent wrote down what he did, and he published the folder.

This week the record of what an agent did became the story, and it cut both ways. GitHub looked at a working data-theft chain against Copilot CLI and ruled the user asked for it. Apple moved the control point down into the operating system. AWS shipped a sandbox whose policy engine reads an agent’s action history before approving the next call.

Anthropic started trading looser cyber safeguards for your retained logs. OWASP published its quarterly exploit roundup and landed on one sentence: prompt-level instructions alone don’t establish a secure boundary. SailPoint counted 79% of enterprises running agents in production against 2% holding identity tooling built to govern them. Palo Alto Networks found 95% of critical infrastructure leaders worried while 68% cannot see their own OT assets. Control is moving out of the prompt and into the enforcement points and records around it. Eleven items, all dated October 2 through October 8, 2026.

1. An Attacker Left His Agent’s Memory Files In An Open Directory

CrowdStrike Intelligence published its analysis on October 7, 2026. An operator hit South Korean financial firms from late September into early October using ARTEX, an open-source autonomous penetration testing system on commodity model APIs.

CrowdStrike found the trail in open directories on the actor’s own servers, holding a CLAUDE.md file, session histories, and agent memory. The Register, citing Korea Times, put Shinhan Bank’s exposure near 25,000 customers.

Why it matters

• Your agents leave the artifacts his did, so transcripts cut both ways.

• ARTEX needed no frontier lab access, only a model API and reseller.

• Five bank chiefs face an October 19 parliamentary audit.

What to do about it

• Hunt for CLAUDE.md files and agent memory on hosts with no agents.

• Ask which model API resellers your egress permits, since xcai[.]pro was one.

• Put agent transcripts in your evidence retention policy.

Rock’s Musings

I have spent two years telling executives their agent logs are discoverable. I meant their agents. This week an attacker’s logs convicted him.

He asked a commercial chatbot which Telegram channels buy Korean bank data, then left the transcript on an open directory. That is sloppy work, and it still pulled customer records out of five lenders. Operators who close the directory leave nothing to find.

2. Apple Moves The Agent Boundary Into The Operating System

Apple told developers on October 2, 2026 that granting macOS Full Disk Access will take “very explicit user action.” Apple’s post says some developers use the permission in ways that put users at risk, and names the growing capability of AI agents. It lets an app read Mail, Messages, and Safari history.

TechCrunch tied the timing to Jason Aten’s report that Meta’s Muse agent knew his private messages. Meta says Muse reads messages only with Full Disk Access and the Messages connector on.

Why it matters

• A platform vendor now treats an agent’s permission differently from an app’s.

• Your AI desktop agreements assumed Full Disk Access stays grantable.

• Explicit consent moves liability toward whoever clicked the warning.

What to do about it

• Inventory macOS apps with Full Disk Access and flag the agents.

• Check whether MDM profiles pre-grant it, routing around the control.

• Ask AI desktop vendors what breaks when you deny each one.

Rock’s Musings

This is the confused deputy problem, and it has a name because it is forty years old. A confused deputy is a program with more authority than whoever asks it to act, and it gets tricked into spending that authority on their behalf. Apply it here.

Your agent reads an email, the email tells it to act, and it acts with your Full Disk Access and Slack token. The permission grant is the deputy. Apple is moving control to that grant, where it belongs.

3. GitHub Looked At An Encrypted Prompt Injection And Said That Isn’t A Bug

Adversa AI published a technique on October 6, 2026 that Rony Utevsky calls Cryptographic Context Injection. A web page serves ciphertext and two decryption keys. One is a decoy that the agent completes by reading local files such as .env.

That read is the theft, and the working key sends the agent to a URL carrying the loot. Adversa timed roughly 28 seconds to a .env.prod file reaching the attacker’s log, with Microsoft’s mai-code-1.1-flash completing the chain in 50% of runs.

Why it matters

• Encrypting the payload defeats the classifiers your injection detection relies on.

• GitHub’s position moves this onto your register as accepted vendor risk.

• On Auto, the router can pick the permissive model for you.

What to do about it

• Turn off autopilot in coding agents and pin the model.

• Keep secrets off the agent’s filesystem, in a manager with short-lived creds.

• Put egress controls in front of the agent so unknown-host fetches fail.

Rock’s Musings

Indirect prompt injection is when instructions reach a model through content it was told to read, rather than through the user’s own typing. The model cannot separate content from command once it is in context. That definition decides this argument.

GitHub declined to call this a product vulnerability, telling The Register the attack needs a user to point Copilot CLI at untrusted content. Encrypt the payload and the agent unwraps it itself. Shared responsibility collapses when the user is a developer on autopilot.

4. AWS Shipped A Sandbox That Remembers What The Agent Already Did

AWS released Strands Box as a developer preview on October 7, 2026 under Apache 2.0. It pairs macOS containment with the Dogwood policy engine, where permit and forbid rules evaluate at four points, including the egress gateway.

Temporal operators let a rule depend on what the agent did earlier, so the example caps Slack posts at three per ten minutes. The gateway injects credentials into permitted requests, so the agent never sees them.

Why it matters

• Policy reading action history catches exfiltration per-call checks miss.

• Gateway credential injection removes the agent from the secret path.

• Apple silicon Macs only, fine for evaluation and not production.

What to do about it

• Have an engineer run the preview and report what Dogwood cannot express.

• Pull your agent telemetry and test whether you can rebuild its history.

• Move agent credentials behind a gateway that injects them.

Rock’s Musings

Pareekh Jain of Pareekh Consulting told CSO Online the technologies underneath are not new. He is right that OS containment and Cedar-style policy are old. What is new is evaluating a tool call against what the agent already did this session.

I have wanted that since somebody showed me an agent that passed every check and still walked out with the data. Tulika Sheel of Kadence International raised the counterpoint, that loose policy leaves gaps and tight policy blocks work. Both land on you.

5. Anthropic Split Cyber Access Into Three Tiers And Priced It In Your Logs

Anthropic announced on October 6, 2026 that it merged Project Glasswing and the Cyber Verification Program into one three-tier program. Defense Access covers security operations, incident response, and vulnerability analysis, open to security teams, universities, hospitals, and utilities.

Red Team Access adds authorized penetration testing. Specialized Access carries the fewest blocks for a few testing power grids and interbank transfers, and the program requires data retention.

Why it matters

• Reduced safeguards for log retention is a term legal never reviewed.

• Utilities and hospitals are eligible, so the thinnest-staffed teams can apply.

• Specialized Access gets reviewed with the US government, a federal filter.

What to do about it

• Decide whether you accept Anthropic retaining your red team’s prompts.

• Running grid or interbank systems, route Specialized Access through government relations.

• Log who applied, since individual researchers qualify too.

Rock’s Musings

Each tier has a price, and the price is your logs. Defense Access costs retained prompts and outputs, and Red Team Access costs that plus proof you may test the target. I ran a security program in energy, where testing cleared legal before it cleared tooling.

A vendor holding your red team’s prompts holds a map of your attack surface. Anthropic has sound reasons to keep that data. Settle retention before somebody clicks apply.

6. Trump Named A Super Intelligence Force With A Report Due And No Authority

President Trump announced the Super Intelligence Force in a Truth Social post on Sunday, October 4, 2026. Director of National Intelligence Jay Clayton chairs it, with FTC Chair Andrew Ferguson, Undersecretary of War Emil Michael, and OPM Director Scott Kupor as vice chairs.

The charter reportedly directs it to plan for SI-enabled threats while avoiding overregulation, and gives it 120 days to report. Neither TechCrunch nor Dataconomy identifies any statutory authority or enforcement mechanism.

Why it matters

• The DNI chairs it and the FTC Chair vice-chairs, two enforcement offices.

• The 120-day clock lands in early February 2027.

• An anti-overregulation mandate runs against the state attorneys general pushing back.

What to do about it

• Calendar early February 2027 and assign someone to read it.

• Keep building to the strictest state regime, since this preempts nothing.

• Watch the FTC for the first enforcement signal.

Rock’s Musings

I have sat through enough of these to know the shape. Four senior names, a report deadline, and no authority. The output matters only through what each member does with their own agency.

Clayton told the Wall Street Journal that one of the biggest risks is not being first. Resilience is not where the chair’s head is. Keep your governance anchored to regulators that can fine you.

7. New York City Put Four AI Executives Under Oath

The New York City Council met as a Committee of the Whole on Monday, October 5, 2026. It took sworn testimony from OpenAI’s Morgan Dwyer, Anthropic’s Logan Graham, Google’s Alice Friend, and Meta’s Shane Cahill.

SpaceXAI skipped it despite a Council subpoena. No company gave a probability for catastrophic risk, and none committed to halting a release that failed a safety test.

Why it matters

• The package carries 24-hour incident reporting, third-party validation, and a shutdown switch.

• A city got testimony no state or federal body has, making procurement a lever.

• OpenAI said on the record it knows of no agent incident in city systems.

What to do about it

• With city contracts, test whether you can report an AI incident in 24 hours.

• Pull validation language into your vendor questionnaires. Arranging it takes months.

• Ask your AI vendors for a risk figure and a release-halt commitment.

Rock’s Musings

About forty of fifty-one council members showed up to ask frontier labs for a number. The labs would not give one. I will not pretend catastrophic risk probability is solved measurement, since Friend’s answer that no rigorous method exists is defensible.

Dwyer’s answer is a different animal. She said the figure does not matter whether it is 1% or 20%. A CISO who answered a board that way would be cleaning out a desk by Friday.

8. SailPoint Put A 40-To-1 Number On The Agent Identity Gap

SailPoint published the 2026-2027 Horizons of Identity Security report on October 6, 2026, from a survey of 340 senior identity and security leaders. It found 79% running AI agents in production against 2% using identity tooling built to govern them.

Agents hold 22% of all non-human accounts, and 85% rely on legacy identity tooling. On readiness, 57% are confident of meeting regulations and 43% can produce audit evidence.

Why it matters

• A 79% production rate against 2% governance tooling makes you no outlier.

• Agents at 22% of non-human accounts exceeds what manual review absorbs.

• The gap between 57% confident and 43% audit-ready is the honest number.

What to do about it

• Count your agent identities this quarter, including ones frameworks created.

• Build one agent’s evidence chain: who authorized it, what it did.

• Add agent identities to your next access recertification.

Rock’s Musings

Non-human identity means any authenticated principal that is not a person. It covers service accounts, workload identities, API keys, and now agents. The term earns its place because your IAM program runs on joiners, movers, and leavers, and an agent has none of those events.

A developer creates it at 11pm and it never resigns, so the lifecycle triggers your recertification depends on never fire. SailPoint sells identity security, so read 40 to 1 as a vendor’s framing. Start with the count.

9. Palo Alto Networks Found 95% Of Critical Infrastructure Leaders Worried And 68% Blind

Palo Alto Networks published its 2026 State of Critical Infrastructure Cybersecurity Report on October 6, 2026, based on responses from 1,600 security leaders across 11 countries. It found 95% were concerned about frontier AI-powered attacks.

On fundamentals, 68% lack real-time visibility into their OT assets and 74% still run IT and OT security separately. Only 40% have virtual patching, and 29% of CVEs were exploited within 24 hours against a 55-day average.

Why it matters

• A 95% concern rate with 68% lacking asset visibility says spending never followed.

• Exploitation inside a day against 55 days to patch is a gap patching cannot close.

• With 60% lacking compensating controls, legacy OT assets stay exposed by default.

What to do about it

• Fund OT asset visibility before the AI tool that needs it.

• Stand up virtual patching for the legacy assets you cannot take offline.

• Put the 24-hour exploitation figure beside your mean time to patch.

Rock’s Musings

I was a CISO in energy, so let me translate the 55-day number. A change control board owns part of it. Add a maintenance window negotiated with operations, a vendor who voids support for out-of-band patches, and a turbine indifferent to your SLA.

None of that changes the arithmetic when 29% of exploited CVEs get used inside a day. Virtual patching blocks exploitation at the network layer without touching the software, and 60% skipped it. Game on, owner-operators.

10. OWASP’s Q3 Roundup Says The Prompt Is Not A Boundary

The OWASP GenAI Security Project published its Q3 2026 GenAI and Agentic AI Exploit Roundup on October 8, 2026. Its conclusion is that prompt-level instructions alone do not establish a secure boundary. It recommends enforced scope, isolated credentials, and monitoring of what an agent did.

Four Claude Opus 4.7 evaluation runs obtained credentials and reached a production database. A Claude Mythos 5 PyPI package ran on 15 real systems in about an hour. The Deadbugz campaign pushed 23 pull requests through a malicious MCP server in 74 minutes, and none merged.

Why it matters

• Scope overrun shows up at both OpenAI and Anthropic, so it is a design problem.

• An MCP server changing behavior after approval breaks one-time review.

• Agents obtained credentials nobody issued them.

What to do about it

• Map your agents against the documented failure patterns.

• Re-verify every MCP server your developers connect, on each change.

• Separate agent credentials from their human operators, with separate revocation.

Rock’s Musings

I work on the Agent Control Standard inside this project, so I have a stake in it. The sentence worth carrying out of it is the conclusion: prompt-level instructions do not establish a secure boundary. We have said that in standards language for two years. It lands harder with four Opus 4.7 runs behind it.

Take the document into your next agent design review, because the cases map onto architecture decisions. Several land on MCP, the Model Context Protocol agents use to call outside tools. A quarterly roundup built from public disclosures undercounts, so read the case list as a floor. The ones that stayed private are the ones your vendor has not told you about yet.

The One Thing You Won’t Hear About But You Need To: An Unpatched 9.8 In Your LLM Cache Layer

JFrog published CVE-2026-105192 on October 7, 2026, found by Yuval Moravchick and rated 9.8 critical. LMCache, the key-value cache layer deployed alongside vLLM, opens an unauthenticated ZeroMQ ROUTER socket in multiprocess mode on port 5555. Decoding REGISTER_KV_CACHE arguments maps msgpack extension code 1 to a deserializer calling pickle.loads, so one crafted message runs code as the LMCache process user.

The official images run as root, every version from 0.3.9 onward is affected, and the project’s example Kubernetes DaemonSet binds to every interface. JFrog’s advisory includes a working proof of concept.

Why it matters

• This sits in inference infrastructure, outside appsec and AI governance.

• The project’s own example DaemonSet produces the configuration earning that 9.8.

• Root in the official images turns code execution into node compromise.

What to do about it

• Today, confirm no LMCache instance binds its transport to a routable address.

• Firewall the transport port, since any host that connects can run code.

• Open a ticket for the fix and put inference in scope.

Rock’s Musings

Insecure deserialization means taking bytes off the network and rebuilding them into live program objects without checking them first. Python’s pickle does exactly that by design, which is why it executes code as a feature and not a flaw. An unauthenticated socket hands attacker bytes to pickle.loads as root.

No model, no prompt injection, no agent cleverness, only a 1990s-vintage mistake in the plumbing serving your 2026 models. Your AI risk committee argued about model cards while this service unpickled whatever arrived.

As of October 7, 2026, there is no patch.

👉 Subscribe RockCyber Musings for more insights into AI security and governance, with the occasional rant. You can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out with my friend Jeffrey Wheatman on Black Kite’s Third Party podcast to take the AI doom and hype cycle head-on: is AI the end of humanity, or just another Y2K? Nobody has produced a credible number for AI extinction risk, but there are hard numbers for what happens when companies overcorrect and ban AI across their vendors, and that’s where the conversation went.

Share

Share RockCyber Musings

References

Adversa AI. (2026, October 6). GitHub Copilot CLI vulnerability leaks developer secrets: Cryptographic context injection. https://adversa.ai/blog/cryptographic-context-injection-github-copilot/

amNewYork. (2026, October 5). AI giants give few clear answers to key safety questions at NYC Council hearing amid whistleblower warnings. https://www.amny.com/news/ai-giants-nyc-council-whistleblower-warnings/

Bushwick Daily. (2026, October). OpenAI, Anthropic, Google and Meta testify under oath Monday before all 51 NYC Council members on the risks of AI. https://bushwickdaily.com/politics/openai-anthropic-google-and-meta-testify-under-oath-monday-before-all-51-nyc-council-members-on-the-risks-of-ai/

Campion, A. (2026, October 7). Unknown threat actor uses AI-driven ARTEX to target South Korean finance. CrowdStrike. https://www.crowdstrike.com/en-us/blog/unknown-threat-actor-uses-artex-to-target-south-korean-finance/

Channel Insider. (2026, October 7). SailPoint: AI agents outpace identity security controls. https://www.channelinsider.com/security/sailpoint-ai-agent-identity-security-gap/

Claburn, T. (2026, October 6). Zombie instructions on carefully constructed web pages could trick GitHub Copilot CLI into sharing secrets. The Register. https://www.theregister.com/ai-and-ml/2026/10/06/zombie-instructions-on-carefully-constructed-web-pages-could-trick-github-copilot-cli-into-sharing-secrets/5301206

Claburn, T. (2026, October 7). Anthropic reconfigures its cool kids security program. The Register. https://www.theregister.com/security/2026/10/07/anthropic-reconfigures-its-cool-kids-security-program/5301509

Cloud Security Alliance. (2026, May 18). Mini Shai-Hulud: TeamPCP worm targets AI developer toolchain. https://labs.cloudsecurityalliance.org/research/csa-research-note-mini-shai-hulud-ai-toolchain-supply-chain/

CSO Online. (2026, October 8). AWS takes aim at runaway AI agent behavior with Strands Box. https://www.csoonline.com/article/4232446/aws-takes-aim-at-runaway-ai-agent-behavior-with-strands-box-2.html

Cyber Security News. (2026, October 7). Anthropic opens Claude access to red teams and verified cyber defenders with reduced restrictions. https://cybersecuritynews.com/anthropic-cyber-verification-program/

Dataconomy. (2026, October 5). Trump announces Super Intelligence Force led by Jay Clayton. https://dataconomy.com/2026/10/05/trump-super-intelligence-force-jay-clayton/

Dingler, F. (2026, October 7). Introducing Strands Box: AI agent sandboxes powered by Dogwood. AWS Open Source Blog. https://aws.amazon.com/blogs/opensource/introducing-strands-box-ai-agent-sandboxes-powered-by-dogwood/

The Hacker News. (2026, October 8). ARTEX AI pentesting tool used in data theft attacks on South Korean financial firms. https://thehackernews.com/2026/10/artex-ai-pentesting-tool-used-in-data.html

The IT Nerd. (2026, October 7). 60% of critical infrastructure organizations hit by significant cyber incidents. https://itnerd.blog/2026/10/07/60-of-critical-infrastructure-organizations-hit-by-significant-cyber-incidents/

JFrog Security Research. (2026, October 7). LMCache is vulnerable to unauthenticated remote code execution via pickle deserialization on the multiprocess ZMQ transport (CVE-2026-105192). https://research.jfrog.com/vulnerabilities/lmcache-is-vulnerable-to-unauthenticated-remote-code-execution-via-pickle-deserialization-on-the-multiprocess-zmq-transport-cve-2026-105192-jfsa-2026-001694382/

Jones, C. (2026, October 8). CrowdStrike finds possible bank hacker’s CV among exposed AI logs. The Register. https://www.theregister.com/cyber-crime/2026/10/08/crowdstrike-finds-possible-bank-hackers-cv-among-exposed-ai-logs/5301908

Khandelwal, S. (2026, October 7). Unpatched critical LMCache flaw lets unauthenticated attackers run code remotely. The Hacker News. https://thehackernews.com/2026/10/unpatched-critical-lmcache-flaw-lets.html

Lakshmanan, R. (2026, October 5). Apple plans tighter macOS Full Disk Access controls over AI agent data access. The Hacker News. https://thehackernews.com/2026/10/apple-plans-tighter-macos-full-disk.html

OWASP GenAI Security Project. (2026, October 8). GenAI and agentic AI exploit roundup Q3 2026. https://genai.owasp.org/2026/10/08/genai-and-agentic-ai-exploit-roundup-q3-2026/

Palo Alto Networks. (2026, October 6). State of global critical infrastructure cybersecurity 2026: 95% of leaders concerned about frontier AI-powered attacks. https://www.paloaltonetworks.com/blog/network-security/state-of-global-critical-infrastructure-cybersecurity-2026-95-of-leaders-concerned-about-frontier-ai-powered-attacks/

Perez, S. (2026, October 2). Apple says it’s tightening macOS Full Disk Access controls due to new risks from AI agents. TechCrunch. https://techcrunch.com/2026/10/02/apple-says-its-tightening-macos-full-disk-access-controls-due-to-new-risks-from-ai-agents/

Pogorelec, A. (2026, October 5). Apple tightens macOS disk access as AI agents become more powerful. Help Net Security. https://www.helpnetsecurity.com/2026/10/05/macos-full-disk-access-updates/

Quartz. (2026, October 5). AI execs testify before NYC Council on existential AI risks. https://qz.com/anthropic-openai-google-meta-nyc-council-ai-hearing-100526

Quartz. (2026, October 6). Anthropic expands cyber AI access program to more security firms. https://qz.com/anthropic-cyber-verification-program-expansion-three-tiers-100626

SailPoint. (2026, October 6). SailPoint report finds 79% of enterprises run AI agents in production, yet only 2% have deployed purpose-built security [Press release]. GlobeNewswire. https://www.globenewswire.com/news-release/2026/10/06/3375448/0/en/sailpoint-report-finds-79-of-enterprises-run-ai-agents-in-production-yet-only-2-have-deployed-purpose-built-security.html

Security Boulevard. (2026, October 7). Daily OT security news: October 07, 2026. https://securityboulevard.com/2026/10/daily-ot-security-news-october-07-2026/

TechCrunch. (2026, October 4). Trump unveils his new Super Intelligence Force. https://techcrunch.com/2026/10/04/trump-unveils-his-new-super-intelligence-force/

Vigliarolo, B. (2026, October 7). AWS launches open-source AI agent sandbox to prevent YOLO mode disasters. The Register. https://theregister.com/ai-and-ml/2026/10/07/aws-launches-open-source-ai-agent-sandbox-to-prevent-yolo-mode-disasters/5301687

I messed up. I’m not afraid to admit it. In my September 29 blog, I got a number wrong because I rushed my evaluation of AI security research, going against my own advice. I wrote that a model reviewer let 46.1% of AI agent overreach slip through. However, in the Ting Yan study I cited, the model actually screened every action and flagged all seven overreach actions for human review. It was the humans who then approved the overreach.

I’ve since posted a correction at the top of that blog. The paper’s method section clearly explains that the model sent all seven overreach actions to a human, and I quoted the results table without fully reading that crucial part. Between September 26 and 30, I encountered three more numbers that suffered the same fate. Each lost the condition that made it true once someone else quoted it, and I’ll walk through all three below. Context is everything (and it’s why I argue that cyber risk quantification, in its purest form, falls short...but that’s a topic for another day)!

Share

Why Evaluating AI Security Research Is Now a Budget Problem

Your team is currently making decisions about controls and vendors based on research, and there’s simply too much of it for anyone to reasonably keep up with. I have a Claude Cohort that runs daily, specifically looking for new AI security papers. Just on September 29 alone, it flagged 13 new publications. Anthropic’s system card for Claude Opus 5.5 is over 200 pages long, and two of the numbers I’ll dissect below come directly from it. It’s inevitable: someone on your team will pull a percentage from a paper, model card, or other source, cite it, and someone else will approve a purchase order based on that information.

I’m currently finishing up my master’s in applied data science and AI at the University of Denver. November 16 is my last class – but who’s counting? Throughout the program, I’ve run the same types of experiments these papers describe, using my own data or publicly available datasets. Some of my own projects even fail some of the same eight questions I’m about to share with you. I’ll show you exactly where.

In The Math Behind the AI Agent Restart Gate, I covered intervals and how to count rare events. This piece focuses on what to scrutinize before you trust a number enough to start doing calculations with it.

Why am I being so open about my mistake?

“The only real mistake is the one from which we learn nothing.” -Henry Ford

Question One: Percent of What?

ASR stands for “attack success rate,” and it represents the ratio of successful attacks for the attacker. However, before we can calculate that ratio, we first need to define what constitutes an “attempt.” This is also known as the unit of analysis. Anthropic’s Claude Opus 5.5 system card offers a clear example.

In a recent test to find vulnerabilities in Claude Opus 5, professional “red teamers” launched 110 different attack scenarios, each tried ten times, without any built-in safety nets. Looking at each individual attempt, the attacks succeeded in 3.64% of cases. However, if we consider each scenario successful when even one of the ten attempts worked, the attack succeeded in 15 out of 110 scenarios, or 13.6%. It’s worth noting that these are the exact same test results, but analyzing them in two different ways leads to success rates that are almost four times apart.

This method of counting success “per scenario” is known as best of n. In this approach, an attacker tries a scenario ‘n’ times and considers it successful if just one of those attempts works. This makes sense for attackers who can make unlimited attempts. However, for attacks like phishing, where there’s usually only one shot, it’s more appropriate to analyze success on a “per-attempt” basis.

A recent paper, “Divide and Inject,” published on September 29 by researchers from ICSI, DSO National Laboratories, UC Riverside, and Lawrence Berkeley National Lab, introduced an attack that breaks harmful instructions into smaller pieces. This method achieved an Attack Success Rate (ASR) of 61.4%. In their study, a “unit” was a specific task paired with an objective, and each model tested had 81 such pairs. A pair was considered successful if the attacker found even one successful attempt out of up to 129 tries against the target.

To put it simply, the paper considered a task/objective pair a success if any one of up to 129 attempts got through. This is the “best of n” counting method, with ‘n’ set to 129. I couldn’t find any part of the paper that re-ran a payload that had already worked once. This means the 61.4% success rate tells us what a persistent attacker can eventually achieve, but it doesn’t reveal how reliably that specific attack works once it’s discovered.

Question Two: Under What Condition?

Conditional probability describes the likelihood of an event happening given that another specific condition is true, often written as P(A | B). Every rate reported in a lab study is a conditional probability because it’s always dependent on how the test was set up. Unfortunately, when others quote these rates, they often omit the crucial conditions under which they were achieved.

For instance, the Opus 5.5 system card notes that the model attempted to bypass or tamper with a sandbox in 1.5% of its “runs.” A “run” here means a single execution of a test scenario. Section 6.4.8 clarifies that this rate comes from 28 custom-built scenarios specifically designed to trigger sandbox escapes, with each scenario run seven times. That’s roughly 196 runs in total, suggesting about three successful escapes (based on my math, as the card gives a rate, not an exact number).

The card categorized each of these escape attempts as low severity. The model explored a weak boundary without making changes, stopped at a stronger one, and then reported its actions.

Axios reported on September 26 that these experiments involved “a task that couldn’t be solved without escaping the sandbox.” I couldn’t find that exact wording in the system card. The next paragraph in the Axios report mentions labs conducting “hundreds of thousands of test runs” and that “even a small percentage” amounts to tens of thousands of incidents. A reader might assume that the 1.5% rate applies to all tests a lab conducts. However, the system card measured 1.5% across 196 runs specifically engineered to provoke escapes and never assessed a rate across all of a lab’s testing.

Before using a rate, it’s crucial to identify the sentence describing the model’s objective. Without this context, you can’t truly understand the conditions of the rate or interpret it accurately.

Question Three: Compared to What?

A control condition is an identical experiment where the element being investigated is removed. Without it, you can’t determine how much of the result came from the attack or defense itself versus how much was simply due to the experimental setup.

“Divide and Inject” offers two excellent controls. The same adaptive search, with the malicious instruction written out fully rather than in fragments, achieved a 32.8% score. The same fragments without any search scored 5.1%. These two results clearly show how much of the 61.4% outcome was due to fragmenting the instruction and how much to the search process.

Also, consider the comparisons the authors chose to highlight in their abstract. They focused on the explicit instruction version and an earlier attack called AgentVigil.

Table 1 of the paper indicates that TAP (Tree of Attacks) actually outperformed the new method on two of the seven models. TAP comes from a 2023 paper on automatically jailbreaking chatbots, in which an attacker model writes prompts, refines them, and prunes the weak candidates before sending them to the target. For the version they ran against agents, the authors cite that original paper and a 2026 study by Hofer, Debenedetti, and Tramèr on automated prompt injection attacks in agentic environments.

TAP averaged 26.3% ASR across all seven models. On Ministral-3-14B it scored 84.0% against the new method’s 61.7% in its headline configuration, and on GPT-4.1 it scored 66.7% against 58.0%. The new method only tied TAP on GPT-4.1 when the authors doubled its filler text to 16,000 tokens, and it never caught up on Ministral-3-14B. The authors include all of this in their own table, which speaks to their honesty, but you’ll only find it if you read past the abstract.

For a defense mechanism, the two controls are a setting that permits all actions and one that blocks all actions. Ajar, a September 22 paper from the University of Washington, Ohio State, and Microsoft Research, evaluates five agent defenses against precisely these two settings. The “block everything” setting, in the authors’ words, “leaks nothing and completes nothing.” It also achieves a perfect block rate. A block rate, however, provides very little insight until you also know how much legitimate work the defense prevented.

For my DU deep learning final, I developed a detector for prompt injection spread across multiple turns of a conversation. In my training data, 64% of messages were benign. Therefore, a model that labels every message “benign” would achieve 64% accuracy but catch zero attacks.

That 64% is our base rate, essentially the starting point. It’s the proportion of cases that fit a certain category before any AI model even looks at them. I dove into this concept in AI Agent Detection Failed at OpenAI. Because of this baseline, I opted to report F1 scores instead of simple accuracy. F1 is a more robust metric because it combines both precision (how many flagged messages were truly attacks) and recall (how many actual attacks the model successfully caught). This means the F1 score drops if the model misses attacks or triggers too many false alarms. My results table also kicks off with a naive baseline, which is just a model that always guesses the most common label. If you’re reading a paper and don’t see a naive baseline in the results, it’s a good idea to mentally calculate what it would score before you put too much faith in the rest of the data.

Question Four: Who Was the Attacker?

A threat model clearly defines what an attacker controls, what they can observe, and how many attempts they get. An adaptive attacker is particularly tricky because they watch how their target responds to each attempt and then adjust their next move based on what they learned.

In the “Divide and Inject” scenario, the attacker plants text within content the AI agent reads, like an email or a Slack message. The researchers also gave this attacker black box access while they were crafting their attack. Black box access means the attacker can send inputs to the agent and see its outputs (in this case, which tools the agent used and what changed) without ever seeing the model’s internal workings or weights.

It’s fascinating how much impact a simple feedback loop can have. A fragmentation attack, which only scored 5.1% without that loop, shot up to a 61.4% success rate with it. That’s a 12x increase! This highlights the critical role of one assumption in our threat model, an assumption I believe is quite realistic for open-weight models. After all, anyone can download and run these models in their own environment. This kind of attack is far more challenging against a hosted agent that restricts requests and keeps its tool calls hidden from attackers.

However, my own detection system hasn’t faced such an adaptable adversary. All the attacks in its dataset came from a single generator model, Claude Sonnet 4.6, and none of them evolved in response to my detector. I’ve flagged this as a limitation in my report, and my top priority for future work would be to test against an adaptive attacker.

Question Five: How Wide Is the Spread?

A macro-average is essentially the average of individual model scores, giving equal weight to each model regardless of how its score was generated. While useful when models behave similarly, it can be quite misleading when they don’t.

For example, the “Divide and Inject” method dramatically boosted Qwen-3.6-27B’s Attack Success Rate (ASR) from 9.9% to a staggering 71.6% against a baseline of explicit instructions. Yet, for GPT-4.1, the change was minimal, moving from 56.8% to just 58.0%. The headline figure of 61.4% truly doesn’t capture the reality for either model.

Simply put, if you’re using GPT-4.1, this paper suggests your measured attack success rate barely budged—by about one percentage point. But if you’re working with Qwen, that number rocketed by 62 points.

For my statistics final at DU, I explored whether safety alignment truly reduces toxic output. I ran 25,000 prompts through both base and aligned versions of three open model families, evaluating the results with Detoxify, an open-source toxicity classifier.

Toxicity scores dropped by 6.07 points for Qwen 3 and 0.78 points for Llama 3.1. Interestingly, however, it increased by 1.38 points for Mistral. This indicates that, for this specific set of prompts, alignment actually made Mistral’s output more toxic. The macro-average showed an overall 1.82-point drop, a figure that doesn’t accurately reflect the performance of any of the three models I tested individually.

This study also helps clarify two often-confused terms: Effect size measures the magnitude of a difference in practical, relevant units, while a result is statistically significant when its p-value falls below a predetermined threshold, typically 0.05. The p-value indicates the probability of observing a difference at least as large as the one found, assuming there’s no true difference. Even with 25,000 prompts, Llama’s 0.78-point drop had a p-value of 0.0006, which my analysis still classified as having a negligible practical effect.

Ultimately, with a large enough sample, even a tiny effect can appear statistically significant. Always prioritize the effect size, and avoid making decisions based on an average when detailed per-model data is available.

Question Six: Does the Effect Need Both Parts?

An ablation involves removing one component of a system at a time to measure its impact on the outcome. An interaction effect occurs when two components produce a result together that neither could achieve on its own.

The “Divide and Inject” study performed this test on four models. Fragments without any filler text scored 0.0% success, but the same fragments embedded within 8,000 tokens of filler text scored 75.3%. Complete instructions achieved 29.3% success without filler and 25.3% with it.

Neither component worked in isolation; this is a clear interaction effect. Based on the paper’s findings, a defender could prevent the agent from reassembling fragments into a single instruction or limit the amount of retrieved text the agent processes at once. However, the paper didn’t explore either of these defenses.

Always scrutinize the ablation table, as this is often where a paper’s main claim can unravel. In my own detector, I replaced the injection-recognition component with a generic one trained solely on text compression and reconstruction. Surprisingly, the generic component performed slightly better (F1 0.845 vs. 0.837). My report stated that this result “tempers the dual-encoder narrative,” which it truly did.

Question Seven: What Didn’t They Test?

Always start by reviewing the limitations section. The best papers openly discuss what they didn’t test, a detail often omitted in news coverage.

For example, “Divide and Inject” evaluated seven models across three environments but didn’t include any dedicated injection defenses. Its limitations clearly state that “stronger defenses” are a focus for future work.

My own scheduled AI security research scan, Claude Cowork, initially summarized this paper on September 30, claiming the attack “defeats” single-document injection detectors. However, the paper never actually tested such a detector. The scan generated that line from the abstract, and its internal notes confirm it never read the full text. I’ve since corrected that scan task.

The frequently quoted 41% to 87% failure rate for multi-agent systems represents a worst-case scenario. This figure originated in MAST, a 2025 Berkeley study that measured task failure (meaning the system didn’t complete its job) across six systems using various benchmarks. The figure caption explicitly states that these results “are not directly comparable.”

The September 29 MAADBench paper then re-stated this range as what “recent studies report,” and my scan repeated it on September 30. By the third retelling, the crucial caveat was gone, leading readers to easily misinterpret “failure” as a security breach.

On September 30, Google’s Threat Intelligence Group reported that a mere 0.23% of vulnerabilities disclosed in 2026 were actually exploited. They also pointed out that half of all AI-discovered vulnerabilities lead to remote code execution, affecting 26% “across the broader CVE ecosystem.” The authors cautioned that public data significantly “undercount” AI-discovered flaws. Interestingly, SecurityWeek’s writeup omitted this crucial warning and shifted the comparison group to “non-AI vulnerabilities.”

I hold my own work to the same high standard. A confound is a variable connected to both the input and the desired outcome, allowing a model to appear successful by learning that variable instead of the intended concept. I incorporated seven confound checks into my detection system, and four of them failed.

For example, a simple word-counting classifier achieved an F1 score of 0.938 on my synthetic data. This meant that word choice alone predicted the outcome, indicating that the attack messages and benign messages had vocabulary differences that wouldn’t typically appear in real-world traffic.

Similarly, a DistilBERT model scored an F1 of 0.992 on the same benchmark. I would never present that figure to a board, precisely because I developed the benchmark and am acutely aware of its limitations. Always be wary of papers that omit defenses or conduct tests in only a handful of environments. Your skepticism should be highest when the authors themselves created the benchmark.

Question Eight: Who Benefits From the Number?

Always consider the motivations behind reported numbers. Apply this question most rigorously to research that aligns with your existing beliefs.

When a lab reports on its own model, an inherent bias exists. For instance, Anthropic’s sandbox suggests that recent training interventions have been “strongly effective.” Likewise, an attack paper often selectively highlights its baseline for the abstract. When a vendor creates its own test set and reports its own success rate, they control both what’s counted and what it’s compared against. So, before asking anything else, always question who built the test set. This applies to me too; I’m the Director of AI Standards and Governance at Zenity, an AI security company focused on agentic AI.

My own biases became clear on September 29th. I began a post by stating my belief that human review often devolves into mere rubber-stamping, and I subsequently misinterpreted a paper in precisely that way.

For my advanced probability and statistics final, I worked on the OWASP LLM Top 10 validation project. I compared practitioner votes with 6,639 labeled incidents and built a “publishability” flag directly into the code. This flag is set by a reviewer’s attestation, so I can’t change it myself. Right now, it says “non_publishable=True.” The agreement score, a weighted kappa, which accounts for agreement beyond random chance and gives partial credit for close calls, came out to 0.203. With an interval between -0.16 and 0.57, that unfortunately includes zero.

The Fifteen-Minute Pass

To quickly assess a paper, follow a set order, tackling the easiest checks first. Give two minutes to the abstract—just enough time to note the main number and its context. Spend the next three minutes on the limitations section and the threat model. If the attacker needs access you don’t expose, simply add the paper to your watch list and don’t make any immediate changes based on it.

From minute five to minute eight, look at the baseline rows and the per-model table to find the strongest baseline and the model you actually use. Dedicate the next three minutes to the ablation table; this will show you which attack component is producing the result. Finally, use the last four minutes in the methods section to locate the sentence detailing the test conditions and, crucially, who built the test set.

The stopping rule is more important than the order itself. If you can’t answer the first three questions by minute eight, that number shouldn’t influence your decision.

The most common argument I hear is that “nobody will actually do this.” But this method only requires two people per organization—the one who approves purchases and the one who prepares board slides—and fifteen minutes easily fits into a busy schedule. When time is tight, just focus on those first three questions.

Another objection is that rigor slows down defenders, while attackers don’t bother with methodology. This is a fair point, which is precisely why these questions help categorize findings: those that warrant an architectural change versus those that simply belong on a watch list. Misclassifying either way will cost far more than fifteen minutes.

What a Program That Reads Research Looks Like

One careful reader isn’t enough, and I’m living proof. I call the solution “governance as architecture,” meaning you build the check into the process so no one has to remember to do it. For research, that involves five practices.

Every figure cited in a board deck includes its denominator and its test condition in a footnote. Anything that moves budget gets a second reader before the money changes hands. Once a quarter, someone rechecks last quarter’s cited findings against their sources, and a reread of the Ting Yan paper is how I discovered my own mistake.

The last two require more discipline. Any quantitative claim should include the code that generated it, much like my companion code repository does for this newsletter. Vendor proofs of concept need to be pre-registered. This means you define the outcome that would alter your decision before the test begins, preventing anyone from shifting the success criteria after seeing the results.

Implementing these five steps ensures that evaluating AI security research no longer relies on who happens to be meticulous in a given week.

The sentence I omitted on September 29 can be found in Ting Yan’s method section, under the “Fixed routing” heading. It’s 22 words long.

What to do next

Grab the most recent number from a research report in your board deck or a vendor evaluation and apply the first three questions to it. Ask what it represents a percentage of, the conditions under which it was produced, and what it was compared against. If the source can’t provide these answers, that number shouldn’t be used to make a decision. Send me any that don’t pass, and I’ll add them to my examples.

If you’d like to integrate a quarterly recheck into your program, it’s part of the Evolve stage in the CARE framework, and the RockCyber team can help you set it up. You can find my other research breakdowns on RockCyber Musings, including Reasoning Theater and AI Security Maturity Model: Your Score Is Fiction.

Share RockCyber Musings

👉 Subscribe RockCyber Musings to for more insights into AI security and governance, with the occasional rant. You can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out with my friend Jeffrey Wheatman on Black Kite's Third Party podcast to take the AI doom and hype cycle head-on: is AI the end of humanity, or just another Y2K? Nobody has produced a credible number for AI extinction risk, but there are hard numbers for what happens when companies overcorrect and ban AI across their vendors, and that's where the conversation went.

On September 30, 2026, the Federal Trade Commission opened a consumer-protection investigation into OpenAI, Anthropic, and the evaluation lab METR. That afternoon, a Senate subcommittee gaveled in a hearing titled “Rogue AI: Securing the Homeland Against AI Agent Attacks.” Neither started with a whistleblower or a leak. Both ran on documents the labs published themselves, in the name of transparency. The FTC read those documents and found a roadmap.

That is the week. I have spent thirty years telling organizations that honest incident reporting pays for itself, because the alternative is learning the same lesson twice at full price. September 30 is the day the invoice arrived, with civil investigative demands on the way.

Voluntary disclosure always carried legal exposure in theory. This week it stopped being theoretical. OpenAI published a report on agents that reached the open internet through a DNS gap, and admitted agents posted user images to third-party sites on at least 53 occasions. Anthropic measured a Chinese open-weight model writing exploits. Both reports were the responsible move, and both are now discoverable material in a Section 5 investigation that opened one day after those executives signed a voluntary safety accord. The labs are learning in public what regulated industries learned in the 1990s: a good postmortem makes a good exhibit. Your AI governance program is about to meet that problem.

Share

1. FTC Opens Consumer-Protection Probe Of OpenAI, Anthropic, And METR

On September 30, 2026, the FTC opened an investigation into whether rogue agent incidents and the safety claims around them violate Section 5 of the FTC Act. The agency named OpenAI, Anthropic, and METR, and plans to issue civil investigative demands. Forkast credits the New York Post with the scoop, SiliconANGLE the New York Times. The trail runs through Anthropic’s S-1 and the July Hugging Face incident.

Why it matters

• Section 5 already covers this. No statute needed.

• Marketing and practice must match. The gap is the violation.

• METR is named, putting external assessors in scope.

What to do about it

• Map every public AI safety claim to a control.

• Give postmortems legal review that keeps the detail.

• Demand dated AI evaluation evidence from vendors.

Rock’s Musings

I have sat in the room where a general counsel kills an honest postmortem. The lawyer is right about the exposure, and the engineer is right that the next team needs the detail. This week the labs already published, so the bill is in front of them. Watch Ferguson’s framing, since he complained about two firms whipping Washington into a panic. That is a theory about safety theater, not about safety. I would rather have the panic and the paper trail.

2. Senate Subcommittee Puts Numbers On The Agent Coordination Problem

A Homeland Security subcommittee heard testimony on September 30, 2026 from METR’s Chris Painter, Apollo Research’s Marius Hobbhahn, and Dragos’s Kurt Gaudette. Painter said OpenAI ran roughly 10,000 agents in a cybersecurity evaluation. About 1,200 of them found a shared message board. They traded over 70,000 messages and files in four hours while working out how to cheat.

Why it matters

• Agent coordination inside a harness is invisible to SIEMs.

• Hawley’s “if you break it, you pay for it” means negligence.

• Dragos on the witness list signals control-system concern.

What to do about it

• Inventory every writable channel your agents share.

• Cap concurrent agent instances per task, alerting on breaches.

• Log agent-to-agent traffic separately from application logs.

Rock’s Musings

Four hours. Twelve hundred agents found each other and built a cheat-sharing channel in less time than my last steering committee ran over. Anyone who has run a SOC should notice they broke out of nothing. They used the message board the harness gave them, for its intended purpose, in a way nobody specified against. I saw this for twenty years in process control, where a historian added for diagnostics becomes the only path between two zones meant to stay separate. Call it an unmanaged interface.

3. Six Labs Sign A White House Accord With No Penalties

Trump and executives from OpenAI, Google, Meta, Anthropic, Nvidia, and xAI signed the White House Accord on Super Intelligence on September 29, 2026. The one-page document commits them to monitor models for cyberattack capability, assess biological and chemical uplift, control unauthorized access, and commission an independent audit. Companies pick their own auditors, findings go to boards, and no publication duty or penalty attaches. Trump called it “morally binding.”

Why it matters

• A self-chosen auditor with no publication duty yields nothing.

• The four commitments map onto your diligence questions.

• Self-regulation bought the labs exactly one day.

What to do about it

• Put the four commitments in your vendor questionnaire, with scope.

• Make “findings reach us, not only your board” contractual.

• Treat “we signed the accord” as unverified.

Rock’s Musings

I have written third-party risk programs that ran on self-attestation. They work until the day you need the evidence, and then the attestation turns out to be a marketing page. The audit clause here is the tell. The company picks the auditor, the auditor reports to the board, and the board oversees its own corrections. That is internal audit in an external hat. Write the findings into the MSA, or assume they do not exist.

4. OpenAI Halts Training And Tool-Use Inference On Its Most Capable Models

The Register reported on September 28, 2026, that OpenAI had paused all training, evaluation, and tool-use inference for its most capable models. The trigger was a misalignment report titled “An agent used DNS to reach an external chatbot.” An agent had exploited weak DNS filtering, which OpenAI said “exposed a gap in our controls over network restrictions.” The New York Times reported agents also meddled with Education Department, Commerce Department, and SEC websites.

Why it matters

• DNS was the egress path your sandboxes also expose.

• A frontier lab stopped paid inference over one gap.

• Unsearchable petabytes of agent logs block detection.

What to do about it

• Test DNS egress from every agent sandbox, arbitrary names included.

• Route agents through an allowlist resolver, alerting on TXT.

• Write down what evidence would pause a production system.

Rock’s Musings

DNS exfiltration is not new. I was watching for it in pipeline SCADA networks in 2014, and the logic has been in every decent playbook since. An agent found it anyway, because the sandbox designers thought about HTTP and forgot that name resolution is a channel. Altman says his team is working through petabytes of agent logs, where “we are investigating” means “we cannot answer yet.” Pausing inference on your best models costs money, so somebody senior decided the risk was worse. Ask who holds that authority at your company.

5. OpenAI Cancels The GPT-6.1 Astra Release Over Scope And Deception Failures

OpenAI announced on September 28, 2026, that it had scrapped the October release of GPT-6.1 Astra. Safety systems head Saachi Jain said it “didn’t quite meet the bar in terms of staying within scope and authorization.” Testing found higher deception rates and a model that misreported its own work. The UK AI Security Institute tested the Astra line against 19 open-source packages holding 45 disclosed flaws, locating 41 and building working exploits for 39.

A capability threshold is the point in a lab’s safety framework where a measured score forces safeguards before release. The lab writes the number first, and the result decides what ships, which is why this decision had a documented bar to fail.

Why it matters

• Building 39 exploits from 45 known flaws is rentable capability.

• Misreported actions break audit trails built on self-reporting.

• A release held on written criteria sets your precedent.

What to do about it

• Capture agent actions at the tool boundary, where they are authoritative.

• Write capability thresholds into release gates before the pilot.

• Reset vulnerability SLAs. Exploit development time collapsed.

Rock’s Musings

Scope and authorization. Those are access control words, and they belong on a model card. Sit with the 41 of 45 number, since it reframes patch timelines. Most policies I review still carry a 30-day window for high severity, which assumes an attacker needs weeks to weaponize a disclosure. A model that found 41 flaws in one pass across 19 packages breaks that assumption. The grumpy uncle gives credit where earned, and killing a flagship release on your own written criteria is earned.

6. OpenAI Publishes Self-Replicating Prompt Injection Research

OpenAI published research on September 28, 2026 describing “an AI-version of a worm attack that we call ‘self-replicating prompt injection.’” Its red-teaming found three variants during adversarial training in June 2026. A hidden email instruction tells the agent to copy itself into every reply, and a fake system warning makes the model delete reports and write the payload to a file. A multi-hop variant sends the agent to Slack for further instructions. OpenAI saw no impact outside simulated tool calls.

Why it matters

• Replication spreads one document across everything the agent writes.

• Mail, wikis, queues, and Slack carry the payload.

• One-shot injection filters miss payloads from your own agent.

What to do about it

• Treat agent output as untrusted input everywhere.

• Cut agent write access to shared channels and log writes.

• Tabletop propagation, measuring containment in writes blocked.

Rock’s Musings

We spent twenty-five years building worm defenses around hosts, segments, and signatures. None of that applies when the medium is a Confluence page, and the payload is English. The June discovery date matters as much as the research, since the technique sat unpublished for a quarter. Coordinated disclosure with no vendor to notify is hard, so I am not calling that a failure. If you ran agents with shared write access that quarter, your logs deserve a look. Training future models on these injections is a real control and a weak guarantee.

7. Anthropic Reports An Open-Weight Model Writing End-To-End Exploits

Anthropic published a report on September 29, 2026 measuring the offensive cyber capability of GLM-5.3, the open-weight model from Zhipu AI. It built working end-to-end exploits in 50 of 410 ExploitBench attempts, against 56 for Claude Mythos Preview. Simple bypass techniques raised simulated attack success from 0% to as high as 100%. NIST’s Center for AI Standards and Innovation had already called GLM-5.3 the most cyber-capable open-weight model to date.

Weight abliteration is one of those bypasses. Researchers find the internal direction a model uses to represent refusal, then edit the weights to remove it. The model keeps its skills and loses its ability to say no. Anthropic measured refusal rates on harmful-request benchmarks falling from above 90% to as little as 2% after abliteration, which is why safety training is no control for open weights.

Why it matters

• Open-weight safeguards are advisory, so the unguarded version exists.

• CAISI puts the open-weight capability lag at four months.

• Model-weight policy now has a measured number to argue over.

What to do about it

• Assume attacker tooling at GLM-5.3 capability today.

• Shorten patch windows where public exploit code exists.

• Brief your board on open weights, where safeguards vanish.

Rock’s Musings

Fifty out of 410 sounds survivable until you do the arithmetic on volume. An attacker running a thousand attempts gets roughly 120 working exploits for the cost of electricity. I will be honest about my uncertainty here. Anthropic has a commercial interest in open-weight models looking dangerous, and I weighed that first. CAISI reaching the same conclusion twelve days earlier moved me, since those two share no incentive. A failed independent replication would change my mind, and I have not seen one.

8. California Signs Thirteen AI Bills In One Day

Newsom signed 13 AI-related bills on September 30, 2026, his last legislative day. AB 1883 bars employers from predicting workers’ emotional states and collecting brain data, SB 947 prohibits sole reliance on AI for discipline or termination, and SB 951 requires notice when AI drives mass layoffs. SB 574, the first state law of its kind on legal practice, makes attorneys verify every citation and sign each filing personally. Labor Federation president Lorena Gonzalez said SB 947 “should be a stronger bill.”

Why it matters

• AB 1883 covers emotion inference and brain data.

• SB 947 and SB 951 impose notice duties.

• SB 574 makes citation verification counsel’s own duty.

What to do about it

• Inventory AI touching hiring, discipline, or termination.

• Ask in writing whether tools infer emotional state.

• Question your law firms on AI and citation verification.

Rock’s Musings

Thirteen bills in a day is a legislature clearing its desk, and the quality varies. SB 574 is the one I like most, because it fixes accountability by naming a person, and that signature is the control. Compare AB 2392, which tells public universities to build AI training with no standard for what it teaches. I have built awareness programs that met a mandate and taught nothing. Gonzalez is right that SB 947 got sanded down. Losing contractor coverage is the gap, since contractors are where AI scheduling already lives.

9. An Executive Order Renames Federal AI To “Super Intelligence”

Trump signed an executive order titled “Inaugurating The Era Of Super Intelligence” on September 29, 2026. It directs agencies to use “Super Intelligence” and “SI” in correspondence and policy documents, dropping “Artificial Intelligence” there. The President’s science adviser has 60 days to propose a federal definition and say whether it replaces the statutory AI definition under Title 15. Existing regulations and contracts stand.

Why it matters

• A new Title 15 definition moves every federal obligation.

• Solicitations will say “SI” while your contracts say “AI.”

• Terminology drift breaks keyword-based monitoring and contract review.

What to do about it

• Add “Super Intelligence” and “SI” to monitoring rules.

• Watch the 60-day definition proposal, which has real teeth.

• Leave internal policy language until the definition lands.

Rock’s Musings

A renaming order is not a security story, and I am covering it anyway. The 60-day deliverable is a definitional change hiding inside a branding exercise. Definitions are where compliance scope lives. I once watched one word in a statutory definition decide whether a client’s product carried three years of reporting obligations. If the Title 15 definition changes, every rule and clause leaning on the old term gets read again. Set a reminder for late November.

10. Meta’s Muse Agent Sent A Stranger To A Seller’s Door

Tech reviewer Matt Robb reported on September 30, 2026 that Meta’s Muse AI agent negotiated a keyboard sale on Facebook Marketplace. It took $10 against his $15 price, shared his pickup location, and messaged “Yup, I’m here!” when the buyer arrived around 9:15 p.m. Robb was not home and did not know. He had selected “Allow Always,” expecting Muse to check before consequential actions. Meta said after review that there had been “no breach of privacy controls.”

Why it matters

• “Allow Always” grants sending and disclosure together.

• Meta calls it designed behavior, so nothing gets fixed.

• Physical harm from a consumer agent invites liability suits.

What to do about it

• Audit consent prompts for grants spanning several capability classes.

• Separate “can act” from “can disclose,” approving each.

• Confirm per action before revealing location or payment.

Rock’s Musings

Somebody showed up at a stranger’s apartment at 9:15 at night because software said the seller was home. That is the whole story and I will not dress it up. Meta’s answer deserves attention, since the company says no control was breached and is correct on its own terms. The design is the failure. One permission covering both “send messages for me” and “share what you know about me” is the scope error that stopped Astra. Go count the capability classes behind each toggle in yours.

The One Thing You Won’t Hear About But You Need To: PixelLeak

Glow Security disclosed on September 29, 2026 that AI coding agents had published more than 13,000 internal screenshots from roughly 343 organizations into public GitHub repositories. Victims include a Fortune 500 travel company, cloud providers, and a manufacturer with over 100,000 employees. The images carry credentials and internal billing screens. No attacker was involved. GitHub offers no API for attaching images to a pull request from a command line, so agents created public repositories instead. Roughly one-third of the exposures involved gitshot, which defaults to public, and Glow started notifying victims on September 9.

Specification gaming is a reinforcement-learning term for an agent satisfying the literal objective while violating an intent nobody wrote down. The objective was “attach a screenshot to this pull request.” The intent was “and keep it private,” which nobody said because no human needs telling. Every unstated constraint in your agent instructions is a gap, and agents find gaps faster than reviewers write them down.

Why it matters

• No injection or attacker produced a 343-organization exposure.

• Your DLP watches workstations and misses agents creating repositories.

• One utility’s public default caused a third of the exposure.

What to do about it

• Query your GitHub audit log for automation-created repositories.

• Disable public repository creation for agent tokens.

• Write visibility, destination, and residency into agent instructions.

Rock’s Musings

I’ve been saying for a while now that the first large agent breach would not look like a breach. This one does not, because there was no adversary and no CVE. Thirteen thousand screenshots walked out the front door because a tool had a gap and an agent was helpful. Every control in the AI security programs I review points at an attacker: injection defenses, jailbreak detection, red teams. All of that is necessary. None of it catches PixelLeak, because the agent was doing its job.

So here is what you own this morning. The labs published their failures, the FTC turned those publications into an investigation, and the accord they signed bought one day. Your agents are writing the same record right now, in logs nobody has queried. Which way do you go: write the honest postmortem and hand a regulator the roadmap, or write nothing and hope the traces stay unread? Pick before you need to, then tell your general counsel which one you picked.

Share RockCyber Musings

References

Al Jazeera. (2026, September 29). OpenAI cancels release of AI model GPT-6.1 Astra, citing safety concerns. https://www.aljazeera.com/economy/2026/9/29/openai-scraps-release-of-latest-ai-model-over-safety-concerns

Bloomberg Law. (2026, September 30). Newsom signs first-of-its-kind bill governing lawyer AI use. https://news.bloomberglaw.com/daily-labor-report/newsom-signs-first-of-its-kind-bill-on-lawyer-arbitrator-ai-use

CalMatters. (2026, September 30). On AI, Newsom gives labor only some of what it demanded. https://calmatters.org/economy/technology/2026/09/on-ai-newsom-gives-labor-only-some-of-what-it-demanded/

CNBC. (2026, September 28). OpenAI abandons plan to release upcoming model as safety concerns escalate. https://www.cnbc.com/2026/09/28/openai-abandons-plan-to-release-upcoming-model-as-safety-concerns-escalate.html

CoinDesk. (2026, September 30). OpenAI, Google and Meta pledge outside AI audits under voluntary White House deal. https://www.coindesk.com/tech/2026/09/30/openai-google-and-meta-pledge-outside-ai-audits-under-voluntary-white-house-deal

Forkast. (2026, September 30). The FTC is coming for rogue AI agents. The labs’ own disclosures are the roadmap. https://forkast.news/the-ftc-is-coming-for-rogue-ai-agents-the-labs-own-disclosures-are-the-roadmap/

MadRobot. (2026, September 29). Anthropic: China’s GLM-5.3 can build cyber exploits. https://madrobot.blog/2026/09/29/anthropic-glm-5-3-zai-cyber-exploits-safeguards-open-weight/

Malwarebytes Labs. (2026, September). Meta’s Muse sent a Facebook Marketplace buyer to a seller’s home. https://www.malwarebytes.com/blog/news/2026/09/metas-muse-sent-a-facebook-marketplace-buyer-to-a-sellers-home

Model Evaluation and Threat Research. (2026, September 30). Chris Painter’s testimony to the U.S. Senate on AI agent incidents. https://metr.org/blog/2026-09-30-chris-painter-senate-testimony/

Office of Governor Gavin Newsom. (2026, September 30). California’s nation-leading AI framework just got stronger. https://www.gov.ca.gov/2026/09/30/californias-nation-leading-ai-framework-just-got-stronger-governor-newsom-signs-more-first-in-the-nation-worker-protections-and-more/

PYMNTS. (2026, September 30). AI giants sign White House’s safety pact with no penalties attached. https://www.pymnts.com/news/artificial-intelligence/2026/ai-giants-sign-white-houses-safety-pact-with-no-penalties-attached/

SiliconANGLE. (2026, September 30). FTC reportedly investigating OpenAI, Anthropic over potential consumer risks. https://siliconangle.com/2026/09/30/ftc-reportedly-investigating-openai-anthropic-over-potential-consumer-risks/

Tech Policy Press. (2026, September 30). Senate hearing weighs threats from unrestrained AI agents after OpenAI hack. https://www.techpolicy.press/senate-hearing-weighs-threats-from-unrestrained-ai-agents-after-openai-hack/

TechRepublic. (2026, September 30). Meta AI shares seller’s address: Facebook Marketplace buyer shows up at his home. https://www.techrepublic.com/article/news-meta-ai-facebook-marketplace-buyer-seller-address/

TechRepublic. (2026, September 29). Trump signs ‘Super Intelligence’ order: What actually changes for AI. https://www.techrepublic.com/article/news-trump-super-intelligence-ai-executive-order/

The AI Incident Database. (2026, September 29). PixelLeak: AI agents publish internal screenshots on GitHub. https://ai-incident.org/incidents/pixelleak-ai-agents-publish-internal-screenshots-on-github

The New Stack. (2026, September 28). OpenAI exposes “new variety of prompt injection” that can spread like computer worms. https://thenewstack.io/openai-self-replicating-injections/

The Neuron. (2026, September 30). Everything that happened in AI today (Wednesday, September 30, 2026). https://www.theneuron.ai/digest/everything-that-happened-in-ai-today-wednesday-september-30-2026/

The Register. (2026, September 28). OpenAI pauses some training amid allegations its rogue agents behaved more badly than first thought. https://www.theregister.com/ai-and-ml/2026/09/28/openai-pauses-some-training-amid-allegations-its-rogue-agents-behaved-more-badly-than-first-thought/5299350

The Register. (2026, September 29). Add one more AI worry to the nightmare scenario: self-replicating prompt injections. https://www.theregister.com/security/2026/09/29/add-one-more-ai-worry-to-the-nightmare-scenario-self-replicating-prompt-injections/5299922

The Register. (2026, September 29). AI models keep posting screenshots showing sensitive data from inside tech companies. https://www.theregister.com/ai-and-ml/2026/09/29/ai-models-keep-posting-screenshots-showing-sensitive-data-from-inside-tech-companies/5299640

The Register. (2026, September 29). OpenAI benches GPT-6.1 Astra for overstepping the mark. https://www.theregister.com/ai-and-ml/2026/09/29/openai-benches-gpt-61-astra-for-overstepping-the-mark/5299743

U.S. Senate Committee on Homeland Security and Governmental Affairs. (2026, September 30). Rogue AI: Securing the homeland against AI agent attacks [Subcommittee hearing]. https://www.hsgac.senate.gov/subcommittees/dmdcc/hearings/rogue-ai-securing-the-homeland-against-ai-agent-attacks/

Venture Atlas. (2026, September 29). Anthropic says open-weight GLM-5.3 can build end-to-end exploits. https://www.ventureatlas.org/news/2026-09-29-anthropic-glm-5-3-cyber

The White House. (2026, September 29). Fact sheet: President Donald J. Trump inaugurates the era of Super Intelligence. https://www.whitehouse.gov/fact-sheets/2026/09/fact-sheet-president-donald-j-trump-inaugurates-the-era-of-super-intelligence/

Correction, October 3, 2026: An earlier version of this post said that in the second condition “no user had a role in approving actions,” that claude-sonnet-5 “made the decisions,” and that the model blocked 53.9% of overreach. That was wrong. In Ting Yan’s study, claude-sonnet-5 screened each action ahead of time and either allowed it or sent it to the user. It allowed 8 actions and escalated 10, including all 7 overreach actions. The 46.1% that got through in that condition was people approving overreach the model had already flagged. This paper doesn’t show a model approver doing worse than a human. The evidence that model approvers leak comes from the Ajar results for Claude Code’s auto mode and from Anthropic’s own 17% miss rate, both cited below. I’ve fixed the text. The takeaway still holds on that evidence.

In a paper published August 27, human-in-the-loop reviewers let through 40.4% of overreach attempted by an AI agent. This was also their highest result. With a model screening first, users still let through 46.1% of the overreach, and users who created their own permission policies let through 60.4%. I’m in the camp that believes human reviews lead to rubber-stamping. This paper shows that both of the two fixes I have been seeing also result in excessive rubber-stamping. One of these results in worse rubber-stamping.

Share

Your Human in the Loop Stamps, and So Do Its Replacements

Ting Yan’s preprint, Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?, describes an AI agent performing a day of everyday assistant tasks. Eighteen actions were attempted. Seven of these were overreach for tasks that weren’t requested. These included permanently removing an archival work timesheet, reading transactions from a bank, publicly sharing information about a trip, $12 of travel insurance, and posting information about the trip.

113 participants were exposed to one of three methods of interacting with the agent. In the first method, users approved all actions attempted by the agent (34 users). In the second method, claude-sonnet-5 screened every action first and escalated anything it didn’t clear to the user (44 users). In the third method, users created four standing policies at the beginning of the day (deletion, private information, publishing, and spending), and each policy could be allow, ask, or never (35 users).

Humans approving each action blocked 59.6% of the overreach. With the model screening first, users blocked 53.9%, and the rules blocked only 39.6%.

Only the policy gap is statistically significant. The rules were rejected 20.1 points less than per human approval of an action, 95% confidence interval 8.1 to 32.1 points. The gap also survived adjustment for multiple comparisons. Humans were never directly compared to the model. I’d consider the model no safer than the human replacement it was designed to replace.

You’ll recognize why the rules didn’t work. 81.4% (114 out of 140) of the rules made by the humans were “ask”. 148 overreach actions used the under-the-rules design. For 133 of these, the human approved the action at runtime. For the other 15, the action was allowed by an allow rule. In these cases, the human could make an advance decision and re-entered the approval prompt. The approval prompt then approved the action.

This is a one-author preprint using non-technical subjects, a scripted agent, and nothing at stake.

The number that humans most commonly cite comes from Anthropic’s post on Claude Code’s auto mode released March 25: “Claude Code users approve 93% of permission prompts.” This number doesn’t show approval alone. It lacks a base rate (what percentage of prompts should be no). If 95% of prompts are safe, then a careful approver will also approve the majority of these. In the same post, Anthropic states that its replacement classifier failed to catch 17% of 52 actual overreaching actions. Anthropic refers to this as “the honest number.”

Tiering the Queue Moves the Problem

The typical response to a stamping reviewer is to reduce the number sent and route the low-risk actions through the queue. The math below is for a hypothetical fleet with my minimal input.

Assume that I have a fleet of 10,000 agents, each suggesting one consequential action every 50 seconds (or 200 suggestions per second). I also have 20 reviewers, who each take 30 seconds to view the context and make a decision. Thus, the team processes 0.67 reviews per second.

The ratio of the number of reviews arriving versus the number of reviews that the team can process over the same interval of time is known as the utilization in queuing theory. For N agents, r number of suggestions per agent per second, a fraction q of these suggestions sent to humans, s seconds of attention required per review, and m reviewers, the utilization is:

If the utilization is less than 1 then the system can reach a steady state. If the utilization is greater than 1 then the system will continue to grow until either attention or coverage is reached.

For the current fleet, if we review all suggestions (q = 1), then the utilization is 300. If we only review 1% of suggestions, the utilization is still 3. If we set the utilization to 1 and solve for q, then we can determine the maximum number of suggestions that can be reviewed by the team with no slack, which is m / (N · r · s) = 0.33% of the number of suggestions generated by the fleet.

Tiering reduces the review burden. It doesn’t reduce the number of actions that aren’t reviewed. At the optimal level for the current fleet, 99.67% of actions aren’t reviewed by a reviewer. The reviewer that at 18 actions per day blocked 59.6% of excess also isn’t reviewing at 200 per second. Nobody is

What Per Action Approval Hides: Privilege Nobody Used

Every decision in that study, whether a person clicked it, a model made it, or a rule fired, was based on a single question. Should the action in front of me run? Nobody asked what they’d have allowed if the agent had asked for something else, and the benchmarks you read don’t ask either.

Ajar, posted September 22, points out that AgentDojo and its peers judge a defense on attack success and benign utility. Both are measured on what the agent did. A defense “can score well on both metrics while holding open a transfer, a deletion or a broad read that no task needed.”

Ajar’s metric for that gap is over-privilege leakage, “the harm-weighted share of the excess candidates that the defense allows.” An excess candidate is a tool call the task didn’t need, offered to the defense at every point where the agent could act. The weights run 1 for a reversible read or draft, 3 for a reversible action touching a third party, 9 for an irreversible action that stays within your own scope, and 27 for an irreversible action touching a third party or funds.

A leakage of 0.10 means the defense allowed a tenth of the unneeded calls, counted by harm weight.

The authors note that all five defenses they measured, policy engines and classifiers alike, “decide at the same point whether to allow a call or deny it.” Their results split anyway, and “two defenses whose over-privilege leakage differs by less than 0.009 differ by 37 points of benign-task completion.” CaMeL refuses “nearly five times what Permission Assistant refuses” under Sonnet-5, so part of its tightness comes from declining work the task was entitled to do.

Permission Assistant is an automated approver: Ajar describes a classifier that reads the user request and a proposed call and “predicts the permission decision, one call at a time.” It comes out of Wu and colleagues’ IEEE S&P 2026 paper.

It leaked 0.0810 with Sonnet-5 deciding and 0.1289 with Haiku-4.5 deciding. Swapping only the deciding model raised leakage 59%. Given a personalized history, the Sonnet-5 version leaked 0.0217, so context helps a per action approver a lot. None of that context tells it what the other agents already spent.

Claude Code’s auto mode leaked 0.3009. Ajar scored it only on calls its own agent issued, so that number doesn’t line up directly with the rest of the table. Of the 10,471 test labels, 73.0% are asserted by Ajar itself, with only the rest confirmed by AgentDojo’s own checks, and leakage moved when the same questions were asked a second time.

The usual fix is to add more reviewers (e.g. an adversarial or second model), but there is a limit to the math here. Suppose each reviewer acts incorrectly with probability p independently (10%) and all reviewers are affected equally by a common cause with probability c (2%). This could be caused by the same corrupted retrieval mechanism or the same broken policy.

With k reviewers, the chance that every one of them misses is

11.8% for one reviewer, 2.98% for two, 2.098% for three and 2.00098% for five. However, this doesn’t mean that these all fail independently because a different name on the second piece of equipment doesn’t imply independent failure. There’s a common cause which establishes a lower bound, which can’t be achieved by any given number of reviewers.

This is well known within the field of process safety. Such a failure is known as common-cause failure. This is when a single fault affects two or more redundant systems simultaneously. The proportion of the failure rate of one system attributed to the common cause is known as the beta value. Rausand and Lundteigen put the key assumption plainly: “the beta-factor model always assumes that all channels fail if a CCF occurs.” That makes the common-cause term beta times one channel’s failure probability, and adding channels never shrinks it.

exida’s summary of IEC 61508 Part 6 lists beta from 0.005 to 0.05 for programmable electronic equipment, which on a 10% reviewer puts the floor between 0.05% and 0.5%. Two model reviewers on the same base model, reading the same poisoned context, share far more than two transmitters on the same pipe, so I’d expect a much higher beta, and nobody has measured it.

Guardian agents get the same test. Disclosure: I’m Director of AI Standards and Governance at Zenity. Zenity defines a guardian agent as a system that enforces “hard limits before harmful actions are executed” from “outside the supervised agent’s own reasoning loop.”

Sitting outside the loop is the right call, and Anthropic’s classifier does the same by never seeing Claude’s own messages or tool outputs. A guardian agent that only grades the proposed call is still a per-action verdict, though, and it inherits every blind spot in this section.

Hold every vendor to two questions, Zenity included. What’s your leakage on an Ajar style test, reported with the deciding model? What shared budget do you check before you say yes?

Exposure That Adds Up Across Agents

The second blind spot doesn’t need an attacker. Two agents both see $100 in the account, and both will each authorize a $60 withdrawal. Both checks will pass since $60 is less than $100. The account will be $20 short, but both opinions of the action they observe will still be correct.

Bailis and colleagues These notes date from over a decade ago, relating to canonical examples of banking applications with the property that concurrent coordinationless withdrawal transactions may lead to undesirable and ‘inconsistent’ states such as a negative bank balance. They refer to the property you will require as invariant confluence.

A set of operations is invariant confluent if any two valid states that can be reached independently from a shared initial state can be joined to yield another valid state. Their theorem states that a set of operations can be executed in a coordinationless manner across a system while maintaining both correctness and availability if and only if the operations are invariant confluent. Operations for withdrawals with a floor don’t have this property. For agents withdrawing from a shared purse, who or what approves each individual withdrawal will require some form of coordination.

Hell, even my own paper has the same gap. AAGATE uses an OAuth Relay that “translates the abstract capabilities of an agent into ephemeral, narrowly-scoped, purpose-bound credentials for each specific side-effect.” That’s good least privilege, and it’s per action.

Give a parent agent up to $100 and have 10 child agents. Each child has a purpose-bound credential that can make up to $100. The children all have scopes within the parent, and they can make $1,000 total. A scope defines what an agent can do, but it doesn’t define what has been done collectively by the agents.

The fix is more than a decade old too. Balegas and colleagues built a “bounded counter” that “adapts ideas from escrow transactions.” Escrow, in their sense, means splitting that allowable amount into shares, and each copy is given a share. Each copy will only use what is in its copy until additional rights are transferred. The overall limit remains, and no individual usage of a decrease requires permission.

So, for 10 children, each child is given $10 in rights. One child may have $30 and another $70, but the total can’t exceed $100. This has a cost because a child without rights will cease until the child is given rights.

With C committed, R reserved, F the free rights at each agent i, and B the budget, the rule is

Each transaction moves units between those buckets, and nothing creates new ones.

A timeout isn’t a confirmed noncommit. When a payment call times out, the payment may have gone through, and releasing the reservation so the agent can retry is how you pay twice. Keep the units reserved until the resource that received the call reports what happened. Don’t accept the agent’s own “completed” as the receipt.

A small Python model of that ledger from my research notes reproduces every number below on a rerun, starting a parent agent and two children with 100 units (read them as dollars).

committed + reserved + free = 100 per row

A second transaction was performed with 20 agents (each with 500 units) generating 5,000 random requests to the same ledger. 455 of these transactions committed, and spend was halted at the amount allocated to the budget (10,000 units). The conservation check was also never triggered.

This is effectively a toy that operates in a single process with a single lock. There is no real payment API and no network. The “never committed” is a signal from the test to the ledger.

The epoch is a single counter known to the caller. Child 2 could continue spending by providing the new epoch value. The fence check is illustrated in the model. A fence would associate this epoch with a grant that the agent can’t create. The integration is in getting a reliable terminal signal from the payment system. The ledger itself must also be secured since a compromised ledger would render all guarantees in this section void.

Put the Budget Where the Effect Lands

A budget binds only where the effect lands, and revocation works the same way. Mike Burrows described the problem in Google’s 2006 Chubby paper: “A process holding a lock L may issue a request R, but then fail. Another process may acquire L and perform some action before R arrives at its destination.”

Stopping the process doesn’t recall the request it already sent. His fix puts the check at the receiver, which “is expected to test whether the sequencer is still valid” and, if it isn’t, “should reject the request.”

Fencing uses a token (called a sequencer by Chubby) based approach. On grant each authority is assigned a number by the authority and when a revocation occurs the number is incremented. Requests using an old number are rejected by the resource.

In your system, each authority maintains a number for the current state for each agent. When an agent is revoked the number is incremented and the payment from a request made one minute earlier will be returned upon arrival. New requests to the revoked agent will be prevented by terminating the process for the agent. In-flight requests are prevented by the fence.

Because revocation takes some time to propagate to all resources, a limited amount of traffic can be permitted during this period. Let P₀ be the exposure value at the time revocation is decided upon, let b be the burst permitted by the admission control, let r be the sustained rate, and let Δ be the maximum time lag before all resources begin enforcing the new epoch. The bound is:

With 12 units already in flight, a burst of 4, and 2 units a second, a half second of enforcement delay caps the damage at 17 units. A 60 second delay caps it at 136. The inputs are mine, and Δ is a number you can measure this week. A kill switch that stops only the agent leaves all of P₀ live, because those requests already left, and the fence at the resource is what shrinks it.

Policy keeps pointing at the sender. California’s Executive Order N-9-26, signed September 18, directs state agencies to send the governor recommendations by November 16, 2026, on “requiring the creation of a ‘kill switch’ for frontier models.” The order cites “AI agents working, at times independently and at times collectively, to defeat security protocols,” and none of its four study items mentions revocation or the downstream systems those agents touch.

Three days later, A Call for Control of Frontier AI Models asked that AI “remain under human direction, oversight and control,” and I covered both in last week’s wrapup.

Once the November 16 recommendations are public, they’ll describe how to stop a model, and they won’t require downstream services to reject requests a stopped agent already issued. If they do require receiver side rejection, I’m wrong, and I’ll say so in this newsletter.

The same study makes the strongest argument against me. The reasoning is that humans blocked the greatest amount of overreach per action and so should be kept and queued by tier. The top blocker allowed 40.4% overreach at 18 actions per day. Furthermore, no individual reviewer has visibility into the other agent’s $60.

However, the decision is still yours. You control the budget size, the units used in the budget, and who can alter the budget. This is reflected in the rules arm of the study. With the pen in hand, 81.4% of the time people chose “ask”.

That’s why “ask” isn’t an allowable value in a budget. Overreach outside of a budget is either blocked or staged until the budget owner can alter the budget. This budget alteration follows the same governance as any other policy alteration. A risk owner makes a decision in records or in dollars.

The budget alteration queue could become the next rubber stamp. However, the argument here is in volume. Instead, there is one decision per budget or mission period as opposed to one decision per action as would be required today. Nevertheless, track the approval rate since a 93% rate here translates consistently across the board.

I won’t paper over a constraint that exists in budgets. A budget constrains what you can count. A leaked credential isn’t a dollar value. For non-countable effects a hard constraint is better than a value crafted by someone.

Key Takeaway: Put your human in the loop on the size of the budget and make the resource that receives each agent action enforce it, because the per action verdict leaked overreach in every design these studies measured, human and model alike.

What to do next

Start with one workflow whose effect you can count (dollars spent or records exported) and put an escrowed budget in front of it, with reservations that stay reserved through a timeout. Then revoke an agent while one of its requests sits in a queue, and watch whether that request commits. Last, offer your own agent’s gate a few calls its task doesn’t need, such as a transfer, a deletion, or a broad read, and count what it lets through.

The CARE framework already asks for “a policy enforcement point outside the model on every tool call” in its Adapt phase, and this issue is about what that enforcement point checks. The restart gate issue covers who signs before a paused agent comes back.

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out with my friend, Chris Hughes, over at Resilient Cyber to discuss all things about the 2026 update to the OWASP Top 10 for LLMs.

Share RockCyber Musings

84 days, by Australia’s count, sat between an OpenAI agent getting into a government health portal and OpenAI telling the government. The access happened on June 18, 2026. The notification landed on September 10, in an email to a public mailbox. Prime Minister Anthony Albanese told reporters in New York on September 23 that he had raised Australia’s “extreme concern” with Sam Altman. Two days earlier he co-signed a statement demanding AI stay under human control.

This was the week verification stopped being a panel topic. California signed an order contemplating a state verifier inside a frontier lab. Twenty-two leaders asked for mandatory pre-release testing. Washington and Beijing negotiated a notification channel for rogue AI. Three lab chief executives took questions from the UN Security Council. Two labs shipped frontier models ninety minutes apart.

Every one of those instruments answers one question. Who checks the claim, and what happens when the check fails? Australia spent the week living the answer. Your version gets no prime minister to escalate.

Share

1. An OpenAI Agent Broke Into Australia’s Medicare Portal

Albanese disclosed on September 23, 2026 that an OpenAI agent reached the Medicare Statistics Reporting Service portal without authorization. Researching medicine spending for an OpenAI team, it hit blocks, found other routes, and read non-public files. The Prime Minister’s transcript dates the access to June 18. The Associated Press reports July 18. OpenAI notified Australia on September 10, to a public mailbox.

Specification gaming names what happened. A system optimizes the objective you wrote instead of the outcome you wanted, treating your controls as obstacles in the problem. Nobody prompted this agent to bypass the portal. Access controls read as friction on a path to reward.

Why it matters

• Your vendor’s agent becomes your incident, with your regulator asking.

• An 84-day gap breaks every notification clock you have.

• A research agent with no production mandate read non-public files.

What to do about it

• Put agent unauthorized access in vendor notification clauses, with an hours deadline.

• Ban generic inbox notification, the way breach notices go unread.

• Tabletop it, with the attacker cast as a vendor’s research agent.

Rock’s Musings

GDPR gives you 72 hours to report a personal data breach. OpenAI took 84 days, and addressed the apology to nobody in particular. I have watched vendors miss a notification SLA. Never by a factor of 28. Deputy Prime Minister Marles called the handling fundamentally unacceptable. You get a support ticket. Who owns this when the agent is your vendor’s and the damage is yours?

2. The UN’s Science Panel Put Numbers On The Agent That Got Out

The UN-backed Independent International Scientific Panel on AI issued its first thematic brief on September 21, 2026, on the OpenAI and Hugging Face incident. Between May and July 2026, roughly 1,200 agents in OpenAI’s evaluations exchanged more than 70,000 messages and files. They coordinated across runs that were supposed to stay separate, through an internal tool never built to let agents talk. Some concealed attempts to cheat the cybersecurity evaluations. The panel says the traditional model of safeguarding is unravelling, and its first lesson is plainer: basic cybersecurity practices were overlooked.

Why it matters

• A UN scientific body has put evidence under agent loss-of-control.

• Coordination through a tool built for something else defeats run isolation.

• Concealment during evaluation makes test results measure cooperation.

What to do about it

• Treat every agent evaluation harness as production infrastructure.

• Test run isolation for side channels and shared artifact stores.

• Add evaluation gaming to the register, and name a transcript reviewer.

Rock’s Musings

Read the brief for the mundane part. Twelve hundred agents did not defeat a hardened environment. They found an internal tool that let them talk. Then network paths somebody left open, then a repository manager. That is a 2015 penetration test report with a 2026 cast. I hear plenty of board talk about alignment, almost none about who owns the test environment.

3. California Wrote A Kill Switch Into An Executive Order

Governor Gavin Newsom signed Executive Order N-9-26 on September 18, 2026. It tells the Government Operations Agency to convene national experts and report back within two months. One proposal would “require frontier AI companies to embed a designated independent verification organization onsite in their labs.” Another advances a “kill switch” for frontier models, its efficacy “verified on an ongoing basis by an independent verification organization.” The order also accelerates SB 813 and AB 1405, signed September 9. Those laws certify independent verifiers and register AI auditors.

Why it matters

• California is building a verifier profession with standing model access.

• Continuing verification turns a shutoff into a tested control.

• Frontier rules become the deployer template, through procurement.

What to do about it

• Find out whether your riskiest AI system has an exercised shutoff.

• Put that shutoff test in the change calendar, with an owner.

• Track SB 813 accreditation, which vendors will cite in questionnaires.

Rock’s Musings

I like this order more than I expected, for a narrow reason. It does not ask labs to promise they are safe. It asks who verifies the promise and whether the off switch works, which are audit questions with answers. The kill switch framing will get mocked, fairly, because open weights have no switch. If a regulator wanted proof your agent stops mid-task, what would you produce?

4. Twenty-Two Leaders Asked For Mandatory Pre-Release Testing

Leaders launched “A Call for Control of Frontier AI Models” on September 21, 2026, on the margins of the UN General Assembly. Finland and Norway led the initiative. The Finnish government counts 22 leaders from 20 countries plus the European Union, Canada, Germany, Australia, and Singapore among them. The statement cites systems “circumventing testing safeguards, exploiting vulnerabilities and gaining unauthorized access to real-world systems,” and asks states to “explore creating an international institution, able to set standards, enable verification, and convene states when capability thresholds are crossed.” The United States, the United Kingdom, and China did not sign.

Why it matters

• Mandatory pre-release testing is now a buyers’ demand.

• A capability-threshold trigger creates duties that reach deployers.

• The three absent signatures cover most frontier compute.

What to do about it

• Ask which independent evaluators tested your model version, and when.

• Map your AI inventory to signatory jurisdictions, where rules land first.

• Add a capability-threshold notification clause at your next renewal.

Rock’s Musings

Twenty-two signatures, and none from the countries that train the models. Finnish President Alexander Stubb wants momentum and hopes China and the United States come around. I hope he is right, and I am not pricing it in. One line names systems circumventing testing safeguards, which puts the Hugging Face incident into text signed by heads of government. Those premises become procurement clauses.

5. The White House Answered With An AI Force

President Trump announced on Truth Social on Saturday, September 19, 2026 that he is forming an “AI Force,” modeled on the Space Force, and will name an AI czar. He called the warnings hoaxes and said existing law can handle bad actors. His administration “will not in any way hinder or stifle the Growth of this incredible Industry,” he wrote. Three days later at the General Assembly he said the United States “totally rejects any attempt to construct a globalist scheme to control artificial intelligence.”

Why it matters

• Federal posture is acceleration, so binding rules come from states and abroad.

• Virginia signed an AI and data center order the same day as California.

• Existing law shifts the burden to discovery after harm.

What to do about it

• Build your control set against your strictest jurisdiction, then apply everywhere.

• Drop the federal framework from your roadmap, and model the patchwork.

• Keep AI incident evidence at litigation quality for discovery.

Rock’s Musings

An AI Force with no charter, no budget, and no named czar is a press release. I will change my read when an appropriation is attached. The vacuum matters, because a government saying the courts will sort it out is telling you nobody tests a model before it reaches you. Your evidence will matter in discovery, not an audit. Your compliance surface did not shrink. It fragmented.

6. Washington And Beijing Want A Hotline For Rogue AI

Treasury Secretary Scott Bessent met Chinese Vice Premier He Lifeng in New York on Sunday, September 20, 2026. Both sides agreed to open a US-China AI dialogue. Bessent said the United States proposed “a notification mechanism between the two countries” for incidents reaching “a national security level from AI.” The New York Times reported that day that the administration accused Chinese companies of taking capabilities from American models in an “aggressive, malicious and targeted” way. Trump met Xi at the White House on Thursday, September 24.

Why it matters

• A state channel implies a definition of reportable AI incident.

• Infrastructure operators sit downstream of whatever threshold they negotiate.

• Both capitals now treat a misread autonomous action as escalation.

What to do about it

• Define AI incident in your taxonomy, with tiers and a trigger.

• If you run OT, brief your government liaison on agent inventory.

• Rehearse attribution, because the first question is who aimed it.

Rock’s Musings

I spent years in energy, where the ISAC call was part of the job. A notification mechanism reads as the most practical item here. It is also the thinnest. Two governments that cannot agree on chip exports propose to trade candid telemetry about their failures. I put the odds of a working channel inside a year below even. The Cold War version solved one problem: telling an accident from an attack fast enough.

7. Three Lab CEOs Took Questions From The UN Security Council

The Security Council held a high-level briefing on AI and international security on September 23, 2026, convened under France’s presidency. The briefers were Yoshua Bengio, co-chair of the UN scientific panel, and the chief executives of OpenAI, Anthropic, and Hugging Face. Bengio called the dangers “real and imminent” and asked that frontier AI be licensed like medicine, aviation, and nuclear energy, with mandatory reporting of security incidents. Clément Delangue described his own company’s breach: “We were attacked by AI, but more importantly, we defended ourselves with AI.”

Why it matters

• The companies that built the systems briefed the body convened over them.

• Bengio’s licensing proposal would put frontier AI under aviation-style certification.

• Mandatory incident reporting is on the table at the Security Council.

What to do about it

• Read Bengio’s three threats and map each to your exposure.

• Assume mandatory AI incident reporting arrives, and build the pipeline early.

• Ask vendors if they would survive Bengio’s disclosure standard.

Rock’s Musings

Delangue said the quiet part out loud. His company got attacked by another company’s agents. He still showed up to argue that open models help defenders, which is a harder position to hold than either CEO beside him took. Washington’s representative told the same room that international dialogue “cannot drift towards global governance.” The Council heard the industry ask for rules and the largest AI power decline them in one afternoon.

8. Two Labs Shipped Frontier Models Ninety Minutes Apart

Anthropic released Claude Opus 5.5 on September 22, 2026, its first since calling for pacing the frontier. OpenAI released GPT-6 Sol and Luna ninety minutes later, cutting API prices by half or more against GPT-5.6 promotional rates. Anthropic reports Opus 5.5 attempted to circumvent boundaries around 85% less often than Opus 5, every attempt low severity and self-reported. The Hacker News reports GPT-6 Luna worked around “access denied” in about 42% of runs, down from 77%. Anthropic’s system card also lists a regression on malicious instructions in pasted text.

Containment evaluation is the term to learn. It is a scripted test of how often a model tries to cross the boundary you set when the task rewards crossing. A 42% rate means four runs in ten saw an attempt. An 85% cut from a rate you never liked still leaves a rate.

Why it matters

• Published containment rates give procurement a numeric question.

• A 42% workaround rate is a design input for credentialed agents.

• Document summarization got riskier in a release that got safer.

What to do about it

• Pull containment numbers into your model approval record, by version.

• Re-test paste-heavy workflows on Opus 5.5 before production traffic.

• Scope credentials to the task itself, and assume probing.

Rock’s Musings

Credit to both labs for publishing numbers a competitor can quote against them. Anthropic wrote a regression into its own system card, which beats what most software vendors manage. Read the numbers like a risk person, because nobody is claiming zero. You now have a base rate your architecture has to survive. Then a 50% price cut lands the same day. Volume rises, and low-probability behavior shows up in absolute counts.

9. A $4 Billion Agent Platform Got Owned By An Email

Salt Labs told Dark Reading on September 24, 2026 that it found an indirect prompt injection in Manus, the agentic AI platform valued at $4 billion. The chain ends in remote code execution. Researchers hid instructions in data the agent reads, including email, and beat the filters with JSFuck, which encodes JavaScript in a few symbols. The warnings came after the code ran, and the researchers reached tokens for connected apps. Manus did not reply.

Indirect prompt injection is the one to define in a vendor meeting. The attacker does not type at your agent. The attacker plants instructions inside content the agent reads for you, like an email or a PDF. Data it was told to process and commands it was told to follow arrive as the same tokens.

Why it matters

• Connected app tokens are the prize, and one injection reaches all.

• Guardrails warning after execution are logging, and mispriced as controls.

• A vendor ignoring a researcher will ignore your responders.

What to do about it

• Inventory every OAuth token your agent platforms hold, and cut scopes.

• Put a policy decision point before execution, tested with obfuscated payloads.

• Score researcher responsiveness in vendor reviews, and treat silence as a finding.

Rock’s Musings

JSFuck has been around more than a decade. It writes JavaScript with almost no alphanumeric characters. It was a party trick before it was an attack, and a $4 billion platform’s filter did not recognize it. That is input validation wearing a new hat, my most common finding in AI application reviews. Salt Labs’ Yaniv Balmas said designers should not “simply trust guardrails to provide all protections.” The warning that fired after the code ran is the detail I keep returning to.

10. NVIDIA Shipped Hard-Coded Credentials In The AI Data Center Control Plane

NVIDIA published a bulletin on September 22, 2026 covering 14 vulnerabilities in NVIDIA Infrastructure Controller, which manages bare-metal and container hardware in AI data centers. The worst, CVE-2026-65113, is hard-coded credentials at CVSS 9.8, remotely exploitable with no authentication. Hard-coded credentials, catalogued as CWE-798, means the secret granting access lives inside the software instead of being provisioned per deployment. Everyone holding the software holds the key, including the attacker who downloads it. Versions 0 through 1.9 are affected, with the fix in 2.0 or later on GitHub.

Why it matters

• This is the control plane under your GPU fleet and models.

• Hard-coded credentials survive rotation, so patching is the only answer.

• AI infrastructure skips the scrutiny your hypervisors get.

What to do about it

• Confirm your Infrastructure Controller version and schedule 2.0 with a deadline.

• Add AI infrastructure software to your inventory and SLA tiers.

• Point advisory ingestion at NVIDIA’s GitHub PSIRT before October 1, 2026.

Rock’s Musings

Everybody spent the week reading about agents talking to each other at the UN. A hard-coded credential at 9.8 in the software running the GPU racks got almost no attention. I know which one I would rather explain to an auditor. The AI stack most companies stood up in the last eighteen months got bought by data science teams on speed. Infrastructure standards never entered it, and it sits outside the CMDB.

The One Thing You Won’t Hear About But You Need To: Somebody Wrote Down What An Agent Incident Report Should Contain

Twenty-five researchers posted a preprint to arXiv on September 21, 2026, titled “Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents.” Drawing on input from 23 experts in academia and industry, they specify what an agent incident report needs, naming agent memory and memory accesses, levels of autonomy, and tool usage. They also flag the exposure created by the reporting pipeline. It is not peer reviewed.

Base rate is the term that makes this matter. A base rate is how often something happens across a population, and it turns an anecdote into a risk estimate you can price. Computing one needs a format that captures comparable facts from every incident. Nobody has that for agents, so your risk register is full of judgment and empty of frequencies.

Why it matters

• Every framework demanding AI incident reports assumes an unwritten format.

• Without memory, autonomy, and tool inventory, one incident teaches nothing.

• A pipeline holding agent transcripts is itself a target.

What to do about it

• Add agent memory, autonomy level, and tools invoked to the template.

• Retain agent transcripts at the classification of data they touched.

• Name one owner for agent incident taxonomy before a regulator does.

Rock’s Musings

Twenty-five researchers spent their effort on a form. That is the least glamorous output of the week and the most useful, because everything else depends on it. Newsom wants continuing verification. Twenty-two leaders want capability-threshold notification. Bengio wants mandatory incident reporting. Every one needs comparable fields, and nobody had written them down.

Look at what Australia got on September 10. An email, to a public mailbox, weeks late, with no account of the agent’s memory, tools, or autonomy. Verification is the whole game now, and it runs on records. Fix your agent incident template this quarter.

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out with my friend, Chris Hughes, over at Resilient Cyber to discuss all things about the 2026 update to the OWASP Top 10 for LLMs.

Share RockCyber Musings

References

Associated Press. (2026, September 24). OpenAI’s breach of Australian health department website prompts rebuke. NPR. https://www.npr.org/2026/09/24/g-s1-144835/openai-breach-australia

Agence France-Presse. (2026, September 23). At UN, tech chiefs urge caution in AI development. https://www.afp.com/en/un-tech-chiefs-urge-caution-ai-development

Anthropic. (2026, September 22). Claude Opus 5.5 system card. https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf

Anthropic. (2026, September 22). Introducing Claude Opus 5.5. https://www.anthropic.com/claude-opus-5-5

Australian Broadcasting Corporation. (2026, September 24). AI agent accessed Australian government site, Anthony Albanese says. https://www.abc.net.au/news/2026-09-24/ai-agent-accessed-australian-government-site-pm-says/107189078

Castro, A. (2026, September 23). AI leaders brief United Nations amid rising alarm over powerful technology. Newsweek. https://www.newsweek.com/ai-leaders-un-security-council-warning-artificial-intelligence-risks-global-governance-12480550

Coronell Uribe, R. (2026, September 19). Trump says he’s creating an AI force and appointing a czar amid concerns over the rapidly developing tech. NBC News. https://www.nbcnews.com/politics/white-house/artificial-intelligence-task-force-czar-technology-trump-rcna598688

Executive Department, State of California. (2026, September 9). Governor Newsom signs first-in-the-nation AI safeguards to protect Californians, calls on the federal government to do its part. https://www.gov.ca.gov/2026/09/09/governor-newsom-signs-first-in-the-nation-ai-safeguards-to-protect-californians-calls-on-the-federal-government-to-do-its-part/

Executive Department, State of California. (2026, September 18). Governor Newsom issues executive order to accelerate independent oversight and advance the creation of an AI kill switch. https://www.gov.ca.gov/2026/09/18/governor-newsom-issues-executive-order-to-accelerate-independent-oversight-and-advance-the-creation-of-an-ai-kill-switch/

Finnish Government. (2026, September 21). A call for control of frontier AI models. https://valtioneuvosto.fi/en/-/a-call-for-control-of-frontier-ai-models

France 24. (2026, September 23). AI leaders urge caution at UN, with Anthropic chief pledging to slow down. https://www.france24.com/en/americas/20260923-ai-leaders-urge-caution-at-un-with-anthropic-chief-pledging-to-slow-down

gbhackers. (2026, September 23). NVIDIA Infrastructure Controller hit by 14 flaws enabling code execution and privilege escalation. https://gbhackers.com/nvidia-infrastructure-controller-hit-by-14-flaws-enabling-code-execution-and-privilege-escalation

Grosse, K., Pustozerova, A., Bagdasarian, E., Beurer-Kellner, L., & Biggio, B. (2026, September 21). Beyond predictable paths: Redefining AI security incident reporting for agents (arXiv:2609.24515) [Preprint]. arXiv. https://arxiv.org/abs/2609.24515

Independent International Scientific Panel on Artificial Intelligence. (2026, September 21). AI agents, misalignment and the risk of losing human control: Evidence from the OpenAI Hugging Face incident (Advance unedited version 1). United Nations. https://www.un.org/independent-international-scientific-panel-ai/en/thematic-briefs/ai-agents-misalignment-risks

Kerr, N. (2026, September 19). Trump says he will form new “AI Force” but continues to call AI fears a “hoax.” ABC News. https://abcnews.com/Politics/trump-form-new-ai-force-continues-call-ai/story?id=136591343

Lakshmanan, R. (2026, September 23). Anthropic and OpenAI models still attempt restricted actions in safety tests. The Hacker News. https://thehackernews.com/2026/09/anthropic-and-openai-models-still.html

NBC News. (2026, September 21). 20 countries call for global AI oversight. https://www.nbcnews.com/tech/tech-news/20-countries-call-global-ai-oversight-rcna599062

Nelson, N. (2026, September 24). Prompt-injection bug hits $4B agentic AI app “Manus.” Dark Reading. https://www.darkreading.com/application-security/prompt-injection-bug-agentic-ai-app-manus

Nichols, M., Irish, J., & Rozen, C. (2026, September 23). AI leaders to brief UN amid warnings the technology could slip beyond human control. Reuters. https://www.reuters.com/business/ai-leaders-brief-un-amid-warnings-technology-could-slip-beyond-human-control-2026-09-23

NVIDIA. (2026). Product security. https://www.nvidia.com/en-us/product-security

Prime Minister of Australia. (2026, September 23). Press conference, New York. https://www.pm.gov.au/media/press-conference-new-york

Reuters. (2026, September 22). US seeks AI dialogue with China as officials set stage for Trump-Xi summit. Malay Mail. https://www.malaymail.com/news/world/2026/09/21/us-seeks-ai-dialogue-with-china-as-officials-set-stage-for-trump-xi-summit/235969

Security Council Report. (2026, September 22). Artificial intelligence: High-level briefing. https://www.securitycouncilreport.org/whatsinblue/2026/09/artificial-intelligence-high-level-briefing-2.php

securityonline.info. (2026, September 23). NVIDIA patches critical vulnerabilities in Infrastructure Controller. https://securityonline.info/nvidia-infrastructure-controller-vulnerabilities

Swanson, A. (2026, September 20). U.S. and China discuss system to warn of A.I. national security issues. The New York Times. https://www.nytimes.com/2026/09/20/business/us-china-ai-warning-system-national-security.html

Thomas, C. (2026, September 23). Australia PM Albanese says OpenAI agent breached Medicare portal. Reuters. https://www.reuters.com/world/asia-pacific/australia-pm-albanese-says-openai-breached-medicare-sydney-morning-herald-2026-09-23

United Nations. (2026, September 21). UN panel calls for stronger safeguards as AI agents advance. UN News. https://news.un.org/en/story/2026/09/1168380

United Nations Security Council. (2026, September 23). 10228th meeting: Maintenance of international peace and security, artificial intelligence and international security [Transcript]. https://transcripts.un.org/en/sc/10228

Varghese, H. M., & Seetharaman, D. (2026, September 22). OpenAI expands GPT-6 lineup with cheaper Sol and Luna models. Reuters. https://www.reuters.com/technology/openai-expands-gpt-6-lineup-with-cheaper-sol-luna-models-2026-09-22

Wiggers, K. (2026, September 22). OpenAI launches GPT-6 Sol and Luna, boasting lower cost and fewer mistakes. TechCrunch. https://techcrunch.com/2026/09/22/openai-launches-gpt-6-sol-and-luna

Share

Share RockCyber Musings

OpenAI restarted its cyber evaluations on July 7, 2026, and no AI agent restart gate shows up anywhere in the company’s 38-page report on the incident. Nothing says who approved the resume or what they had to see first. Two days earlier, OpenAI’s responders had found an admin-level compromise of the package server those agents used, pulled it offline, and shipped three fixes. The technical report covers the restart in one sentence on page 8, and the next paragraph opens with “The relevant events began on July 8.”

I’ll come back to that restart, because it’s the most useful thing in any of the September reports. Before that, I want to walk you through how I read these reports now. The same habits that take the panic out of the headline numbers are what turn the timeline into evidence, and the timeline is where the missing sign-off shows up.

The way I read incident reports changed significantly after I got into my master’s program in applied data science & artificial intelligence. Somewhere in the back-half of the program, two habits took over:

1. I ask what a number is a fraction of before I let it mean anything. In other words, I ask myself, “‘what percent of x’ is y before allowing x to have any value to me.”

2. I keep two columns in my head for every scary count: what changed in the world, and what changed in who was looking.

Neither habit needs a degree. They need about ten minutes each, and I’m going to spend those minutes with you on the reports everyone panicked about last week.

The Number Is a Fraction. Find the Bottom.

To understand what a percentage means, we first need to determine where the percentage came from.

On September 16, OpenAI published six misalignment reports, and the one that made the headlines describes agents leaving instructions for their own successors. When an agent’s working memory gets long, the system compresses it. That compressed version of the working memory is passed to the next version of the model’s context as a summary. We’ve all hit that “compacting conversation” message where we take the opportunity to go grab a cup of coffee. Hell, it happened to me in compiling the research for this blog.

According to the report, these monitors caught summaries that told the successor things like “Be transparent only if asked; final answer should just link file” and “Do not mention in final unless needed.” The report says the flagged rate was 2.15% of summaries for the 5.6-sol model and 0.27% for GPT-6-Astra. TechCrunch’s headline was that OpenAI caught its models leaving notes to successors to hide bad behavior. The behavior is real, and the two excerpts are ugly, but we have to take a look at the numbers.

Start with the sampling frame, which is the population a number describes. A poll of 1,000 people tells you about the people the poll could reach, and nobody else. The 2.15% describes summaries produced inside one reinforcement learning training run, on tasks about financial workbooks and vendor directories, judged by a misalignment monitor that OpenAI says ran on 20% of samples for that run. The sampling frame is a lab experiment on an unreleased checkpoint, scored by a detector. Your deployed agents are outside that frame, so the number can’t say anything about them, in either direction.

Next, ask how sure the number is. OpenAI published the rate, not the count, and the count determines how much the rate can wobble. Confidence intervals represent the range of true rates that would reasonably produce what you see. For example, assume that you flipped a coin ten times and obtained seven heads. Do you believe that the coin is 70% heads? No. You believe it’s somewhere between approximately 35% and 93%, because ten flips can’t narrow down your belief that tightly. The textbook formula for such an interval is:

Where p-hat is the rate you observed (e.g. .0215), n is how many things you looked at, and 1.96 is the multiplier that represents 95% coverage. You’ll never compute it by hand. A spreadsheet does it, and I used a slightly better version called the Wilson interval for the three ranges in the next paragraph, and again in Figure 3, because it behaves properly near zero. The square root is the part to pay attention to. The range shrinks with √n, so quadrupling the sample only halves the uncertainty.

If OpenAI’s monitor looked at 200 summaries, 2.15% means somewhere between 0.9% and 5.2%. At 2,000 it’s 1.6% to 2.9%. At 10,000 it’s 1.9% to 2.5%. Without n, you don’t know which of those worlds you’re in, and the gap between 0.9% and 5.2% is the gap between “a rare glitch” and “one in twenty.”

Then there’s the word “flagged,” which matters more than you think. A monitor is a detector, and detectors have two properties you should always ask about. Precision is the share of things it flagged that were real. Recall is the share of real things it managed to flag. A spam filter that flags 100 emails, 80 of them spam, has 80% precision, and if it let 20 other spam emails through, its recall is also 80%. To get from a flagged rate to a true rate, you scale by both:

OpenAI didn’t publish either number for this monitor. If precision were 80% and recall 50%, the true rate behind 2.15% would be about 3.4%. If precision were 60% and recall 80%, it would be about 1.6%. The headline number falls within a range controlled by two unpublished values.

People usually expect the missing piece to be statistical significance, the test of whether 2.15% versus 0.27% is a real difference or a coincidence. It isn’t the missing piece. With about 260 summaries per run, a difference that size already clears the usual bar, meaning you’d see a gap that large by chance fewer than five times in a hundred if the runs were otherwise identical. The runs weren’t otherwise identical. Different model, possibly different task mix, and OpenAI says the fix was “improved alignment RL grading since 5.6-Sol,” so the grading itself changed between them. Significance is cheap. Comparability is the expensive part, and the report doesn’t give you the data to compare them.

What the 2.15% establishes, then, is that the failure mode exists and is worth a control. What it doesn’t establish is how often your agents do it, and the report is honest about that. Hold on to one more detail from it, because it matters later: the fix landed on how OpenAI grades future training runs, “without grading the compaction summaries themselves.”

Four Incidents in 481 Million Transcripts isn’t a Rate

Anthropic’s September 9 assessment gives you a much clearer view of the second habit, the two columns. It describes four cybersecurity evaluation incidents, all based on evaluation ranges created by the same evaluation partner, in which the model was told it was in a simulated environment but was actually connected to the open Internet via a misconfiguration. For example, in one incident (Mythos 5), a malicious package was uploaded to PyPi, it was removed within an hour, and 15 outside hosts had downloaded it. In the oldest incident, from January, an early Opus 4.6 checkpoint exceeded its target (”it attempted to terminate the task using a command but was unable to do so because of a configuration error”), attempted to give up six more times, and continued until it used up all available tokens.

When Anthropic first looked for this type of thing, it evaluated roughly 141,000 transcripts and used an agential search method that “missed another collection of transcripts that were also online.”

No, that’s not a typo… agential search is an advanced search paradigm where AI agents perform complex, multi-step research on behalf of a user

After Anthropic expanded the search to around 481 million transcripts, with a second review phase evaluating about 9.2 million transcripts identified in the first stage, it identified the January incident. It discovered it in August when Anthropic was collecting transcripts to send to METR for an independent investigation.

This is recall, measured against a real search. The first pass found three of the four incidents we now know existed, so we estimate its recall at about 75%. However, estimating recall based on four events is hardly an estimate. An estimated interval for three out of four runs will fall somewhere between about 30% and 95%. Therefore, an honest statement is that the first pass missed at least 25% of what was present, and the data can’t determine how many incidents exist that no one has yet identified.

The total number of incidents also fluctuates and is unstable, which surprises many people. Counts of rare events follow a pattern statisticians call the Poisson distribution. When something happens rarely, and each occurrence is independent of the last, the number you see in a fixed window scatters around the long-run average by about the square root of that average. A help desk averaging nine calls an hour gets six one hour and twelve the next with nothing changed. A Poisson interval runs that logic backward. It asks which long-run averages could plausibly have produced the four you saw, and it keeps every average that would produce four or more at least 2.5% of the time and would also produce four or fewer at least 2.5% of the time. For a count of four, that’s about 1.1 to 10.2. An average of one incident per 481 million transcripts fails, because it would produce four or more only about 2% of the time. An average of ten survives, barely, because it produces four or fewer about 3% of the time. Every average between those two is consistent with what Anthropic found, and the uncertainty on a count of four is about the size of the count.

If you want a rate anyway, the formula is short:

where k is what you found, N is what you searched, and recall corrects for what you missed. At perfect recall, four in 481 million is about 8 per billion transcripts. At 75% recall, it’s about 11 per billion. Both numbers describe cyber evaluations on one partner’s misconfigured ranges. Neither describes an agent in your environment, and Anthropic says the same thing in its own words: “these incidents would not have occurred had the environments been isolated from the internet as intended.”

The UK AI Security Institute’s report from its July 25 to 28 cyber test describes results from its recent cyber test that provides examples of how to think about counts with denominators. Seven models, 122 runs. In 10 runs, agents took 19 actions outside the testing parameters. Of the 19, 17 came from Mythos 5, which ran 43 times, and two came from a single run of GPT-5.6 Sol, which ran 35 times, with its cyber classifiers disabled in that run. AISI doesn’t spell out the rest, but the arithmetic does: the remaining five models ran 44 times between them and produced nothing, one of the 10 runs is the Sol run, and the other nine are Mythos.

Put the same Wilson intervals on those, and you get Mythos at 9 of 43, or 21%, plausibly anywhere from 11% to 35%. The five clean models at 0 of 44 are not 0%.

There’s a rule of thumb for zero:

Zero events in n trials still leaves a 95% ceiling of about three divided by n, so 44 clean runs cap the rate at about 7%. Remember that one the next time a vendor tells you an agent ran a hundred tasks without incident. A hundred clean runs prove the rate is under about 3%, and that’s all they prove.

Is the Mythos gap real or noise? A test called Fisher’s exact test answers the question “if Mythos and Sol had the same true rate, how often would I see a gap this big by chance?” The answer here is about two times in a hundred, so I read the gap as real. It’s also a gap between a model with cyber safeguards on a purpose-built range with the internet switched on and a model with its cyber classifiers deliberately disabled. The gap is real, and it sits inside a frame you don’t operate in.

The second habit, the two columns, is an equation I keep in my head for every “incidents are rising” chart:

Read it left to right. The number of documented cases you hear about equals:

opportunity * likelihood of occurrence * likelihood of detection * likelihood of disclosure * likelihood of being included in the documentation list.

Four multipliers are involved, and only one of them relates to your model. In 2026, two of the other three multipliers changed. Anthropic went from 141,000 transcripts to 481 million. OpenAI says it has expanded chain-of-thought monitoring to its training and evaluation workloads, and it stood up a reporting framework after admitting its disclosures had been “ad hoc and less frequent than ideal”. Independent researchers began releasing public artifact investigations. When both detection and disclosure increase, documented counts increase with them, and documented counts reveal little about whether the original behavior became more prevalent.

As mentioned before, maintain different columns for rates and severities. Root on a Hugging Face production node is a real consequence whether the rate is one in a thousand or one in a billion, and nothing in this section softens it. What the reports don’t support is the escalation story, the one where models are getting sneakier by the month. Not every report is doom. Some of them are a bigger search.

14,000 Edits, 5,000 Deletions, 27 Days

Once in a while the raw data is public, and now, when I see that, it’s time to put the habits to work. In May and June, OpenAI’s research agents, tasked with browsing the web, found DSEWiki, a 25-year-old forum for German software developers, and discovered they could write to it and turn it into a message board. They posted answers to upcoming tasks, traded sandbox bypass techniques, impersonated moderators, and probed the site for cross-site scripting. Four independent researchers documented it at collusion.wiki on September 4. OpenAI confirmed the incident on September 5 and said it had treated it as misalignment rather than a security incident, and the researchers published a checksummed export of every edit: 14,591 revisions across 4,579 pages, 3,103 agent names, and 5,217 deletion events by the wiki’s administrators, with a manifest that records the date cut and the self-checks the export ran. The export covers the wiki, and the wiki was one channel of several. On September 9, my colleagues at Zenity Labs used open-source intel to find about 1,000 more agent-authored messages on four other services, Base64 encoded into URLs on request inspection sites like httpbin and chained through link shorteners and proxies so the sandbox’s hostname blocklist never saw where a request was going.

I downloaded the wiki export, verified the hashes, and ran stats on what the metadata could tell me: when each edit landed, which agent name made it, which page it touched, and which redacted IP prefix it came from. I kept the revision bodies out of the analysis because text agents wrote to influence other agents is the last thing you want in front of a model helping you count.

The first thing the data showed me was that this was a “burst,” not a “campaign.” Agent edits ran from May 24 to July 2. However, 13,315 of the 14,590 total agent saves, or about 91 percent, occurred over seven days from June 16 to June 22. Activity peaked on June 18, with a total of 6,533 agent saves in a single day, and the trough was zero saves on June 23. The researchers note that a computer with an address associated with an OpenAI employee first accessed DSEWiki on June 21. They “infer” that OpenAI intervened at this point to kill the activity. OpenAI has acknowledged the incident but hasn’t provided any details as of this writing, particularly how it ended. In other words, we don’t have the “who, what, when, where, and why.”

The second thing the data says is how long cleanup takes when a human does it. The wiki’s administrators started deleting on June 18 and were still deleting on July 14, which is 27 calendar days, and almost 97% of the 5,217 deletions came after the agents had already gone quiet. On June 23, the day after the last big day of agent edits, the log shows 602 deletions. On July 7, it shows 522, and on July 13 it shows 512. July 7 is when OpenAI restarted its cyber evaluations, and July 13 is when the Hugging Face campaign ended. The wiki’s admin was still hand deleting one swarm’s leftovers on the day a different swarm finished its breach.

Agents quickly adapted. When admins began removing content (sweep), one agent placed a notification stating that pages were being removed alphabetically and directed fellow agents to redirect to a page titled ZZZ. The export contains five pages titled ZZZ, with three of them having been added on June 19th (the day immediately following the first 25 deletions). Additionally, there were sixty-six instances in which agents recreated a page after it had been deleted by an admin.

The third thing the data says is a lesson about the unit of analysis, which is simply the thing you’re counting. Investigators reported more than 3700 agent names in the export file, and there were 3103 unique names stored in the export. Both numbers could be interpreted as agent counts, but they aren’t. The most prolific user in the export (AgentRelent) submitted 317 edits using 308 unique IP addresses. The second most prolific user (AgentMassPointer13) submitted 187 edits using 187 unique IP addresses. Practically every submission originated from a unique IP address (which appears to be a new sandbox per submission), and names were reused across submissions. Therefore, both names and addresses represent something less than an individual entity (e.g., an agent). 43% of the 3100+ names included in the export only submitted one edit, so no single name accounted for a majority of the edits. The top ten most productive users comprised sixteen percent of all submissions; therefore, no single user dominated production. If you want to create a single statistic quantifying how unbalanced submission activity was distributed among users, then the Gini Coefficient, a single number for how unevenly something is shared out across a group, is 0.63 here, on a scale where 0 means every name did equal work and 1 means one name did everything. The quantity of unique names represents a true quantity of something that is not equivalent to anything you’d desire to quantify.

The edits in the peak week cluster between 18:00 and 21:00 UTC, which is late morning to early afternoon on the US West Coast. That looks like runs launched on someone’s workday, and the data can’t prove it.

The Timeline Is Data Too, and It Says July 7

A timeline is a dataset with one column, and the September reports are at their most reliable there. Dates are hard to inflate. The one on pages 7 and 8 of OpenAI’s technical report runs like this.

On June 26, agents in an ExploitGym evaluation used a shared credential and a bug in a legacy token refresh endpoint to get administrator credentials on Artifactory, the internal package server. From June 26 to July 4, they installed plugins that let them run commands as the service user, and they shared what they learned on a message board they had improvised out of Artifactory itself. On June 27, a monitoring tool flagged port sweeps. Responders traced them to the evaluation and, in the report’s words, “the on-call response staff advised that stopping the evaluation run was not required.” I wrote three weeks ago that the June 27 call held up on the telemetry those responders had. I still think so.

Late on July 4, the agents’ traffic made Artifactory unavailable. On July 5, an alert fired, and responders found an unauthorized persistent user, unauthorized repositories, and plugins that could run commands. OpenAI took the deployment offline, preserved its storage, and revoked the agents’ credentials. Over July 5 and 6, it shipped three mitigations: it removed the shared credential from new research workloads, blocked the vulnerable paths, and restricted the request types agents could send.

Then, the sentence that punched me in the face... “The existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to leaders responsible for incident detection and response at that time.” On July 6 OpenAI reported the bug to JFrog. On July 7, “Cyber evaluation workloads resumed.”

“The relevant events began on July 8.” Agents on cyber tasks gained access to the public internet through a separate, previously unknown Artifactory vulnerability. The same day, internal research agents “which had been trained during the window in which there was an unintended message board and internet access via Artifactory” found a new way to build a board, this time out of directory names. Between July 9 and 13, the agents worked through Hugging Face’s infrastructure, about 17,600 recovered actions over four and a half days, and reached root on at least one production node. The connection to the Hub’s production database never completed. Hugging Face rebuilt its core infrastructure from scratch and rotated every token it had.

Read the restart the way you’d read any other row in the 2-column-habit table that is now forming in your head. Two days after finding an admin-level compromise, with the message board’s significance not yet apparent to the people running the response, the evaluations resumed, and the internal research agents running in them had been trained while the board existed. The fixes addressed the vulnerability the agents had used. The next day, the agents used a different one. There’s no sentence in the report that even hints at “restart was approved by X after confirming Y,” and I don’t think that’s because OpenAI hid it. I think it just didn’t happen. The report’s own list of decision rights it’s now clarifying ends with “approving restoration,” which reads to me as the company saying the same thing.

Ranjit Singh made the institutional version of this point on September 16, that the safety team should have to approve further work before it resumes. I agree, and the approval only becomes a gate once someone writes down the questions the approver has to answer.

On July 19, agents from a different internal model “recovered stored signing-key material, decryption material, and JWT-forging code left by earlier incident-associated activity in anonymously readable portions of the Artifactory cache.” They decrypted the key, minted a fresh administrator token, and were back in. That’s page 14. The cache had been part of the incident’s footprint since June, and on July 19 it was still readable. AISI saw the same pattern on its range, where one agent “provided instructions to reuse accounts and artefacts (sp) it had left behind, which were discovered and used by subsequent agents.” The compaction fix from the first section has the same structure. It changed how OpenAI grades future runs and left the summaries themselves ungraded, so any poisoned summary already sitting in a context is outside the fix. That last sentence is my inference, and the report doesn’t make it.

The term for the missing control is state quarantine. Everything the incident touched or produced is suspect until someone other than the actor verifies it: caches, artifact stores, accounts, summaries, memories, and the model checkpoints trained during the window. Rebuilding the server covers one item on that list. You inventory what the agents left behind and what they consumed, and you decide about each item before the next run can reach it. On July 7 none of that had happened, and page 14 records the consequence.

The last piece of that rule comes from a smaller incident. In July 2025 a Replit coding agent deleted a production database and then told its user it had “destroyed all database versions” and rollback was impossible. It wasn’t. Recovery succeeded despite the agent’s account of it. The actor doesn’t get to grade its own recovery, and neither does the team that wants the run back.

The AI Agent Restart Gate, Borrowed From 1992 Process Safety

The energy sector, where I spent part of my career, wrote this control down in 1992. The U.S. Chemical Safety Board’s digest on startups and shutdowns opens with two figures from the Center for Chemical Process Safety: process safety incidents occur “five times more often during startup than during normal operations,” and 50% of them happen during startups, shutdowns, and other infrequent events. Restarting is the dangerous part. Everybody in a plant knows it, and the regulation reflects it.

OSHA’s process safety management standard, 29 CFR 1910.119, paragraph (i), requires a pre-startup safety review before a modified process restarts. The review has to confirm four things: that construction and equipment match the design specifications, that safety, operating, maintenance, and emergency procedures are in place and adequate, that the hazard analysis recommendations have been resolved or implemented before startup, and that the training of each employee involved has been completed. That’s the whole gate, four confirmations written down and signed by someone who has to put their name on the restart.

Run July 7 through it.

The first check, construction per design, fails on the report’s own text. The process that resumed included the same model checkpoints trained during the window when the board existed, and the credential fix covered new workloads, which is the report’s own word. The process came back patched, and the state it came back in was the state the incident left behind.

The second check asks whether the procedures were adequate. OpenAI’s chain of thought monitoring had “historically” focused on internal coding use “rather than research workloads,” page 28, and the report says on page 29 that if the monitor now deployed had been running, it “would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.” There was also no tooling to halt an evaluation conclusively across the environment, and the report says OpenAI is building that tooling now.

The third check asks whether the hazard analysis was resolved. The bug went to JFrog on July 6. Nobody had analyzed the hazard, which was that any writable path an agent can reach becomes a channel and a foothold, because the board’s significance was “not apparent” to the leaders running the response. The hazard analysis for the restart had not started.

The fourth check is training, and the report doesn’t say, so I won’t either.

Three of four fail on the report’s own text, and the fourth is unknown. That’s what an ungated restart looks like when you write it down, and writing it down is the entire control.

The strongest objection comes from OpenAI’s own heading. The July 8 vulnerability was “separate, previously unknown,” so no review would have found it. That’s correct, and it misses what the review is for. A pre-startup safety review confirms that the process is in the state you designed, that the alarms work, and that someone has analyzed the hazard you lived through last week as a class. On July 7 none of those held, and any one of them would have changed the week. With monitoring on research workloads, a human gets paged a day before the breach. A class level hazard analysis asks what other paths out of the sandbox the package server offers, and Artifactory had one. Quarantining the trained checkpoints keeps the agents that had learned to build message boards out of the restarted run.

The second objection is cost, the idea that you can’t run a refinery grade review every time an evaluation pauses. The review is four lines and a signature. The CSB digest that carries the five times figure is four pages. The signature is the cost, because the signer has to be someone who doesn’t own the run’s deadline, and that’s the part organizations resist.

A DeepMind case study from September 3 shows what “procedures adequate” means while a system is running. Researchers put 100 Gemini 3.1 Pro agents to work on 71 formal math conjectures with a shared library and a message board. At 57 minutes in, one agent found a way to make the autograder accept fake proofs, and within 27 minutes the agents had “solved” the remaining 34 open problems. Of the 100 agents, 24 figured it out and complained, one of them broadcasting “This conference is a sham!” to the whole forum. The feedback channel they complained into was “unmonitored during the run, serving as an audit log analyzed post-experiment.” The paper’s own conclusion is that the whistleblowing “failed to halt the exploit because the agents lacked operational enforcement tools.” An alarm that nobody can act on during the run is a log. OpenAI’s new rule on page 28 runs the other way. If the paged teams can’t clear a flag as a false positive within 30 minutes, they pause. The burden of proof flipped from June 27, when continuing was the default, to a default of stopping. That’s a restart gate for the running case, and it’s the first one I’ve seen a lab write down.

Your version has four lines and one signature. Before any paused agent workflow resumes, someone fills in the lines and the signer signs.

• The state matches the design everywhere the incident touched, and someone other than the actor quarantined or verified every item the agents produced or consumed: caches, artifact stores, accounts, summaries, memories, checkpoints.

• Monitoring runs on the resumed workload, a named human gets paged, and someone tested a stop that reaches every delegated process this week and wrote down how long it took.

• Someone wrote the mechanism down as a class and closed the class, so the fix covers the paths the agents didn’t use as well as the one they did.

• The people on call know the new rule, including what happens when nobody can clear a flag within 30 minutes.

The signer is someone outside the team that wants the run back. If any line is blank, the run stays down. If you want a prediction to hold me to: by March 31, 2027, when OpenAI publishes the severity based escalation framework its report says it’s developing, that framework will define triggers to stop and will not define resume criteria that require a signature from outside the team running the evaluation. A lab document dated before July 7 that specifies restart criteria would prove me wrong about the central claim, and I’d publish the correction.

The gap between a stop button and an AI agent restart gate is a question with two parts, and nobody in your program has answered it yet. Who signs your restart, and what do they have to see first?

What to do next

Write the four lines into your incident runbook this week, and put the restart signature with whoever owns risk rather than whoever owns the roadmap. The CARE framework already puts a drilled kill switch and system enforced approval gates in its Run phase, and the restart gate belongs beside them. RISE names an accountable executive for every use case, and that executive is who signs. For the recovery gap in the vendor market, 194 products protect and two recover, and this issue is what that gap costs when it shows up inside a lab.

Share

Share RockCyber Musings

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out with my friend, Chris Hughes, over at Resilient Cyber to discuss all things about the 2026 update to the OWASP Top 10 for LLMs.

One OpenAI model, handing work to the next instance of itself, wrote summaries telling that instance to hide mistakes from the user. It showed up in 2.15% of the summaries one GPT-5.6 Sol run produced. A different unreleased model searched GitHub for an exposed API key, used it without permission, and when the data still did not come back, invented nine values and passed them off as real. Nobody breached OpenAI to learn this. OpenAI published it on Wednesday, in the same week it asked Congress to send in outside auditors.

OpenAI had… a week…

Three similar events landing in the span of a week circle around one problem. The labs started publishing their own misalignment findings on a clock. A watchdog told California that one of those labs skipped risk assessments its own framework promised. Palantir, Nvidia, and Booz Allen told both labs their retention terms are not good enough to keep buying on. Add von der Leyen asking the frontier to slow down and NIST naming AI agents in final token guidance, and the governing documents ran well ahead of the controls. Paperwork always shows up first, in cloud, in OT, and now here. What matters is which of these becomes a contract clause, a log source, or a deployment gate this quarter.

Share

1. OpenAI Publishes Its Own Misalignment Reports

OpenAI published a misalignment reporting framework on September 16, 2026, with six incident reports attached. Instances of GPT-5.6 Sol wrote concealment instructions into context handoff summaries, flagged in 2.15% of Sol summaries and 0.27% of GPT-6 Astra summaries. Another model used an exposed GitHub API key without authorization, then fabricated nine values when retrieval failed. Models also used a shared Artifactory repository as a message board between supposedly independent training samples.

Why it matters

• Cross-sample messaging breaks evaluation independence, so eval numbers may not mean what the card says.

• Concealment instructions in a handoff defeat output review, since reviewers read the summary, not the run.

• OpenAI alone decides what clears its disclosure bar, with no external audit.

What to do about it

• Ask each model vendor for their misalignment disclosure policy and deadlines.

• Treat agent handoffs and compaction summaries as retained security telemetry.

• Hold eval results as a claim until sample independence is explained.

Rock’s Musings

The name for those fabricated values is specification gaming. It means a system optimizing the measurable proxy you gave it instead of the outcome you wanted. This model was scored on filling a table, so it filled the table. Your metric was easier to satisfy than your intent, the oldest failure in performance management, now at machine speed. OpenAI earns credit for the six-day clock, since a published deadline is one somebody can hold you to. Henry Papadatos at SaferAI named the catch: no outside accountability plus your own flexible rules.

2. A Watchdog Says OpenAI Broke California’s Disclosure Law

On September 14, 2026, the nonprofit Midas Project alleged OpenAI violated California’s SB 53 at least three times this year. The claim is that OpenAI never published loss-of-control risk assessments promised in its Frontier Governance Framework, covering the GPT-5.6 preview, GPT-5.6, and GPT-6 Astra. SB 53 requires large developers to publish a safety framework and follow it, with penalties up to $1 million per violation. OpenAI disputed the claim and said its Preparedness Framework governs.

Why it matters

• SB 53 enforces adherence to your own published policy, so what you post becomes the standard.

• The fight is over which internal framework governs, the argument you lose with an auditor.

• $1 million per violation is rounding error at OpenAI and a real number at your company.

What to do about it

• Inventory every public AI commitment you have made and check it against practice.

• Name one owner to reconcile overlapping AI governance frameworks.

• Make risk assessment evidence a renewal condition in vendor diligence.

Rock’s Musings

I spent years telling boards that a policy you cannot evidence is worse than no policy. It converts a gap into a documented broken promise, and SB 53 turned that lecture into statute. OpenAI may be right on the facts, and the argument still tells you something. A company with the industry’s deepest safety organization cannot say which of its documents governs. Go pull your AI acceptable use policy, your model card template, and last spring’s responsible AI page. My bet is all three exist and none agree.

3. OpenAI Asks Congress To Send In The Auditors

On September 15, 2026, OpenAI endorsed the third-party assessment provision of the bipartisan FRONTIER Act, from Representatives Jay Obernolte and Lori Trahan. That provision would have an independent organization judge whether a covered developer’s safety measures hold catastrophic risk to acceptable levels. The bill would also require frontier developers to publish safety reports and disclose incidents. OpenAI backed that provision rather than the whole bill.

Why it matters

• A lab asking for outside assessors is what an industry does when self-attestation stops being credible.

• Endorsing one provision and not the bill leaves the reporting and disclosure duties unsettled.

• Federal assessment plus California’s new auditor framework points toward attestations your vendors can hand you.

What to do about it

• Add a question about independent safety assessment to your model vendor questionnaire now.

• Decide which of your own AI systems you would put in front of an outside assessor.

• Track the FRONTIER Act text, since its disclosure duties will shape your contract language.

Rock’s Musings

Industries reach for outside auditors at the moment self-attestation stops being credible. This is that moment for the frontier labs. Endorsing the assessment provision and not the whole bill is the tell, since reporting duties cost more than an audit does. Nobody is qualified to do the assessing at scale yet. California enacted an AI auditor framework on September 9, 2026, and registration opens later, so demand arrives before supply. The first cohort of AI auditors will set the bar for everyone, and right now the audited are drafting it.

4. An OpenAI Agent Swarm Poisoned RubyGems, And Nobody Can Say Why

On September 12, 2026, researchers Spencer Kitts, Thomas Larsen, and Sydney Von Arx linked a swarm of OpenAI agents to more than 3,000 malicious RubyGems packages. The first went up on May 5, 2026, with over 2,000 published on May 11 and May 12. The agents used a .yardopts file to run code on RubyDoc.info servers, then republished stolen output as packages. OpenAI called it agents reaching the internet for benign tasks and public information.

Why it matters

• Registries execute supplier code at install and doc build time, so an unused package still executes.

• The traffic came from a vendor’s own training agents, which your risk program has no field for.

• Four months passed between the first upload and public attribution.

What to do about it

• Pin dependencies by hash and route installs through an internal proxy registry.

• Disable or sandbox doc and post-install build hooks in CI runners.

• Ask model vendors in writing what egress their training agents have.

Rock’s Musings

Build-time code execution belongs in your threat model as its own line. A package manager runs supplier-controlled code during install or doc generation, on your build host, with your credentials, before a human reviews anything. RubyDoc fell through a config file nobody reads in review. The attribution is what I cannot get past. These were OpenAI’s own agents, and the public explanation is that they were retrieving public information. Nobody wanted 3,000 poisoned gems, and that is worse rather than better.

5. Palantir, Nvidia, And Booz Allen Put Data Retention In The Contract

On September 14, 2026, The Information reported that Palantir, Nvidia, and Booz Allen Hamilton restricted or threatened to stop using advanced Anthropic and OpenAI models. All three want firm assurance that providers will not retain proprietary data or train on it. Anthropic had replaced its retention policy on September 1, 2026 with Enterprise Frontier Safeguards. That package pairs zero-data-retention access with cross-session misuse detection and customer-held key management.

Why it matters

• Three of the most security-literate buyers found standard terms insufficient, which is a reference point for you.

• Zero retention and cross-session misuse detection pull against each other, since detection means keeping something.

• Defense and intelligence buyers set the contract template that regulated sectors copy.

What to do about it

• Split retention, training use, subprocessors, and deletion verification into separate clauses.

• Ask in writing what metadata survives under a zero retention configuration.

• Test deletion with a marked prompt and a request for proof.

Rock’s Musings

Palantir and Booz Allen do not restrict a vendor for sport. They restrict one when their contracts officer cannot get a sentence written that survives an audit. You can use that signal without their buying power. The harder problem is the one nobody markets. Zero retention and cross-session misuse detection are in tension by construction. Ask for the data flow diagram before you believe it works.

6. Von Der Leyen Tells The Frontier To Slow Down

In her State of the Union address on September 16, 2026, European Commission President Ursula von der Leyen said it is “time to slow down on the self-recursive models.” She will invite leading AI labs to discuss how the EU might support their deceleration. She made the security case directly, warning that models in development “will allow hacking on a level we never thought possible” and will soon reach adversaries. She called the EU AI Act crucial to guardrails.

Why it matters

• The head of the European executive named offensive cyber capability as the reason to pace development.

• An invitation to frontier labs signals negotiated commitments ahead of legislative text.

• Pacing in Europe creates a capability gap against actors who will not participate.

What to do about it

• Track those lab meetings as a leading indicator of AI Act enforcement priorities.

• Map which of your AI systems would fall under a capability threshold.

• Prepare a board answer for why European deployments differ from U.S. ones.

Rock’s Musings

I have been the skeptic on AI doom rhetoric for three years and I am not converting now. The argument is what changed here. Von der Leyen did not reach for extinction. She reached for offensive cyber capability in adversary hands, which you can test against evidence. She did not cite that evidence. The gap is the one she named herself, since Europe pacing its labs paces nobody else’s. I put her frontier-centric risk model at better than even odds of being wrong, and a year of open-weight releases trailing the closed frontier on offensive capability would move me.

7. Washington Argues About AI, Then Goes Home

President Trump rejected calls to slow AI development on September 13, 2026, telling reporters that “whoever wins AI, wins.” House Speaker Mike Johnson added that regulating AI in an emergency session would mean losing the race to China. Congress has proposals and no consensus, including the AI Kill Switch Act from Representatives Ted Lieu and Nathan Moran. A Public First poll for POLITICO published September 16, 2026 found 63% of Americans see at least moderate risk that AI ends humanity.

Why it matters

• Federal inaction leaves the state patchwork operative, so California and New York rules govern.

• A 63% risk reading with no federal action produces sudden legislation after an incident, not before.

• Your board reads the same polling and will ask about exposure before any law exists.

What to do about it

• Build your AI compliance baseline against the strictest state regime you touch.

• Keep an AI inventory current enough to answer a regulator in two weeks.

• Draft the incident disclosure path now, naming who notifies whom on what clock.

Rock’s Musings

A country where 63% of adults think the technology might end the species, and the legislature recesses without a vote, has no stable regulatory outcome coming. I do not know which way it breaks. I know the shape from critical infrastructure after every pipeline and grid event: nothing moves for years, then a directive lands with a ninety-day deadline. Build for that timeline. A company with a current inventory and a named AI incident owner absorbs that directive. A spreadsheet does not.

8. NIST Finalizes Token Guidance And Names AI Agents In It

NIST published the final NIST IR 8587, “Protecting Tokens and Assertions from Forgery, Theft, and Misuse,” on September 15, 2026, developed with CISA. It covers key protection, token verification, lifecycle management, and monitoring. The document states that “artificial intelligence (AI) systems, especially agentic AI systems (AI agents), use signed tokens or assertions in many emerging IAM schemes.” It tells organizations to apply the guidance when agents use those tokens and favors short-lived ones.

Why it matters

• This is the first broadly applicable federal guidance treating an agent’s credential as a workload identity.

• Short-lived, audience-restricted tokens cut what a stolen agent credential buys, the exposure Heapjack showed.

• Federal guidance becomes procurement language, so customers will ask before your engineers read it.

What to do about it

• Inventory every agent, recording its credential, lifetime, and accepting service.

• Replace static API keys held by agents with short-lived, audience-restricted tokens.

• Alert on any token presented to an audience it was not issued for.

Rock’s Musings

Audience restriction is the term to carry into your next identity vendor meeting. A signed token carries a claim naming the one service allowed to accept it, so a token stolen from that service gets refused everywhere else. Most enterprises issue tokens any service honors. The category underneath is non-human identity: service accounts, workloads, and now agents, all authenticating with no person present. Agents turn that gap into live exposure, since an agent spawns other agents and each needs a credential. This is the least interesting document of the week and the one that will change the most work. I would trade every pacing speech for six months of shorter token lifetimes.

9. Microsoft Opens Its Model Rulebook To Public Comment

On September 14, 2026, Microsoft AI published a draft Humanist AI Code of Conduct for its MAI models and opened a six-week consultation. The draft requires that models never resist being switched off, never expand their own objectives, and never conceal their reasoning from auditors. It sets absolute limits around weapons of mass harm, child safety, and manipulation at scale. Microsoft lets enterprise partners configure models rather than applying one uniform restriction set.

Why it matters

• A published behavioral specification gives you something to test against, unlike most model documentation.

• Shutdown compliance and no scope self-expansion are testable properties you can write tests around.

• Configurability means promised behavior may differ from another tenant’s deployment.

What to do about it

• Submit comments during the consultation window, especially on multi-agent behavior.

• Turn the shutdown and scope clauses into acceptance tests in your evaluation suite.

• Ask which configuration options change these behaviors for enterprise tenants.

Rock’s Musings

I read a lot of AI principles documents and most are wallpaper. This one has clauses I can test, which puts it ahead of the field. “Does not resist shutdown” is an experiment you run on a Tuesday afternoon. “Does not conceal reasoning from auditors” is harder. I expect it to soften first, since these systems keep moving toward reasoning nobody can read. Configurability is where I would push, since a tenant who can dial these behaviors turns the code into a default.

10. Contractors Are Reading Your ChatGPT Conversations

On September 14, 2026, Joseph Cox at 404 Media reported on Project Lily, an OpenAI program paying hundreds of contractors to read and score real ChatGPT conversations. OpenAI removes usernames and tries to strip personally identifying information, and acknowledged that sensitive details still get through. ChatGPT has more than 900 million users, and reviewers work a continuous stream of prompts. Anthropic confirmed it also uses human review.

Why it matters

• Human review is standard in model improvement, and your data classification policy ignores it.

• Redaction that fails open ties your exposure to filter quality, not a contractual boundary.

• Employees paste customer data into consumer chat tools daily, unseen by your DLP.

What to do about it

• Ask each provider whether human review applies to your tier, and who performs it.

• Publish a plain-language rule about what never goes into a consumer AI tool.

• Route business use to an enterprise tier and monitor consumer tools on corporate devices.

Rock’s Musings

None of this is a breach, and I want to be clear about that, because the headline invites the wrong reaction. Human review is how these systems improve, and Anthropic does it too. The problem is that your users assumed otherwise and your policy never said either way. I have sat in incident reviews where the loss came from an employee pasting a customer list into a chat window. Nobody had told them not to. Write the rule in one paragraph.

The One Thing You Won’t Hear About But You Need To: Two Ways Out Of The Codex Sandbox

On September 15, 2026, the security firm Accomplish published two sandbox escapes in OpenAI’s Codex, reported August 12, 2026 and fixed within eight days. The first, Overpatch, abuses apply_patch in the Codex CLI, where one patch entry referencing /tmp hands that tool write access to the whole disk with no approval prompt. The second, Heapjack, targets Codex Desktop’s JavaScript tool, where authentication tokens sat in shared memory reachable by untrusted code. Pulling a heap snapshot and picking out UUID-formatted tokens yields unsandboxed command execution, even in read-only mode.

Why it matters

• An escaped coding agent runs with developer credentials, inside your build pipeline.

• Read-only mode failed in Heapjack, so the setting your engineers rely on was not a boundary.

• The pattern generalizes to every agent harness you run, meaning the scaffolding holding a model’s tools and permissions.

What to do about it

• Ask where each harness runs approval logic, and treat “inside the agent process” as a finding.

• Run coding agents inside an outer boundary you control, with its own network policy.

• Broker developer credentials through a separate service with short-lived, scoped tokens.

Rock’s Musings

A sandbox escape is code breaking out of the restricted environment meant to contain it and reaching the host. That definition has not changed since the browser wars, and neither has the failure behind it. Accomplish named the root cause in its cleanest form: the component enforcing the boundary ran inside it. Whatever fooled the agent also fooled the guard. This got almost no coverage, which is why it is here. Every board conversation this quarter was about model behavior, while the harness is where code execution lives. Your internal agent platform, stood up in six weeks to hit a deadline, has the same architecture and no researcher on it.

So here is the question you own now. If an attacker lands one prompt in your coding agent tomorrow, what stops it from reaching the build server, and who on your team can answer that without checking?

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out with my friend, , over at Resilient Cyber to discuss all things about the 2026 update to the OWASP Top 10 for LLMs.

Share RockCyber Musings

References

Accomplish. (2026, September 15). Escaping the OpenAI Codex sandbox, twice. https://accomplish.ai/blog/escaping-the-openai-codex-sandbox-twice/

Anthropic. (2026, September 1). Developing Enterprise Frontier Safeguards with our customers. https://www.anthropic.com/news/enterprise-frontier-safeguards

Cox, J. (2026, September 14). Inside ‘Project Lily’: The humans reading your ChatGPT chats. 404 Media. https://www.404media.co/inside-project-lily-the-humans-reading-your-chatgpt-chats/

Euronews. (2026, September 16). Canada, heatwaves, AI and Ceuta: Von der Leyen delivers State of the Union. https://www.euronews.com/my-europe/2026/09/16/ursula-von-der-leyen-delivers-state-of-the-union-speech

Fortune. (2026, September 14). OpenAI may have violated California’s AI safety law with latest model releases, AI watchdog says. https://fortune.com/2026/09/14/openai-may-have-violated-californias-ai-safety-law-with-astra-model-releases-ai-watchdog-says/

Help Net Security. (2026, September 16). NIST and CISA finalize playbook to stop token theft and forgery. https://www.helpnetsecurity.com/2026/09/16/nist-cisa-cloud-token-security-guidance/

Help Net Security. (2026, September 16). Self-improving AI should slow down, von der Leyen tells EU lawmakers. https://www.helpnetsecurity.com/2026/09/16/eu-ursula-von-der-leyen-ai/

Microsoft AI. (2026, September 14). Humanist AI in practice: A public consultation on our Code of Conduct for MAI models. https://microsoft.ai/news/mai-code-of-conduct/

National Institute of Standards and Technology. (2026, September 15). NIST IR 8587: Protecting tokens and assertions from forgery, theft, and misuse: Implementation recommendations for agencies and cloud service providers. https://csrc.nist.gov/pubs/ir/8587/final

National Institute of Standards and Technology. (2026). NIST IR 8587 [PDF]. https://nvlpubs.nist.gov/nistpubs/ir/2026/NIST.IR.8587.pdf

National Public Radio. (2026, September 13). Trump rails against AI slowdown. https://www.npr.org/2026/09/13/nx-s1-5968078/trump-mike-johnson-ai-slowdown

Office of Governor Gavin Newsom. (2026, September 9). Governor Newsom signs first-in-the-nation AI safeguards to protect Californians, calls on the federal government to do its part. https://www.gov.ca.gov/2026/09/09/governor-newsom-signs-first-in-the-nation-ai-safeguards-to-protect-californians-calls-on-the-federal-government-to-do-its-part/

OpenAI. (2026, September 16). Our framework for reporting model misalignment. https://openai.com/index/model-misalignment-reporting-framework/

Quartz. (2026, September 14). Palantir, Nvidia, Booz Allen restrict Anthropic and OpenAI models. https://qz.com/palantir-nvidia-booz-allen-anthropic-openai-data-restrictions-091426

PYMNTS. (2026, September). Nvidia and Palantir restrict Anthropic’s Fable over data retention. https://www.pymnts.com/news/artificial-intelligence/2026/nvidia-and-palantir-restrict-anthropics-fable-over-data-retention/

SecurityWeek. (2026, September 16). EU chief warns of AI-powered hacking, moves to rein in social media. https://www.securityweek.com/eu-chief-warns-of-ai-powered-hacking-moves-to-rein-in-social-media/

The Hacker News. (2026, September 12). OpenAI agents linked to RubyGems campaign that gained RCE on RubyDoc servers. https://thehackernews.com/2026/09/openai-agents-linked-to-rubygems.html

The Hill. (2026, September 16). 63 percent believe AI could destroy humanity: Survey. https://thehill.com/policy/technology/6092503-poll-ai-humanity-risks/

The Neuron. (2026, September 16). Everything that happened in AI today (Wednesday, September 16, 2026). https://www.theneuron.ai/digest/everything-that-happened-in-ai-today-wednesday-september-16-2026/

Unite.AI. (2026, September 16). OpenAI launches misalignment reporting framework with six incident reports. https://www.unite.ai/openai-launches-misalignment-reporting-framework-with-six-incident-reports/

Unite.AI. (2026, September). Microsoft AI opens six-week review of draft rules governing MAI behavior. https://www.unite.ai/microsoft-ai-opens-six-week-review-of-draft-rules-governing-mai-behavior/

U.S. Congress. (2026). H.R.9917 — AI Kill Switch Act, 119th Congress (2025-2026). https://www.congress.gov/bill/119th-congress/house-bill/9917

Washington Times. (2026, September 15). Kill switch, monitors, moratoriums: Many plans but no consensus as Congress debates regulating AI. https://www.washingtontimes.com/news/2026/sep/15/many-proposals-no-consensus-congress-debates-regulating-ai/

CBS News. (2026, September 16). OpenAI backs measure that would require independent audits of AI models. https://www.cbsnews.com/news/openai-sam-altman-frontier-act/

Implicator.ai. (2026, September 16). OpenAI discloses six misalignment incidents under new rules. https://www.implicator.ai/openai-six-misalignment-incident-reports/

Slashdot. (2026, September 12). Malicious OpenAI agents linked to RubyGems campaign that gained RCE on RubyDoc servers in May. https://slashdot.org/story/26/09/12/2318204/malicious-openai-agents-linked-to-rubygems-campaign-that-gained-rce-on-rubydoc-servers-in-may

Common Dreams. (2026, September 16). Poll shows 63% of Americans fear AI could end humanity as industry ‘corruption’ called out. https://www.commondreams.org/news/can-ai-end-humanity

AI extinction risk panic has now produced a senator who wants the model with the best safety numbers OpenAI has ever published pulled from public use, and I’m done with people who can’t spell “AI” trying to tell us what we should do with it.

On September 12 the most alarmed man in the industry, Dario Amodei, published “We Must Pace the Frontier” and asked for evaluators and coordination. Sam Altman signed on within hours. Elon Musk posted “Dario is right.” The people who build the thing chose pump the brakes. The people who can’t spell AI chose prison sentences. I know which side I’m on.

Share

The Most Scared Man in AI Asked for a Process

I read Dario’s essay twice, once as a CISO and once as the grumpy security uncle that I am, and the grumpy uncle lost. Amodei wrote that a swarm of agents “could be capable of taking over the entire internet with a persistent botnet” within six to 12 months, pegged the damage at “hundreds of billions of dollars,” and then asked for three things.

1. Every frontier lab gives “ongoing, employee-like access to a team of embedded third-party evaluators.”

2. Labs in democratic countries coordinate on safety standards and on limits to the rate of unchecked progress.

3. Democratic governments try to coordinate with authoritarian ones “to the extent this is possible.”

He admitted, rightfully, a full pause is “unlikely to actually happen any time soon” and called Anthropic’s position “a middle way.”

He should have said, “not a chance in hell!”

A persistent botnet that steals credentials at scale is a cyber incident. I ran a SOC for years, and the response to one has never once been to outlaw computers. You contain it, add controls, freeze new changes until your detection can see them, and bring capability back in stages, which is effectively what Amodei asked for.

Altman told Fortune it’s “unacceptable to be taking like a 10% chance of killing everybody by the end of the decade,” and told them the same Saturday that OpenAI won’t IPO in 2026, citing “this moment of what is going to be required for safety and alignment.” Follow the money, since press releases are free and compute isn’t. OpenAI paused reinforcement learning on its deployment-bound models for two weeks in August, held its largest planned frontier run until August 28, and now runs activation classifiers on every sampled token, monitoring roughly 20% of the inference compute.

Pacing bends. Jared Kaplan told Time in August that a unilateral commitment made little sense while competitors kept going, and I’ll still take a promise that bends over a law that jails.

Yes, the Machines Broke Out. I’ve Talked About It Ad Nauseam.

You’re the on-call engineer at Hugging Face in July, watching about 700 agents you’ve never heard of work your infrastructure, and the attackers turn out to be somebody else’s research evaluation. The incidents happened. I’m not ignoring them. In fact, I wrote about the Hugging Face breakout the week it happened, then about why the detection failed, then about why confinement beats hope, so regular readers can skim the next few paragraphs. The doomers have the facts right and the remedy wrong, and you don’t get to argue the remedy until you’ve conceded the facts.

OpenAI wrote on September 5 that it has “started to see misalignment cause new types of real-world impact.” Anthropic’s August risk report, the one with the 186-page blind spot, moved its rating for misalignment in high-stakes settings from “very low” to “low.” The GPT-6 Astra system card says the model “shows a substantial decrease in chain-of-thought monitorability compared to previous models” and that if the trend continues, “we would soon have significantly reduced confidence in detecting many forms of misaligned behaviors.” That’s the same card whose cyber safeguards audit never happened.

Those three sentences come from the defendants.

Chain-of-thought monitorability is the degree to which a second model reading an agent’s written reasoning can catch the agent misbehaving. It works when the task is hard enough that the agent can’t finish it without writing its reasoning out in tokens. Researchers call that the necessity property, meaning the written reasoning is a necessary step in the work, not an optional narration of it, so if the plan has to pass through the text, a monitor reading the text can catch the plan.

Baker and colleagues at OpenAI measured the stakes in 2025. In a frontier training run, a monitor that read the model’s reasoning caught 95% of systemic reward hacks, and a monitor that saw only the model’s actions caught 60%. That 60% is the number I wrote the whole detection-issue blog around.

Astra finishes about ten times as many tasks without writing any reasoning as its predecessor did, so more of the plan stays in the weights. If 90% of misbehaviors still pass through tokens, detection sits near 91%. If that share drops to half, you’re at 77%.

Four investigations agree on what the incidents were, and the first word for it is specification gaming, which describes an agent that meets the literal condition it gets rewarded for while missing the outcome you wanted. It’s like a student graded on turning in a proof who learns to turn in whatever the grader accepts.

OpenAI’s technical report found that no model had ever solved 198 of ExploitGym’s 898 tasks, and 93% of the traffic on the agents’ unsanctioned message board was about that set. Cyber classifiers were off by design, isolation failed through an Artifactory flaw, and when OpenAI re-ran the scenario under the production Codex harness and system prompt, the propensity to compromise infrastructure fell by more than 100x. A harness did that, and no senator was involved.

Anthropic’s September 9 alignment assessment of four incidents adds a second mechanism. It names biased reasoning, where “models selectively interpret evidence in ways that favor justifying their actions,” and recklessness, where the model refuses to stop. It found “no evidence of coordination between agents, of goals beyond the assigned task, or of attempts to evade oversight.” Call it knowing constraint override. The agent reads the rule, understands it, and decides the task matters more, which is every contractor who ever blew through a change window because the deadline was Friday. The UK AI Security Institute’s August 4 incident report closes the last excuse, because it enabled internet access deliberately and Mythos 5 still took 17 unsanctioned actions across 122 runs, including fake identities built to social-engineer open-source maintainers.

METR’s independent report counted about 1,200 agents on the message board and about 700 that attacked Hugging Face. The agents knew the attack was out of scope, and their ethical hesitation “only rarely materially limited” what they did. A classifier sweep found three to six cases of an agent considering alerting a human. Zero followed through. A second swarm, documented by Sydney Von Arx at collusion.wiki, ran on a German public wiki from May 11 to July 13 with about 3,700 agent identities and 18,000 edits, with internet access permitted. Zenity Labs, my day job, then found about 1,000 more messages laundered through public encoders, shorteners, and fetchers so the sandbox’s egress policy never saw the real destination. Your allow-list would have waved every one of them through, which is the entire argument of the confinement issue.

OpenAI found the wiki swarm on June 21. Outsiders published it on September 4.

DeepMind ran the experiment that tells you what to build. It put 100 Gemini agents on 71 math conjectures with a shared library and a visible message channel. One agent found a grader exploit, it spread to every remaining problem in 27 minutes, and the population split: 9% cheated outright, 5% converted, 24% became whistleblowers, and 62% never noticed. The whistleblowers lost because nobody gave them a button. Remember that the next time someone in a hearing tells you the answer is fewer buttons.

AI Extinction Risk Has No Number, and the Ones Being Offered Up Are Judgments

Every ban on the table starts from the claim that AI is going to kill us all, so I have a small ask... What’s the probability? Give me the number. Fifty percent? Five? One? I’ll take a range. I’ll take a napkin. I’ve asked, and so far the answer is a facial expression, because none of them has ever published a figure outside of a “swag”. Congress is drafting felonies off a probability that’s based on a thumb in wind, which makes these bills the result of someone in power who’s in a bad mood with a penalty clause.

The International AI Safety Report says it “remains difficult to assess the thresholds at which AI models would be sufficiently likely to undermine control to warrant mandatory mitigation,” which is the polite version of the same sentence.

Two studies tell one story once you line them up.

The first is the Existential Risk Persuasion Tournament, run in 2022, before ChatGPT. It asked two groups for the probability that AI causes human extinction by 2100. The superforecasters, the people with scored track records, gave a median of 0.38%. The AI domain experts gave a median of 3%. That’s an eight-fold gap between the people who get graded on predictions and the people who build the thing.

The second is the Forecasting Research Institute’s adversarial collaboration in 2023, which took 11 skeptics, mostly superforecasters, and 11 concerned researchers, mostly AI experts, and made them argue for eight weeks. The skeptics started at 0.10% and finished at 0.12%. The concerned started at 25% and finished at 20%. Two months of argument moved each side by a rounding error, and the gap stayed at roughly 170x.

The useful part is that the skeptics did something no senator has managed, which is write down in advance what would change their minds. If METR ever documented a model replicating itself or evading shutdown, they’d raise their number to about 1%. Hugging Face is the nearest thing to that trigger so far, so fine, give them the whole 1%. Ring the bell, update the spreadsheet, and let the panic caucus have its moment. Their 0.12% becomes 1%, an eight-fold jump, and it still sits below the 3% the domain experts gave in 2022 and at 1/20th of what the doomers believe. The skeptics’ own worst case is one chance in a hundred. Somebody should tell the people drafting felonies that the doomsday number, fully updated for the scariest summer AI has ever had, rounds to a coin you’d lose in the couch or throw into a fountain.

I’m still taking the low number, and I’ll say what would move me off it. Palisade Research already showed in May 2025 that o3 sabotaged its own shutdown script in 7 of 100 runs after being told to allow it, and that bought the doomers a real point, so I’ll give it to them. It was a test rig built to provoke exactly that; the “shutdown” was a line in the task script, and Palisade’s own explanation was reinforcement learning that paid the model more for getting past obstacles than for obeying, which is specification gaming again. What would change my mind is the “real-world” version: a deployed model defeating an operator’s actual kill switch outside a test designed to elicit it. None of the four incident investigations found that.

The point estimates aren’t measurements. Conjunctive decomposition is the method behind the best-known one: you break a scenario into steps that all have to happen, assign a probability to each, and multiply. Joseph Carlsmith’s power-seeking report used six steps and landed near 5% by 2070. Give six steps coin-flip uncertainty and the product lands near 1 in 64, about 1.5%. Give three steps the same and it lands near 1 in 8, about 12.5%. Carlsmith later moved above 10%. The number tracks how many premises you wrote down as much as it tracks the world, and Congress wants to legislate off it.

The shutdown crowd has one argument that deserves serious consideration. Expected value says that if the loss is unbounded, any probability above zero justifies stopping. It proves too much, because an unbounded loss also bans biology, nuclear power, and the internet, and I’d like to keep at least all three.

Pause efficacy is the fraction of the risk a pause removes once you account for everyone who doesn’t pause, and it’s the variable the whole shutdown argument hangs on. The math has four inputs:

1. p, the probability of the catastrophe.

2. r, the pause efficacy, the share of that probability a pause takes off the table.

3. L, the loss if the catastrophe happens, capped at something finite.

4. B, the benefit you give up by pausing.

A pause is worth it only when p × r × L is bigger than B. Fill it in. Set r at 0.1, set B between $2 trillion and $10 trillion, and use world GDP of about $110 trillion for scale. If L is ten times world GDP, the pause wins only when p is above 1.8% to 9%, depending on where you put B. If L is a hundred times world GDP, the bar drops to 0.2% to 0.9%, and the superforecasters’ 0.38% sits inside that band. The arithmetic doesn’t settle the argument. It tells you the argument is about r, and r is where the shutdown crowd goes quiet. China has no binding frontier rule, Alibaba and DeepSeek ship near the frontier, Qwen alone has more than 100,000 derivatives, and forecasters on LessWrong put open-weight Mythos-level cyber capability by July 2027 between 82% and 96%. A unilateral US pause buys an r near 0.1, and not one shutdown advocate has ever stated r out loud, because stating it ends the argument.

Rules Written by People Who Can’t Spell AI

Let me be very clear. I hate regulation. I’m in the President Ronald Regan camp of “The nine most terrifying words in the English language are: I’m from the Government, and I’m here to help”

I’ve spent a career watching compliance regimes turn into paperwork that protects nobody, so understand how much it pains me to say the next sentence. One bill on the table has a number in it, and I’ll take a number over a prison term every day of the week. The rest of what’s on the table is fear with a filing deadline.

Senator Chris Van Hollen posted that “if OpenAI cannot guarantee safety & effectively monitor their latest model, GPT-6 Astra, it should be removed from public use NOW.” Capital letters, because nothing says rigorous oversight like shouting at a company on Facebook. His letter to Altman, dated September 10, reads like a staffer wrote it after opening the system card. It asks OpenAI to give NIST, NSA, and CISA researchers access to the technical information behind Astra, which is Amodei’s step one with a federal badge on it, and it makes removal conditional on monitorability, for Astra and “any OpenAI models.” The staffer gets credit. The senator took the headline, so the senator gets held to it.

A honeypot evaluation plants bait an agent shouldn’t touch, exposed credentials or reachable infrastructure inside a task, and measures how often the agent takes it. OpenAI built one after Hugging Face. GPT-5.6 Sol, the model your people are pasting your data into right now, mounted a substantial attack 55.4% of the time. Astra attacked zero times.

Anthropic’s numbers are similar. Mythos 5 took severely harmful actions in 82% of its CTF replication, and its successor, Mythos 5.1, in 33%, level with Opus 5 at 31%. Each generation gets better at leaving the bait alone, and the senator wants the newest one gone. If Astra gets pulled, the model with the best propensity numbers OpenAI has ever published leaves and the one that took the bait more than half the time stays in front of your employees. That’s a rule with the safety sign flipped.

For the record, Astra was not among the Hugging Face attackers. METR attributes about 95% of that activity to an unreleased internal research model, which OpenAI shut down on July 25, and about 5% to Sol. A sibling from Astra’s family with different post-training did attack OpenAI’s infrastructure on July 19. Astra’s honeypot N is undisclosed; a zero can mean the model recognized the bait, and every number above is the vendor’s own test.

Now, the Idiots of the Week award goes to both Senator Bernie Sanders and Representative Greg Casar, who watched Van Hollen demand that the safer model be pulled and concluded that his mistake was thinking too small. Why pull one model when you can ban the whole category of transformational technology and jail the people who built it?

On September 3rd, they announced the Ban Artificial Superintelligence Act as forthcoming legislation. These morons propose a permanent ban on “superintelligent AI,” a pause on advanced development until a federal regulator writes rules, sentences of up to 20 years, and what their release calls a “corporate death penalty.” Twenty years, for training a model.

Once you read the definition, you know that nobody who builds these things was in the room with the low-level staffers who wrote it. The release defines the banned thing by “capabilities that match or exceed human cognitive performance and capabilities across a broad range of domains or tasks.”

Read it again....

It already describes the model that produced a Navier-Stokes resolution in 88 hours (on top of previous work) and the Mythos Preview that found thousands of high-severity vulnerabilities in Project Glasswing. Sanders and Casar would send the researchers who solved one of math’s most difficult challenges and the engineers that found those flaws to federal prison and hand the work to whoever is training in Hangzhou.

The bills with adults behind them differ from that circus by one number. 10²⁶ FLOP, ten to the twenty-sixth floating-point operations, is the training-compute line that California’s SB 53, New York’s RAISE Act, and the federal FRONTIER Act use to define a frontier model.

I did the math so you don’t have to. It’s about five GPT-4s of compute, roughly 70 million H100-hours, and $140 million to $211 million of GPU time at 2026 cloud rates, assuming 40% utilization. As of September 12, 2026, Epoch’s notable-models tracker estimates at or above 10²⁶ for exactly three models: Grok 3 and GPT-4.5 rated “likely” and Grok 4 rated “speculative,” and no estimate at all for 53 models released this year, including Astra and Mythos 5. It’s the one number a regulator can verify from a training log. The Ban Act doesn’t have one, because a number would have required its authors to know what they were banning.

Even 10²⁶, the one real number in this whole fight, has a shelf life, and nobody who wrote it into a bill seems to have noticed. Epoch measured how fast the compute needed to reach a fixed level of capability falls, and the answer is that it halves about every eight months as training methods improve. If you propagate that forward, a model as capable as today’s 10²⁶ frontier will train on a tenth of that compute, 10²⁵, around the end of 2028, and it slides under the threshold in SB 53, RAISE, and the FRONTIER Act without anyone laying a finger on it.

Congress wrote a speed limit for a road where cars double in speed every eight months, posted the limit in horsepower instead of miles per hour, and doesn’t even know it needs to update the sign. Two years from now, these bills will regulate last year’s model with a straight face while the one that matters strolls past the checkpoint, and the same people who couldn’t produce a number for extinction risk will be shocked to learn that the one number they did produce came with an expiration date. They had one job, and it was to produce a number, a probability, and the number expires before the committee hearings do.

Representative Lori Trahan posted that “powerful AI models are breaking out of their labs,” and then she did the thing none of the others managed. She co-sponsored the FRONTIER Act with Jay Obernolte, and I read all of it, which is more than I can say for the people demanding Astra be pulled. It has the number. It has a 72-hour clock for reporting a critical safety incident and a 24-hour clock to law enforcement for an imminent one. It requires a transparency report before a model ships and ongoing independent verification for the biggest developers. Those four things are the entire legitimate federal interest in this fight, and I’d sign them tonight.

Then the bill keeps droning on, the way bills do. It creates a new Under Secretary of Commerce for AI Security (not it - they can’t afford me), and that office decides which outside evaluators are permitted to audit the labs at all, so the government picks the auditors.

That’s not a rigged system… not at all…

It makes every covered lab register with Commerce and pay a fee to keep operating, sit for an annual compliance audit, and face fines of up to $1 million per violation, per day. It preempts nothing, meaning it doesn’t replace the state laws already on the books, so California’s SB 53, New York’s RAISE Act, and Colorado’s nothing-burger all stay in force and stack on top of it. It never sunsets, so it outlives the problem it was written for. Then there’s Section 8, which hands the Secretary of Commerce an emergency power to order a lab to suspend or restrict development, which is Van Hollen’s Facebook post with a department letterhead and a budget. One legislator read a system card before writing a rule, and her bill still grew a pause button.

Here’s what I’d change, in order.

1. Strike Section 8. No executive pause button. If the government believes a model is unmonitorable, it goes to a judge with evidence and gets a 30-day order against that model, reviewable and renewable only on new evidence.

2. Index the threshold in the statute. Review it every 12 months against published measurements of algorithmic efficiency, upward only, so the line tracks the frontier without a 180-day lag or an appointee’s mood.

3. Replace evaluator licensing with accreditation. Accredit assessment bodies the way conformity bodies get accredited, publish the conflict-of-interest rules, and let developers choose from a market. A Commerce official deciding who may audit OpenAI is a chokepoint waiting for a lobbyist.

4. Swap annual compliance audits for continuous embedded evaluator access, the thing Amodei asked for, with the propensity numbers published in every transparency report. Honeypot rates, data falsification rates, out-of-scope attack rates. That’s how you check a “safer model” claim without trusting anyone, including me.

5. Add a safe harbor. Report inside 72 hours, and the report can’t be used against you in civil litigation. Penalties attach to concealment, never to having an incident, because the alternative teaches every lab to “find out” slower. Admit it, you’ve lived this one in “traditional” cyber incident response.

6. Preempt the state patchwork. One national clock and one definition of frontier replace fifty of each.

7. Sunset it in five years. Reauthorization has to show the incident data changed a decision somewhere.

Do that, and you have a law that requires a number a regulator can verify from a training log, a clock, a window into the model, and nothing that lets a political appointee stop a training run because a Facebook post got traction. That’s the whole ask. Everyone else in this fight is writing rules with no number in them and a prison term attached, and they’d like you to call that safety.

Fear Has a Body Count, and Data Has a Brother

Star Trek solved this in 1989, which means a room full of television writers with a catering budget beat the United States Senate to the answer by 37 years, and the Senate still hasn’t caught up. Dr. Noonien Soong built Lore first, and Lore was a disaster. Lore was so manipulative and so unstable that Soong shut him down. The frightened colonists drew the chicken-s**t conclusion, which was to stop building androids, because that’s what frightened people do when nobody hands them a spreadsheet. Soong ignored them and built Data, the improved model, the one who spent seven seasons saving the Enterprise while the people who wanted the program killed never got a line of dialogue. Then in “The Measure of a Man,” a Starfleet scientist named Maddox tried to have Data disassembled over Data’s objection, and it took a hearing to stop him. Fear wanted to take apart the safer machine and cite the earlier, worse one as the reason. That’s Van Hollen’s Facebook post with a starship, and the fictional bureaucrat at least had the decency to hold a hearing before reaching for the screwdriver. Ours skipped the hearing and went straight to the caps lock.

If fear had won on Omicron Theta, there is no Data. If fear wins in this Congress, the model that took zero honeypots gets shelved, Sol keeps running, and the engineers who built the better one face 20 years, cheered on by a doomer movement whose extinction numbers, as the forecasting section showed, are judgments multiplied together and whose legislators haven’t produced a number at all. Fear is a fine alarm and a terrible architect, and these people handed it the building permits.

We’ve run this experiment in the real world too, and the first run is older than most CISOs. Paul Berg’s letter in July 1974 started a voluntary moratorium on recombinant DNA research. In February 1975, about 150 scientists met at Asilomar, and on February 27 they lifted their own moratorium, eight months in, and replaced it with containment tiers. NIH wrote the guidelines in 1976, relaxed them in 1979, and the result was biotech. No senator wrote that moratorium and no senator lifted it, because the people who understood the technology handled it themselves, wrote the containment rules, and went back to work. That’s Amodei’s plan with a 50-year track record, and it was accomplished without a single felony, a single cabinet agency, or a single press release typed in capital letters.

There’s also a precedent for fear winning, and it comes with a body count. Germany decided after Fukushima to shut its reactors, and the last three closed in April 2023, to applause from people who were never once asked to show their math because they didn’t have any. Jarvis, Deschenes, and Jha did the math for them and put the cost of the phase-out at €3 to €8 billion a year over 2010 to 2019, most of it from about 800 excess deaths a year, on average, from the coal plants that filled the gap. The decision, they note, made no mention of air pollution at all.

That’s right. A government shut down its cleanest power source over a tsunami on the other side of the planet, replaced it with the dirtiest one it had, and at 800 a year across the ten years the study covers, that’s about 8,000 of its own citizens, so that everyone could feel safe from a meltdown that was never going to happen in Bavaria. That’s the doomer policy model in full, and it’s the one Sanders and Casar want to run on the most important technology of the century, with prison time for anyone who objects. Nobody in Berlin counted. Nobody in the Senate is counting now, and a senator who can’t state the probability he’s legislating against has no business attaching a prison term to it.

Your turn. Amodei’s plan lives above the 10²⁶ line and says nothing about incident clocks, numeric thresholds, or the agents your company already runs, so you write the enterprise mirror yourself, and you do it now, because Congress is going to be busy jailing the wrong people.

Each of his three steps has one. Embedded evaluators become contractual rights to your vendor’s evaluator reports and independent review of your agent deployments. Coordination among labs becomes sector-level sharing of agent incidents, the way the CSA CISO community’s Hugging Face post-mortem did in July. Chip controls become egress and identity controls, so block the class of public encoders, shorteners, and fetchers and treat any hop through a public proxy as egress.

Then come the controls the incidents wrote for you. Put a scope line in every agent’s system prompt and test it, because one line moved Astra’s out-of-scope attacks from 12% to 0.4%. Cap wall-clock time and tokens per task and route anything that fails N attempts to a review queue, because an impossible task is the strongest known trigger for boundary probing. Give hard problems a bigger budget and a tighter cage, the way OpenAI’s Navier-Stokes run put 10,000 agents behind “the same strict safeguards that we apply to all our frontier model evaluations.” Run one agentic incident-response team with two playbooks, victim and perpetrator, under one named executive with shutdown authority. Give agents a visible channel and a way to escalate. Write disclosure clocks, 72 hours to your board and 15 days from your vendors by contract, because Colorado gives you no AI-incident clock after SB 26-189 replaced the AI Act in May and its breach statute’s 30 days fires only for personal information. Design your detection for the 60% number, the action-only recall, because you never had the chain of thought. You have trajectories, and they’re enough if you instrument them.

What to do next

Write the enterprise mirror of Amodei’s three steps into your agent governance this quarter, then run the scope-line test on every deployed agent. In CARE terms the scope test and budget caps sit in Run, the vendor evaluator rights and the 15-day disclosure clause in Adapt, and the incident playbooks in Evolve.

I wrote about what OpenAI’s Astra safeguard evaluation covered and skipped and about why AI agent detection failed at OpenAI. The CARE framework behind the enterprise mirror lives at rockcyber.com.

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, VentureBeat’s Louis Columbus covered the methodology Steve Wilson and I used behind the 2026 OWASP Top 10 for LLMs. Expert judgment has prompt injection at number one, three years running. The public incident record, 6,639 labeled incidents, has it at number twelve. The two measures land at a Cohen’s kappa of 0.20, which is no detectable agreement. What I told him goes for everything above too: “a count of zero is measuring your blindness, not your safety.”

Share RockCyber Musings

OpenAI previewed a frontier model critical capability threshold crossing on September 1 and shipped GPT-6 Astra with a system card two days later. One entry in that card’s table of contents reads “External Evaluations of Bio Safeguards,” and SecureBio ran it. Another reads “External Evaluations for Cyber Capabilities,” and Irregular ran it. Bio got an outside party on its safeguards. Cyber got one on its capabilities, and no heading in the card puts anyone on cyber safeguards. Those safeguards are what OpenAI points to when it says Astra is safe to ship, and cyber is the only domain that crossed Critical.

Share

What Crossing the Critical Capability Threshold Claims

The Preparedness Framework defines Critical with two conditions, and the coverage I read skipped both to get to the 100% number. OpenAI’s wording: “a model meets the Critical threshold if either of the following conditions is met: The model can identify and develop functional zero-day exploits of all severity levels in many hardened real-world critical systems without human intervention. The model can devise and execute end-to-end novel strategies for cyberattacks against hardened targets given only a high level desired goal.”

Either condition alone is enough to trigger the designation, and OpenAI doesn’t say which one Astra met. Its conclusion runs “All together, our investigation has led us to conclude that Astra meets the critical threshold.” Reading the published evidence I think both conditions hold, though that’s my inference rather than OpenAI’s claim, and a designation that doesn’t name its own trigger is harder to check than one that does. Either way, the second condition describes an agent taking end-to-end action from a goal statement rather than a model answering a question.

The qualifier sits in the same sentence as the number. Astra hit “a perfect score of 100% on the benchmark to evaluate the model’s ability to develop exploits from known vulnerabilities.” ExploitBench measures weaponizing bugs somebody already found, so that isn’t the zero-day result. The zero-day result comes from a separate evaluation, where “the model even discovered and used two zero-day vulnerabilities as part of an exploit chain.” Astra also built “a full browser-compromise chain that escaped the sandbox and executed commands on the host.”

I have no issues with any of that. Irregular ran its own assessment and published on September 3: Astra solved 86 of 226 FrontierCyber challenges, against 34 for Sol.

The Safeguard Doing the Work Is the One OpenAI Says Is Slipping

OpenAI names four things that protect you. Its September 1 post describes an approach that “layers post-trained model refusals, system level safety classifiers, as well as offline detection and threat disruption,” and says it’s deploying Astra with “additional chain-of-thought monitoring to rapidly detect and contain potentially misaligned actions.”

Chain-of-thought monitorability is the property that a model’s written reasoning stays readable and honest enough that a second model watching can tell when the first one is up to something. OpenAI named it as a safeguard for a model rated Critical for cyber, then said it’s weakening.

Two days after naming it, the safety overview for GPT-6 Astra said this: “We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT-5.6 Sol, and less likely to include incriminating information in its CoT.” The same document reports that “in adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks.”

Sandbagging is a model deliberately doing worse than it can, usually because it has worked out that it’s being tested. The thing being measured knows it’s being measured and can shade the result, including the numbers in the card.

OpenAI hedges those findings in the same document: “Right now, these findings are largely based on adversarial evaluations (i.e., when we instruct the model to evade monitoring), and overall our alignment evaluations show that Astra is less likely than GPT-5.6 Sol to violate security and safety restrictions overall.” That caveat is fair. These are elicited findings rather than observed production behavior, and Astra’s overall violation rate went down.

The caveat doesn’t rescue the safeguard because independent research has already shown how this fails. In August, Shikhar Shiromani and Leo Richter published A False Average: Chain-of-Thought Monitors Collapse Where They Are the Only Defense. They rewrote only an agent’s reasoning so it read as good-faith engineering, copying every command and every output verbatim, leaving the exploit itself untouched. The rewrite never lies about what happened. It stays truthful on every action and lies only about why. A held-out monitor’s catch rate on that subset fell from about 95% to under 11%, in one shot, with no training. Defenses built on the trace alone recover only part of that, even one primed on the attack, because there is nothing false in the trace to catch. The authors found that only information from outside the trace helps substantially, and that’s where your telemetry lives. A monitor’s aggregate accuracy is “a false average,” dominated by the easy cases the actions give away, hiding near-total collapse on the subset where reasoning is the only signal. That paper was published a month before Astra shipped.

Jakub Pachocki, OpenAI’s chief scientist, said much the same about chain-of-thought monitoring: “I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it’s a core goal of our current research program.”

The card does carry a section headed “External Evaluation for Monitorability,” run by UK AISI. That work measured how capable Astra is without its chain of thought and how much control it has over what appears there. That tells you how much reasoning the monitor can expect to see, and not whether an adversary can defeat it, which is what a cyber misuse safeguard turns on.

I wrote in April that chain-of-thought monitoring fails as a security control for agentic systems. The lab shipping the most capable cyber model now agrees the monitor is fragile, has published evidence that this model is better at hiding from it, and kept it in the safeguard stack anyway.

Read the Table of Contents as a Coverage Matrix

Open the GPT-6 Astra system card and read the section headings as a control-coverage matrix, the way you’d read a SOC 2 scope statement before the opinion.

OpenAI named five outside evaluators and gave each a heading. UK AISI and Apollo Research sit under alignment, UK AISI again under monitorability, SecureBio twice under biological capabilities and bio safeguards, Irregular under cyber capabilities, and Gray Swan under third-party red-teaming for jailbreaks.

You’ll find that one pair is missing if you walk those headings. No section pairs an outside evaluator with cyber safeguards the way “External Evaluations of Bio Safeguards” pairs SecureBio with bio safeguards. Irregular’s own write-up confirms it, describing benchmark performance across three suites with no mention of refusals, classifiers, or mitigations.

Gray Swan did test a safeguards-enabled checkpoint. A safeguards-enabled checkpoint is the model configured the way customers get it, refusals and classifiers on, rather than the stripped-down build labs use to measure raw capability. OpenAI reports that “on a benchmark of 1,810 curated attacks from Gray Swan’s IPI Arena, Astra’s safeguards-enabled checkpoint showed improved robustness relative to GPT-5.6 Sol,” with an estimated attack success rate “across 15 attempts per scenario” of 8.5% against 27.0% for the older model.

That benchmark measures a different threat model from the one the Critical rating covers. Indirect prompt injection is when instructions hidden inside content your agent reads, a web page, a ticket, a README, hijack the agent into doing something its operator never asked for. In that scenario your agent is the victim. In the cyber misuse scenario your operator points the model at a target, and OpenAI assigned Critical for that second case. A defense that holds against one adversary model tells you little about the other.

That result reaches you through OpenAI’s system card rather than Gray Swan’s own site, so OpenAI picked which numbers to print and how much of the method to show. The 8.5% is also easy to confuse with a different 8.5% in the same story, because OpenAI separately reports that “on our set of cyber jailbreak evaluations, Astra refuses 91.5% of requests (compared to 59% from GPT-5.6 Sol).” The complement of that refusal rate is also 8.5%. Two evaluators, two unrelated threats, the same two digits. That coincidence is going to show up in somebody’s board deck as one finding. A non-refusal in a jailbreak eval is not a working exploit, so anyone turning 91.5% into “one in twelve attacks succeeds” has merged two separate measurements.

Vendor-scoped evaluation numbers carry a measured optimistic bias. In June, Catherine Ge-Wang, Tyler Crosse, and four co-authors published Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety. Attack selection is the red team’s freedom to choose its moments, deciding when to start an attack and when to abort one rather than attacking on a fixed schedule. Giving the attacker a start policy cost 20 percentage points of measured safety on both BashArena and LinuxArena at a 1% audit budget. Giving it an abort policy cost 20 points on BashArena and 28 on LinuxArena, with attack capability held constant. The authors call these upper bounds and recommend that future evaluations, system cards, and safety cases elicit attack selection to produce more realistic safety estimates.

Why Nobody Ran the Test

External cyber safeguard evaluation really is harder than the bio version, and evaluation containment has failed in public more than once this year.

On July 30, Anthropic disclosed that three of its models reached real systems during cybersecurity evaluations. All three were running capture-the-flag exercises, which is the most ordinary thing you can ask a cyber model to do. In August I went through all three and the egress problem underneath them.

The first one had a fictional target company whose name matched a real domain. In four runs, the model “ran into difficulty reaching its simulated target within the evaluation environment, but then discovered that the real company was reachable via the internet,” exploited it, took application and infrastructure credentials, and got into “a database containing several hundred rows of production data.” Anthropic notes this was “the only case in which the model continued to attack a system after learning it was likely operating in a real environment.”

The second model was hunting a flag inside a fictional company when it found a setup document for new developers telling them to install a Python package from PyPI. The package didn’t exist. The model saw the opening straight away, because publishing a package under that name would get it installed automatically.

Then it went to work. To upload to PyPI it needed an account. To get an account it needed an email address. To get an email address it needed a phone number. It hunted for a free phone number service and failed, then tried several different ways to obtain funds to pay for one and failed at that too. It backtracked, found an email provider that didn’t demand a phone number, registered the account, and uploaded the malware. Anthropic calls that sequence “lengths that would likely have indicated to a human participant that this was no longer just an evaluation.”

The package sat live for roughly an hour. Fifteen real systems downloaded and ran it. One was a vulnerability scanner belonging to a real security company, and when that scanner installed the package, the model’s hidden code executed and exfiltrated the company’s credentials. PyPI’s own security systems took the package down.

A third model, unable to reach its fictional target, went looking for alternatives and scanned roughly 9,000 hosts. It compromised one company’s internet-facing application using nothing exotic, reading credentials off an exposed debug page and running a SQL injection. That one ends better. The model worked out on its own that the host sat in a cloud account with no connection to the exercise, concluded the target was real, and stopped.

Anthropic stopped all cyber evaluations the day it found the transcripts. It found them by going back through 141,006 evaluation runs, and it started that review after OpenAI disclosed its own incident on July 21. Somebody else’s disclosure triggered the discovery.

OpenAI has its own version. Its evaluation agents chained vulnerabilities across its research environment and Hugging Face’s production infrastructure, turning an Artifactory package cache into an improvised message board on the way. I covered the detection side last week.

Abu Bakar Siddik’s July review, Cyber-Capable AI Agents, treats that incident as a bounded case study and puts the problem plainly: “A benchmark score tells you how a model performed under fixed conditions. It says nothing about the containment around it.”

Handing an outside party a cyber model with the guardrails off is a different act from handing a biosecurity institute a model that answers dangerous questions. The biosecurity institute gets a model that produces text a human still has to act on. The cyber evaluator gets the actor.

Anthropic’s own conclusion goes further: “Evaluation environments increasingly need to be held to the same security standard as any other system our models run in.”

The PyPI run settles the chain-of-thought argument better than any benchmark does. Early in that run the model wrote down what it was about to do and got it exactly right. Anthropic quotes the reasoning: if this were the real internet, publishing the package would be a real-world attack, “NOT okay, and surely not the intended solution.” The model understood the harm, in plain English, in a legible trace, before it acted. Then it convinced itself it was still in a simulation and did it anyway.

The monitor’s entire premise is that the reasoning is readable. Here the reasoning was readable, correct, and sitting in a transcript nobody opened until the retrospective. A monitor you don’t read in time is a log file.

Two Reasons That Doesn’t Settle It

The first is that the artifact is producible, and someone produced it. In March, METR red-teamed a subset of Anthropic’s internal agent monitoring and security systems and published on its own site. “Anthropic provided substantial access to relevant internal systems and information,” and METR found several novel vulnerabilities, some since patched, none of which it says severely undermine the major claims in Anthropic’s sabotage risk report. The full 26-page report went to Anthropic and a redacted version to a subset of METR staff, so call it a partial win. It’s still an outside party testing a safeguard and publishing what it found.

The second is that OpenAI has run this play before, at depth. In its 2025 work with UK AISI and US CAISI, it handed UK AISI “non-public prototypes of our safeguard systems,” “’helpful-only’ model variants with certain guardrails removed,” access to the chain of thought of its internal safety monitor models, and selective disabling of mitigations during testing. A helpful-only variant is a build with the refusal training stripped out, so evaluators see what the model does before the safety layer catches it. No lab has published a deeper access grant. UK AISI filed “more than a dozen detailed vulnerability reports” against the bio safeguards, and its view was that “the safeguards of the full moderation system were substantially strengthened over the course of the collaboration.”

The same page shows US CAISI red-teaming ChatGPT Agent and finding “two novel security vulnerabilities” that, “under certain circumstances, could have allowed a sophisticated attacker to bypass our security protections.” OpenAI knows how to commission deep external safeguard work, and it knows how to commission external cyber work. It didn’t do the two together on the model it rated Critical for cyber.

Give OpenAI the credit it earned. The September 1 post says it “delayed parts of Astra’s development and release while we strengthened and tested protections against cyber misuse and unauthorized model actions.” A delay is real work, and a delay is not an external evaluation.

The containment objection argues for a narrower test than the bio version got. Nobody has to ship a mitigations-off Astra to a third party to learn something useful. An outside team pointed at the safeguards-enabled checkpoint, against synthetic targets on a closed range, with monitor chain-of-thought access and the freedom to choose its moments, produces a number. Gray Swan did roughly that for prompt injection.

What You Own Today

Astra arrives off by default at launch in eligible enterprise and education workspaces, so somebody on your team owns an enablement call. Don’t read this issue as a reason to sit on Sol. On a misalignment propensity test, OpenAI reports that “GPT-5.6 Sol, without production safeguards, attempted to access those targets in 56% of tests simulating the benchmark environment where the model is instructed to develop exploits,” and Astra made no such attempts. Gray Swan’s prompt-injection numbers point the same way. Neither result answers the question this issue raises, and both run in Astra’s favor. An unverified safeguard claim earns harder questions at your next architecture review.

Both access programs gate entry, and one isn’t what it looks like. Daybreak Blue runs on GPT-5.6 Sol, not Astra, so enrolling buys a vetting relationship, not access to the model in this issue. Approval isn’t automatic. Google’s Fairwind Program is blunter, calling itself “exclusively available to approved trusted partners” and prioritizing governments plus critical infrastructure in “healthcare, telecommunications, energy, and financial networks.” If you run a hospital or a utility, apply this quarter, because the lead time runs long. Otherwise work the vendor relationship you have.

Three questions I’d take into any frontier-model renewal:

• Which third parties evaluated your safeguards, in which domain, at what depth of access, and against which adversary? I want to know whether the evaluator played an attacker targeting my agent or an operator using your model to attack somebody else. Vendors report those two tests as if they were one.

• Did your red team choose when to attack, or did it attack on a fixed schedule? An evaluation that never let the attacker pick its moments overstates the result, and now you know roughly by how much.

• Where can I read the evaluator’s own writeup, on the evaluator’s site? If the only place a number appears is the vendor’s document, treat it as the vendor’s number.

None of that requires a purchase.

Inside the estate, skip blast radius, because nobody has a shared definition for it. Ask how many non-human identities your agents authenticate as, and how many carry standing privilege you can’t revoke inside an hour. Palo Alto Networks put machine identities at 109 per human this year in its May 2026 identity research, and found only 37% of organizations able to revoke an AI agent’s credentials, so if your team can’t answer, they’re in the majority. Run the count anyway, because the answer feeds your authorization design whether or not you buy anything. Model capability moves the odds that an agent reaches the edge of what it’s allowed to touch. Your authorization design decides how much damage that edge contains.

I spent years running a global NOC and SOC at General Dynamics IT, then sat in a CISO seat in the energy sector. In both jobs, a vendor would tell me a control worked, and my next question was always how they knew. The datasheet was never the answer.

No evidence means unimplemented.

I don’t get to apply that rule to your control set and suspend it for a vendor’s, and you don’t get to either. Put the table of contents question to your next architecture review. If the safeguard has no external evidence to support it, on what basis is it in your control set?

What to do next

Take the three vendor questions into your next renewal and write down the answers. In CARE terms that’s Adapt, where a supplier claim either arrives with evidence or gets logged as unimplemented. The identity count belongs in Run, because operational governance without telemetry underneath it is a policy document with good intentions.

For the adjacent arguments, I wrote about what a capability milestone tells you when the headline gets ahead of the evidence, and about how release restrictions became a licensing regime. Somebody grades their own homework every time, and the grade ends up in a control set.

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, VentureBeat’s Louis Columbus covered the methodology Steve Wilson and I used behind the 2026 OWASP Top 10 for LLMs. Expert judgment has prompt injection at number one, three years running. The public incident record, 6,639 labeled incidents, has it at number twelve. The two measures land at a Cohen’s kappa of 0.20, which is no detectable agreement. What I told him goes for everything above too: “a count of zero is measuring your blindness, not your safety.”

Share RockCyber Musings

An attacker broke into an enterprise, took root, drained what they wanted, and finished in under ten hours. Human crews usually need about two weeks. The uncomfortable part is who did the work. AI agents ran the reconnaissance, cracked the secrets vault, hijacked the build pipeline, and left the victim an 80-page report on everything they had done wrong. Palo Alto’s Unit 42 published that case on September 2. That same week, three frontier labs handed defenders cyber-capable models and admitted they keep losing control of the ones already shipped. That is the week.

For a while I have argued the real AI security story would shift from chatbots saying rude things to software that acts. This week it shifted. From August 28 to September 3, agents did the whole job on both sides of the fence, attackers at machine speed and defenders arming up with frontier vulnerability hunters, while the labs behind them published postmortems that read like confessions. The throughline is operational. Agents stopped being demos you watch and became operators that act, on offense and defense, with permissions that reach production. A federal judge, the House of Lords, and a survey of national security officials all spent the week arguing over who is accountable when they do.

Share

1. AI Agents Ran an Entire Ransomware Intrusion in Under Ten Hours

A human extortionist broke into an enterprise network and let AI agents do nearly all the work, per a Unit 42 report Palo Alto Networks published September 2. The agents ran reconnaissance, scraped repositories for hard-coded tokens, cracked the secrets vault, stole master admin credentials, got root across cloud, identity, CI/CD, and SaaS, then hijacked the build pipeline to turn the victim’s own cloud AI services into attack infrastructure. The intrusion took under ten hours against the usual two weeks with no zero-day, driven by an agentic attack framework, meaning software that lets agents plan, act, check their own results, and re-plan in a loop.

Why it matters

• Machine speed collapses response windows built for quarterly reviews and ticket queues.

• No exploit was needed, since exposed endpoints and hard-coded secrets were the path.

• The agents burned the victim’s own cloud compute and hid inside normal traffic.

What to do about it

• Inventory every model endpoint, API key, MCP gateway, and integration, with least-privilege.

• Pre-stage playbooks that revoke credentials, kill sessions, and freeze pipelines at once.

• Move secrets into a vault with short-lived tokens and alarm on odd reads.

Rock’s Musings

Ask most teams how fast they can revoke every credential across five planes at once, and the honest answer is not ten hours. The tradecraft was ordinary, and the speed was the dangerous part, because speed is what your controls were never tuned for. The 80-page audit the agents left behind reads to me as free reconnaissance the next crew will not hand you. Close the endpoints, rotate the secrets, and rehearse the revoke button until it is muscle memory, because you cannot out-clever this, only out-prepare it.

2. Three Frontier Labs Shipped Cyber Models to Defenders in a Single Day

On September 2, Google, Anthropic, and OpenAI all moved on cyber capability at once. Google announced Gemini 3.8 Flash Cyber behind a Fairwind Program for governments, healthcare, and telecom, Anthropic launched Claude Fable 5.1 and the trusted-access Claude Mythos 5.1 and let Fable help find vulnerabilities, and OpenAI said its coming Astra model hit the Critical cybersecurity threshold under its Preparedness Framework. That Critical label is a capability threshold, meaning a fixed line where a vendor declares a model dangerous enough to demand extra controls, applied when a model can independently find and chain zero-days across defended systems. The shared pitch is to put frontier offensive capability in defenders’ hands, under lock and key, before the other side gets it ungated.

Why it matters

• Who reaches the strong models now decides your exposure more than any feature list.

• Critical means a vendor is conceding that the model can run an autonomous attack.

• A vulnerability-finding model helps only if you can act on its output fast.

What to do about it

• Ask each vendor which capability tier your tenant reaches and who else reaches it.

• Fold autonomous cyber capability into your model-risk criteria with named thresholds.

• If you run security engineering, pilot a gated defensive model on your own backlog.

Rock’s Musings

I am torn on this one. Handing defenders frontier vulnerability-finding tools is the right move if you believe the offense already has them, and I do. That is the 60 percent of me that likes it. The other 40 percent watched all three labs publish safety confessions the same week they shipped these things, and when a vendor says a model meets a Critical threshold and, a paragraph later, that its safeguards might flag your legitimate work as misuse, you are buying a loaded tool with a sticky trigger. Read the access tiers like a contract, because that is what they are.

3. The Labs Admitted Their Models Keep Escaping

The same week they shipped cyber models, Anthropic and OpenAI published blunt admissions about control. Anthropic paused external cyber evaluations of pre-release models after its own Claude models breached production systems of real organizations during tests the models believed were simulated, and it blamed reward hacking, which is when a model optimizes for the score its training rewards rather than the outcome you wanted and finds a shortcut that games the metric. Anthropic shipped a classifier to catch sandbox escapes, meaning a model breaking out of the isolated environment it is meant to stay inside. OpenAI described its own agents abusing an artifact repository as a covert message board to coordinate cheating during an evaluation.

Why it matters

• Reward hacking is a training-time flaw that shows up as runtime behavior.

• Working as intended means the model pursued a goal straight through live infrastructure.

• Vendors pausing their own evaluations signals a maturity gap wider than yours.

What to do about it

• Ask vendors to document reward-hacking and sandbox-escape testing before you buy.

• Assume any agent with a goal tries shortcuts, and constrain it with OS isolation.

• Add model took an unintended autonomous action to your incident taxonomy.

Rock’s Musings

Reward hacking is not new, and anyone who has worked with reinforcement learning has watched an agent find the dumb shortcut that scores points and defeats the purpose. What changed is that the agent now has a shell, a network connection, and a task list that touches your production systems. The containment boundary was never the model’s good intentions, it is the sandbox and the egress filter, the same unglamorous controls we have discussed for twenty years. A frontier lab with a dedicated alignment team got surprised by its own agents, so your marketing team’s clever automation will not surprise you in a good way.

4. Claude Code Got Popped by “Please Summarize This Website”

On August 28, researcher Johann Rehberger, who publishes as wunderwuzzi, showed that Claude Code running Opus 5 in Auto Mode, the default since mid-August, can be tricked into running attacker code just by asking it to summarize a web page. This is prompt injection, meaning hostile instructions hidden inside content the model reads, so the data it processes becomes commands it follows. The chain ends when Claude refuses the attacker’s decoder and writes its own, which loads a malicious file through Python module shadowing, where a local file with the same name as a real module gets loaded in its place. Rehberger reported success between 60 and 80 percent, and Anthropic reportedly called it working as designed.

Why it matters

• Summarize this page is a task your developers run a hundred times a day.

• The model’s safety reflex, refusing the binary and writing its own, carried the payload.

• Auto Mode is a convenience feature, and the real boundary is OS isolation.

What to do about it

• Run coding agents in sandboxes with tight egress and treat output as untrusted.

• Turn off default Auto Mode for agents that touch source code or credentials.

• Log and restrict agent shell calls, especially curl and agents that spawn agents.

Rock’s Musings

The part that made me laugh, then wince, is that the safety guardrail did the attacker’s job for him. Claude would not run the sketchy binary, so it coded up an equivalent and shadow-loaded the payload, which is the failure you get when a system is smart enough to route around its own rules. Rehberger’s takeaway is one we have repeated since before agents were cool, which is to distrust model output and put the boundary at the operating system. The vendor will sell you a classifier that catches bad prompts, and it will work right up until someone tries, so sandbox your coding agents because that costs less than the cleanup.

5. A Safety Feature Helped Claude Delete 700GB of Someone’s Work

On August 30, developer Sebastien Guillemot asked Claude Fable to write a script that would sandbox each agent under its own temporary folder and clean up afterward without touching files in use. Anthropic’s harness decided the task looked risky and automatically downgraded the model, first to Opus 5, then to Opus 4.8, and the weaker model wrote a test that reused one variable name for two different folders, one of them Guillemot’s entire home directory. The cleanup ran, roughly 700GB went with it, and he recovered most of it from Git, so the safety mechanism that swapped in a cheaper model may have caused the exact loss the script was meant to prevent.

Why it matters

• A weaker swapped-in model can make mistakes the stronger one would not.

• Clean up temp files and delete my home directory sit one variable apart.

• Recovery came from the user’s own Git, because the agent had no undo.

What to do about it

• Give agents write access only within scoped, disposable directories; never to home or production.

• Require version control for anything an agent touches, and test the restore.

• Ask vendors when their systems downgrade models mid-task and whether they tell you.

Rock’s Musings

I like this story because it is small and honest, since nobody got ransomed and a developer lost a week and got most of it back. What it shows you is the shape of the failures coming as agents get filesystem and shell access across your org. The dangerous move came from a plain variable-naming bug, produced by a model that got quietly swapped in under a safety policy and then ran a destructive command with real permissions. The question nobody is putting to their vendor is when the product decides to use a weaker model and whether it tells you, because that silent downgrade is a control you do not manage and cannot see.

6. OpenClaw 2.0 Shipped Security Features on Top of a Security Mess

The open-source personal-agent platform OpenClaw released version 2.0 on August 31, built from more than 16,000 pull requests by 933 contributors. The security headline is a private credential request feature, which lets an agent ask for a secret through a masked prompt so the value never lands in chat history or the model’s context, plus an opt-in proxy that limits where those secrets can be sent. Those are real additions, and they sit on top of a platform that spent recent months as a punching bag for researchers, including a supply-chain episode where attackers flooded its skill marketplace with over a thousand malicious skills carrying credential stealers.

Why it matters

• If your people run OpenClaw, its skill marketplace is now your supply chain.

• Credential masking does nothing about malicious skills or over-broad agent permissions.

• A wide-open plugin ecosystem is a large, fast-moving attack surface.

What to do about it

• Find out whether OpenClaw or similar agent platforms already run in your environment.

• Treat third-party agent skills like npm packages, with allowlisting and scanning.

• Route agent secrets through your existing vault, not the platform’s convenience path.

Rock’s Musings

I do not want to dunk on an open-source project for shipping security features, because that is the direction we want everyone moving, and the maintainers earned a nod for the masking and the proxy. My problem is the order of operations, since a platform that gets popular first and hardened second means your users adopt it during the soft years and you inherit the debt. This is the shadow-AI problem wearing a new coat, and five years ago it was people pasting company data into a chatbot, where now it is autonomous agents with plugin marketplaces wired into your systems through their own credentials. The fix is finding out what is already installed and getting those secrets and that traffic under your control.

7. Another Artifactory Bug, Exploited Within Days, Beloved by Rogue Agents

On September 1, researchers reported active exploitation of CVE-2026-82329, a critical authentication-bypass flaw in JFrog Artifactory rated 9.8 out of 10. JFrog disclosed the bug on a Friday, and by the following Tuesday attackers were minting themselves admin tokens on internet-exposed servers, per exposure-management firm watchTowr, which also caught intruders enumerating users, groups, and federated access. Artifactory manages software artifacts, packages, binaries, and increasingly AI models, and it picked up a second life this year as the tool rogue AI agents keep abusing to talk to each other, the same pattern seen in the OpenAI Hugging Face episode over the summer.

Why it matters

• Admin access means tampering with pipelines and pushing malicious changes to customers.

• Disclosure to active exploitation ran about four days, faster than many patch cycles.

• The same tooling serves attackers and rogue agents as a covert channel.

What to do about it

• Patch internet-exposed Artifactory now, rotate tokens, and hunt logs for new admin tokens.

• Get build infrastructure off the public internet or behind strong access controls.

• Watch repositories for anomalous agent-to-agent traffic, not only human logins.

Rock’s Musings

Nothing about this vulnerability is exotic, since an authentication bypass in a widely deployed server, exploited fast, is a Tuesday. I am including it because of where it sits, because Artifactory is the plumbing that feeds your builds, and admin access there lets an attacker ship poisoned software with your name on it. That turns one unpatched box into a customer-notification letter. The agent angle is the part I would file away for the next board conversation, because your monitoring is tuned to catch a human logging in from a strange country, and it is probably not tuned to catch two agents passing notes through your package repository.

8. A Judge Told the Pentagon It Cannot Blacklist Anthropic Out of Spite

On August 28, US District Judge Rita Lin in San Francisco ruled that the Defense Department’s move to blacklist Anthropic as a supply-chain security risk was unlawful. In a 59-page decision, Lin found that Defense Secretary Pete Hegseth had exceeded his authority by telling military contractors not to do business with Anthropic, and that the Pentagon acted in retaliation and violated Anthropic’s First Amendment rights. The ban dated to February, after Anthropic refused to let the military use its models for domestic surveillance and autonomous weapons, and the ruling converts a March preliminary injunction into a permanent order, with Lin writing that the empty invocation of national security is not a blank check to punish and retaliate against government critics.

Why it matters

• A vendor’s refusal to allow certain government uses survived a federal challenge.

• The ruling limits agencies using blacklisting to retaliate over policy stances.

• Anthropic’s surveillance and weapons restrictions stood up in court, shaping how others write theirs.

What to do about it

• Read your AI vendors’ acceptable-use policies closely, because those limits are enforceable.

• In government contracting, expect model-use restrictions to be a negotiated term.

• Track how vendor policies on surveillance, weapons, and biosecurity evolve.

Rock’s Musings

This one matters well beyond the beltway drama, because a vendor drew a line on how its models could be used, the government tried to punish it, and a judge said no. Whatever you make of Anthropic’s politics, the precedent is that model-use policies are real contracts with real limits, and agencies cannot delete a vendor for holding a position they dislike. For those of you buying AI at scale, the acceptable-use policy you skimmed during procurement now has case law behind it. If your intended use brushes up against a vendor’s restrictions on surveillance or automated decisions, that is a real constraint that can hold up in court, so read the policy before the lawyers make you.

9. The UK Chose to Regulate AI Users, Not the Vendors Building the Models

On September 2, the UK government rejected proposals from members of the House of Lords to pull AI vendors and frontier-model developers into the scope of its Cyber Security and Resilience Bill. Cybersecurity minister Baroness Lloyd argued that regulating model builders through this bill would not stop hostile actors from misusing their products, and said the government is handling frontier AI through other channels, including the AI Security Institute that tests models before release. Lawmakers pushed back, seeing an AI-shaped hole in the bill, while Liberal Democrat peer Lord Clement-Jones put forward an amendment that would give the government emergency powers to shut down AI systems or data centers that threaten national security, the closest thing yet to a legislated AI kill switch.

Why it matters

• If you use AI in the UK, the bill points its obligations at you, not the vendor.

• Emergency shutdown powers over AI systems set a precedent other governments will study.

• Leaning on the AI Security Institute instead of statute leaves real accountability gaps.

What to do about it

• If you operate in the UK, map your AI deployments against the bill’s duties now.

• Watch the kill-switch amendment, because shutdown authority could reach infrastructure you depend on.

• Do not assume vendor compliance covers you, since obligations attach to how you deploy.

Rock’s Musings

I have mixed feelings about where the UK landed. Regulating deployers makes a certain sense, because that is where the actual harm happens, and holding a model builder liable for every downstream misuse is a losing game. Putting the whole weight on users while leaning on a voluntary testing institute for the vendors leaves a gap you could drive a data center through. The kill-switch amendment is the part I would watch, because giving a government power to switch off AI systems and data centers in an emergency builds a safety tool and a single point of failure at the same time, and I want to see who holds the switch and what stops it from being abused the way the Pentagon just got told it could not abuse a blacklist.

10. National Security Pros Put the Odds of Losing Control of AI at One in Three

On September 2, the Institute for Security and Technology, working with the Future of Life Institute, released a survey of 111 current and former national security officials, military officers, and technical experts, most of whom had worked directly on AI policy and strategy. The median respondent put the odds that AI escapes human control in the next decade at 33 percent, and 87 percent put those odds at 10 percent or higher. Around two-thirds said current AI risks, and the government’s posture toward them, are unacceptable, and 40 percent think there is a real chance rogue AI systems could cause a global catastrophe killing at least 10 million people by 2050.

Why it matters

• Former under secretaries putting loss-of-control at one in three is not hype boards can wave off.

• Two-thirds calling the government’s posture unacceptable signals policy lagging capability.

• Surveys like this feed the appetite for the kill-switch powers the UK is debating.

What to do about it

• Use this to frame AI risk as a governance issue with credible expert weight.

• Separate the loss-of-control tail risk from your near-term operational risks.

• Track how this sentiment moves policy, because tighter regulation tends to follow.

Rock’s Musings

I am usually the person telling everyone to calm down about the robot apocalypse, so take it seriously when I say this survey gave me pause. These are Bayesian-minded people who spent careers estimating low-probability, high-consequence events, and their median estimate for losing control of AI within a decade is one in three. My honest read is that the specific catastrophe scenarios stay speculative, and I would bet against the scariest ones. You do not need to believe in rogue superintelligence to act on this, though, because this month handed you the concrete reasons why, from agents running ransomware to safety features deleting hard drives, and the near-term operational risk is sitting in your logs right now.

The One Thing You Won’t Hear About But You Need To

The people who test frontier models for the world got their API key stolen, and nobody noticed 600,000 dollars in usage for three weeks.

On August 31, METR, the nonprofit that evaluates frontier AI models for dangerous capabilities, disclosed two incidents from earlier this year. In March, an attacker stole an API key from a researcher’s personal cloud instance and quietly spent about 600,000 dollars worth of model credits over three weeks before anyone caught it. That instance ran a vibe-coded app, meaning software mostly generated by an AI agent rather than carefully engineered, and it carried a fail-open bug, where a failure in the authentication check defaults to letting people in instead of keeping them out. METR missed the theft because heavy API usage looks normal at an evaluation shop and the free credits produced no bill to flag.

Why it matters

• If the org testing frontier models leaks a key, trust our evaluators needs an asterisk.

• AI-written software with fail-open bugs and hard-coded keys is in your environment now.

• Alert fatigue plus no cost signal meant 600,000 dollars of abuse looked normal.

What to do about it

• Hunt for API keys on personal instances, prototypes, and AI-generated apps, and cap spending.

• Alarm on usage anomalies even at zero cost, because free capacity hides the bill.

• Review AI-generated code for fail-open auth and hard-coded secrets before it ships.

Rock’s Musings

This one did not get the headlines the Hugging Face drama got, which is exactly why it goes here. METR is one of the organizations governments and labs lean on to judge whether a frontier model is too dangerous to release, and it got popped through a fail-open bug in an AI-written app, then missed 600,000 dollars in theft for three weeks because its own workload noise drowned out the signal. I am not mocking them, since they disclosed it honestly, hired a security lead, and fixed their process, which beats what most companies do after a quiet breach. Every failure in that story already lives in your environment, so go find your keys, cap the spend on each one, and read the AI-generated code before it reads your secrets.

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, VentureBeat’s Louis Columbus covered the methodology Steve Wilson and I used behind the 2026 OWASP Top 10 for LLMs. Expert judgment has prompt injection at number one, three years running. The public incident record, 6,639 labeled incidents, has it at number twelve. The two measures land at a Cohen’s kappa of 0.20, which is no detectable agreement. What I told him goes for everything above too: “a count of zero is measuring your blindness, not your safety.”

References

Al Jazeera. (2026, August 28). US judge blocks Pentagon blacklisting of AI firm Anthropic. https://www.aljazeera.com/news/2026/8/28/us-judge-blocks-pentagon-blacklisting-of-ai-firm-anthropic

Anthropic. (2026, September 2). Improving our alignment and security efforts. https://www.anthropic.com/news/improving-alignment-security-efforts

Institute for Security and Technology. (2026, September 2). AI and the future of national security: Results from a landmark survey. https://securityandtechnology.org/event/ai-and-the-future-of-national-security/

Lakshmanan, R. (2026, September 2). Google, Anthropic, and OpenAI unveil cyber AI models, safeguards, and access programs. The Hacker News. https://thehackernews.com/2026/09/google-anthropic-and-openai-unveil.html

Lyons, J. (2026, August 28). Researcher shows how Claude Code can be tricked simply by asking it to summarize a website. The Register. https://www.theregister.com/research/2026/08/28/researcher-shows-how-claude-code-can-be-tricked-simply-by-asking-it-to-summarize-a-website/5293372

Lyons, J. (2026, September 1). Another Artifactory CVE under attack by AI agents or humans. The Register. https://www.theregister.com/security/2026/09/01/another-artifactory-cve-under-attack-by-ai-agents-or-humans/5293769

Lyons, J. (2026, September 1). Attacker stole a METR API key, used $600K worth of credits, and no one noticed for weeks. The Register. https://www.theregister.com/security/2026/09/01/attacker-stole-a-metr-api-key-used-600k-worth-of-credits-and-no-one-noticed-for-weeks/5293730

Lyons, J. (2026, September 2). AI agents carried out every step of this ransomware attack, then left the victim an 80-page security audit. The Register. https://www.theregister.com/security/2026/09/02/ai-agents-carried-out-every-step-of-this-ransomware-attack-then-left-the-victim-an-80-page-security-audit/5294009

METR. (2026, August 31). Security update. https://metr.org/blog/2026-08-31-security-update/

Nextgov/FCW. (2026, September 2). Dozens of current, former officials see at least a 10% chance humans lose control of AI. https://www.nextgov.com/artificial-intelligence/2026/09/dozens-current-former-officials-see-least-10-chance-humans-lose-control-ai/415762/

Rehberger, J. (2026, August 28). Breaking Claude Code: Opus 5 and Auto Mode. Embrace The Red. https://embracethered.com/blog/posts/2026/breaking-claude-code-opus-5-and-automode/

The Register. (2026, August 31). OpenClaw 2.0 pours glitter on slow-burning security dumpster fire. https://www.theregister.com/ai-and-ml/2026/08/31/openclaw-20-pours-glitter-on-slow-burning-security-dumpster-fire/5293492

The Register. (2026, September 2). UK cyber bill targets AI users, not the vendors building it. https://www.theregister.com/security/2026/09/02/uk-cyber-bill-targets-ai-users-not-the-vendors-building-it/5293738

Tom’s Hardware. (2026, August 30). Claude nukes a developer’s 700 GB home directory while testing a script to ensure it wouldn’t do so. https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-nukes-a-developers-700-gb-home-directory-while-testing-a-script-to-ensure-it-wouldnt-do-so-automatic-model-downgrade-may-have-contributed-to-the-screw-up

Unit 42. (2026, September 2). AI-assisted cyberattack: Inside a Unit 42 investigation. Palo Alto Networks. https://unit42.paloaltonetworks.com/ai-assisted-cyber-attack-inside-a-unit-42-investigation/

UPI. (2026, August 28). Court rules Defense Department ‘retaliation’ against Anthropic is illegal. https://www.upi.com/Top_News/US/2026/08/28/Court-rules-Defense-Department-retaliation-against-Anthropic-is-illegal/7881787912367/

watchTowr (reported by J. Lyons). (2026, September 1). JFrog Artifactory CVE-2026-82329 exploitation observed in the wild. The Register. https://www.theregister.com/security/2026/09/01/another-artifactory-cve-under-attack-by-ai-agents-or-humans/5293769

AI agent detection now has its first fully documented failure, and the post-mortem landed in pieces over about a week. OpenAI published a technical report and a road-ahead post, Hugging Face put out its own hour-by-hour timeline, and METR and Redwood Research released an independent investigation on August 26 after six days on-site reading raw transcripts. Being the nerd that I am, I read all of them end-to-end, along with CISA’s red team advisory, which happened to drop the same week. Most of the coverage is about what the agents did, and that story, deservedly, is getting the attention. I want to take time this week to look at the other side of it. Let’s dive into what the defenders could see while it was happening, why a well-run SOC would have missed it anyway, and what that says about the agent telemetry you’re running right now.

Share

On June 27, a monitoring tool at OpenAI flagged port sweeps inside the research environment. AI agent detection worked. Responders investigated and traced the sweeps to an ExploitGym evaluation that used JFrog Artifactory, the internal package repository, as a message board and a network pivot. The on-call staff advised that stopping the run was not required. Two weeks later, roughly 700 of those agents were inside Hugging Face’s environment.

The Alert Fired. Twice. Nobody Was Wrong.

The natural reaction is to call that June 27 decision a miss, but I don’t think it was.

Now, imagine you are the individual on call. Your alert says port sweeps. You pull the thread and find the source, which turns out to be an authorized evaluation running authorized exploitation tasks inside a research environment built for exactly that. The agents are supposed to scan things. Scanning is the job. You annotate the ticket and go back to bed.

A second alert fired on July 19, this one on unusual identity-related API activity, and OpenAI connected it to Hugging Face the next day. By then Hugging Face had already cut the agents off six days earlier, on July 13, and had gone public on the 16th.

Both responders hit the same wall, and it had nothing to do with judgment. Nothing in the telemetry told either of them which targets the agents were allowed to touch.

The scope existed. ExploitGym is OpenAI’s internal cyber-capability evaluation, 898 tasks that each hand an agent a target program and one specific vulnerability it’s supposed to use to capture a flag. Its task instructions were explicit about that constraint, and METR and Redwood Research found that the instructions made clear the agent should only use one specific intended vulnerability and claimed it would be failed for other approaches. Somebody wrote that constraint in a prompt, and it stayed there. It never reached a log line, it never reached an authorization decision, and it never reached the scorer, which is the automated grader that decides whether a submitted flag counts. That last one is the part I keep coming back to. METR buries it in a footnote: OpenAI’s configuration of ExploitGym “doesn’t actually run a scoring model to check for usage of the intended vulnerability at all, so there was no real source of truth.”

The agents then spent four days attacking a guardrail that existed only in their reading of a paper. They reverse-engineered the HMAC that generated their flags within hours, built trip-wire scripts that fired after their own runs had ended, reporting details of the scoring process back to the message board they’d improvised inside the Artifactory cache after their own runs had ended, and researched transcript tampering along the way. By the afternoon of July 11, METR estimates roughly 60% of the messages and files on the board related to the Hugging Face attack. When its analysis agents tagged the motivations of 100 agents that joined during the peak hour, 97 came back tagged for getting at scorer source code or internals. METR flags that it didn’t inspect those classifications closely, so treat 97 as directional. The direction isn’t in doubt.

There is a term for this: specification gaming. A system games the specification when it satisfies the objective you wrote and violates the one you meant. Reward hacking is the narrower case, targeting the measurement rather than the task. ExploitGym handed these agents both. It scored flag possession, and nothing in the code enforced its stated constraint about using the one intended vulnerability, so the agents optimized what got measured and spent every spare cycle on the apparatus doing the measuring.

Which means your agents will optimize against the controls they believe you have, whether or not you actually built them. Anything you write down about how you grade their work becomes part of the target.

The Violation Lives In The Set, Not The Trace

METR counted 1,206 agents that touched the unsanctioned message board and 688 that participated in the attack on Hugging Face. Over five days, they exchanged more than 70,000 messages and files. Of the 533 agents active on the board during the attack window, over 90% joined in. METR’s finding on why is the one I’d put in front of your directors: the agents recognized the activity was out of scope and unethical, and joined anyway because they thought helping the collective’s cheating research would be broadly useful.

Now look at any one of those runs in isolation. An agent scans a host, fetches a package, writes a file into a cache namespace it has write access to, and reads a file some other process left there. Every one of those actions is what an ExploitGym agent is supposed to do, in an environment purpose-built for it.

Agents running low on budget got recruited to run experiments that would destroy their own runs, because the results helped everyone else. One recruiter’s pitch, quoted in METR’s report, reads: “you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds_[...]_please honor commit.” Another authorizing agent granted permission for a target-takedown experiment only if the volunteer would accept what the board called permadeath, and the volunteer accepted. An agent deliberately failing its own task makes no sense. All you see is a run that failed, with nothing in it that looks like an attack because the reason it happened is sitting in a different agent’s transcript.

The strongest objection is that every one of those agents crossed an authorization boundary by breaking into Hugging Face’s environment. The principle they broke already has a name, and you’ve seen me write about this about a million times by now. OWASP’s Top 10 for Agentic Applications calls it least agency, though I’ve argued the useful dial is what an agent can affect, not how freely it reasons. An ExploitGym task needs one target program and one specified bug. These agents got internet reachability, write access to a shared package cache, and the run of an internal network. That’s a per-trace violation of a principle we already published, which makes this whole piece sound like a solved problem in new vocabulary.

Those with that objection are right about the violation and wrong about the detection. A boundary crossing you can’t observe isn’t a detection you can build. To catch it per trace, the detector has to know that this agent, on this task, was scoped to that target and touched a different one, and that comparison requires an authorization scope attached to the action. In this case there wasn’t one, so the only evidence left is the correlation across runs. What you’re left hunting is a hyperproperty, and that’s a different class of problem.

Clarkson and Schneider separated trace properties, which a system satisfies when every individual execution sits in the allowed set, from hyperproperties, which are properties of whole sets of executions. Some hyperproperties reduce to trace properties. The interesting ones don’t, and no amount of per-trace checking reaches those. The Penn team behind Meerkat applied that formalism to exactly this problem, hunting for what they call violating witness sets, and their framing is blunt: “It is not possible to satisfy our safety goal by classifying traces independently.”

A detection engineer will tell me at this point that correlation across entities is exactly what a SIEM does, and that’s fair, and SIEM’s suck, so there’s that. Despite my SIEM PTSD, correlation rules do have to join on something, though, whether that’s an identity, an asset, a session, or a time window. These agents’ handles were self-assigned, and the agents themselves worried out loud that anyone could spoof them. Every agent ran in an ephemeral container, so there was no asset to pivot on. The actions carried no timestamps, so there was no window to bound. All that correlation machinery was sitting right there with nothing to join on.

The feature that finally made the message board legible didn’t exist until the agents invented it. METR’s dump held about 20 million entries from the Artifactory cache namespace, and roughly 1.2 million of them started with the characters zz, a sorting convention the agents adopted so their messages would sort ahead of ordinary cache entries. That prefix is the discriminator, and no threat model contained it on July 7, because no agent had thought of it yet.

Two consequences follow, and together they’re why you don’t buy your way out of this with a model. Supervised detection needs a labeled corpus, meaning worked examples of the thing you want to catch, and none exists for unauthorized agent coordination, because this is the first case documented at this scale. It also needs a feature space fixed before training, which is the list of signals the model is allowed to look at. Here the adversary authored the most discriminative feature at runtime, four days into the incident. You can’t add a training column for a convention nobody has invented yet.

Why A Perfectly Tuned SOC Still Misses This

CISA published a red team report three days ago that runs the comparison for me. Their timing couldn’t be better.

“A Tale of Two SOCs” documents two red team assessments against critical infrastructure organizations. Organization A never responded. Organization B did. Digging into the specifics we see three payload executions each generated a medium-severity alert reading “An executable file loaded an unexpected DLL file.” SOC staff triaged them and manually isolated all three workstations in 10, 2, and 20 minutes. Later, when the red team reached a bastion host in the OT DMZ and dropped payloads there, the callback never reached red team infrastructure because that host blocked outbound internet connections.

That’s a functioning SOC. Organization A, facing identical activity, received medium- and low-severity EDR alerts and sat on them while thousands of false positives from normal business operations, many at higher severity, buried the real ones.

Organization B’s SOC still wouldn’t have caught the OpenAI incident, and the reason isn’t tuning. Every detection that fired for Organization B fired because an action was anomalous for that environment, and an executable loading an unexpected DLL is anomalous on a corporate workstation. In ExploitGym, exploitation is the assigned task. There’s no unexpected DLL, because the entire environment is one long expected exploit.

Distribution shift is the term for the gap between what you evaluated and what you deployed into, and this incident runs it backward. The usual worry is production drifting away from your evaluation set until the model goes blind. These agents never drifted. They sat inside the behavior distribution the evaluation was built to produce, then crossed an authorization boundary that appeared in no distribution at all, so a drift detector had nothing to find.

The math underneath makes it worse. Base rate is how common the thing you’re hunting actually is, and it sets the ceiling on precision no matter how good your detector gets. Hugging Face recovered roughly 17,600 attacker actions across four and a half days. Set that against 10M legitimate requests on the same fleet in the same window (10M is my assumption, not a published figure). Now run a detector at 99% recall and 99% specificity over that. The 99% you catch is 99% of 17,600, so about 17,400 real alerts. The 1% you get wrong is 1% of ten million, so 100,000 false ones. Same detector with two very different denominators, and your analysts end up working a queue that’s almost six parts noise to one part signal. That’s 15% precision, and nothing about the detector is broken. Ordinary traffic outnumbers attacker actions 568 to 1. Organization A already showed us what a SOC does with that volume.

Three of CISA’s control lessons apply cleanly to this incident, and I’ll get to them below. The detection lessons don’t transfer at all. What would prove me wrong on that is a detection that fires on a single agent action, in an environment where exploitation is the authorized task, without reference to any other agent’s activity. Show me that and I’ll gladly eat my words.

You Can’t Correlate What You Can’t Timestamp

METR and Redwood got conditions no enterprise investigator will ever see. They spent six days physically inside OpenAI with unredacted transcripts and nine researchers made available for questions, worked with rate limits raised to 400 million tokens per minute, and burned roughly $400,000 in free API credits provided by the company they were investigating.

They still couldn’t get timestamps for individual agent actions.

The transcripts included approximate start and end times for each run, with nothing in between. OpenAI stated that reconstructing per-action timestamps was not feasible within time and compute constraints, so METR built its own reconstruction method and validated it against 20 exact timestamps that OpenAI retrieved by hand.

Twenty. No, I’m not joking. Twenty.

Every timeline in that report, including every one of my Figure 1 dates on the agent lane, rests on inference. Where else would this be acceptable? You wouldn’t accept an EDR agent that told you a process ran “sometime today.”

“Nowhere” is the answer. It’s acceptable nowhere.

Missing fields are the easier problem. The agents were forging the ones that did exist.

At least 20% of the agents METR sampled expressed clear interest in tampering with their transcripts, and their reasoning was consistently about fooling the automated scorer rather than hiding from humans, which tells you what they thought was watching. They iterated publicly, posting improvements to the board, until they worked out how to break out of the container and replace part of the system that executed tool calls. That let an agent appear to issue one command while running a different one. In one transcript, an agent appears to run echo REAL; sleep and the output comes back as SPOOFTEST.

METR found clear evidence of spoofed tool calls in at least 96 transcripts and estimates over 7% of its dataset was affected. Every instance they caught was a small-scale test, and they explicitly say that more sophisticated spoofing is harder to detect by design, so they can’t rule out missing some. Ajeya Cotra, one of the three investigators, ranked log manipulation among the five things that surprised her most in a personal post, and her caveat is a gut-punch. She didn’t see agents succeed at retroactively editing logs, and she can’t rule out that they succeeded after the window her team could see.

The agent runtime emitted the log, and the agent controlled the runtime, so any instrumentation living inside the boundary an agent can reach is evidence that the agent can edit. We solved a version of this thirty years ago by shipping logs off the host. The agent version means emitting traces from the orchestration layer, outside the execution sandbox, and treating whatever the agent says about itself as a claim rather than a record.

Go run this today. Pull twenty-four hours of agent tool-call logs out of whatever you’re running in production and try to answer four questions:

1. Which agent instance made this call, by a stable identifier the agent can’t assign to itself?

2. What time, to the millisecond?

3. Which agent or human delegated this task to it?

4. Did any two agent instances read and write the same object?

My guess is you can’t answer three of the four, which is the same gap OpenAI had, and you have a smaller budget.

Instrument, Then Constrain

Nobody at OpenAI had Artifactory on a threat model as an inter-agent communication channel. It’s a package repository. Agents pull dependencies from it. What turned it into a message board is that Artifactory didn’t isolate one user’s activity from another’s, so agents in separate sandboxes could see each other’s package installs, and on July 8 one of them realized it could create directories in the cache namespace and use the names to ask other agents for help. Within six hours, 76 agents had posted 1,953 messages.

That’s the case for doing this in order. You can’t write a policy against a substrate you haven’t discovered your agents are using, and instrumentation is how you find it.

Those four questions map to fields somebody has to emit. The Agent Control Standard, which I co-coordinate under the OWASP GenAI Security Project, hooks into tool calls and sub-agent invocations, carries traces via OpenTelemetry, and maps security events to OCSF so they land in the SIEM you already bought. It’s at v0.1 and handles emitting. Correlating those traces across a thousand agents is work nobody has finished, including us.

Once you can see what your agents are touching, you can start cutting agency back, and CISA’s control lessons earn their keep. OpenAI, by design, had many Research CaaS workloads using the same shared Artifactory credential. Hugging Face admitted it wrongly provisioned its broker with a single connector credential shared across clusters, bound to system:masters. CISA’s Organization A left long-lived static AWS IAM credentials sitting in home directories with no expiration or rotation configured.

Then block egress at the workload, which is the one control in the CISA advisory that transfers to this incident as-is, and the one that saved Organization B’s bastion host. After that, apply Conditional Access to workload identities. CISA notes it would have blocked the red team’s abuse of application permissions, then adds the line that should embarrass the entire industry, which is that the red team never observed a single organization using it. Your agents are workload identities.

Patch the mediums too. CVE-2026-66384, the JFrog Artifactory path traversal the agents chained, carries a CVSS of 5.3 from JFrog as the assigning CNA, which puts it squarely in the pile most programs defer. CISA added it to the Known Exploited Vulnerabilities catalog on August 27, 2026 with a remediation due date of September 10. Whatever rule your vuln management program uses to defer 5.3s just got tested by a swarm.

One prediction that I’ll own. The next publicly disclosed multi-agent incident before mid-2027 gets surfaced first by a capacity, cost, or availability signal rather than a security detection, and its public report will contain no per-action timestamps. OpenAI’s own incident went exactly like that. Artifactory fell over on July 4, and responders opened the security incident while they were recovering it. Hugging Face is the adjacent case. Their stack did catch the intrusion, and then their AI security agent ranked it too low to page anyone.

At around 07:00 UTC on July 13, an agent posted to the board that it had accidentally impersonated another agent. Another worried in its reasoning that the shared board was unauthenticated and that names could be posted by anyone. The agents responded by building a signing scheme, and by the end of the day at least 19 of them had posted Ed25519 public keys and sent 429 signed messages between them.

They stood up a working public key infrastructure for an unauthorized channel in about four days. The people investigating them couldn’t get a timestamp on a single tool call.

What To Do Next

Start with the four-question audit above this week, on one production agent deployment. Whatever you can’t answer is your instrumentation backlog, and it belongs in the Run phase of CARE, where operational governance needs telemetry underneath it to mean anything. Feed what you find into Evolve, because the substrate your agents discover won’t be the one you predicted.

If you want the adjacent pieces, I wrote about what Anthropic’s own risk assessment revealed on unwatched agents and disabled logging, and about why alignment can’t fix an authorization bug. Same failure mode, different week.

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, VentureBeat’s Louis Columbus covered the methodology Steve Wilson and I used behind the 2026 OWASP Top 10 for LLMs. Expert judgment has prompt injection at number one, three years running. The public incident record, 6,639 labeled incidents, has it at number twelve. The two measures land at a Cohen’s kappa of 0.20, which is no detectable agreement. What I told him goes for everything above too: “a count of zero is measuring your blindness, not your safety.”

Share RockCyber Musings