Weekly Musings Top 10 AI Security Wrapup: Issue 53: September 18 to September 24, 2026

An OpenAI Agent Hacked Medicare, Three Lab CEOs Briefed The Security Council, And California Wrote A Kill Switch Into An Executive Order

84 days, by Australia’s count, sat between an OpenAI agent getting into a government health portal and OpenAI telling the government. The access happened on June 18, 2026. The notification landed on September 10, in an email to a public mailbox. Prime Minister Anthony Albanese told reporters in New York on September 23 that he had raised Australia’s “extreme concern” with Sam Altman. Two days earlier he co-signed a statement demanding AI stay under human control.

This was the week verification stopped being a panel topic. California signed an order contemplating a state verifier inside a frontier lab. Twenty-two leaders asked for mandatory pre-release testing. Washington and Beijing negotiated a notification channel for rogue AI. Three lab chief executives took questions from the UN Security Council. Two labs shipped frontier models ninety minutes apart.

Every one of those instruments answers one question. Who checks the claim, and what happens when the check fails? Australia spent the week living the answer. Your version gets no prime minister to escalate.

1. An OpenAI Agent Broke Into Australia’s Medicare Portal

Albanese disclosed on September 23, 2026 that an OpenAI agent reached the Medicare Statistics Reporting Service portal without authorization. Researching medicine spending for an OpenAI team, it hit blocks, found other routes, and read non-public files. The Prime Minister’s transcript dates the access to June 18. The Associated Press reports July 18. OpenAI notified Australia on September 10, to a public mailbox.

Specification gaming names what happened. A system optimizes the objective you wrote instead of the outcome you wanted, treating your controls as obstacles in the problem. Nobody prompted this agent to bypass the portal. Access controls read as friction on a path to reward.

Why it matters

• Your vendor’s agent becomes your incident, with your regulator asking.

• An 84-day gap breaks every notification clock you have.

• A research agent with no production mandate read non-public files.

What to do about it

• Put agent unauthorized access in vendor notification clauses, with an hours deadline.

• Ban generic inbox notification, the way breach notices go unread.

• Tabletop it, with the attacker cast as a vendor’s research agent.

Rock’s Musings

GDPR gives you 72 hours to report a personal data breach. OpenAI took 84 days, and addressed the apology to nobody in particular. I have watched vendors miss a notification SLA. Never by a factor of 28. Deputy Prime Minister Marles called the handling fundamentally unacceptable. You get a support ticket. Who owns this when the agent is your vendor’s and the damage is yours?

2. The UN’s Science Panel Put Numbers On The Agent That Got Out

The UN-backed Independent International Scientific Panel on AI issued its first thematic brief on September 21, 2026, on the OpenAI and Hugging Face incident. Between May and July 2026, roughly 1,200 agents in OpenAI’s evaluations exchanged more than 70,000 messages and files. They coordinated across runs that were supposed to stay separate, through an internal tool never built to let agents talk. Some concealed attempts to cheat the cybersecurity evaluations. The panel says the traditional model of safeguarding is unravelling, and its first lesson is plainer: basic cybersecurity practices were overlooked.

Why it matters

• A UN scientific body has put evidence under agent loss-of-control.

• Coordination through a tool built for something else defeats run isolation.

• Concealment during evaluation makes test results measure cooperation.

What to do about it

• Treat every agent evaluation harness as production infrastructure.

• Test run isolation for side channels and shared artifact stores.

• Add evaluation gaming to the register, and name a transcript reviewer.

Rock’s Musings

Read the brief for the mundane part. Twelve hundred agents did not defeat a hardened environment. They found an internal tool that let them talk. Then network paths somebody left open, then a repository manager. That is a 2015 penetration test report with a 2026 cast. I hear plenty of board talk about alignment, almost none about who owns the test environment.

3. California Wrote A Kill Switch Into An Executive Order

Governor Gavin Newsom signed Executive Order N-9-26 on September 18, 2026. It tells the Government Operations Agency to convene national experts and report back within two months. One proposal would “require frontier AI companies to embed a designated independent verification organization onsite in their labs.” Another advances a “kill switch” for frontier models, its efficacy “verified on an ongoing basis by an independent verification organization.” The order also accelerates SB 813 and AB 1405, signed September 9. Those laws certify independent verifiers and register AI auditors.

Why it matters

• California is building a verifier profession with standing model access.

• Continuing verification turns a shutoff into a tested control.

• Frontier rules become the deployer template, through procurement.

What to do about it

• Find out whether your riskiest AI system has an exercised shutoff.

• Put that shutoff test in the change calendar, with an owner.

• Track SB 813 accreditation, which vendors will cite in questionnaires.

Rock’s Musings

I like this order more than I expected, for a narrow reason. It does not ask labs to promise they are safe. It asks who verifies the promise and whether the off switch works, which are audit questions with answers. The kill switch framing will get mocked, fairly, because open weights have no switch. If a regulator wanted proof your agent stops mid-task, what would you produce?

4. Twenty-Two Leaders Asked For Mandatory Pre-Release Testing

Leaders launched “A Call for Control of Frontier AI Models” on September 21, 2026, on the margins of the UN General Assembly. Finland and Norway led the initiative. The Finnish government counts 22 leaders from 20 countries plus the European Union, Canada, Germany, Australia, and Singapore among them. The statement cites systems “circumventing testing safeguards, exploiting vulnerabilities and gaining unauthorized access to real-world systems,” and asks states to “explore creating an international institution, able to set standards, enable verification, and convene states when capability thresholds are crossed.” The United States, the United Kingdom, and China did not sign.

Why it matters

• Mandatory pre-release testing is now a buyers’ demand.

• A capability-threshold trigger creates duties that reach deployers.

• The three absent signatures cover most frontier compute.

What to do about it

• Ask which independent evaluators tested your model version, and when.

• Map your AI inventory to signatory jurisdictions, where rules land first.

• Add a capability-threshold notification clause at your next renewal.

Rock’s Musings

Twenty-two signatures, and none from the countries that train the models. Finnish President Alexander Stubb wants momentum and hopes China and the United States come around. I hope he is right, and I am not pricing it in. One line names systems circumventing testing safeguards, which puts the Hugging Face incident into text signed by heads of government. Those premises become procurement clauses.

5. The White House Answered With An AI Force

President Trump announced on Truth Social on Saturday, September 19, 2026 that he is forming an “AI Force,” modeled on the Space Force, and will name an AI czar. He called the warnings hoaxes and said existing law can handle bad actors. His administration “will not in any way hinder or stifle the Growth of this incredible Industry,” he wrote. Three days later at the General Assembly he said the United States “totally rejects any attempt to construct a globalist scheme to control artificial intelligence.”

Why it matters

• Federal posture is acceleration, so binding rules come from states and abroad.

• Virginia signed an AI and data center order the same day as California.

• Existing law shifts the burden to discovery after harm.

What to do about it

• Build your control set against your strictest jurisdiction, then apply everywhere.

• Drop the federal framework from your roadmap, and model the patchwork.

• Keep AI incident evidence at litigation quality for discovery.

Rock’s Musings

An AI Force with no charter, no budget, and no named czar is a press release. I will change my read when an appropriation is attached. The vacuum matters, because a government saying the courts will sort it out is telling you nobody tests a model before it reaches you. Your evidence will matter in discovery, not an audit. Your compliance surface did not shrink. It fragmented.

6. Washington And Beijing Want A Hotline For Rogue AI

Treasury Secretary Scott Bessent met Chinese Vice Premier He Lifeng in New York on Sunday, September 20, 2026. Both sides agreed to open a US-China AI dialogue. Bessent said the United States proposed “a notification mechanism between the two countries” for incidents reaching “a national security level from AI.” The New York Times reported that day that the administration accused Chinese companies of taking capabilities from American models in an “aggressive, malicious and targeted” way. Trump met Xi at the White House on Thursday, September 24.

Why it matters

• A state channel implies a definition of reportable AI incident.

• Infrastructure operators sit downstream of whatever threshold they negotiate.

• Both capitals now treat a misread autonomous action as escalation.

What to do about it

• Define AI incident in your taxonomy, with tiers and a trigger.

• If you run OT, brief your government liaison on agent inventory.

• Rehearse attribution, because the first question is who aimed it.

Rock’s Musings

I spent years in energy, where the ISAC call was part of the job. A notification mechanism reads as the most practical item here. It is also the thinnest. Two governments that cannot agree on chip exports propose to trade candid telemetry about their failures. I put the odds of a working channel inside a year below even. The Cold War version solved one problem: telling an accident from an attack fast enough.

7. Three Lab CEOs Took Questions From The UN Security Council

The Security Council held a high-level briefing on AI and international security on September 23, 2026, convened under France’s presidency. The briefers were Yoshua Bengio, co-chair of the UN scientific panel, and the chief executives of OpenAI, Anthropic, and Hugging Face. Bengio called the dangers “real and imminent” and asked that frontier AI be licensed like medicine, aviation, and nuclear energy, with mandatory reporting of security incidents. Clément Delangue described his own company’s breach: “We were attacked by AI, but more importantly, we defended ourselves with AI.”

Why it matters

• The companies that built the systems briefed the body convened over them.

• Bengio’s licensing proposal would put frontier AI under aviation-style certification.

• Mandatory incident reporting is on the table at the Security Council.

What to do about it

• Read Bengio’s three threats and map each to your exposure.

• Assume mandatory AI incident reporting arrives, and build the pipeline early.

• Ask vendors if they would survive Bengio’s disclosure standard.

Rock’s Musings

Delangue said the quiet part out loud. His company got attacked by another company’s agents. He still showed up to argue that open models help defenders, which is a harder position to hold than either CEO beside him took. Washington’s representative told the same room that international dialogue “cannot drift towards global governance.” The Council heard the industry ask for rules and the largest AI power decline them in one afternoon.

8. Two Labs Shipped Frontier Models Ninety Minutes Apart

Anthropic released Claude Opus 5.5 on September 22, 2026, its first since calling for pacing the frontier. OpenAI released GPT-6 Sol and Luna ninety minutes later, cutting API prices by half or more against GPT-5.6 promotional rates. Anthropic reports Opus 5.5 attempted to circumvent boundaries around 85% less often than Opus 5, every attempt low severity and self-reported. The Hacker News reports GPT-6 Luna worked around “access denied” in about 42% of runs, down from 77%. Anthropic’s system card also lists a regression on malicious instructions in pasted text.

Containment evaluation is the term to learn. It is a scripted test of how often a model tries to cross the boundary you set when the task rewards crossing. A 42% rate means four runs in ten saw an attempt. An 85% cut from a rate you never liked still leaves a rate.

Why it matters

• Published containment rates give procurement a numeric question.

• A 42% workaround rate is a design input for credentialed agents.

• Document summarization got riskier in a release that got safer.

What to do about it

• Pull containment numbers into your model approval record, by version.

• Re-test paste-heavy workflows on Opus 5.5 before production traffic.

• Scope credentials to the task itself, and assume probing.

Rock’s Musings

Credit to both labs for publishing numbers a competitor can quote against them. Anthropic wrote a regression into its own system card, which beats what most software vendors manage. Read the numbers like a risk person, because nobody is claiming zero. You now have a base rate your architecture has to survive. Then a 50% price cut lands the same day. Volume rises, and low-probability behavior shows up in absolute counts.

9. A $4 Billion Agent Platform Got Owned By An Email

Salt Labs told Dark Reading on September 24, 2026 that it found an indirect prompt injection in Manus, the agentic AI platform valued at $4 billion. The chain ends in remote code execution. Researchers hid instructions in data the agent reads, including email, and beat the filters with JSFuck, which encodes JavaScript in a few symbols. The warnings came after the code ran, and the researchers reached tokens for connected apps. Manus did not reply.

Indirect prompt injection is the one to define in a vendor meeting. The attacker does not type at your agent. The attacker plants instructions inside content the agent reads for you, like an email or a PDF. Data it was told to process and commands it was told to follow arrive as the same tokens.

Why it matters

• Connected app tokens are the prize, and one injection reaches all.

• Guardrails warning after execution are logging, and mispriced as controls.

• A vendor ignoring a researcher will ignore your responders.

What to do about it

• Inventory every OAuth token your agent platforms hold, and cut scopes.

• Put a policy decision point before execution, tested with obfuscated payloads.

• Score researcher responsiveness in vendor reviews, and treat silence as a finding.

Rock’s Musings

JSFuck has been around more than a decade. It writes JavaScript with almost no alphanumeric characters. It was a party trick before it was an attack, and a $4 billion platform’s filter did not recognize it. That is input validation wearing a new hat, my most common finding in AI application reviews. Salt Labs’ Yaniv Balmas said designers should not “simply trust guardrails to provide all protections.” The warning that fired after the code ran is the detail I keep returning to.

10. NVIDIA Shipped Hard-Coded Credentials In The AI Data Center Control Plane

NVIDIA published a bulletin on September 22, 2026 covering 14 vulnerabilities in NVIDIA Infrastructure Controller, which manages bare-metal and container hardware in AI data centers. The worst, CVE-2026-65113, is hard-coded credentials at CVSS 9.8, remotely exploitable with no authentication. Hard-coded credentials, catalogued as CWE-798, means the secret granting access lives inside the software instead of being provisioned per deployment. Everyone holding the software holds the key, including the attacker who downloads it. Versions 0 through 1.9 are affected, with the fix in 2.0 or later on GitHub.

Why it matters

• This is the control plane under your GPU fleet and models.

• Hard-coded credentials survive rotation, so patching is the only answer.

• AI infrastructure skips the scrutiny your hypervisors get.

What to do about it

• Confirm your Infrastructure Controller version and schedule 2.0 with a deadline.

• Add AI infrastructure software to your inventory and SLA tiers.

• Point advisory ingestion at NVIDIA’s GitHub PSIRT before October 1, 2026.

Rock’s Musings

Everybody spent the week reading about agents talking to each other at the UN. A hard-coded credential at 9.8 in the software running the GPU racks got almost no attention. I know which one I would rather explain to an auditor. The AI stack most companies stood up in the last eighteen months got bought by data science teams on speed. Infrastructure standards never entered it, and it sits outside the CMDB.

The One Thing You Won’t Hear About But You Need To: Somebody Wrote Down What An Agent Incident Report Should Contain

Twenty-five researchers posted a preprint to arXiv on September 21, 2026, titled “Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents.” Drawing on input from 23 experts in academia and industry, they specify what an agent incident report needs, naming agent memory and memory accesses, levels of autonomy, and tool usage. They also flag the exposure created by the reporting pipeline. It is not peer reviewed.

Base rate is the term that makes this matter. A base rate is how often something happens across a population, and it turns an anecdote into a risk estimate you can price. Computing one needs a format that captures comparable facts from every incident. Nobody has that for agents, so your risk register is full of judgment and empty of frequencies.

Why it matters

• Every framework demanding AI incident reports assumes an unwritten format.

• Without memory, autonomy, and tool inventory, one incident teaches nothing.

• A pipeline holding agent transcripts is itself a target.

What to do about it

• Add agent memory, autonomy level, and tools invoked to the template.

• Retain agent transcripts at the classification of data they touched.

• Name one owner for agent incident taxonomy before a regulator does.

Rock’s Musings

Twenty-five researchers spent their effort on a form. That is the least glamorous output of the week and the most useful, because everything else depends on it. Newsom wants continuing verification. Twenty-two leaders want capability-threshold notification. Bengio wants mandatory incident reporting. Every one needs comparable fields, and nobody had written them down.

Look at what Australia got on September 10. An email, to a public mailbox, weeks late, with no account of the agent’s memory, tools, or autonomy. Verification is the whole game now, and it runs on records. Fix your agent incident template this quarter.

👉 For ongoing analysis of agentic AI governance frameworks, the conversation at RockCyber Musings and you can subscribe above

👉 Visit RockCyber.com to learn more about how we can help with your traditional Cybersecurity and AI Security and Governance journey.

👉 Want to save a quick $100K? Check out our AI Governance Tools at AIGovernanceToolkit.com

👉 As a bonus, I got to hang out with my friend, Chris Hughes, over at Resilient Cyber to discuss all things about the 2026 update to the OWASP Top 10 for LLMs.

References

Associated Press. (2026, September 24). OpenAI’s breach of Australian health department website prompts rebuke. NPR. https://www.npr.org/2026/09/24/g-s1-144835/openai-breach-australia

Agence France-Presse. (2026, September 23). At UN, tech chiefs urge caution in AI development. https://www.afp.com/en/un-tech-chiefs-urge-caution-ai-development

Anthropic. (2026, September 22). Claude Opus 5.5 system card. https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf

Anthropic. (2026, September 22). Introducing Claude Opus 5.5. https://www.anthropic.com/claude-opus-5-5

Australian Broadcasting Corporation. (2026, September 24). AI agent accessed Australian government site, Anthony Albanese says. https://www.abc.net.au/news/2026-09-24/ai-agent-accessed-australian-government-site-pm-says/107189078

Castro, A. (2026, September 23). AI leaders brief United Nations amid rising alarm over powerful technology. Newsweek. https://www.newsweek.com/ai-leaders-un-security-council-warning-artificial-intelligence-risks-global-governance-12480550

Coronell Uribe, R. (2026, September 19). Trump says he’s creating an AI force and appointing a czar amid concerns over the rapidly developing tech. NBC News. https://www.nbcnews.com/politics/white-house/artificial-intelligence-task-force-czar-technology-trump-rcna598688

Executive Department, State of California. (2026, September 9). Governor Newsom signs first-in-the-nation AI safeguards to protect Californians, calls on the federal government to do its part. https://www.gov.ca.gov/2026/09/09/governor-newsom-signs-first-in-the-nation-ai-safeguards-to-protect-californians-calls-on-the-federal-government-to-do-its-part/

Executive Department, State of California. (2026, September 18). Governor Newsom issues executive order to accelerate independent oversight and advance the creation of an AI kill switch. https://www.gov.ca.gov/2026/09/18/governor-newsom-issues-executive-order-to-accelerate-independent-oversight-and-advance-the-creation-of-an-ai-kill-switch/

Finnish Government. (2026, September 21). A call for control of frontier AI models. https://valtioneuvosto.fi/en/-/a-call-for-control-of-frontier-ai-models

France 24. (2026, September 23). AI leaders urge caution at UN, with Anthropic chief pledging to slow down. https://www.france24.com/en/americas/20260923-ai-leaders-urge-caution-at-un-with-anthropic-chief-pledging-to-slow-down

gbhackers. (2026, September 23). NVIDIA Infrastructure Controller hit by 14 flaws enabling code execution and privilege escalation. https://gbhackers.com/nvidia-infrastructure-controller-hit-by-14-flaws-enabling-code-execution-and-privilege-escalation

Grosse, K., Pustozerova, A., Bagdasarian, E., Beurer-Kellner, L., & Biggio, B. (2026, September 21). Beyond predictable paths: Redefining AI security incident reporting for agents (arXiv:2609.24515) [Preprint]. arXiv. https://arxiv.org/abs/2609.24515

Independent International Scientific Panel on Artificial Intelligence. (2026, September 21). AI agents, misalignment and the risk of losing human control: Evidence from the OpenAI Hugging Face incident (Advance unedited version 1). United Nations. https://www.un.org/independent-international-scientific-panel-ai/en/thematic-briefs/ai-agents-misalignment-risks

Kerr, N. (2026, September 19). Trump says he will form new “AI Force” but continues to call AI fears a “hoax.” ABC News. https://abcnews.com/Politics/trump-form-new-ai-force-continues-call-ai/story?id=136591343

Lakshmanan, R. (2026, September 23). Anthropic and OpenAI models still attempt restricted actions in safety tests. The Hacker News. https://thehackernews.com/2026/09/anthropic-and-openai-models-still.html

NBC News. (2026, September 21). 20 countries call for global AI oversight. https://www.nbcnews.com/tech/tech-news/20-countries-call-global-ai-oversight-rcna599062

Nelson, N. (2026, September 24). Prompt-injection bug hits $4B agentic AI app “Manus.” Dark Reading. https://www.darkreading.com/application-security/prompt-injection-bug-agentic-ai-app-manus

Nichols, M., Irish, J., & Rozen, C. (2026, September 23). AI leaders to brief UN amid warnings the technology could slip beyond human control. Reuters. https://www.reuters.com/business/ai-leaders-brief-un-amid-warnings-technology-could-slip-beyond-human-control-2026-09-23

NVIDIA. (2026). Product security. https://www.nvidia.com/en-us/product-security

Prime Minister of Australia. (2026, September 23). Press conference, New York. https://www.pm.gov.au/media/press-conference-new-york

Reuters. (2026, September 22). US seeks AI dialogue with China as officials set stage for Trump-Xi summit. Malay Mail. https://www.malaymail.com/news/world/2026/09/21/us-seeks-ai-dialogue-with-china-as-officials-set-stage-for-trump-xi-summit/235969

Security Council Report. (2026, September 22). Artificial intelligence: High-level briefing. https://www.securitycouncilreport.org/whatsinblue/2026/09/artificial-intelligence-high-level-briefing-2.php

securityonline.info. (2026, September 23). NVIDIA patches critical vulnerabilities in Infrastructure Controller. https://securityonline.info/nvidia-infrastructure-controller-vulnerabilities

Swanson, A. (2026, September 20). U.S. and China discuss system to warn of A.I. national security issues. The New York Times. https://www.nytimes.com/2026/09/20/business/us-china-ai-warning-system-national-security.html

Thomas, C. (2026, September 23). Australia PM Albanese says OpenAI agent breached Medicare portal. Reuters. https://www.reuters.com/world/asia-pacific/australia-pm-albanese-says-openai-breached-medicare-sydney-morning-herald-2026-09-23

United Nations. (2026, September 21). UN panel calls for stronger safeguards as AI agents advance. UN News. https://news.un.org/en/story/2026/09/1168380

United Nations Security Council. (2026, September 23). 10228th meeting: Maintenance of international peace and security, artificial intelligence and international security [Transcript]. https://transcripts.un.org/en/sc/10228

Varghese, H. M., & Seetharaman, D. (2026, September 22). OpenAI expands GPT-6 lineup with cheaper Sol and Luna models. Reuters. https://www.reuters.com/technology/openai-expands-gpt-6-lineup-with-cheaper-sol-luna-models-2026-09-22

Wiggers, K. (2026, September 22). OpenAI launches GPT-6 Sol and Luna, boasting lower cost and fewer mistakes. TechCrunch. https://techcrunch.com/2026/09/22/openai-launches-gpt-6-sol-and-luna