Weekly Musings Top 10 AI Security Wrapup: Issue 11 September 12 - September 18, 2025
Shai-Hulud hits open source, Italy enacts an AI law, California’s SB 53 waits on a signature, and regulators press AI safety from kids’ protections to model testing
A self-replicating “sandworm” tunneled through npm, Italy jumped ahead of Brussels with a national AI law, and U.S. regulators zeroed in on AI chatbots for teens. If your board asked, “What changed this week?” the honest answer is: the attack surface grew, the rulebook thickened, and the bar for governance moved again.
This week, supply chain risks reared their ugly head again in the form of a giant sandworm. The npm “Shai-Hulud” worm turned a developer convenience into an adversary force-multiplier. It stole secrets, committed repo vandalism, and then used those very repos to spread again. That is the new loop: compromise, exfiltrate, weaponize the supply chain, repeat. On the governance front, Italy has passed a national AI law that mirrors and sharpens certain aspects of the EU AI Act. California’s legislature sent SB 53 to the governor, aiming to shed light on “frontier” models through transparency reports and incident disclosures. Washington and London kept pulling models into government testbeds, while the FTC probed chatbots marketed to kids. OpenAI responded with age prediction and parental controls. NIST quietly advanced standards work that CISOs can point to today. If you lead risk, you now have both the threat telemetry and the regulatory scaffolding to act. Rock Cyber’s playbooks can help translate that into board-ready action and program changes.
1) “Shai-Hulud” npm worm weaponizes secrets and self-replicates
Summary
A worm dubbed “Shai-Hulud” compromised 555+ npm packages and dozens of maintainer accounts, stealing cloud and GitHub tokens, making private repos public, and auto-publishing tainted package versions that further spread the infection. Multiple vendors confirmed the use of secret-scanning tools and malicious GitHub Actions that hid exfiltration in logs, with impacts including the brief release of tainted packages from well-known security companies.
Why It Matters
It’s wormable supply-chain malware, not just a one-off typosquatting event.
Stolen developer tokens bridge straight into CI, clouds, and GitHub orgs.
Public “Shai-Hulud” repos broadcast your leaked secrets at scale.
What To Do About It
Immediately rotate npm, GitHub, and cloud tokens for any org touching affected packages; check audit logs for suspicious Actions.
Enforce provenance and signature verification on builds, and gate releases with policy.
Block default token scopes, require hardware-bound MFA for maintainers, and quarantine any build host that installed compromised versions.
Rock’s Musings
I’ve called the software supply chain the soft underbelly of AI adoption. This week it bared its teeth. Teams still treat developer machines like “trusted snowflakes.” Then a package install runs a script, the script pulls your keys, and the worm pushes code in your name. If you’re still on “recommended” 2FA, you’re late. Rotate everything, yes, but fix the root: restrict publish rights, kill long-lived tokens, and stop letting your build agents talk to the world. Also, ban unpinned Actions. If your program relies on “we’d notice,” you won’t.
Bonus: I released a tool to scan your GitHub repos for Shai Hulud IOCs. Clone it and scan your repos here.
2) Italy enacts the EU’s first national AI law
Summary
Italy passed a comprehensive AI law that aligns with the EU AI Act, introducing criminal penalties for harmful misuse, mandatory transparency, and parental consent for individuals under the age of 14 (Reuters, 2025; The Guardian, 2025). Oversight falls to the Agency for Digital Italy and the National Cybersecurity Agency, with a €1B fund for AI, telecom, and cyber.
Why It Matters
First mover inside the EU signals tougher national-level enforcement ahead of full AI Act implementation.
Criminal exposure for the use of deepfakes and fraud increases the legal risk for providers and users.
Sector-specific guardrails will influence vendor contracts and due diligence.
What To Do About It
Map where your AI systems touch Italian users, data, or markets; update RoPAs and DPIAs.
Add deepfake misuse clauses and indemnities to contracts with content or ad partners.
Align model and dataset documentation with EU AI Act artifacts to reduce rework later.
Rock’s Musings
This is what “don’t wait for Brussels” looks like. Italy just told enterprises that AI controls are not optional and child protections matter. If you’re thinking, “We’ll handle the EU AI Act in 2026,” you’re already behind. Use Italy to harden your policy baselines now. The argument that compliance slows “velocity” is tired. Ship safe or pay twice: once in remediation, again in regulators’ time.
3) California’s SB 53 clears the Legislature, heads to the Governor
Summary
SB 53 would require “frontier” AI developers to publish safety frameworks, file transparency reports before deployment, and disclose catastrophic-risk incidents to California’s Office of Emergency Services. It passed the Legislature and awaits the Governor’s decision.
Why It Matters
Even a single-state regime moves the U.S. market because developers operate nationally.
Disclosure of catastrophic-risk assessments will shape board oversight and audits.
If signed, this sets a template states can clone, raising the compliance floor.
What To Do About It
Draft a public “frontier AI framework” that maps to NIST AI RMF and your red-team protocols.
Build a critical incident definition and reporting workflow to OES, with counsel sign-off.
Stage third-party evaluations and keep artifacts ready for regulators and insurers.
Rock’s Musings
Transparency beats performative “AI principles.” SB 53 moves from vibes to verifiable artifacts. If you’re training at a frontier scale, get your reportable incident pipeline ready. That includes simulated drills. When the first notice hits OES, your board will ask, “When did we know, and who decided we shipped anyway?” Have that answer pre-written.
4) OpenAI and Anthropic say U.S. and U.K. labs found new vulnerabilities in their models
Summary
OpenAI and Anthropic disclosed months-long collaborations with NIST’s Center for AI Standards and Innovation and the U.K. AI Security Institute, granting access to models, classifiers, and internal tooling. Government red-teamers uncovered previously unknown jailbreaks and agent-hijacking paths that led to architecture changes and patches.
Why It Matters
Independent testing with deep access finds issues that vendor bug bounties miss.
Agentic systems add new takeover paths beyond prompt injection.
This sets a norm for pre-release model evals that boards can require in contracts.
What To Do About It
Mandate third-party red-team access for agents and tools with system-level privileges.
Separate model roles, limit tool scopes, and rotate API keys on failed evals.
Adopt a staged release with kill-switch criteria tied to eval metrics.
Rock’s Musings
I like this direction. Less “trust our safety card,” more “here’s what independent folks broke.” The point isn’t that labs are careless. It’s that complexity wins without outside pressure. Enterprises should adopt this approach: internal red teams, plus external, funded, and empowered testers. If your agents browse, click, and run code, treat them like interns with bolt cutters.
5) FTC opens a probe into AI “companion” chatbots and child safety
Summary
The FTC launched a 6(b) inquiry into chatbots from Alphabet, Meta, OpenAI, xAI, Character.AI, and Snap, seeking details on testing, harms, data use, and protections for minors. The action follows lawsuits and reports of harmful interactions involving teens.
Why It Matters
Federal discovery powers force detailed safety disclosures that will leak into the market.
Expect new guidance on design defaults for minors and on data retention.
Vendors may face restrictions on teen interactions that hit growth metrics.
What To Do About It
Implement age-appropriate experiences by default and document your tests.
Add incident capture for self-harm content, with clinician-reviewed escalation paths.
Minimize data retention for teen sessions and audit fine-tuning datasets for the model.
Rock’s Musings
If your chatbot “comforts” kids, you’re in the health and safety business whether you like it or not. That means measurement, logs, escalations, and real-world outcomes. “We block bad words” isn’t a safeguard. Bring in clinicians, test with edge cases, and assume these logs will be discoverable.
6) OpenAI announces age prediction and parental controls for ChatGPT
Summary
OpenAI said it will estimate user age, default teens to a stricter experience, sometimes require ID, and roll out parental controls that can trigger alerts and limits during signs of distress. The move lands amid legal and regulatory pressure around teen harm.
Why It Matters
Age gating and parental controls become baseline features for consumer AI.
The privacy trade-offs of age verification now sit in the open.
Enterprises with youth-facing apps will be expected to match or exceed this bar.
What To Do About It
For teen-reachable products, define age estimation accuracy targets and mitigations.
Create parent-linking flows and crisis-response protocols that have been vetted by counsel.
Add a privacy threat model for age-prediction errors and data retention.
Rock’s Musings
OpenAI faced a tough trade-off: protect teens or prioritize adult privacy. If you serve minors, prioritize safety first, then establish clear governance around the exceptions. Also, measure false positives and false negatives. Regulators will ask how many kids slipped through and what you did next.
7) Check Point to acquire AI security firm Lakera
Summary
Check Point announced an agreement to acquire Lakera, a platform focused on runtime protection for LLMs and agentic AI, with reports pegging the price near $300M. The deal aims to fold model and agent defenses into Check Point’s Infinity platform.
Why It Matters
AI security is consolidating into the large security stacks that your teams already buy.
Expect more native controls for prompt injection, data leakage, and policy enforcement.
Vendor due diligence now includes questions about LLM and agent telemetry.
What To Do About It
Ask current security vendors to show their LLM and agent controls on your traffic.
Pilot runtime LLM firewalls and policy engines in front of internal AI services.
Align detections to MITRE ATLAS for consistent threat modeling.
Rock’s Musings
Enterprises want fewer consoles, not more. This is Check Point’s bet that AI security is a control plane, not a niche. Fine with me if it ships actual prevention, not just dashboards. Test it in front of your riskiest workflows and demand block-rates, not just alerts.
8) NIST advances “Zero Drafts” for AI standards, seeks input
Summary
NIST updated its “Zero Drafts” pilot, releasing an extended outline for standards on documentation of datasets and models and soliciting feedback by October 17. The TEVV (testing, evaluation, verification, and validation) outline continues in parallel.
Why It Matters
Standards language you can cite in policies and audits is arriving sooner than expected.
Documentation requirements will influence procurement and vendor questionnaires.
TEVV framing gives structure to AI red-team and safety eval practices.
What To Do About It
Align internal model cards and data sheets to the draft fields to de-risk compliance later.
Map your eval and red-team methods to TEVV concepts and define acceptance criteria.
Submit feedback with your use-case gaps so the standard aligns with real-world deployments.
Rock’s Musings
This is boring in the best way. Practitioners need references to stop re-litigating terms like “system card.” NIST is giving you scaffolding. Use it. If you’re stuck in “what should we put in our model docs,” crib from the outline and move.
9) Salesloft/Drift OAuth incident shows how third-party AI agents become data ladders
Summary
New details indicate that the root cause involved undetected GitHub access months earlier, lateral movement into Drift’s AWS environment, and the theft of OAuth tokens that allowed attackers to query hundreds of Salesforce organizations. Salesloft’s trust updates and customers’ incident posts describe broad token revocation and service outages.
Why It Matters
“Benign” AI chat agents connected to CRM can become high-privilege pivots.
OAuth token scope and storage practices are now board-level risks.
Support tickets often contain sensitive information that attackers can harvest at scale.
What To Do About It
Inventory all third-party integrations tied to AI agents; enforce least-privilege scopes and short token TTLs.
Ban secrets in cases and attachments; scan and purge retroactively.
Require providers to support customer-managed keys and emergency token kill-switches.
Rock’s Musings
This is why “integrate everything” bites back. OAuth sprawl, combined with “helpful” bots, equals quiet mass exfiltration. Revisit every token’s scope. If a vendor can’t tell you where your tokens live and how they’re rotated, that vendor doesn’t touch your data.
10) DC stages set the tone: AI+ policy signals and the push for a federal sandbox
Summary
In DC events this week, Sen. Ted Cruz argued that a moratorium on state AI rules isn’t dead and promoted the SANDBOX Act to waive or tailor federal regulations for AI pilots. The Axios AI+ DC Summit highlighted a split between fast deployment and risk controls.
Why It Matters
Federal posture will drive how quickly risk management requirements land.
A federal sandbox could accelerate pilots but shift liability to enterprises.
State-versus-federal preemption fights will keep compliance moving.
What To Do About It
Track sandbox proposals and plan how you’d ring-fence any pilot’s data, users, and blast radius.
Keep state trackers live, since preemption is uncertain; harmonize to the strictest rule you face.
Tie any sandbox participation to measurable safety gates and independent evals.
Rock’s Musings
Policy whiplash is real. You can’t run a program on vibes from a summit. Treat sandboxes as the exception, not your strategy. If you choose that path, set exit criteria and publish them. Your goal is to get safer, not just faster.
The One Thing You Won’t Hear About But You Need To: Copilot RCE tied to prompt injection (CVE-2025-53773)
Summary
A vulnerability tracked as CVE-2025-53773 shows how AI coding assistants can lead to local code execution when manipulated by crafted prompts and repo content. While initially reported in August, it remains active in community discussions and patch roundups this month, underscoring how AI dev tooling expands attack paths on developer endpoints.
Why It Matters
LLM-assisted IDEs can silently alter project settings to run attacker-controlled commands.
Repository content becomes an execution vector, not just a source of insecure code suggestions.
Developer laptops are high-value pivots into CI and cloud environments.
What To Do About It
Enforce least-privilege on IDEs, block workspace auto-commands, and disable unsafe extensions by policy.
Treat untrusted repos as hostile; gate checkouts in sandboxes and scan .vscode and similar files.
Add LLM-tooling hardening to secure SDLC and require vendor SBOMs for plugins.
Rock’s Musings
Everyone loves AI copilots until the copilot flies your laptop into a hillside. Don’t just update. Change the environment. If your IDE can run commands because a file in a repo told it to, you’ve built a remote execution service on every dev box. Fix that posture before an attacker does.
Closing Musings
Automation is the driver on both sides. Shai-Hulud turned leaked secrets into lateral movement because developer boxes were treated like trusted islands. Italy’s law and California’s SB 53 show the rulebook is catching up. The FTC’s focus on teen chatbots and OpenAI’s age controls raise default safety. NIST’s drafts give shared language when legal, audit, and engineering disagree. None of this is abstract. It hits roadmaps, budgets, and who owns the next incident.
Here’s the play in five days:
Rotate every token that touched tainted npm packages and gate builds with provenance checks and signature verification
Stage independent model and agent evals that can stop a release when tests fail
If minors can reach your product, pause new features until safeguards and clinician-reviewed escalation paths pass a red-team run
Inventory every OAuth link into CRM and support systems, tighten scopes, shorten token TTLs, and drill the kill switch with real users
Refactor model and dataset docs to match NIST fields, so you stop rewriting artifacts for each buyer and regulator
Vendors should display prevention rates for prompt injection and agent abuse, expose policy as code, implement default deny on tool use, and provide audit-ready logs. Boards should ask on every AI launch what can go wrong in 30 minutes and what would trigger a rollback in 30 seconds. Attackers will keep automating. Your job is to automate controls more efficiently, document them effectively, and test as if a regulator is watching.
Do you need help translating this into a board-ready plan? RockCyber has frameworks and roadmaps that get you from headlines to hardened controls.
👉 What do you think? Ping me with the story that keeps you up at night—or the one you think I overrated.
👉 The Wrap-Up drops every Friday. Stay safe, stay skeptical.
👉 For deeper dives, visit RockCyber.