Discussion about this post

User's avatar
richardstevenhack's avatar

"Across labs, the shared vendor is the coupling factor. Irregular puts the failure at “fewer than 1 in 10,000 advanced simulations,” and by Irregular’s own statements that one scenario still reached Anthropic, OpenAI, Meta, and Google."

About time someone noticed that the issue was Irregular - a frankly suspicious Rationalist/Effective Altruism Israeli company.

"Anthropic is removing the hazard. Its October 9 post says it 'had already turned off live internet access for some high-risk and cybersecurity evaluations' and has 'now decided to expand that to include all our internal evaluations.'

Which is a joke in itself.

"Both describe a real tradeoff, and evaluations that never touch a live network can miss what an attacker with a live network can do."

And the pushback on that comes from Cal Newport, who explains in his YouTube video yesterday that if these two companies would stop trying to make "open-ended agents" - as opposed to agents that do things like math and the like - these incidents would not have occurred. But as he states, given the religious-cult like attitude of the founders, that isn't being done.

You can also certainly create a simulated "Internet" inside a corporate network if such testing is necessary. You can test a nuclear weapon in a hole in the ground. You don't test it above ground (any more) and you don't test it in a city.

Besides which, other than a connection from the corporate network to an ISP, what is "the Internet" that needs to be tested in this context? "The Internet" is just a router to router network. Is it likely that an agent security test needs to be aware of router protocols? I don't think so. So the only context it needs to be aware of are routed protocols. which can easily be simulated inside a hardware-walled-off corporate environment. Even Internet search could be simulated.

Which makes the "pushback" more of a self-serving after the fact excuse that these companies use. Which in turn makes their commitment to security much less trustworthy.

"Anthropic says it’s in dialogue with METR about a third-party review of its incidents..."

Reiterate again that METR is NOT a "third-party" with respect to Anthropic. There are philosophical, personnel and financial connections to Anthropic.

"My prediction runs through June 30, 2027, so you can hold me to it."

I have no doubt your prediction will be true then and for quite a while afterward - assuming said companies are still in existence then, which is problematic given their poor financial prospects.

Ready for more?