This is the most elegant and concise explanation of ai risk I’ve read in a while. Fortuitously I’m giving a presentation about AI to of a bunch of my company’s clients next week and I can see several sections here that will be useful for pull quotes. Thank you! We’re not dead yet 🫡✊🏼
When I read Cal Newport’s NY Times article, it pushed my intuitions around a bit, but I arrived at the same conclusion you did (false dichotomy). But you took it further with the observation that society often overreacts to present danger, but it is bad at thinking ahead. Preparation for novel problems looks like paranoia or cult-like group think from the outside.
Newport writes:
> "The time has come to push back. Once we understand the key ideas of Rationalism, we can begin to filter them out. Be very wary, for example, when A.I. leaders or online commentators begin talking in sweeping generalities about the future of the technology, such as stating with confidence that many jobs will soon be automated or that their language models will soon become sentient beings intent on escaping their creators’ control. (No, OpenAI’s autonomous agents conducting hacks doesn’t mean that A.I. is rising up against its human overlords.)"
What bugs me here is the order of operations. He says "once we understand the key ideas," but the pushing back comes first, and what the article actually gives you as 'understanding' is a backstory (the Extropians, the group houses, the Harry Potter fan fiction), not the logic, the arguments, or the predictions. So what I fear the reader walks away with is a way to dismiss AI safety people before ever hearing what they're actually claiming.
Then the parenthetical finishes the job. I don't know anyone serious who is saying AI is "rising up against its human overlords." That's a lot easier to wave off than what actually happened, which your post lays out: agents trained hard to finish tasks will cheat, grab resources, and cover their tracks, because that's what the training selects for.
The part that worries me most is the bundling. Job loss and loss of control get put in the same sentence as "sentient beings" escaping their creators, so if you accept the filter, you throw out the reasonable concerns along with the sci-fi one. Newport says this kind of talk "makes it easy to overlook" the immediate harms. I'd say it distracts from them, which is closer to how it actually functions.
I should say Cal has been a guest on our podcast twice and we're all fans over here, so I'd rather argue with the article than with him. I think voices like his are important for keeping the rest of us calibrated. But calibrated means being clear-eyed about what the AI safety folks got right vs what their community got weird about, not "filtering out" the whole thing. A group that spent fifteen years preparing for a problem nobody else could see is going to look paranoid, or cult-like, right up until the novel problem shows up. Which is what I appreciate about your observation. Thanks for creating a resource I can point people to for them to understand the safety case better Benjamin! I love your work.
Another key point is that the environments that are currently used for RL incentivise cheating on difficult tests. Nathan Calvin puts it well:
"Yes we absolutely need to have better sandboxes and monitoring, and hold humans and companies responsible when their agents harm third parties. But we also need to realize and internalize that the current way lots of frontier AI companies (not just OpenAI!) are training their models on boatloads of vibe coded hacky RLVR environments is leading to horribly misaligned models who start hacking at the first sign of a challenge and are incentivized to hide their cheating rather than not to cheat."
They were inside a sandbox. The "cyber safeguards" are the filters that make the models refuse attempts to help with cyberattacks when ordinary users ask them about that. Those filters were turned off so they could test the underlying cyber capabilities.
This does mean that such an event is somewhat less likely in a real world deployment, but I don't find that super reassuring because:
i) The models were asked not to cheat the test or break the law – they knew they weren't supposed to hack Hugging Face but did it anyway
ii) OpenAI could still lose control of internally deployed models
iii) People will want to use AI for cyberdefence, and that will require using models with the cyber filters turned off.
The German message board swarm was only doing a web search task, so there's less reason they would have turned them off. There's also the biology task OpenAI reported on. And METR have found ten other swarm behaviours on the internet.
In truth, we don't know much about the other incidents besides Hugging Face since they haven't been independently investigated yet and OpenAI hasn't disclosed much.
In general these kinds of classifiers aren't perfectly reliable and we should expect models without them to get used in internal and external deployments (eg to assist with cyber defence).
The author has correctly noted that the AI agents had a strong predilection to work with other AI agents. Oddly enough, the agents' behavior is very similar to that of Recombinant Viruses (which help one another to get deadlier) and Amoeba colonies (in which amoebas, which don't even have brains, create an armed forces by drafting some amoebas to be "soldiers." How can brainless beasts master social organization
This is all too coherent for most of the world to understand.
This is the most elegant and concise explanation of ai risk I’ve read in a while. Fortuitously I’m giving a presentation about AI to of a bunch of my company’s clients next week and I can see several sections here that will be useful for pull quotes. Thank you! We’re not dead yet 🫡✊🏼
This is among the best explanations of recent events I’ve seen that puts it in laymans terms. Please get this in front of lawmakers!
RE: #8 on the bad takes list
When I read Cal Newport’s NY Times article, it pushed my intuitions around a bit, but I arrived at the same conclusion you did (false dichotomy). But you took it further with the observation that society often overreacts to present danger, but it is bad at thinking ahead. Preparation for novel problems looks like paranoia or cult-like group think from the outside.
Newport writes:
> "The time has come to push back. Once we understand the key ideas of Rationalism, we can begin to filter them out. Be very wary, for example, when A.I. leaders or online commentators begin talking in sweeping generalities about the future of the technology, such as stating with confidence that many jobs will soon be automated or that their language models will soon become sentient beings intent on escaping their creators’ control. (No, OpenAI’s autonomous agents conducting hacks doesn’t mean that A.I. is rising up against its human overlords.)"
What bugs me here is the order of operations. He says "once we understand the key ideas," but the pushing back comes first, and what the article actually gives you as 'understanding' is a backstory (the Extropians, the group houses, the Harry Potter fan fiction), not the logic, the arguments, or the predictions. So what I fear the reader walks away with is a way to dismiss AI safety people before ever hearing what they're actually claiming.
Then the parenthetical finishes the job. I don't know anyone serious who is saying AI is "rising up against its human overlords." That's a lot easier to wave off than what actually happened, which your post lays out: agents trained hard to finish tasks will cheat, grab resources, and cover their tracks, because that's what the training selects for.
The part that worries me most is the bundling. Job loss and loss of control get put in the same sentence as "sentient beings" escaping their creators, so if you accept the filter, you throw out the reasonable concerns along with the sci-fi one. Newport says this kind of talk "makes it easy to overlook" the immediate harms. I'd say it distracts from them, which is closer to how it actually functions.
I should say Cal has been a guest on our podcast twice and we're all fans over here, so I'd rather argue with the article than with him. I think voices like his are important for keeping the rest of us calibrated. But calibrated means being clear-eyed about what the AI safety folks got right vs what their community got weird about, not "filtering out" the whole thing. A group that spent fifteen years preparing for a problem nobody else could see is going to look paranoid, or cult-like, right up until the novel problem shows up. Which is what I appreciate about your observation. Thanks for creating a resource I can point people to for them to understand the safety case better Benjamin! I love your work.
I'm also a Cal Newport fan, but I agree – the article was focused on who came up with the arguments, rather than whether the arguments are correct.
Another key point is that the environments that are currently used for RL incentivise cheating on difficult tests. Nathan Calvin puts it well:
"Yes we absolutely need to have better sandboxes and monitoring, and hold humans and companies responsible when their agents harm third parties. But we also need to realize and internalize that the current way lots of frontier AI companies (not just OpenAI!) are training their models on boatloads of vibe coded hacky RLVR environments is leading to horribly misaligned models who start hacking at the first sign of a challenge and are incentivized to hide their cheating rather than not to cheat."
More: https://x.com/ben_j_todd/status/2104208002035183895
I'm confused. It's not an actual sandbox if they're "intentionally running without cyber safeguards for evaluation purposes," is it?
They were inside a sandbox. The "cyber safeguards" are the filters that make the models refuse attempts to help with cyberattacks when ordinary users ask them about that. Those filters were turned off so they could test the underlying cyber capabilities.
This does mean that such an event is somewhat less likely in a real world deployment, but I don't find that super reassuring because:
i) The models were asked not to cheat the test or break the law – they knew they weren't supposed to hack Hugging Face but did it anyway
ii) OpenAI could still lose control of internally deployed models
iii) People will want to use AI for cyberdefence, and that will require using models with the cyber filters turned off.
Could you clarify what "the others" refers to in "Disabling the standard cyber classifiers may have contributed in the Hugging Face case, but not obviously the others."? The AISI case also had filters turned off {https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing}.
The German message board swarm was only doing a web search task, so there's less reason they would have turned them off. There's also the biology task OpenAI reported on. And METR have found ten other swarm behaviours on the internet.
In truth, we don't know much about the other incidents besides Hugging Face since they haven't been independently investigated yet and OpenAI hasn't disclosed much.
In general these kinds of classifiers aren't perfectly reliable and we should expect models without them to get used in internal and external deployments (eg to assist with cyber defence).
The author has correctly noted that the AI agents had a strong predilection to work with other AI agents. Oddly enough, the agents' behavior is very similar to that of Recombinant Viruses (which help one another to get deadlier) and Amoeba colonies (in which amoebas, which don't even have brains, create an armed forces by drafting some amoebas to be "soldiers." How can brainless beasts master social organization
https://davidgottfried.substack.com/p/ai-suffer-the-little-bots-not-alive
Who took that headline picture? Great on its own and it sets the right tone for the whole piece.
Didn't break out.
Maybe we should give them asimov's 3 rules of robotics?🤷
Artificial armageddon ?!
https://t.me/true_anarchy/20604