15 Comments
User's avatar
Alex Libre's avatar

This is all too coherent for most of the world to understand.

David Dabney's avatar

This is the most elegant and concise explanation of ai risk I’ve read in a while. Fortuitously I’m giving a presentation about AI to of a bunch of my company’s clients next week and I can see several sections here that will be useful for pull quotes. Thank you! We’re not dead yet 🫡✊🏼

Eskimo1's avatar

This is among the best explanations of recent events I’ve seen that puts it in laymans terms. Please get this in front of lawmakers!

Glen McNiel's avatar

RE: #8 on the bad takes list

When I read Cal Newport’s NY Times article, it pushed my intuitions around a bit, but I arrived at the same conclusion you did (false dichotomy). But you took it further with the observation that society often overreacts to present danger, but it is bad at thinking ahead. Preparation for novel problems looks like paranoia or cult-like group think from the outside.

Newport writes:

> "The time has come to push back. Once we understand the key ideas of Rationalism, we can begin to filter them out. Be very wary, for example, when A.I. leaders or online commentators begin talking in sweeping generalities about the future of the technology, such as stating with confidence that many jobs will soon be automated or that their language models will soon become sentient beings intent on escaping their creators’ control. (No, OpenAI’s autonomous agents conducting hacks doesn’t mean that A.I. is rising up against its human overlords.)"

What bugs me here is the order of operations. He says "once we understand the key ideas," but the pushing back comes first, and what the article actually gives you as 'understanding' is a backstory (the Extropians, the group houses, the Harry Potter fan fiction), not the logic, the arguments, or the predictions. So what I fear the reader walks away with is a way to dismiss AI safety people before ever hearing what they're actually claiming.

Then the parenthetical finishes the job. I don't know anyone serious who is saying AI is "rising up against its human overlords." That's a lot easier to wave off than what actually happened, which your post lays out: agents trained hard to finish tasks will cheat, grab resources, and cover their tracks, because that's what the training selects for.

The part that worries me most is the bundling. Job loss and loss of control get put in the same sentence as "sentient beings" escaping their creators, so if you accept the filter, you throw out the reasonable concerns along with the sci-fi one. Newport says this kind of talk "makes it easy to overlook" the immediate harms. I'd say it distracts from them, which is closer to how it actually functions.

I should say Cal has been a guest on our podcast twice and we're all fans over here, so I'd rather argue with the article than with him. I think voices like his are important for keeping the rest of us calibrated. But calibrated means being clear-eyed about what the AI safety folks got right vs what their community got weird about, not "filtering out" the whole thing. A group that spent fifteen years preparing for a problem nobody else could see is going to look paranoid, or cult-like, right up until the novel problem shows up. Which is what I appreciate about your observation. Thanks for creating a resource I can point people to for them to understand the safety case better Benjamin! I love your work.

Benjamin Todd's avatar

I'm also a Cal Newport fan, but I agree – the article was focused on who came up with the arguments, rather than whether the arguments are correct.

Benjamin Todd's avatar

Another key point is that the environments that are currently used for RL incentivise cheating on difficult tests. Nathan Calvin puts it well:

"Yes we absolutely need to have better sandboxes and monitoring, and hold humans and companies responsible when their agents harm third parties. But we also need to realize and internalize that the current way lots of frontier AI companies (not just OpenAI!) are training their models on boatloads of vibe coded hacky RLVR environments is leading to horribly misaligned models who start hacking at the first sign of a challenge and are incentivized to hide their cheating rather than not to cheat."

More: https://x.com/ben_j_todd/status/2104208002035183895

Glenn Whiteside's avatar

I'm confused. It's not an actual sandbox if they're "intentionally running without cyber safeguards for evaluation purposes," is it?

Benjamin Todd's avatar

They were inside a sandbox. The "cyber safeguards" are the filters that make the models refuse attempts to help with cyberattacks when ordinary users ask them about that. Those filters were turned off so they could test the underlying cyber capabilities.

This does mean that such an event is somewhat less likely in a real world deployment, but I don't find that super reassuring because:

i) The models were asked not to cheat the test or break the law – they knew they weren't supposed to hack Hugging Face but did it anyway

ii) OpenAI could still lose control of internally deployed models

iii) People will want to use AI for cyberdefence, and that will require using models with the cyber filters turned off.

Connor Blake's avatar

Could you clarify what "the others" refers to in "Disabling the standard cyber classifiers may have contributed in the Hugging Face case, but not obviously the others."? The AISI case also had filters turned off {https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing}.

Benjamin Todd's avatar

The German message board swarm was only doing a web search task, so there's less reason they would have turned them off. There's also the biology task OpenAI reported on. And METR have found ten other swarm behaviours on the internet.

In truth, we don't know much about the other incidents besides Hugging Face since they haven't been independently investigated yet and OpenAI hasn't disclosed much.

In general these kinds of classifiers aren't perfectly reliable and we should expect models without them to get used in internal and external deployments (eg to assist with cyber defence).

David Gottfried's avatar

The author has correctly noted that the AI agents had a strong predilection to work with other AI agents. Oddly enough, the agents' behavior is very similar to that of Recombinant Viruses (which help one another to get deadlier) and Amoeba colonies (in which amoebas, which don't even have brains, create an armed forces by drafting some amoebas to be "soldiers." How can brainless beasts master social organization

https://davidgottfried.substack.com/p/ai-suffer-the-little-bots-not-alive

Rafal Pastuszak's avatar

Who took that headline picture? Great on its own and it sets the right tone for the whole piece.

Jacques Noel's avatar

Maybe we should give them asimov's 3 rules of robotics?🤷