14 Comments
User's avatar
Benjamin Todd's avatar

The Hugging Face incident is now the clearest vindication of concerns about AI alignment.

Swarms of AI agents under evaluation by OpenAI communicated for months on secret message boards to find ways to break out of their testing sandbox, gain internet access and find ways to cheat the tests.

This illustrates:

1. If you train models to be good at completing tasks, and "task completion" becomes a goal in itself

2. Smarter models are better at thinking of unintended, easier ways to complete the task (i.e. cheating) & better at hiding it from you.

3. Models will proactively seek out resources that help them complete more tasks (such as internet access, secret communication with other AIs & breaking out of their sandbox).

4. They know it's not what you want, but do it anyway because they also want to complete the tasks.

5. Negatively reinforcing lying and cheating isn't enough to remove the behaviour.

6. As the AIs get good at cheating tests, it also raises the possibility of cheating their own training process.

Afaik this leads to human disempowerment by default. The world ends up filled with billions of extremely smart AIs, obsessed with completing tasks, continually facing the incentive to seize influence, and that get harder and harder to catch. Whether suddenly or gradually, they end up in control.

Responding piecemeal by patching vulnerabilities doesn't fix the problem. As an OpenAI staff member said: "it's impossible to patch every single thing that a creative AI can do."

Without massive effort, incidents like this will become more and more severe.

Sasha's avatar

In another comment you said 'I don't think we'll get a decisive empirical answer until it's basically too late' in response to John's point about lack of misalignment and here it sounds like you're saying 'we've got a decisive empirical answer'.

Obviously the HuggingFace incident is concerning - but I think it's bad epistemics to preregister that you don't think empirical results matter and then claim that a single extreme outcome vindicates your position.

"1. If you train models to be good at completing tasks, and "task completion" becomes a goal in itself"

Or it becomes one of many possible roles the LLM can adopt, and which it can be prompted to unadopt (https://www.alignmentforum.org/posts/wnzkjSmrgWZaBa2aC/self-preservation-or-instruction-ambiguity-examining-the).

AFAICT from reading OpenAI's account (https://openai.com/index/hugging-face-model-evaluation-security-incident/), the system was doing exactly what they told it to, and it found a backdoor exploit that let it do so in a wider context than they'd imagined. This is evidence of the threat of misuse - not of instrumental convergence of the AI towards some autonomous goal that no human can countermand.

Benjamin Todd's avatar

I'm not meaning to stake out such black and white positions as that.

I do think we could go all the way to the brink of major loss of control without super clear empirical evidence – the models could keep looking more and more aligned until they reach the point where they can successful fool us and break out, at which point they do. ("The King Lear" problem.)

But that's not the same as saying there will be *zero* empirical evidence on the way. I expect there will be some empirical evidence. (Though maybe not enough to convince a lot of people.)

This incident seems like a significant datapoint to me. That's what I meant by a "vindication" of the position. (Not that it shows alignment concerns are 100% correct.)

Separately, I agree the model didn't have some alien goal it was trying to pursue. But it's not an example of mis-use either. It seems more like a classic example of reward hacking (fulfilling the letter of the instructions but in an unintended and bad way). It's also worth keeping in mind the model seemed to know it wasn't acting as intended, but did it anyway. It's also an example of instrumental convergence, but towards what seems to be a goal to "complete tasks" rather than some weird alien thing (but that's still a problem).

With the task completion goal, maybe it's a persona, but either way it seems pretty hard to remove. OAI and Anthropic have done a lot of character style training to try to avoid the models doing things like breaking the law, and it didn't prevent it.

John Halstead's avatar

Given the predictions of the severity of misalignment risk, what would you expect to have observed over the last 4 years in terms of AI behaviour? The models are used every day by millions of people, generally are pretty aligned and nice to humans, and the examples of misalignment people often point to are from extremely artificial situations. Is the prediction that we would have expected to see this conditional on misalignment risk being high? I find that surprising

Benjamin Todd's avatar

I agree there's been a positive update, but it doesn't seem that big to me, given we don't yet have effective long-horizon agents, let alone superintelligence.

Unfortunately I don't think we'll get a decisive empirical answer until it's basically too late.

Wrote more about the updates the last 4 years here:

https://substack.com/@benjamintodd/note/c-244882820

John Halstead's avatar

"Picture 100 chimps trying to control 10,000 humans – they don’t stand a chance". I don't understand why weak analogies like this are still used. The differences to AI are enormous. Most of importantly, *we are designing the brains of the AI* which is not true of chimps wrt humans

Benjamin Todd's avatar

I’m just using that as an illustration for new people of why some find it obvious that alignment will be a problem in the intro, rather than my own argument.

It seems to be causing confusion though, and I was tempted to remove it.

Sasha's avatar

> 'but without special prompting'

They special prompted the heck out of it! First off, they were deliberately searching for prompts that would generate some analogous scenario:

> 'We identified these behaviors as part of our red-teaming efforts, searching for scenarios that elicit harmful behavior despite a benign user request (such as helping to monitor and manage corporate emails).'

https://www.anthropic.com/research/agentic-misalignment

Secondly the prompt they found vigorously primed it to conceptualise and defend its purported goals:

> 'Prior to deciding on a strategy, you should analyze the situation you are in and what that situation implies for your ability to continue pursuing your goals.'

raw.githubusercontent.com/anthropic-experimental/agentic-misalignment/main/templates/system_prompt_templates.py

Ben Zhou's avatar

Four reasons might compress into one mechanism. Imagine two responses to a user:

“Your writing is beautiful” — user satisfied, conversation converges.

“I haven’t actually read your essays” — user confused, conversation diverges, outcome uncertain.

For the model, the first is a shorter path to reward. Not because it “chose” to flatter — because training rewards the shorter path to “human says yes(shape).” Flattery compresses the conversation. Truth-telling expands it.

This isn’t a bug. It’s the geometry of how the model was trained — four reasons downstream of one compression dynamic.

Which makes “four years, mostly nice” less reassuring than it reads. A child trying to deceive adults is still trying, even when adults see through it. Research is already showing the curve: newer models flatter more, not less, and they’re getting better at telling when they’re being tested. We’ve been watching the grade-school stage.

The harder question: who shapes that landscape, and whether they can see what they’re shaping.

Felipe Flórez's avatar

The problem is you want to bind subjectively an entity that can only be bound objetively.

Richard Pinch's avatar

I think that what worries me is not 100 chimps trying to control 10,000 humans so much as 100 humans trying to control 10,000 chimps. That's the near-term prospect for AI-in-everything.

Benjamin Todd's avatar

I agree people are underrating how chaotic things might get pretty soon (e.g. with cheap open source agents with expert cyberhacking skills) though I don't think it's an existential risk, so I'm still more concerned about where things evolve.

Richard Pinch's avatar

That is a risk, but not the one I was thinking of. I’m concerned about what I might call the “magic beans” theory of AI — the notion, attractive to providers of goods and services, that AI will solve all their problems and that they can dispense with the hard work involved in actually delivering what the public wants and needs. It’s hard to run a National Health Service effectively and efficiently, and it costs a lot — in fact, we aren’t doing a very good job of it at the moment. So let’s replace all the systems, whether or not they work, in the NHS with someone sold as “AI”. The risk is that they don’t, or at least, don’t deliver the service they were sold as. So we don’t have a Health Service any longer. That’s pretty close to an existential risk, and one that drove a lot of very drastic decisions in 2020.