Anthropic calls for an AI pause over safety concerns

Leading AI lab makes a public plea for a pause in the development of frontier AI

Misalignment between humans and AI could lead us to “lose control”.

“It would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.”

These are the kinds of warnings you would expect to hear from AI safety advocates like PauseAI, not the leading AI company, Anthropic.

Anthropic fears losing control of AI

In a statement released last week the lab said, “The rare occurrences of misalignment present in today’s models could compound as the models build their successors, growing more frequent but less understood until we lose control of them.”

Alignment refers to the challenge of ensuring AI systems act in accordance with human interests.

Maxime Fournes, CEO of PauseAI, said, “The AI lab that has developed the most-powerful AI model – and is working to develop even more powerful models – is worried that they might lose control and is now calling for a pause. What does this tell us? That AI is safe? That we should go full steam ahead?”

“This is a cry for help. Surely governments and governing bodies can see this and will now begin to take the catastrophic risk of AI seriously.”

Anthropic predicts that AI improvements will continue to compound and “could also be turned to harmful ends, from authoritarian surveillance of whole populations to influence operations that tailor manipulation to each individual and run at a scale no human team could match.”

Anthropic is not alone in its concern. Professor Stuart Russell, the author of the standard textbook used in over 1,500 universities worldwide, told MEPs in February that “eight out of ten top AI researchers are convinced that the creation of artificial general intelligence will lead to a loss of control.”

Humans may soon be obsolete

The International Monetary Fund (IMF) predicts that 60 percent of roles in advanced economies will be replaced by AI.

Now Anthropic is warning that in the very near future the role of AI could “fully eclipse those of humans … it is difficult to predict what the economy looks like if human labor stops being competitive.”

The statement details how the company is becoming more and more reliant on AI to write code and perform other tasks, while “the human role is narrowing at each step in the AI development process.”

The statement includes testimonies from Anthropic researchers. One said, “I started leaning hard into Claudifying about a year ago. That’s been a crazy adventure and it’s now been 5 months since I last wrote any code myself.”

And AI is only becoming more capable. The Anthropic statement points out that “the rate at which AI models improve is accelerating rapidly. The length of tasks that they can reliably complete on their own has been doubling roughly every four months, up from an earlier trend of doubling every seven months.”

According to Fournes, “The AI companies are marching us all towards a human-free workplace; this will have a devastating impact on societies, livelihoods and the economy.”

A pause is entirely possible

In the statement Anthropic suggested how they think a pause might happen:

“A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions. It would also require that each can verify that the others have actually stopped.

“None of this is necessarily impossible in principle—the world has built verification regimes for other complex technologies … but those regimes took decades to build both the infrastructure and the trust. We don’t have that long.”

Just last week Donald Trump signed an executive order to create a voluntary framework for the US government to vet powerful new AI models before they are released.

Fournes said, “This is a positive sign but it needs to go much further – especially with companies themselves now calling for a pause. The US can’t do this alone: a global treaty is required that includes not only the US and China but all technologically-advanced nations and multilateral organisations.”

An international agreement is typically established through a summit, where leaders of countries meet to discuss the issue and make a decision. We have already had four AI summits: the first three - held in the UK, South Korea and the US respectively - were called AI safety summits. The most recent, held in India, dropped safety from its name and focused more on commercial interests than AI risk.

“We need a return to AI safety summits with the objective being a global treaty,” said Fournes.

“Anthropic have said it themselves: we don’t have a lot of time. Superintelligent AI could be just around the corner and once it’s here we will have lost the chance to keep it in check.”

The UN’s AI For Good summit has just kicked off in Switzerland - this represents another chance to make AI safety a real priority.