The Anthropic saga exposes AI's regulatory black hole

Anthropic has developed an AI that is, in its own CEO's words, "incompatible with democratic values" and would put "civilians at risk." So why was this company allowed to build it in the first place?

Last week two stories about Anthropic broke simultaneously. Together they reveal the fundamental flaw in how the world governs – or rather, fails to govern – artificial intelligence.

The first story made headlines. The Pentagon demanded that Anthropic remove two contractual red lines from its military contract: no mass domestic surveillance and no fully autonomous weapons without a human pulling the trigger. Anthropic refused. The Pentagon threatened to invoke the Defense Production Act – wartime powers to commandeer private companies – or label Anthropic a “supply chain risk,” a designation normally reserved for foreign adversaries like Huawei. Anthropic CEO Dario Amodei publicly stated he would rather lose the contract than comply.

The same week, TIME reported that Anthropic had scrapped its flagship safety pledge – a commitment to never train an AI system unless its safety measures were guaranteed in advance. Anthropic co-founder Jared Kaplan admitted: “We didn’t really feel, with the rapid advance of AI, that it made sense for us to make unilateral commitments… if competitors are blazing ahead.”

The real story: Anthropic drops safety pledge

Anthropic deserves credit for holding its two red lines amidst pressure from the Pentagon.

But ensuring that your technology does not cause harm or undermine democracy should be the bare minimum we expect of AI companies. Mass domestic surveillance and fully autonomous weapons are already restricted under existing US law.

Meanwhile, the safety commitment Anthropic quietly abandoned – the promise not to train an AI model unless safety mitigations are guaranteed – was the one that might actually have slowed the race to build increasingly dangerous AI systems.

It is worth noting that in February, Anthropic raised $30 billion in new investments, elevating the company’s estimated value to $380 billion. The endeavour to please investors, it would seem, is winning out over safety.

Maintaining two red lines while dropping the commitment that would actually slow the race to superintelligence is not enough. Anthropic is still rushing to build the machine that the majority of experts say would threaten human existence.

A government wouldn’t actually deploy mass surveillance and autonomous weapons, would it?

A common dismissal of AI risk goes something like this: “No one would actually use AI for mass surveillance or autonomous killing. You’re catastrophizing.”

This week suggested otherwise. The Pentagon openly demanded unfettered access to AI with zero guardrails. A senior defense official said the goal was to “make them pay a price” for even asking questions about how their technology would be used.

If it materialises, the Defense Production Act threat would amount to the quasi-nationalization of an AI lab: Anthropic would be forced to build military AI without any safeguards.

This is still hypothetical but for how much longer?

Others rushed to fill the Anthropic-shaped hole

OpenAI’s deal with the Pentagon to supply AI to classified US military networks seems to be essentially the same deal that Anthropic had refused. Reporting suggests that OpenAI, along with Google and xAI, have signed military deals with “minimal safeguards.”

The backlash was immediate and severe. Demonstrators lined the streets outside OpenAI’s offices in California, chalking pavements with slogans calling on employees to resign. Some did. A petition to boycott ChatGPT has already gathered over 2.5 million signatures as users are abandoning the platform in protest.

Only after this backlash did Sam Altman announce that OpenAI would amend the deal to ensure its systems are not “intentionally used for domestic surveillance of U.S. persons and nationals.” Such a clause, it would appear, had not been included in the original agreement.

If the Pentagon punishes Anthropic for maintaining its red lines while rewarding OpenAI for its compliance, the message to every AI company is clear: hand over the keys and never ask questions.

None of the other labs have spoken up in Anthropic’s defence, despite the obvious precedent being set. Their silence is an implicit acceptance that they will provide AI for any use the state demands. It is an explicit admission of their disregard for human safety and democratic freedom.

Voluntary commitments are worthless

Anthropic once had the strongest voluntary safety framework in the industry.

If the supposedly most safety-conscious lab cannot keep its promises, no lab can. This is not just a failure of Anthropic’s character. It is a failure of the system. Private companies do not and will not unilaterally resist the pull of market competition and the push of government coercion. That is what binding regulation is for.

The regulatory black hole

Anthropic has developed a technology that is, in Amodei’s own words, “incompatible with democratic values” and would put “civilians at risk.” This technology exists today. Why are these companies given free rein to develop AI systems that they themselves consider unsafe? And remember: what AI companies are racing to build next will be many times more powerful, with far greater potential for destruction.

In a statement, Amodei said the “purchase [of] detailed records of Americans’ movements, web browsing, and associations from public sources without obtaining a warrant… is currently legal.” The only reason, he said, is that “the law has not yet caught up with the rapidly growing capabilities of AI.”

Just last week, Stuart Russell, professor of Computer Science at UC Berkeley, addressed Members of the European Parliament during a conference organized by PauseAI: “If AI companies succeed in building a superintelligence, most experts think the chance of human extinction is somewhere between 10 and 50 percent: that’s the equivalent of playing Russian roulette with everyone on the planet.”

Experts think the chance of human extinction could be as high as 50 percent. Yet there is no regulation governing the technologies that Anthropic and its competitors are building.

Neither corporations nor governments can be trusted

The lesson is not “trust Anthropic.” It is that neither corporations nor governments can be trusted as sole stewards of this technology.

In the long term, a global treaty — much like the Treaty on the Non-Proliferation of Nuclear Weapons — is likely the only framework that would reduce the catastrophic risk posed by AI. Multilateral discussions have been proposed by China, but this offer has yet to be accepted.

The International Dialogues on AI Safety (IDAIS) brings together leading scientists from around the world, including from China, to collaborate on mitigating AI risks. But an international governance framework still seems a long way off at a time when it is more urgent than ever.

Binding democratic oversight — national regulation and international treaty frameworks — cannot come soon enough.

The race to the bottom is no longer theoretical. It is front-page news.

Photo credit: Michaël Trazzi