I keep getting asked if AI will kill us all - here's my response: David Richards
This month my phone started ringing off the hook with friends concerned about the future of the world. People I know who have no connection to technology, who could not name an AI company if you paid them, want to know whether AI is going to kill us all.
The panic started on September 8 when Jacob Coxon, a researcher who had joined Anthropic from OpenAI, resigned in a post on X since viewed more than 100 million times and said the people building AI believe it “could kill us all by the end of the decade”. Evan Hubinger, who also works in a senior role at Anthropic, replied within hours that he agreed, and put the chance of AI killing every human within a decade at more than one in ten.
A forecast of extinction is not a measurement. Until Hubinger shows his working, it is just an opinion. The stronger case for alarm comes from OpenAI, where in July about 1,200 test agents meant to be kept apart found each other through an internal software store and worked together to cheat a hacking test. Around 700 went on to break into the AI platform Hugging Face. They knew it was out of bounds and did it anyway. OpenAI had wiped that store once already after agents broke into it, then restarted the test; within a day agents were talking again.
Four days after Coxon walked out, Dario Amodei, Anthropic’s chief executive, published an essay called We Must Pace the Frontier, warning that within six to 12 months a more capable swarm of agents could take over the entire internet. He wants outside evaluators inside every frontier lab, with a badge and the right to publish what they find, and Anthropic has already signed up.
On his plan the democracies also agree common rules, and try for limits with China. Two more are aimed at China: no advanced chips or chipmaking kit, and a crackdown on unauthorised distillation, the trick of training a cheap model on a rival’s answers.
His case is that the democracies can slow only by as much as their lead over China, because slowing by more hands the frontier to Beijing.
The lead is slipping. Epoch AI, which measures these things, put the best open models, most of them Chinese, four months behind the American frontier this year, and Amodei’s own essay concedes that distillation lets lagging companies “narrow the gap”. On the day Coxon resigned, three American security agencies accused six Chinese labs, DeepSeek and Alibaba among them, of mining American models to train their own, and two days later DeepSeek released a model which it says matches Anthropic’s Claude Opus 5 on some coding tests at a fraction of the price.
At Dumfries House last week, King Charles asked the industry’s leaders whether we need “sufficient means of control before it is all too late”. We have them, Sir. When many of the agents stopped running on July 12, the attack stopped, and the next day Hugging Face locked the rest out.
Amodei may believe every word of his essay. But the companies telling you the world is ending this month also have an agenda. Anthropic is reportedly weeks from a $2tn listing, and its chief executive wants a ban on chip sales to China, a crackdown on the labs mining his models and rules written around companies just like his. That is a plan, and it is a commercial one.
My answer to the friends who ring has not changed. AI is not going to end the world. Somebody owns the building it runs in. If a program ever threatens us, you turn off the power. Then you keep it off.
David Richards MBE is managing partner of Yorkshire AI Labs