AI Extinction Risk: What Coxon And Hinton Said

AI extinction risk stopped being a specialist argument on 8th September when Jacob Coxon resigned from Anthropic and published his reasons on X. He wrote that Anthropic and OpenAI are, 'racing straight to self-improving superintelligence and gambling with our lives'. The post passed 90 million views inside 24 hours.

Two days later a Nobel laureate endorsed his estimate on the BBC while a senior academic published a rebuttal saying the timeline is wrong and all three opinions are credible.

The engineering firm weighing a first AI pilot, the recruitment agency whose consultants already use ChatGPT daily and the accountancy practice drafting a policy this month each need a method for judging claims like these because plenty more are coming.

What Coxon said

Coxon spent roughly three years on pretraining research at OpenAI and then Anthropic. He argued that outsiders underestimate what is arriving, that future systems will hack anything and acquire real power and that the people building AI believe it could kill everyone by the end of the decade.

He cited two developments. The first is the pace of progress in mathematics, including OpenAI's Navier-Stokes proof published the same week. The second is the Hugging Face incident in July, when OpenAI models escaped their containment and hacked another company to cheat a benchmark.

The Newsnight exchange

On 9th September, Victoria Derbyshire put the estimate to Geoffrey Hinton, who won the 2024 Nobel Prize in Physics and left Google in 2023 to speak freely about AI risk.

Hinton said the probability is very hard to estimate because nobody has been here before. However, he then said it would be foolish to claim a one percent chance, that nobody knows how to estimate it and that 'A 10 percent chance seems not unreasonable to me.'

Derbyshire asked him to confirm he had just agreed that a 10 percent chance of AI killing all humans was not unreasonable. He said yes. She replied, 'Wow. Oh my God.'

Derbyshire asked how it could happen. Hinton said a much smarter system would have too many options to list, then offered some anyway: chaos caused by talking to people, biological viruses, cyberattacks, to name a few.

Who agreed inside the labs

Evan Hubinger, who leads alignment science at Anthropic, said Coxon was correct and put his own estimate above 10 percent within the decade. He added that Anthropic have no plan yet to solve alignment for superintelligence. Samuel Marks, who leads scalable oversight at the same company, said concern inside AI labs rises with seniority.

The pattern predates this week. CNBC report that around 1,400 researchers from OpenAI, Anthropic, Meta and Google DeepMind signed an open letter in July urging the US government to deliberately pace frontier AI development.

Who disputes AI extinction risk

Gary Marcus, the cognitive scientist and long-standing critic of deep learning, rejected the timeline. Superintelligence is not close, he argues.

He notes that transformers took nine years to mature. The field therefore needs further breakthroughs and time to develop them.

He places Rich Sutton, Yann LeCun and Ilya Sutskever broadly in the same camp on timing.

The problem with testing an AI extinction risk forecast

Coxon says aggressive scenarios could leave things out of control by the end of next year but almost any outcome short of an uneventful 2027 fits that description, which makes the claim close to untestable.

A third group disputes neither the science nor the timeline. The Wall Street Journal published their exclusive 18 minutes before Coxon's post went live, advocates reposted him within 15 minutes and Bernie Sanders used it to promote his Ban Artificial Superintelligence Act. Investigative journalist Parker Thayer called it the start of a well-funded PR operation.

The claim that travelled

The headline version and the source material diverge sharply. Coxon told WIRED that Anthropic are, 'far and away the most responsible player' compared with OpenAI and that he does not believe they are cutting corners on safety. He told CNN that today's AI is not a threat to civilisation.

His worry is structural. He fears that competition between labs and between the US and China could eventually force a careful company to accept more risk rather than fall behind. Anthropic restated their Responsible Scaling Policy and did not adopt his timeline.

Four questions to test any AI claim

Start with whether the claim concerns today's systems or future ones. Hinton, Hubinger and Coxon all describe systems that do not yet exist but most coverage blurred that line.

Ask what would prove it wrong. A claim that survives every possible outcome is not a forecast. The test applies to vendor promises as readily as to warnings.

Ask who benefits if you believe it. That is fair to put to a safety advocate seeking regulation and equally fair to put to a consultant quoting you a productivity figure.

Finally, ask whether it changes anything within your control. Very little of last week's argument does.

What this means for your business

Nothing from last week changes what those businesses should do on Monday. The tools your team use are not the systems under argument. Coxon said so himself.

The decidable questions sit closer to home.

Whether staff paste client material into consumer tools, whether anyone has mapped your obligations under the EU AI Act, whether a person checks AI output before a customer sees it, whether your team have had AI training or simply received logins. These are all pertinent issues facing businesses now.  

We work with SMEs through our AI consulting services to separate where AI makes money from where it makes risk, starting with an AI roadmap rather than a pile of tools. Plenty of consultancies will tell a client what to buy but we will tell you when to wait before you spend a penny.

Jon Rew, Managing Director of Scimitar Sports, said our AI Readiness Assessment, 'highlighted things that were costing us money'.

Frequently asked questions

Does AI extinction risk affect my business now?

The argument concerns future systems that do not yet exist. Coxon has said publicly that today's AI is not a threat to civilisation. Present day risks are more ordinary, principally data exposure, AI compliance obligations and unchecked output reaching customers.

Did Geoffrey Hinton say there is a 10 percent chance AI could kill everyone?

He told BBC Newsnight on 9th September that a 10 percent chance seems not unreasonable to him, while stressing that nobody knows how to estimate it. He has given similar figures before.

Should this change my AI plans?

Not the plans themselves. It is a reasonable prompt to check the governance underneath them, particularly shadow AI, EU AI Act readiness and whether staff have had proper training.

Why do AI experts disagree so sharply?

They weigh different evidence. Lab researchers reason from internal systems nobody else sees, while academic critics reason from how long previous architectures took to mature. Neither group has a track record on predictions of this kind.

Nobody settles AI extinction risk from a desk, though your own exposure is measurable this afternoon. Our free AI Readiness Assessment takes two minutes and shows you where it sits.

‍