Anthropic researcher who quit says AI staff fear for humanity

Jacob Coxon, a 27-year-old researcher who resigned from the artificial intelligence company Anthropic, told the BBC that people working on AI are “genuinely frightened” about the speed of its advances and what they could mean for humanity. He said progress should be slowed in coordination with China.

What did the former Anthropic researcher say?

Coxon, who worked at OpenAI before joining Anthropic, spoke to the BBC’s Sunday with Laura Kuenssberg after his resignation post on the dangers of AI spread widely. “I believe that if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future,” he said.

He said a slowdown would need to be co-ordinated with China to avoid “a race at an international scale”. Coxon said one risk outlined by Dario Amodei — a swarm of bots acting like a supercomputer that could take over the internet — could be realistic within six months to a year, and that his peers feared the danger could arrive within two years.

Coxon said staff at AI companies were “planning what to do with their lives and thinking about the impacts of their work”, and that some were “considering buying land somewhere because they’re so scared of the instability as a result of rapid AI progress”.

How has Anthropic responded?

An Anthropic spokesperson told BBC News: “We have always been transparent that AI will bring both enormous benefits and unprecedented risks. To address these risks, we continue to build models with some of the strongest safeguards in the industry.” The company said it was the first to publish a framework for mitigating the risks posed by model development. It has also said it blocked AI use for biological weapons research.

What has Anthropic’s chief executive said?

Anthropic head Dario Amodei recently called for AI development to slow down, and wrote in an essay on Saturday that the risks associated with the technology were “serious” and that companies and governments must be given time to address them. Sam Altman of OpenAI and Elon Musk of xAI have both said they agree with Amodei’s proposal for industry-wide deceleration and regulation, as well as independent monitoring of AI model development.

Who else has warned about AI risk?

Anthropic scientist Evan Hubinger said: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” Geoffrey Hinton, a computer scientist and Nobel Prize winner, told the BBC on Friday that a 10% chance of AI killing all humans was “not unreasonable”. Former UK prime minister Rishi Sunak, a paid adviser to Anthropic, wrote in the Sunday Times that he was also worried about the risk of AI to humanity.

Who disputes the warnings?

Clement Delangue, chief executive of the AI platform Hugging Face, said: “Sorry, but asking Jacob [Coxon] about AI extinction risk is like asking your AC guy about climate change.” Grindr chief executive George Arison told the BBC that the statements from Anthropic and others were made to drum up investor support, saying “the only way to justify these valuations” was to claim an AI would take over every industry and every job.

Frequently asked questions

Who is Jacob Coxon?

Jacob Coxon is a 27-year-old AI researcher who resigned from Anthropic and previously worked at OpenAI. He told the BBC that people working on AI are “genuinely frightened” about the pace of its advances.

What did Anthropic say about his resignation?

An Anthropic spokesperson told BBC News the company has always been transparent that AI will bring both enormous benefits and unprecedented risks, and said it continues to build models with some of the strongest safeguards in the industry.

Which AI leaders have called for slowing development?

Anthropic head Dario Amodei called for AI development to slow down, and OpenAI’s Sam Altman and xAI’s Elon Musk both said they agree with his proposal for industry-wide deceleration, regulation and independent monitoring of AI model development.

Who has questioned the warnings about AI extinction risk?

Hugging Face chief executive Clement Delangue compared asking Jacob Coxon about AI extinction risk to asking an air-conditioning technician about climate change. Grindr chief executive George Arison told the BBC the statements were made to drum up investor support.

With reporting from BBC News.

Stock image, not from the events described. Atlas Continent — World Maps / Wikimedia Commons (CC0 1.0)