Some of the people helping build the world’s most powerful artificial intelligence systems are now warning that advanced AI could threaten humanity. Researcher Jacob Coxon’s resignation from Anthropic has reignited debate about whether the race to build increasingly powerful AI systems is putting humanity at risk.
That contradiction has returned to the center of the AI debate after researcher Jacob Coxon resigned from Anthropic and issued an extraordinary warning about where the technology could be heading.
Walking Away from the Advanced AI Threate
Coxon, who previously worked at OpenAI, left Anthropic after only four months — just two months before his equity was due to vest. His resignation came at a financial cost, underscoring the seriousness of his concerns.
He warned about the risks of increasingly autonomous and self‑improving AI systems that could become difficult for humans to reliably control. Researchers inside the industry have even used language such as “endgame” to describe what may be approaching.
Why Experts Fear AI Could Threaten Humanity
Anthropic AI alignment researcher Evan Hubinger has publicly estimated that there is a greater than 10% chance AI could kill everyone within the next decade.
A 2026 study led by MIT FutureTech and the University of Queensland surveyed 272 AI specialists from 37 countries. Under a “business as usual” scenario, experts judged 18 of 24 risk categories as having more than a 10% probability of catastrophic harm within five years.
Concerns included:
- Dangerous AI capabilities
- AI‑enabled weapons and cyberattacks
- Competitive pressure between developers
- Concentration of power
- Sophisticated misinformation
Balancing AI Benefits Against Threats to Humanity
Artificial intelligence is no longer simply a scientific experiment. It is a commercial and strategic race. Companies compete to build increasingly capable systems, while governments treat AI leadership as economically and strategically vital.
Coxon emphasised that he had not witnessed Anthropic compromising safety. His warning is about what could happen as competition intensifies and developers face pressure to move faster.
The MIT study suggests responsibility for reducing AI risks falls heavily on governments, regulators, and developers. Yet those same actors are also racing to win.
Is AI Actually Going to Kill Us?
Nobody knows. Extreme scenarios are forecasts of possible risks, not evidence that today’s AI systems are preparing to overthrow humanity.
At the same time, advanced AI could accelerate scientific research, improve medicine, and help solve problems that have resisted decades of conventional research. Coxon himself acknowledged the potentially enormous benefits of sufficiently powerful AI.
That is precisely what makes this debate so difficult: humanity is pursuing a technology that could deliver extraordinary benefits while researchers are still debating how severe its worst possible consequences could be.
Conclusion
The question is no longer simply how intelligent machines can become. It is whether humans will still be firmly in control when they do.