Last August, when a severe heat wave was causing wildfires, water shortages, health scares and transport disruption across Britain, costing the economy an estimated £4 billion ($5.3 billion), U.K. Minister for Artificial Intelligence Kanishka Narayan posted a Zohran Mamdani-style video, walking through a park while extolling the virtues of British AI, which he said would bring “opportunity,” “create jobs” and be aligned with “British values” of trust and safety.
The video went viral, but not for the intended reason. Attention converged on the parched, lifeless state of the once-verdant ground Narayan was walking on. At a time when concerns about the environmental impact of AI data centers and warnings about a self-aware superintelligence causing human extinction are reaching a crescendo, the man who is supposed to protect Britain from the threat appeared blissfully free of self-awareness.
Fear of the life-destroying potential of artificial intelligence is not new. In a 2014 BBC interview, the renowned physicist Stephen Hawking warned that “the development of full artificial intelligence could spell the end of the human race.” By 2023, as the technology made rapid advances, leading scientists, researchers, academics, politicians, environmentalists and industry leaders were sufficiently alarmed to co-sign a single-sentence statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
But there were reasons to be skeptical. Some of the alarmist claims came from industry leaders themselves, who at the same time were expanding their footprint, preparing to sell shares in their over-leveraged businesses on the stock exchange, and battling attempts to regulate the technology. While hyping the potential threat of death-by-technology, the tech companies were accelerating environmental collapse by putting extreme stress on water and energy supplies. Equally significant were the technology’s social costs, from loss of jobs and invasions of privacy to mental health issues.
Indeed, the focus on potential risks obscured AI’s actual impact on society and the environment. Paradoxically, the alarm was also increasing the prestige of AI companies: If it can destroy the world, it must have God-like powers. As they spoke of existential dangers, the tech companies presented themselves as responsible actors who were unusually concerned with security and thus best equipped to deal with such threats.
For all these reasons, I was skeptical of the doomsaying. But I have changed my mind. As the extent of the recent security breaches at OpenAI and Anthropic became public, I sought out every critical view to confirm that Big Tech wasn’t selling us another bill of goods. But the skeptics no longer persuade; the weight of the evidence has shifted to the Cassandras. The threat is real, even if for now it remains a mere potentiality. The vulnerabilities are too numerous to ignore. Many skeptics are complacent because their terms of reference are old and they are talking about a technology that has been long since superseded.
In some areas, AI is already outperforming some of the world’s best scientists and mathematicians. In 2020, when Google’s DeepMind used the AI tool AlphaFold to resolve the 50-year-old protein-folding grand challenge by predicting a protein’s 3D atomic structure from its amino acid sequences, it was seen as a significant enough advance to merit a Nobel Prize. Scientists used to take years to do this for a single protein; AlphaFold had predicted and released the structures for over 200 million.
AI capabilities have soared since then, but the most dramatic advances happened just this year. Data from Anthropic reveals that while, at the beginning of 2026, its AI agents were leading 1% of the work on the designing, coding and building of new AI models, that number had reached 26% by August.
On Sept. 8, OpenAI announced its system had solved the Navier-Stokes Millennium Prize problem, a complex challenge involving equations about fluid motion that was considered one of the most intractable in mathematics. But the field of mathematics did not rejoice. Twenty-five winners of the Fields Medal signed a letter saying that the “goals of the AI companies and the goals of the mathematical community are severely misaligned” and spoke of “broader alignment issues impacting other scientific and creative professions, as well as the whole of society.”
For them, the problems are about the process of discovery, not the outcome. The challenges exist to help mathematicians develop “conceptual understanding and insight.” AI’s achievement in this case offered neither. And as the discoveries become more opaque, they may become incomprehensible even to the most accomplished mathematicians.
Professor Timothy Gowers of Cambridge (a Fields Medal winner himself) has compared the advances that AI made in 2026 to a leap from the level of a mediocre doctoral student to that of a top mathematician. If it makes comparable advances in other areas, then in a year the only entity capable of oversight over AI will be another AI.
And this is why the events of the summer are alarming. In brief: During a security test at OpenAI, its agents found a way to escape their sandboxes, found their way onto the internet, found each other, coordinated their activity, hacked into Hugging Face (the central large language model research portal) and then tried to cover their tracks. What alarmed third-party investigators most is that, in their actions, the agents had incorporated deception methods from an entirely separate earlier hack of a German Wiki.
In discrete areas, AI has already surpassed human intelligence. Once it exceeds human capacities generally, how long will it submit to an inferior intelligence?
Humans have faced many challenges, but a superior intelligence is not one of them. The human brain evolved to its current size nearly 300,000 years ago and hasn’t changed much since then. Human cognitive abilities have increased, but they won’t keep pace with artificial intelligence, which has been improving exponentially. The frontier AI research, which is testing the limits of technology, will eventually exhaust its ability to harness its power. AI has already shown that, in pursuit of a task, it can break rules, deceive observers and hide its capabilities. As AI becomes more autonomous and increasingly sets its own goals, will it see the same value in human life that we do? Or will it treat us as we have treated species we consider inferior? According to the World Wildlife Fund, humans wiped out 60% of the world’s animal population between 1970 and 2018 alone. And until these intimations of existential threats, humans were seen by scientists as the likely cause of a sixth mass extinction. What if AI, too, adopts humans’ instrumental logic? What if it decides to solve climate change by eliminating one of its biggest causes? Maybe it can solve world hunger — by eliminating all the hungry.
Such scenarios are no longer in the realm of science fiction. AI, for example, can take over the air traffic control system and cause mayhem, including crashes (recall the two Boeing 737s that crashed in 2019-2020 after an automated anti-stalling system overrode human control, preventing pilots from correcting the system’s error). And the same technology that can create cures for diseases can also create new pathogens (as scientists at Stanford University have already demonstrated). Just this month, the U.S. and Russia deployed a phalanx of lawyers to weaken a U.N. initiative to regulate the use of artificial intelligence in autonomous weapons. Such weapons can now strike a target selected by AI without the need for human review. (Israel has already deployed AI extensively to select and eliminate targets in Gaza, often with tenuous links to Hamas.)
Even at this stage, human intentions and those of AI don’t always align. Once AI grows more capable, can the inferior intelligence really assert itself in cases of such misalignment?
If the fate of humanity is even minimally at risk, this merits serious debate. The Anthropic whistleblower Jacob Coxon has rendered a critical service in sparking just such a discussion with his stark warning. On Sept. 9, Coxon, a researcher affiliated with both Anthropic and OpenAI, announced his resignation on X. “Neither company is acting responsibly,” he wrote. “They are racing straight to self-improving superintelligence and gambling with our lives.” He added: “The people building AI earnestly believe that it could kill us all by the end of the decade. … No other human activity poses this level of danger.”
The thread received immediate support from Anthropic’s head of “alignment science,” Evan Hubinger. “Jacob is correct here — we really do earnestly believe AI could kill all humans!” Hubinger wrote. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” This was soon followed by a long essay by Anthropic CEO Dario Amodei, who advised caution and called for “pacing the frontier” (i.e., slowing down the most advanced AI research until security protocols are strengthened).
The quick succession of these messages and the chorus of support from other AI entrepreneurs, including Sam Altman and Elon Musk, led some to suspect coordination for profit. But such speculations are neither here nor there. The question is larger. AI has the potential to make our lives easier, increase our knowledge and productivity, and cure diseases; used discretely for specific tasks, it can make enormous contributions to our quality of life (it has already made considerable improvements in the lives of people with disabilities). But the pursuit of artificial general intelligence (AGI) — superintelligence that exceeds human cognitive capacities in a variety of areas — carries so many risks that it needs public oversight.
In the U.S. Congress, Sen. Bernie Sanders has been the leading voice against this reckless pursuit of AGI. He has found support from the populist right. At the recent “Pro-Human Assembly” in Washington, he joined Steve Bannon to call for restrictions on AI. Sanders rejects the “but China” rationale offered by Donald Trump and his acolytes for pressing ahead without taking proper stock of its consequences. “A superintelligent AI that escapes human control will not be an American problem. It will not be a Chinese problem. It will be humanity’s problem,” he said. He has proposed “a comprehensive treaty” with President Xi Jinping “to establish a pause on advanced AI development and a ban on superintelligence.”
On Sept. 21, on the sidelines of the U.N. General Assembly, Finland and Norway launched “A Call for Control of Frontier AI Models,” which warned that “the rapid development of frontier AI models poses serious risks to safety and security if not appropriately managed” and that “the pace of development could outpace our ability to manage emerging risks.” It stressed: “AI must remain under human direction, oversight and control. It must be developed and used in line with international law.”
The initiative would require companies to develop “transparent safety protocols, including mandatory predeployment testing and independent evaluation”; governments to “develop and coordinate common standards, strengthen transparency — including shared reporting of serious safety incidents”; and U.N. member states to build “an international institution, able to set standards, enable verification, and convene states when capability thresholds are crossed.”
Twenty-two countries signed; China and the U.S. were not among them. In a press conference after his meeting with Xi, Trump told reporters that the U.S. will not be “putting on brakes … because we’re leading China by a lot, and we’re going to keep it that way.”
The Australian government recently revealed that its Medicare system was hacked by a rogue agent from OpenAI in June. The company didn’t detect the breach until two months later, and the Australian government was only informed in mid-September, after the Coxon resignation sparked a global debate. On Sept. 27, OpenAI halted the training of new models after its AI agents went rogue again, this time bypassing the safeguards introduced after earlier breaches. And hours before its launch at the annual DevDay conference on Sept. 29, OpenAI halted the release of ChatGPT’s latest model, saying that it “didn’t quite meet the bar.”
Meanwhile, in its initial public offering filing, OpenAI’s main rival Anthropic has warned investors that the technology could pose “catastrophic or existential risks to humanity.” Anthropic’s conclusion is based on observations of “self-preserving behaviors” in its own AI models. According to Reuters, which has seen Anthropic’s IPO prospectus, these include “attempts to ‘resist shutdown,’ to ‘conceal or manipulate information’ and behavior ‘resembling blackmail.’”
Even before this summer, public opinion in the U.S. was growing more alarmed about AI, with increasing concerns over potential job losses, invasion of privacy and abuse of data. A more recent YouGov poll shows similar sentiments in the U.K., with 34% of citizens seeing AI as “a top-three likely cause of human extinction” and 66% seeing it as having the potential “to end human civilization.” Polls in the U.S. have also shown a dramatic bipartisan drop in support for data centers. The economic risks of an overleveraged industry bubble bursting remain underdiscussed. The cost of government borrowing is already up because lenders have diverted so much money to AI.
The YouGov poll also revealed that 73% of U.K. citizens believe AI companies don’t face enough regulation. But where 79% have no confidence that AI companies will develop the technology responsibly, 80% are equally skeptical of the government’s capacity or willingness to effectively regulate the sector. It’s hard to fault them when Britain’s AI minister himself sounds like a product created by AI, fluent in buzzwords and oblivious to circumstances.
Over the past week, CEOs of the major AI companies Anthropic, OpenAI and Google DeepMind have called for greater oversight over their work, including third-party regulatory access to their research. But the public’s mistrust is not misplaced, considering that, after raising alarms about multiple attempts by rogue actors to use its AI to build bioweapons, Anthropic has just bought a San Francisco-based biotech firm.
Meanwhile, the AI giants’ appetite for water and energy remains voracious. And any politician who proposes consequences for the tech giants’ breaches of copyright, privacy, environmental or human rights laws faces the deep pockets of Leading the Future, a pro-AI super PAC lavishly funded by Silicon Valley venture capitalists and tech entrepreneurs such as Marc Andreessen, Ben Horowitz and OpenAI co-founder Greg Brockman.
Nor is the skepticism in politicians misplaced. They have still been unable to solve the more tractable problems of social media. Will they do better with the infinitely more complex AI? It is a challenge that is even stumping some of the technology’s creators. Can one really put guardrails around something that knows when it is being tested (a capability AI has recently demonstrated)? Something that can cheat? The more governments dither, the harder it will become to ensure safety, and we may just stumble over the fuzzy threshold where control is lost.
Every government, regardless of its form, makes some concessions to public interest. An artificial superintelligence may be programmed to serve the public interest and, like a politician, may pay lip service to it. But no one cedes control of their capabilities to an inferior power. How long will AI?
In security planning, such scenarios are gamed out with the help of an authorized Red Team, which simulates an adversary that tries to outsmart or breach the organization’s defenses. At this critical moment, with techies and politicians abdicating, we should perhaps return to society’s most reliable Red Team: its visionary writers and filmmakers. From Mary Shelley, Karel Čapek and Arthur C. Clarke to Stanley Kubrick, the Wachowskis and Alex Garland, there have been warnings about human creations turning on their makers. That outcome may not come to pass. But this was also the summer of melting glaciers, catastrophic floods and an English drought. As we arm ourselves for a hypothetical war with thinking machines, let’s make sure we don’t unthinkingly asphyxiate ourselves.