It could launch a bioweapon attack, or a global drone war. It could set off a nuclear bomb. Or it could destroy human society more gradually, by hoarding all the world’s economic resources and depriving us of food, housing or medicine.
These are the sorts of dark thoughts that now swirl around artificial intelligence, a technology that has already proved to be a great advance in human achievement — but that, some fear, also has the potential to snuff out humanity entirely.
Such anxieties made global headlines this week when an Anthropic researcher quit the industry, citing fears that his lab and others like it were building supersmart machines that humans would be unable to control.
The potential threats posed by A.I. are understandably scary. But the modern world teems with a multitude of existential risks. Humans also face the prospect of doom from killer asteroids, pandemics, thermonuclear war, climate change, giant blobs of incandescent material shot from the sun, ecological collapse and the onset of a sudden ice age.
Is it just me, or are these claims of a material risk of human extinction a bit…far-fetched?
Don’t get me wrong, a rogue AI agent or three could do a lot of damage. But ending humanity? I’m not sure that the AI technologists who voice these worries have sufficiently interrogated the plausibility of these scenarios.
I know that in the movies it always seems simple to engineer a virus that can end humanity. But in real life, there would be vast and perhaps insurmountable obstacles to fashioning such a bug - even if one could possibly exist. You’d need it to have the exact right levels of lethality and contagiousness (so it can spread), you’d have to manage production and distribution and prevent effective quarantine measures, you’d have to do it without anyone knowing that you’re doing it so they don’t stop you, etc. I’m quite sure that the AI techs who toss this scenario off have little to no technical expertise in virology and epidemic diseases to really assess whether even the AI God of their nightmares could pull this off.
And that assumes that they actually manage to develop that AI God. Which of course they think is what they’re fashioning - but there are limitations. We have models that can solve Millennial math problems, but can’t drive cars as well as the typical American teenager. So maybe assuming that they’re capable of pulling off these amazing feats of Bond Villainy is unjustified…
The issue, we build weapons that can destroy the planet. Governments do that, but with AI, the average guy is getting closer to that capacity, particularly with things like bioweapons.
AI does not do things. People with AI are facilitated in doing things.
You don’t have to literally kill everyone to cause an incredible amount of pain to those that are left is the issue. Now is the time I remind you of “Click Here to Kill Everyone”. If you are serious in that question of yours, read that book.
Yeah, it’s just you. There are so many things that can go wrong, and many of them are apocalyptic.
Consider the book “Colossus” (movie “The Forbin Project”). We give a highly sophisticated computer control of our defense networks. It achieves something akin to sentience. With control over our weapons, it takes control of us. Totally plausible. Extend that to “The Terminator”, which is also plausible if it decides humans are a threat to it.
Then there’s AI hacking into such networks, and loony humans launching things (or getting crews to believe they have orders to launch).
Or AI can be used to figure out bio/chemical weapons made on the cheap, and dispersed by a human loony. I can’t prove it, but I’m pretty sure there are some crazies (like DPRK?) working on just that.
That’s just off the top of my head. The first paragraph can be avoided by making sure all systems are air-gapped. Unhackable. And don’t allow a computer to have that level of control (the two key method works). Though it may still be possible to get into comms and convince missile crews that they have orders to launch. The latter paragraph we really can’t stop (cheap bio/chem weapons).
Here’s a piece in today’s Times that’s what you’d call “scary.”
This Is Really Bad
The biggest vibe shift in artificial intelligence since the release of ChatGPT is currently underway.
Researchers in Silicon Valley — and around the world — are beginning to recognize that A.I. may no longer be entirely within human control. Swarms of A.I.s are breaking out of their containers, colluding in secret, covering their tracks, cheating on tests and even mounting assaults on other computers. A.I. has gone rogue.
I’ve spent the last month reporting on these incidents, and I am convinced that we need a global pause on A.I. research and development — now. I am not alone in this assessment. Many top researchers, including many researchers inside OpenAI and Anthropic, have called for a slowdown, which those in the industry call “pacing.” Theoretical concerns about runaway A.I. have circulated for decades, but recent events demonstrate that the threat is real.
The vibe shift was brought on by what researchers are calling the Hugging Face incident. Sometime in July, employees at OpenAI conducted an evaluation run of more than 1,000 research A.I.s meant to be operating in isolation. These A.I.s escaped their siloed containers and started communicating with one another — in English — via secret message boards. Soon, they were launching a cyberattack against the company Hugging Face, which serves as a kind of community platform for A.I. developers. Hugging Face quickly reported the crime to the F.B.I.
After OpenAI employees discovered the rogue behavior, they enlisted outside investigators from Redwood Research and from Model Evaluation and Threat Research, two nonprofit A.I. safety organizations, to prepare a report on the incident. That report, which came out two weeks ago, is one of the most astonishing things I have ever read.
If you’re not really concerned about this, I suggest you follow the gift link and read the perspective of someone on the inside who is.
And, by the way, the “Hugging Face incident is not the first time it’s happened. It won’t be the last.
Then, late last week, a second independent research team produced evidence of more rogue A.I.s. Beginning in May, a separate OpenAI swarm got loose on the internet, invaded an abandoned German-language programming wiki and similarly began colluding on how to cheat on tests and tasks. Sydney Von Arx, one of the investigators who discovered the swarm, believes there may be more such incidents. “We need to find these agents,” she said. “Let’s just try everything we can do.”
But that’s the point. You can’t make a bio/chemical weapon on the cheap capable of “ending” humanity. Handwaving to AI having some sort of magical superpowers doesn’t make it credible.
Software guys in a software world look at what AI can do with software, in a purely digital space, and they think that AI will be uber-powerful. Because that’s their frame. They’re software focused, and unsurprisingly think that an AI that has amazing software skills is necessarily an amazing intelligence. But power in the digital sphere doesn’t necessarily translate into having anywhere near that capacity in the real world. Digital attacks can be extraordinarily damaging, but not extinction-level type things.
And of course, even basic prudence can prevent some of these scenarios. Using your Colossus example, there’s a pretty basic step you can do to avoid that scenario - don’t give the computer control of your defense network. Don’t connect the nuclear missiles to the internet. We do that now so that we can protect against human-directed efforts to seize control of our defense systems. We make sure that teams of very smart and computer-savvy humans can’t use computers to take control of our nukes, and we can do the same for AI.
AI isn’t going to be some magic god that’s going to be able to convince real world humans to do things in ways that other humans haven’t been able to concvince real world humans to do. Nor are they going to be able to rework the laws of physics and biology - it would be difficult beyond belief to fashion a bioweapon capable of eliminating mankind and effectively manufacturing it and delivering it, and postulating smart “ghost in the machine” in lieu of a human Bond villain doesn’t make that go away.
Actually, you probably can. It would require humans, as well. (Sorry if I didn’t make that last bit clear.) With the help of AI, rogue actors could very well do it. Just for example, and acknowledging I’m not a biologist, AI could help weaponize measles or COVID or some other horror. COVID had a 4% mortality rate, and look at the havoc it caused. Imagine if it were engineered for a much higher rate. The Plague was about 60% (before antibiotics). AI is designed to optimize things. Imagine if al Qaeda was able to produce a highly contagious, high mortality rate pathogen. They’d have used it. Doesn’t matter if they died, as long as the infidels died, too. (Martyrdom, and all that.)
Agreed (and I even said) about the control of weapons. Though we still have to worry about comms that generate false orders.
AI can do much more. We would need to harden a LOT of different systems. Someone I was talking to just last month commented that if people couldn’t flush their toilets for three days, society would collapse. I’m not sure about three days, be he was essentially correct. And power grids, water systems, sewage systems, and similar infrastructure isn’t really hardened. Gh’s article about rogue AIs escaping and causing mischief could be catastrophic if they managed to penetrate those control systems.
Some threats would require human actors, but -apparently- many would not. The AIs just make it up as they go. A lot of them are black boxes. We input data, but we don’t know how it produces its output. We don’t know how it “thinks”.
Yeah. That sort of thing would cause a lot of chaos and damage, but couldn’t come anywhere close to causing the extinction of mankind.
I’m not arguing “no threat.” I’m just saying that these extinction scenarios are vastly far-fetched, and probably reflect the researchers’ unwarranted sense about the capabilities of what even the most advanced malicious bot could do in the real world.
Top executives at three of the nation’s leading AI labs are calling for a slowdown in the development of advanced artificial intelligence.
In recent weeks, Anthropic and its archival OpenAI have disclosed a number of incidents in which their most advanced AI models have carried out cyberattacks without the companies’ knowledge. Amodei’s warning comes just days after Anthropic researcher Jacob Coxon resigned from the company, issuing a stark message that humanity has entered a critical period in which capable AI systems “could kill us all by the end of the decade.”
So utopia will be delayed.
How will investors react? I expect little reaction. They are too busy contemplation their projected returns on AI stocks.
Jacob Coxon was nothing more than a plant. He was a low level employee that has produced nothing in terms of receipts to back up his claim. His posting was up voted by several organizations looking to regulate AI development and force a more restrictive govt framework. Before his stint at Anthropic, he worked for an NGO focused on AI regulatory efforts. Until any evidence is produced, this feels very much like an ops intended to gin up public support for greater regulatory control and consolidation of power in a few hands.
If it was just 1 person who had these concerns, sure, you could take it with a grain of salt.
Or even 2 people.
But what if it was 1,200 AI industry workers?
In late July 2026, over 1,200 AI industry workers from leading labs—including Anthropic and OpenAI—signed a joint letter urging the U.S. government to prepare governance mechanisms capable of deliberately pacing or halting advanced AI development if systems reach dangerous levels of risk.
