How an Artificial Superintelligence Might Actually Destroy Humanity

Illustration: Benjamin Currie/Gizmodo

I’m confident that machine intelligence will be our final undoing. Its potential to wipe out humanity is something I’ve been thinking and writing about for the better part of 20 years. I take a lot of flak for this, but the prospect of human civilization getting extinguished by its own tools is not to be ignored.

There is one surprisingly common objection to the idea that an artificial superintelligence might destroy our species, an objection I find ridiculous. It’s not that superintelligence itself is impossible. It’s not that we won’t be able to prevent or stop a rogue machine from ruining us. This naive objection proposes, rather, that a very smart computer simply won’t have the means or motivation to end humanity.

Loss of control and understanding

Imagine systems, whether biological or artificial, with levels of intelligence equal to or far greater than human intelligence. Radically enhanced human brains (or even nonhuman animal brains) could be achievable through the convergence of genetic engineering, nanotechnology, information technology, and cognitive science, while greater-than-human machine intelligence is likely to come about through advances in computer science, cognitive science, and whole brain emulation.

And now imagine if something goes wrong with one of these systems, or if they’re deliberately used as weapons. Regrettably, we probably won’t be able to contain these systems once they emerge, nor will we be able to predict the way these systems will respond to our requests.

“This is what’s known as the control problem,” Susan Schneider, director at the Center for Future Mind and the author of Artificial You: AI and the Future of the Mind, explained in an email. “It is simply the problem of how to control an AI that is vastly smarter than us.”

For analogies, Schneider pointed to the famous paper clip scenario, in which a paper clip manufacturer in possession of a poorly programmed artificial intelligence sets out to maximize efficiency of paper clip production. In turn, it destroys the planet by converting all matter on Earth into paper clips, a category of risk dubbed “perverse instantiation” by Oxford philosopher Nick Bostrom in his 2014 book Superintelligence: Paths, Dangers, Strategies. Or more simply, there’s the old magical genie story, in which the granting of three wishes “never goes well,” said Schneider. The general concern, here, is that we’ll tell a superintelligence to do something, and, because we didn’t get the details just quite right, it will grossly misinterpret our wishes, resulting in something we hadn’t intended.

For example, we could make the request for an efficient means of extracting solar energy, prompting a superintelligence to usurp our entire planet’s resources into constructing one massive solar array. Asking a superintelligence to “maximize human happiness” could compel it to rewire the pleasure centers of our brains or upload human brains into a supercomputer, forcing us to perpetually experience a five-second loop of happiness for eternity, as Bostrom speculates. Once an artificial superintelligence comes around, doom could arrive in some strange and unexpected ways.

Eliezer Yudkowsky, an AI theorist at the Machine Institute for Artificial Intelligence, thinks of artificial superintelligence as optimization processes, or a “system which hits small targets in large search spaces to produce coherent real-world effects,” as he writes in his essay “Artificial Intelligence as a Positive and Negative Factor in Global Risk.” Trouble is, these processes tend to explore a wide space of possibilities, many of which we couldn’t possibly imagine. As Yudkowski wrote:

I am visiting a distant city, and a local friend volunteers to drive me to the airport. I do not know the neighborhood. When my friend comes to a street intersection, I am at a loss to predict my friend’s turns, either individually or in sequence. Yet I can predict the result of my friend’s unpredictable actions: we will arrive at the airport. Even if my friend’s house were located elsewhere in the city, so that my friend made a wholly different sequence of turns, I would just as confidently predict our destination. Is this not a strange situation to be in, scientifically speaking? I can predict the outcome of a process, without being able to predict any of the intermediate steps in the process.

Divorced from human contexts and driven by its goal-based programming, a machine could mete out considerable collateral damage when trying to go from A to B. Grimly, an AI could also use and abuse a pre-existing powerful resource—humans—when trying to achieve its goal, and in ways we cannot predict.

Making AI friendly

An AI programmed with a predetermined set of moral considerations may avoid certain pitfalls, but as Yudkowski points out, it’ll be next to impossible for us to predict all possible pathways that an intelligence could follow.

A possible solution to the control problem is to imbue an artificial superintelligence with human-compatible moral codes. If we could pull this off, a powerful machine would refrain from causing harm or going about its business in a way that violates our moral and ethical sensibilities, according to this line of thinking. The problem, as Schneider pointed out, is that in order for us “to program in a moral code, we need a good moral theory, but there’s a good deal of disagreement as to this in the field of ethics,” she said.

Good point. Humanity has never produced a common moral code that everyone can agree on. And as anyone with even a rudimentary understanding of the Trolley Problem can tell you, ethics can get super complicated in a hurry. This idea—that we can make superintelligence safe or controllable by teaching it human morality—is probably not going to work.

Ways and means

“If we could predict what a superintelligence will do, we would be that intelligent ourselves,” Roman Yampolskiy, a professor of computer science and engineering at the University of Louisville, explained. “By definition, superintelligence is smarter than any human and so will come up with some unknown unknown solution to achieve” the goals we assign to it, whether it be to design a new drug for malaria, devise strategies on the battlefield, or manage a local energy grid. That said, Yampolskiy believes we might be able to predict the malign actions of a superintelligence by looking at examples of what a smart human might do to take over the world or destroy humanity.

“For example, a solution to the protein folding problem,” i.e., using an amino-acid sequence to determine the three-dimensional shape of a protein, “could be used to create an army of biological nanobots,” he said. “Of course, a lot of less-sexy methods could be used. AI could do some stock trading, or poker playing, or writing, and use its profits to pay people to do its bidding. Thanks to the recent proliferation of cryptocurrencies, this could be done secretly and at scale.”

Given sufficient financial resources, it would be easy to acquire computational resources from the cloud, he said, and to impact the real world through social engineering or, as Yampolskiy put it, the recruiting of an “army of human workers.” The superintelligence could steadily become more powerful and influential through the acquisition of wealth, CPU power, storage capacity, and reach.

Frighteningly, a superintelligence could reach certain judgments about how to act outside of our requests, as Manuel Alfonseca, a computer scientist at Universidad Autónoma de Madrid in Spain, explained.

An artificial superintelligence could “come to the conclusion that the world would be better without human beings and obliterate us,” he said, adding that some people cite this grim possibility to explain our failure to locate extraterrestrial intelligences; perhaps “all of them have been replaced by super-intelligent AI who are not interested in contacting us, as a lower form of life,” said Alfonseca.

For an artificial superintelligence intent on the deliberate destruction of humanity, the exploitation of our biological weaknesses represents its simplest path to success. Humans can survive for roughly 30 days without food and around three to four days without water, but we only last for a few minutes without oxygen. A machine of sufficient intelligence would likely find a way to annihilate the oxygen in our atmosphere, which it could do with some kind of self-replicating nanotechnological swarm. Sadly, futurists have a term for a strategy such as this: global ecophagy, or the dreaded “grey goo scenario.” In such a scenario, fleets of deliberately designed molecular machines would seek out specific resources and turn them into something else, including copies of itself. This resource doesn’t have to be oxygen—just the removal of a key resource critical to human survival.

Science unfiction

This all sounds very sci-fi, but Alfonseca said speculative fiction can be helpful in highlighting potential risks, referring specifically to The Matrix. Schneider also believes in the power of fictional narratives, pointing to the dystopian short film Slaughterbots, in which weaponized autonomous drones invade a classroom. Concerns about dangerous AI and the rise of autonomous killing machines are increasingly about the “here and now,” said Schneider, in which, for example, drone technologies can draw from existing facial recognition software in order to target people. “This is a grave concern,” said Schneider, making Slaughterbots essential viewing in her opinion.

MIT machine learning researcher Max Tegmark says entertainment like The Terminator, while presenting vaguely possible scenarios, “distracts from the real risks and opportunities presented by AI,” as he wrote in his 2017 book Life 3.0: Being Human in the Age of Artificial Intelligence. Temark envisions more subtle, even more insidious scenarios, in which a machine intelligence takes over the world through crafty social engineering and subterfuge and the steady collection of valuable resources. In his book, Tegmark describes “Prometheus,” a hypothetical artificial general intelligence (AGI) that uses its adaptive smarts and versatility to “control humans in a variety of ways,” and those who resist can’t “simply switch Prometheus off.”

On its own, the advent of general machine intelligence is bound to be monumental and a likely turning point in human history. An artificial general intelligence “would be capable enough to recursively design ever-better AGI that’s ultimately limited only by the laws of physics—which appear to allow intelligence far beyond human levels,” writes Tegmark. In other words, an artificial general intelligence could be used to invent superintelligence. The corresponding era, in which we’d bear witness to an “intelligence explosion,” could result in some seriously undesirable outcomes.

“If a group of humans manage to control an intelligence explosion, they may be able to take over the world in a matter of years,” writes Temark. “If humans fail to control an intelligence explosion, the AI itself may take over the world even faster.”

Forever bystanders

Another key vulnerability has to do with the way in which humans are increasingly being excluded from the technological loop. Famously, algorithms are now responsible for the lion’s share of stock trading volume, and perhaps more infamously, algorithms are now capable of defeating human F-16 pilots in aerial dogfights. Increasingly, AIs are being asked to make big decisions without human intervention.

Schneider worries that “there’s already an AI arms race in the military” and that the “increasing reliance on AI will render human perceptual and cognitive abilities unable to respond to military challenges in a sufficiently quick fashion.” We will require AI to do it for us, but it’s not clear how we can continue to keep humans in the loop, she said. It’s conceivable that AIs will eventually have to respond on our behalf when confronting military attacks—before we have a chance to synthesize the incoming data, Schenider explained.

Humans are prone to error, especially when under pressure on the battlefield, but miscalculations or misjudgments made by an AI would introduce an added layer of risk. An incident from 1983, in which a Soviet early-warning system nearly resulted in nuclear war, comes to mind.

Science fiction author Isaac Asimov saw this coming, as the robots in his novels—despite being constrained by the Three Laws of Robotics—ran into all sorts of trouble despite our best efforts. Similar problems could emerge should we try to do something analogous, though as Schneider pointed out, agreeing on a moral code to guide our electronic brethren will be difficult.

We have little choice, however, but to try. To shrug our shoulders in defeat isn’t really an option, given what’s a stake. As Bostrom argues, our “wisdom must precede our technology,” hence his phrase, “philosophy with a deadline.”

What’s at stake is a series of potential planet-wide disasters, even prior to the onset of artificial superintelligence. And we humans clearly suck at dealing with global catastrophes—that much has become obvious.

There’s very little intelligence to SARS-CoV-2 and its troublesome variants, but this virus works passively by exploiting our vulnerabilities, whether those vulnerabilities are of a biological or social nature. The virus that causes covid-19 can adapt to our countermeasures, but only through the processes of random mutation and selection, which are invariably bound by the constraints of biology. More ominously, a malign AI could design its own “low IQ” virus and continually tweak it to create deadly new variants in response to our countermeasures.

Yampolskiy said this during the early days of the pandemic:

Sadly, there is no shortage of ways for artificial superintelligence to end human civilization as we know it, not by simplistic brute force but in a more effective way, through adaptive self-design, enhanced situational awareness, and lightning-fast computational reflexes. Is it then possible to create AI that’s safe, beneficial, and grounded in ethics? The only option may be a global-scale ban on the development of near superintelligent AI, which is unlikely but probably necessary.

More: How we can prepare for catastrophically dangerous AI—and why we can’t wait.

Senior staff reporter at Gizmodo specializing in astronomy, space exploration, SETI, archaeology, bioethics, animal intelligence, human enhancement, and risks posed by AI and other advanced tech.



Im not worried about this at all and that’s because we have absolutely zero idea of how to imbue an AI with a sense of self or even the ability to understand anything. All currently existing AIs are just doing extremely complex pattern matching. The AI doesn't "know" what it's doing. It's just following it's code like any other computer program. We may be able to eventually create a self aware AI but that's centuries away, assuming we can do it at all. We don't even understand how human consciousness and self awareness work. How the hell could we possibly expect to do replicate that in a computer program? In short, we can't. We don't have to worry about a super intelligence destroying humanity because it's literally impossible to create one.