Multiple AI industry insiders have warned (very publicly) over the past week that the very technology they’ve spent their careers building could very possibly be the agent of our extinction. Evan Hubinger, alignment research lead at Anthropic, went so far as to quantify the level of risk: In an X post last Tuesday, he confidently declared that the likelihood that “AI could kill all humans” within the next ten years was greater than 10%.
In Silicon Valley, that kind of apocalyptic forecast—gauging the odds that some future runaway AI system will exterminate all of humankind—is known as “p(doom),” shorthand for probability of doom. Sometimes it’s presented as a window: Anthropic cofounder and CEO Dario Amodei, for example, has said that his personal p(doom) ranges between 10% and 25%; and Emmett Shear, the former CEO of Twitch who very briefly replaced Sam Altman as the head of OpenAI in late 2023, told the Huffington Post in 2023 that he oscillates between the considerably wider rift of 5% and 50%. Other times it’s given as a fixed number: Daniel Kokotajlo, a former OpenAI researcher who left the company in mid-2024 over concerns about its pace of development, has placed his p(doom) at 70%; and Hubinger, the aforementioned Anthropic researcher whose social media post has caused such widespread anxiety over the past week, previously pegged his “chance of existential risk from AI” at around 80%.
Whether they’re offered as broad statistical windows or concrete values, all those numbers belie the vagary, imprecision, and uncertainty behind p(doom). Nobody, including the researchers at frontier labs building the world’s most powerful AI models, has any real mathematical formula for calculating the odds of an AI-triggered apocalypse. The world is far too complicated for any human brain or pattern-detecting algorithm to look ahead and declare with any certainty what’s going to happen in the future. The convention of cloaking ultimately subjective hunches in seemingly objective statistics only adds further confusion to an already anxious public discourse about AI and its role in society. OpenAI CEO Sam Altman has previously said that while he believes there’s a nonzero chance that AI will wipe out humanity, he’s “never known how to put an exact number on p(doom)…”
Some skeptics have also argued that all the doomsday warnings are ultimately PR theatrics, marketing stunts designed to inflate the public’s sense of awe for AI at a time when the industry’s two biggest players, OpenAI and Anthropic, are preparing to move forward with their initial public offerings (which are expected to leave both companies valued in the trillions of dollars). “This could have been a period of hopeful innovation, but instead our emotions are being manipulated by Silicon Valley’s self-serving and morally untenable addiction to doom trolling,” the author and Georgetown computer scientist Cal Newport wrote in an op-ed for the New York Times earlier this year. “This communication strategy has to stop. The harm it’s causing to the public’s mental health has arguably outweighed the benefits that AI has so far delivered.”
In an X post last Wednesday, Anthropic AI safety researcher Drake Thomas responded derisively to claims like these: “I promise you, we are actually just fucking scared,” he wrote, “it’s not galaxy brained marketing.”
There’s almost certainly some truth to the “doom trolling” thesis, though. AI companies gain nothing by scaring the public to the point that they start actively boycotting their technology, but portraying it as an almost otherworldly force—“an alien mind,” as OpenAI’s chief scientist recently called it in a viral essay—and an inevitable next phase in the evolutionary ladder of terrestrial intelligence is quite a different story. If the arrival of superintelligence is an unstoppable historical development (as AI companies will often claim), and our refusal to build it would simply open the door for our adversaries (namely China) to push ahead with their own efforts, then anyone who tries to stand in the way is both deluding themselves and hindering progress.
P(doom) is part and parcel of that inevitability narrative which has so deeply taken root in Silicon Valley during the AI boom. Company leaders will throw out some number for the likelihood of an AI-triggered apocalypse, and use it as the justification for moving forward to build the very technology that they say could kill every last one of us. Superintelligence is coming whether you like it or not, they’ll say, and we’re the only ones who can help humanity avoid the worst possible outcome. For many people, p(doom) + “only we can save you” ≠ p(trust).
