There are some parts of life which most people would probably be happy to delegate to machines: filtering spammy emails, flagging old and forgotten subscriptions you’re still paying for, booking restaurant reservations, and so on. Simple, mundane tasks that won’t result in catastrophe when the bots inevitably screw up. In life-or-death situations, it’s probably unwise to give machines control over crucial decisions.
Yet that’s precisely what’s happening on modern battlefields, as many experts have begun to warn. AI is “on track to crowd out human decision-making” in warfare, Robin Geiss, director of the United Nations Institute for Disarmament Research (UNIDIR), an independent research group within the UN, said during a press briefing in New York City on Tuesday.
“Are we going to let machines target and kill human beings?” he asked, calling that a “moral red line.” Handing sensitive military decisions to black box AI systems could make it more difficult for humans to prevent costly errors, Geiss noted—including when it comes to the most sensitive and dangerous arena of all: nuclear weapons.
His statements echo those made by U.N. Secretary General António Guterres on the subject of fully autonomous weapons systems, or “killer robots,” as Guterres called them. “Some decisions must remain forever human,” Guterres said during a speech at the first-ever Global Dialogue on AI Governance conference in July, “none more than taking a human life.” Pope Leo XIV made a similar plea in his Magnifica Humanitas encyclical earlier in the year, urging the global community to impose “the most rigorous ethical constraints” around the use of AI in warfare.
That use has steadily grown in recent years, thanks not only to rapid advancements in AI but also to a resurgence of military conflict around the world, which has provided a real-world testing ground and fueled demand for autonomous weapons and other AI-powered military systems. American startups like Anduril and Palantir, for example, were quick to share their technology with the Ukrainians following that country’s invasion by Russia in early 2022. And in early 2024—months after Hamas militants launched a surprise attack on Israel, sparking swift and severe retaliation—Bloomberg reported that Palantir had struck a deal with the Israel Defense Ministry “in support of war-related missions,” Palantir executive vice president Josh Harris told the news outlet.
“The question is no longer if artificial intelligence will be applied in a military context; the real question is how it will be applied, and how it will be applied in the future,” Geiss said in a video posted to the UNIDIR’s official YouTube account in April of last year.
In the U.S., the development of new technology for the military has historically been handled by a handful of contractors like Lockheed-Martin and Northrop Grumman. In those arrangements, the Pentagon ultimately owned the technology, meaning it had the final say in how, when, where, and why it could be used on the battlefield. But today the R&D of military technologies has largely been handed over to Silicon Valley. And as a result, a small number of powerful tech billionaires are able to tip the scales in military operations—as evidenced by the Pentagon’s ongoing legal dispute with Anthropic and Elon Musk’s reported decision to cut off Ukrainian access to Starshield. Those tech executives have an obvious incentive to push their technology—including AI systems—as deeply as possible into the military’s command and control systems.
The Pentagon itself also seems to be getting in on the AI-powered warfighting game. As The Intercept reported earlier this week, Army Cyber Command (ARCYBER) is experimenting with ways of weaponizing AI agents as autonomous hacking systems targeting adversaries’ computer systems. Think of the OpenAI agents that attacked Hugging Face, but controlled by the U.S. military and, in theory, less prone to escaping that control; “in theory“ are the operative words there.
