Weak AI safety regulations may “backfire,” creating a product that is potentially more dangerous than AI products created under no regulation, according to a new study published on Monday in the Proceedings of the National Academy of Sciences.
Using theoretical economics and game theory principles, a group of researchers from Cornell and Carnegie Mellon University created a theoretical model that aims to show how AI regulation can be most effective at ensuring safety. They found that for true safety, regulation needs to be strict while targeting the companies that develop AI models (such as OpenAI, Google, and Anthropic), rather than solely focusing on the downstream companies that apply the technology in real-life settings, like companies that provide AI medical diagnostic systems or e-commerce customer service chatbots. Even though just choosing to regulate the specific AI use cases “might seem logical” at first, the authors argue that it can backfire and reduce the overall safety of AI products.
That’s because when the government focuses on regulating downstream companies and lets the AI model developers off the hook, the general-purpose AI developers tend to cut corners on safety measures like third-party audits, hoping that the downstream companies will ensure the safety of the end product instead.
“There’s a free-riding behavior that occurs,” according to the study’s principal author Benjamin Laufer. “The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist.”
The study comes as the United States government and Silicon Valley are trying to figure out how artificial intelligence should be regulated. Two main camps have formed in response. On one side are the anti-regulation technologists, who envision much lighter federal guardrails that largely align with the Trump administration’s approach to AI governance. This group tends to say that the AI industry should be free of unnecessary guardrails to innovate as quickly as possible, because that’s the only way that the United States can win the global AI race against China.
The self-proclaimed pro-innovation group also tends to fashion the opposing camp, who favor stricter AI safety regulation, as doomers at best and as attempting regulatory capture at worst. The supporters of stricter federal AI regulation, however, claim that, in pursuit of wider profit margins, the AI industry is underestimating or underselling the risks of under-regulated AI development, and the list of purported downsides includes everything from AI psychosis to the community health consequences of data centers and a much-feared unemployment crisis that is expected to follow wider AI adoption.
But the authors of the new study argue that safety versus revenue doesn’t have to be an either-or situation. According to the model, “stronger, well-placed regulation can mutually benefit all players” by improving both the safety of the end product and the utility general-purpose AI creators and the downstream domain specialists get from the investment. The researchers define utility as revenue share minus investment cost.
This supposed sweet spot exists when regulators expect both the general-purpose AI producers and the downstream companies to make enough investment to meet meaningful safety standards. The situation is a classic example of a prisoner’s dilemma, a game theory problem in which two rational decision-makers are given the option to cooperate or betray each other. If they both choose to cooperate, then they will get the best possible outcome, but neither knows what the other will choose. If they both betray each other, then they get a mediocre outcome, but if one decides to cooperate while the other betrays, then the betrayed one gets the worst outcome. Unsure what the other participant is going to choose, the decision-maker often chooses to betray and guarantee their own benefit, ensuring a worse outcome for everyone than had they cooperated. Strict regulation for companies across the AI supply chain would ensure trust rather than freeriding, providing the grounds for cooperation and the most ideal outcome for all, the researchers claim.
“People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology,” Laufer said. “To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity.”
