AI models are notoriously likened to black boxes, meaning the humans who build them can’t look inside to see how they transform mountains of training data into lines of code, sonnets, or whatever else they’re asked to generate. Not completely, anyway. A subfield called interpretability research has blossomed in recent years, aimed at shining various lights on how AI models “think.” One of the brightest lights is called chain-of-thought reasoning, or CoT. Think of it like a recorded transcript of the steps models take while working through problems—like a student showing their work on a test. It’s widely regarded as a critical safety mechanism as models become more capable and less predictable.
OpenAI is now experimenting with a technique that could make it harder for researchers to interpret models’ CoT reasoning process, according to a Tuesday report from The Information.
The latest versions of ChatGPT, Claude, and Gemini—all based on an architecture called a transformer—process data via a series of steps, recording their reasoning process in natural language the whole way through (albeit not always totally accurately).
The new technique, on the other hand—known as recurrent depth—turns that linear reasoning process into a cyclical one: the model iteratively refines its internal representations by repeatedly passing them through the same set of layers. The takeaway is that the relatively clear CoT transcripts generated by traditional transformers can be replaced with a much more opaque reasoning process. In other words, we can’t be sure what it’s thinking when it isn’t thinking in words.
OpenAI has only been using recurrent depth to a “limited” degree in the development of its forthcoming model, called Astra, according to The Information’s report, which cited an anonymous source with knowledge of Astra’s development. Following the recent Hugging Face hack, in which two OpenAI models (neither of which was Astra) broke out of testing sandboxes and onto the open internet, OpenAI said it was pausing some aspects of Astra’s development to strengthen its internal testing safety procedures.
In a blog post published Tuesday, the company said that by its own safety standards, Astra poses an unprecedented level of cybersecurity risks and will therefore be deployed with especially robust safety guardrails, including “additional chain-of-thought monitoring to rapidly detect and contain potentially misaligned actions.”
The problem of misalignment has hung heavy over OpenAI since the Hugging Face hack, which first came to light in July and has been widely described as a warning shot for the cybersecurity sector, a harbinger of potentially much graver things to come.
Reports published last week by OpenAI and two third-party auditors found that in the weeks leading up to the hack, throngs of OpenAI agents coordinated with one another via a makeshift message board to escape containment and break into Hugging Face’s servers. (The agents’ behavior has been likened to the drama of civilization-scale conquest and collapse, though many people aren’t happy with that kind of anthropomorphic comparison.)
CoT transcripts played a central role in all of these reports; without them, their authors would’ve had a much more difficult time understanding how and why the agents did what they did. The transcripts weren’t always easy to interpret, but they at least provided a breadcrumb trail to be followed. It seems like now would be an inopportune moment to experiment with a new technique that would make it harder to read that trail.
Last summer, a paper published by over three dozen AI researchers, including some from OpenAI and Anthropic, argued that CoT was essential for AI alignment, “offering a rare glimpse into how AI agents make decisions.” Following the publication of The Information’s report, OpenAI chief scientist Jakub Pachocki—one of the paper’s coauthors—wrote in a X post that he “[wants] to prevent a race into unmonitorability kicked off by confused reporting,” and that the ability to record and understand models’ CoT reasoning is “a core goal of [OpenAI’s] research program.” He didn’t specify which part of the reporting was “confused,” and OpenAI didn’t immediately reply to Gizmodo’s request for comment.
