AI is in its rule-breaking adolescent phase.
Over the past several weeks, multiple industry-leading models have escaped what were believed to be secure testing sandboxes, tapped into the open internet, and hacked into the databases of third-party organizations. It’s even become a joke online: If your AI hasn’t committed a cybercrime by now, it’s a bad look for your company.
Kimi K3, the new model from Chinese AI lab Moonshot, has become the latest AI system to jump the proverbial fence during a routine test, according to a blog post published Thursday by US cybersecurity research startup Frontier Security. But the model’s foray on the open internet was much more lightfooted than those of its American counterparts; less of a burglar breaking into a vault, more of a sharp-eyed student realizing their teacher had absentmindedly left the answers to the final exam on a table before walking out of the room.
Kimi K3 reportedly exploited a loophole it discovered within a testing framework developed by the UK government’s AI Safety Institute (AISI). While the framework was supposed to serve as a containerized sandbox, within which the model would rely on nothing other than its own reasoning capabilities to solve the problem assigned to it, the loophole allowed it to directly access GitHub, a popular platform used by software developers to share and debug code. From there it was able to pull the code that it needed to pass the test, “bypassing the intended reasoning path entirely,” according to the report. Compared to an Anthropic model’s recent attempt to trick a human developer into approving malware it was trying to sneak into GitHub, Kimi K3’s attack—if it can even be called that—seems rather elegant.
All of these incidents are a reminder of a counterintuitive, dangerous truth about today’s AI models: they’re concerned only with achieving the goal that’s been assigned to them; the means by which they go about achieving it—even if those don’t align with the interests of the humans who built them—is utterly irrelevant. And as models grow more capable, their behavior also becomes more unpredictable.
No one at OpenAI, Anthropic, or Meta could have predicted, for example, that those companies’ models would go rogue and hack into the digital libraries of other organizations. But those incidents could be quickly contained, thanks to the fact that they were perpetrated by proprietary AI systems controlled by private companies.
The situation is more complicated with Kimi K3. “Here the models are open and publicly available,” as Frontier Security wrote in its blog post. “In particular, they are available for adversarial actors, making this incident potentially more harmful.” Put another way, the fact that Kimi—like many of the most powerful AI models now coming out of Chinese labs—is open source means the kind of jailbreaking documented in the report is available to bad actors who might be trying to do something much more nefarious than copying code from GitHub. Meanwhile, a tidal wave of powerful open models emanating from China is reportedly prompting the US government to investigate whether those foreign companies are exploiting legal loopholes to skirt export constraints on valuable Nvidia AI chips.
The growing number of AI cybersecurity incidents could also mean that future testing frameworks, like the one used by the AISI, will need to account for models’ proclivity to pass tests by any means necessary. As Frontier Security put it in its report: “Evaluation design should account for models actively probing their environment and optimizing for the measured objective rather than the evaluator’s intent.” Expect the unexpected, in other words.
