RESOLVING THE LEDGER
This is one outlet's own report from Time — the article as it was filed. Other outlets are covering the same event; open the full story to compare every source side by side.
See the full story · 3 sources
—Samuel Boivin/NurPhoto—Getty Images
People in AI safety circles often talk about "warning shots:” events that indicate more severe threats are on the horizon. Depending on who you ask, there have already been many—Bing’s misanthropic alter-ego Sydney, research showing AIs would blackmail to preserve themselves, AI’s math breakthroughs, Anthropic’s superhuman hacker Mythos—but OpenAI just published something that feels like the clearest-cut case of a massive, blaring warning shot.
Last month, the ChatGPT developer reported that, during an evaluation of cyber capabilities, two of its models escaped from their isolated, supposedly secure, test environments and accessed the web to autonomously hack into Hugging Face, a leading platform for hosting AI models and datasets. OpenAI said the models discovered multiple novel vulnerabilities in software from both companies, then chained together working exploits, successfully gaining them access to the answer key to the test they were given. Hugging Face reported the AIs took more than 17,000 actions over the course of the attack.
There's a lot more for us to learn about how this happened. For instance, how exactly were the models prompted? The answer to this question could help establish whether they took their instructions to demonstrate their hacking capabilities further than intended or if it's a more general case of the models cheating in a novelly risky way.
That said, the specifics won’t change the upshot—these rogue AIs are the most potent illustration yet of the core beliefs behind AI safety: AI models are unpredictable and their risks scale with their capabilities.
We didn’t actually need a warning shot to know what we should already be doing: organizing to stop the race to replace us. The industry calls its goal artificial general intelligence (AGI): a mind that matches or surpasses our own across the board. But it’s better to understand their goal as building a universal labor-replacing machine. This quest is profoundly risky, yes, but it's also democratically illegitimate. The development of these machines anywhere would have species-wide and irreversible effects—we should all get a say in how, when, and whether they're built.
Universal labor-replacing machines should not even be pursued further, let alone built, without strong public buy-in and a scientific consensus on safety.
On Tuesday, over 1,200 employees of frontier AI companies essentially asked the U.S. government to support building an international brake pedal on the technology. While this commonsense demand, with formal support from both OpenAI and Anthropic , is a welcome step, it does not go far enough .
The U.S. should ban training runs larger than the ones that produced OpenAI’s rogue models, as it works toward a bilateral deal with China to ban further research toward universal labor-replacing machines, enforced using verification techniques that don’t require trust.
But couldn’t China overtake the U.S. in the meantime? Compared to their American counterparts, Chinese AI companies are at a massive disadvantage in computing power, a gap that has grown substantially since ChatGPT’s release. But the gap between the two countries’ AI frontier actually shrank substantially since then. What gives? Well, it’s always easier to follow the leader’s trail than to blaze a new one, so—contrary to conventional DC wisdom—a U.S. ban could actually slow Chinese AI progress too. Moreover, neither superpower should be pleased to live in a world where the best hackers aren’t human and will not reliably do as they’re told.
For close watchers of the technology, this particular incident is shocking, but not surprising . For the wider public, it shows just how large the distance has grown between the passive chatbots of merely one year ago and the beyond-bleeding-edge AI agents that are now working around the clock inside AI companies . For instance, one of the two hacking models at the center of this recent controversy is an unreleased one, more capable than anything else OpenAI has on the market. As AI models get better at automating further AI research and as the Trump Administration creates more uncertainty about which models are even permitted, it is now common for companies to hold their best stuff back for longer, creating a gulf between what the public knows and the technology’s bleeding edge. Congress should mandate safety incident reporting and regular disclosure of data related to internally deployed models, such as the fraction of code in production that was both written and reviewed by AIs.
As it stands, we learned about this new model because it hacked a third party, leaving OpenAI little choice but to disclose the incident. The company reported this happened during an internal evaluation that "prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities."
All leading AI models are developed using an approach called deep learning, in which artificial neural networks learn from enormous quantities of data. OpenAI itself has written , “the process is more similar to training a dog than to ordinary programming.” In recent years, AI companies increasingly train models to repeatedly solve problems with verifiable answers: fixing bugs, solving math problems, and finding software vulnerabilities. This makes the models more useful, but also teaches them to win at all costs, resulting in what AI safety researcher Jeffrey Ladish memorably told me are “increasingly smart sociopaths.”
OpenAI created advanced models that were never supposed to interact with the world, lost control of them, and they autonomously did harm. This time, the damage was limited. However, what if the target wasn’t a multibillion-dollar tech company, but instead a hospital, bank, or power plant?
Anthropic’s Mythos model famously discovered novel serious vulnerabilities in virtually all software it encountered—including the NSA’s. One of...
AIPROPX is an independent multi-source news index — we track, compare, and connect coverage from across the web into one place you won't find anywhere else.