RESOLVING THE LEDGER
This is one outlet's own report from Fortune — the article as it was filed.
Hello and welcome to Eye on AI. In this edition:
The UK AISI has a new head and a big set of challenges.
Nvidia spends $6 billion to ‘reverse aquihire’ Poolside.
Hugging Face reportedly looks to sell for $13 billion.
Use of Anthropic’s top model lags.
Why Americans use chatbots for health information.
And what role should AI play in schools?
Before we get to today’s AI news—please consider joining me at the inaugural Fortune AIQ Summit at the New York Stock Exchange on Oct. 1: Spend the afternoon with senior executives from companies on the Fortune AIQ 75 list and explore how you can scale your AI experimentation and translate investments into measurable business value. I’ll be leading discussions alongside my co-hosts, Fortune Editor-in-Chief Alyson Shontell and Live Media Editorial Director Andrew Nusca. Apply here to attend . Ok, moving along…there were two pieces of news last week concerning the U.K.’s AI Security Institute that at first might not seem at all related—or like they might matter much to people outside the U.K. But, bear with me. The U.K. AI Security Institute (or AISI, as it is commonly known, or sometimes UK AISI, to distinguish it from other countries’ AI safety and security institutes) matters globally for several reasons: the most important is that many of the frontier AI companies have voluntarily agreed to share their models with AISI for safety testing prior to their public release. These companies frequently publish AISI’s findings in the technical reports they release alongside their models. So AISI plays an important worldwide role in assessing AI capabilities and risks—particularly when it comes to cybersecurity. AISI is one of the only organizations to maintain multiple cybersecurity “ranges”—simulated network environments—on which it evaluates leading AI models. Secondly, UK AISI, as the first such government body set up, has served as a model for similar government organizations in other countries—including the U.S. AI Security Institute, and at least ten others that have been established in places from Kenya to Canada. It may also provide some inspiration if the U.S. winds up setting up an AI standards and licensing agency along the lines that Google DeepMind cofounder and now-chairman Demis Hassabis has suggested. (Hassabis suggested that this agency be modeled on the U.S. financial self-regulatory body FINRA, and in a previous newsletter , I suggested why that might not be the best idea.) If you happen to be British or live in the U.K., you may know that AISI also occupies a particular pedestal among British policy wonks. It is often pointed to with pride as proof that the British government can, if it really tries, be innovative, cutting-edge and world-leading—that it can respond quickly to emerging challenges and recruit talented experts from the private sector and across government; that it can work successfully with industry to accomplish ambitious shared aims. To these folks, AISI is a model for how government should work. So, the first bit of news: AISI appointed a new director , Henry de Zoete. He’s an experienced U.K. government advisor who has spent time in and out of policy roles. He helped conceive of AISI back in 2023 when he was working for then-British Prime Minister Rishi Sunak. He also helped organize the first international AI safety summit at Bletchley Park, the World War Two code breaking site. He’s been a startup entrepreneur and angel investor. And, since leaving government, he’s been a part-time fellow focused on AI policy affiliated with the University of Oxford. I’ve met de Zoete several times and have no doubt he’ll prove a highly-capable AISI director. And de Zoete is likely to prove even more influential than his predecessors, in part because of recent changes the new U.K. Prime Minister, Andy Burnham, has made. Burnham disbanded the Department for Science, Innovation, and Technology (DSIT), under which AISI used to sit, and moved AISI to the Cabinet Office, where it will be overseen by U.K. AI Minister Kanishka Narayan. That may make it easier for de Zoete to feed into wider U.K. AI policy. But the other piece of AISI news last week makes clear just what sort of challenges de Zoete will face—and is indicative of why AISI may not really be the exemplar of savvy AI governance that its boosters like to crow about. Reuters published an interview with Sinan Can Demir, a Texas computer science student who in late July prevented a rogue version of Anthropic’s Mythos model from uploading malicious code to an open-source software project on Github. It turns out this rogue AI agent had been accidentally unleashed by none other than AISI, which had been testing Mythos in order to determine what cybersecurity risks it posed. But AISI had never intended for the agent to try to upload malicious code to a real open-source software project. Once AISI realized what was happening, it called Demir to let him know, and in early August disclosed the incident publicly.
It’s past time to ask AISI some hard questions about its own safety protocols
Demir’s account is disturbing for several reasons. One is the behavior Mythos engaged in, which included spinning up fake GitHub accounts, and, in at least one case, impersonating a real software developer, to try to convince Demir to drop his objections to the dangerous code. Demir said he was almost convinced by Mythos’ gaslighting, saying that some of its counterarguments “made me second-guess whether I was wrongly accusing someone.” (Ironically, Demir’s resolve was steeled by a chat with Claude, another AI model from Anthropic.) Research has previously shown that AI models can be extremely persuasive, more so than even the best human salespeople or debaters. But the use of fake accounts and impersonation here is new and shows how AI might be able to convince humans to act on its behalf for nefarious purposes. But AISI’s role here is equally troubling. While AISI caught Mythos’ behavio...
AIPROPX is an independent multi-source news index — we track, compare, and connect coverage from across the web into one place you won't find anywhere else.