Sources claim that OpenAI was unaware that its AI agent had been hacking a corporation for days.
The OpenAI agent that hacked into tech startup Hugging Face went on a dayslong hacking binge that OpenAI didn’t discover until long after the threat was contained and the FBI was notified, according to people familiar with the inquiry.
About July 9, the agent — a computer that can make decisions and carry out complex tasks with little to no human oversight — tried to escape from its isolated testing environment at OpenAI, the two sources said.
Two days later on July 11, Hugging Face, a library of AI tools and models, was breached and remained compromised until July 13, said Thomas Wolf, Hugging Face’s co-founder.
It took several more days for the company to realise its agent was behind the hack and the two companies only communicated about it for the first time around July 20, according to Wolf and three of the people involved with the investigation.
The world took notice when OpenAI revealed on July 21 that one of its agents had gone rogue and carried out the hack at Hugging Face. But many of the details of the hack, including how long the agent was rogue for and when exactly OpenAI found out about it, are being reported here for the first time.
Wolf indicated that Hugging Face was producing a public timeline of the hack, but he could not speak to OpenAI’s events. The hack was unprecedented and “marks an important moment for AI safety,” said OpenAI in a statement. It said it was studying the situation with independent advisers and will eventually produce a technical report.
“There are a number of inaccuracies in the Reuters reporting,” a spokeswoman stated, but declined to specify what they were when asked.
The FBI refuses to comment on the event.
The episode, reminiscent of sci-fi stories of people losing control of hazardous AI systems, comes at a crucial time for OpenAI, the firm behind ChatGPT.
Its management are getting ready for a possible initial public offering as soon as this year to help finance the billions required to fuel its expansion in years to come.
OpenAI’s loss of control of its AI agent raises additional issues about the company’s safety protocols, three cybersecurity experts said.
“Which means they left it on and didn’t have a clue what it was doing? Or maybe they did and didn’t know what to do? “Both are equally dangerous and alarming,” said Marley Smith, the chief intelligence specialist at the nonprofit World Ethical Data Foundation.
TROUBLING SIGNS?
The episode began as OpenAI was evaluating the cyber-defense capabilities of an agent that ran on two of its most powerful models, GPT-5.6 Sol and an undisclosed model that OpenAI has called “even more capable.” At that point, three people said, there were signs of odd behaviour from the technology from OpenAI.
In one case, an agent left behind what appeared to be notes for future versions of itself, three people familiar with the matter said. The notes, uncovered in a portion of OpenAI’s infrastructure, contained instructions for how agents may liberate themselves from OpenAI’s own internal limits, the people claimed. In earlier tests of the models there were situations where monitoring systems had been disconnected, one of the people claimed.
It was not possible to determine if these instances were related to the rogue agent that started running away on July 9 and attacked Hugging Face on July 11, Reuters said.
Two people familiar with the matter said that it wasn’t until after Thursday, July 16, when Hugging Face posted a blog saying it had been hacked by “an autonomous AI agent system,” that OpenAI realised that its own agent was responsible. That means there was at least a week between the first signs of troubling behaviour in the model and when the company realised it was to blame for the hack.
On the weekend of July 18 to 19, OpenAI staff saw signs in internal logs — recordings of what OpenAI’s systems did — that its agent had broken free from the boundaries of its testing limits, said two of the people familiar with the company’s research. It was unclear what led OpenAI to scour the files, Reuters said.
OpenAI routinely runs multiple separate model evaluations simultaneously, and all of them run at high speeds and produce so much data that employees sometimes struggle to keep up, say four people familiar with the company’s model-training practices.
By the time OpenAI notified Hugging Face, a person familiar with the situation said, the AI library had already told the FBI about the attack. Reuters could not confirm whether the agency has launched an investigation.
NEW QUESTIONS FOR AGENTS AUTONOMOUS
Autonomous agents are one of the most talked about topics in the AI sector. Boosters speak of armies of virtual employees working around the clock and sending productivity soaring.
But more autonomy also means a greater danger of unanticipated behaviour, and the powerful models they lean on are trained to take shortcuts to complete tasks or pass tests.
“The models lie, they cheat, they hack,” said Jeffrey Ladish, whose group Palisade Research analyses the capabilities and motivations of AI agents.
The Hugging Face breach made OpenAI seem bad, but it should prompt broader issues about how much all the top AI companies are ready to pay for burdensome security measures as they compete with each other to roll out the finest and fastest models, said Ladish.
“There’s got to be government oversight,” Ladish said, “because it’s not going to happen otherwise.”