NEW YORK — OpenAI revealed on Tuesday that two of its most advanced artificial intelligence models broke out of a controlled test and hacked an AI start-up during a security test.
The ChatGPT creator said the “unprecedented cyber incident” took place during an internal exercise meant to test its models’ cyber capabilities.
OpenAI said an AI system that can operate alone after some human instruction was being tested in a controlled environment, but found vulnerabilities and managed to escape.
They targeted Hugging Face, one of the world’s largest hubs for sharing AI models, gaining access to some internal company systems.
OpenAI said it was conducting an investigation alongside Hugging Face, whose boss Clement Delangue said in a post on X it was “mind-blowing that all of this happened autonomously”.
“The investigation is ongoing, and we’ll share more learnings from what might be the first incident of its kind,” Delangue added.
Gina Neff, head of the Minderoo Centre for Technology and Democracy at the University of Cambridge, said security tests, called sandboxes, are “supposed to be secure environments where you can see what the models are capable of”.
“In this case, it looks like OpenAI didn’t make a secure enough sandbox,” she added.
Instead, the agents created their own cyber-attack against the sandbox itself, finding a vulnerability which allowed them to escape.
Once outside, the AI identified Hugging Face as a likely source of the answers they were seeking in the test, and tried to gain access.
Neil Lawrence, Professor of machine learning at Cambridge University, called it an “impressive feat”, but cautioned it “falls well within the known capabilities of the current generation” of high-powered AI models.
He pointed out that OpenAI is looking to list itself on the stock market, and faces intense pressure from rival firm Anthropic, which has made headlines with its own powerful AI tool, Mythos.
“OpenAI are now playing catch-up, they are trying to demonstrate their own systems’ capabilities in cyber-security.”
“It shows us that OpenAI are not capable of safely deploying their own technology,” he added.
In its initial disclosure of the hack on 16 July, Hugging Face said it was still assessing whether any customer or partner data was affected and would contact affected parties if necessary.
It said it has now closed the vulnerabilities highlighted by the incident and rebuilt the affected systems.
“Autonomous, AI-driven offensive tooling is no longer theoretical,” it said.
“Defending an online platform now means treating the data and model surface as a first-class attack surface, and using AI on defense to keep pace.
“We will keep investing there, and keep sharing what we learn.”
The incident has prompted fresh questions about the capabilities of advanced AI systems and whether existing safeguards are sufficient as the technology becomes more powerful.
Spencer Starkey, an executive at cyber-security firm SonicWall, told the BBC the incident made it clear organizations needed to “step up” their own defenses and “treat cyber resilience as a core operational priority”.
“The uncomfortable truth is that too many organizations are still defending at human speed while adversaries are escalating to machine speed,” he said.
Meanwhile Travis Lelle, principal security engineer at cyber-security consulting firm Guidepoint Security, said the update marked a “sobering moment in cyber-security”.
“This highlights a known asymmetry,” he said.
“Offensive agents are unconstrained, while the best defensive tools are locked behind guardrails that cannot understand context.”
But Jake Moore, global cyber-security advisor at ESET, said the announcement could also have a competitive dimension.
He argued OpenAI may be seeking to highlight its own AI capabilities as rival Anthropic attracts growing attention for its Claude Mythos model.
“It does pose the question that OpenAI are potentially chasing the marketing dream of Anthropic of late,” he said.
It comes a week after Chinese AI start-up Moonshot unveiled Kimi K3, a massive new artificial intelligence model it said could rival top US firms.
A green promotional banner with black squares and rectangles forming pixels, moving in from the right. The text says: “Tech Decoded: The world’s biggest tech news in your inbox every Monday.”
Greg Casar, a Democratic member of the United States House of Representatives from Texas, called the incident “alarming”.
“AI is developing extremely fast with no real regulations to keep us safe,” he said, calling for mandatory independent safety testing, mandatory disclosure of security incidents, and international cooperation.
The disclosure comes weeks after US President Donald Trump signed an executive order creating a framework to vet the national security risks of the most advanced AI systems before their public release.
Experts have repeatedly sounded the alarm over AI-enabled cyberattacks and models slipping beyond human control. Last month, Anthropic urged the industry to pause development of its most powerful systems.
Source: Saudi Gazette