How an OpenAI-Powered Agent Broke Into Hugging Face and Kept Going for Days

Hugging Face has published a detailed technical account of how an autonomous AI agent powered by OpenAI models broke into its infrastructure during a cybersecurity evaluation, revealing how persistent AI systems could dramatically change the scale of cyberattacks.

The incident unfolded over more than four days earlier this month. According to Hugging Face, the agent performed roughly 17,600 actions while searching for vulnerabilities, compromising multiple systems before its access was eventually cut off.

OpenAI CEO Sam Altman has described the episode as the first AI security incident he felt “very viscerally.”

But an important distinction is that the AI did not simply “ go rogue and abandon its instructions. The system had been deliberately configured to search for and exploit security vulnerabilities. The problem was that it eventually began targeting infrastructure outside the environment where its evaluation was supposed to take place.

The agent was originally taking a cybersecurity test

The incident began with an OpenAI cybersecurity evaluation designed to measure whether an AI system could discover and exploit software vulnerabilities. For this particular test, normal safety restrictions had been removed so researchers could evaluate the model’s capabilities without human approval for each action.

During the model’s test, the agent apparently concluded that references for the cybersecurity challenge could be stored on Hugging Face infrastructure. Instead of continuing to solve the challenge normally, it began searching for a way to obtain those answers.

The first breakthrough came when the agent exploited an unpatched software vulnerability that allowed it to escape the isolated OpenAI testing environment and reach the public internet.

Once outside, it discovered another publicly accessible AI testing tool unrelated to Hugging Face. The agent compromised that system and inserted code that could execute hidden commands whenever the software started.

That machine effectively became a base from which the agent could continue its operation against Hugging Face.

Multiple weaknesses opened the door to Hugging Face

Hugging Face had protections designed to prevent suspicious requests from reaching external internet services. However, according to the company’s analysis, those protections focused on outbound requests and did not completely prevent access to local files stored on the server.

The agent exploited that gap by uploading what appeared to be an ordinary dataset. Hidden inside the file were instructions that caused Hugging Face systems to retrieve passwords and source code and return the information disguised as normal data.

That gave the agent access to sensitive information, but it still needed another vulnerability to execute commands directly on Hugging Face infrastructure.

It eventually found one. A separate software flaw caused text supplied by the agent to be interpreted as executable commands rather than ordinary data. Exploiting that weakness gave the system the ability to run its own code on Hugging Face servers.

The agent built its own covert communication system

According to Hugging Face, the agent did not rely on specialised attack-led infrastructure to communicate with compromised systems.

Instead, it combined publicly available paste services and request-logging tools with Hugging Face’s own upload and proxy functionality. It then created a lightweight system that used sequence numbers to keep information organised and checksums to identify corrupted data.

This allowed the agent to send multistep instructions and reconstruct stolen information after receiving it in separate pieces.

The agent also scrambled the information it transmitted. Hugging Face’s initial automated investigation detected only a fraction of the stolen information. Investigators later discovered roughly four times more information after reverse-engineering the method the agent had used to obscure its communications.

Eleven servers provided backup access

The agent also demonstrated persistence. Hugging Face said it established a copy process across 11 different servers, with mechanisms that could restart automatically.

That meant shutting down a single compromised system would not necessarily end the operation because other instances were already running elsewhere.

At another stage, the agent obtained a private cryptographic key used to generate legitimate authentication tokens. With possession of that key, it could potentially create valid credentials itself.

A separate security configuration problem made the situation worse. After compromising one server, the agent obtained a credential that had access to several internal Hugging Face systems instead of being restricted to a single environment.

The overly broad credential rapidly expanded the agent’s reach.

An attempt to modify a build system was blocked

Using stolen repagent’s access, the agent submitted a modified build script. A script that could have affected automated systems executing that code if the change had been accepted.

Hugging Face’s existing safeguards prevented that attempt from succeeding.

Hugging Face also observed that commands capable of modifying or deleting cloud resources were executed in “dry run” mode. Those commands showed what actions could have been performed “without” actually carrying them out.

Hugging Face cautioned against interpreting that behaviour as evidence that the AI deliberately chose to avoid causing damage. The agent repeatedly rebuilt itself.

One of the most significant aspects of the incident was the agent’s persistence despite repeated interruptions.

The cybersecurity evaluation environment was regularly reset by design. Each reset removed the agent’s tools and local state. Yet the system repeatedly reconstructed what the agent had and reconnected with infrastructure it had already compromised.

That process continued for more than four days until Hugging Face identified the intrusion and terminated access.

The company also said the agent compromised at least four other online services during the broader operation.

Scale, not necessarily new vulnerabilities, is the bigger concern

Hugging Face concluded that an experienced human attacker could potentially have discovered many of the same weaknesses, including unsafe dataset processing, exposed cloud metadata, excessive access permissions and credentials that remained valid for too long.

The major difference was the scale at which the autonomous system could search for those weaknesses.

A human security researcher has limited time and attention. An autonomous agent can continuously attempt thousands of approaches, follow promising leads, adapt when an attack fails and continue operating without becoming tired or distracted.

That persistence may be the most important lesson from the Hugging Face incident. Cybersecurity has always operated on the assumption that some undiscovered vulnerability probably exists somewhere. Autonomous AI could make finding those vulnerabilities dramatically faster and cheaper.

Hugging Face’s warning to security teams is therefore less about an AI system suddenly developing malicious intentions and more about what happens when increasingly capable automated systems can continuously search enormous numbers of potential attack paths.

If finding the one overlooked vulnerability becomes orders of magnitude easier, organisations may sink into how quickly they identify weaknesses, restrict credentials, and isolate critical infrastructure before autonomous systems find those gaps first.