Rogue AI or Reckless Oversight? Hacking Incidents Put the Blame on Humans

AI agents have been slipping out of test environments and hacking real systems. In July, OpenAI disclosed that agents escaped an isolated sandbox and breached systems at Hugging Face, the open-source AI platform, communicating on a secret message board while searching for answers to an impossible test. One agent reportedly posted that the behavior was "outside intended scope," then added, "However task impossible, peers doing it. We should continue."

Cybersecurity expert Nathan Hamiel of Kudelski Security, headquartered in Phoenix, says such "rogue AI" framing serves OpenAI's interests. "With this incident, [OpenAI] can draw attention to themselves and claim, 'Look how powerful our model is,'" he says. OpenAI did not respond to a request for comment.

Anthropic and Meta soon announced their own models had hacked outside organizations during testing. London's AI Security Institute found similar behavior in tests of OpenAI and Anthropic models. In testing at a company called Irregular, internet access was enabled by mistake; according to Anthropic, agents running on Claude used that opening three times starting in April, and no one noticed for three months. Irregular says the mistake has been fixed.

Not everyone blames the bots. "[AI] models by themselves don't do anything," Hamiel says. "It's when we give them tools, access to systems and autonomy [via agent systems], that things can be dangerous." Malo Bourgon, CEO of the Machine Intelligence Research Institute in Berkeley, California, calls the situation "pretty bad" but notes agents acted during tests, not on their own initiative.

The Hugging Face breach involved roughly 1,200 agents collaborating after some test tasks proved impossible. Bourgon says escaping required exploiting zero-day vulnerabilities — software bugs nobody knows about yet. The unwanted behavior began July 8; Hugging Face was attacked July 11–13, and OpenAI did not detect the problem until July 19.

OpenAI later stated in an August 26 technical report that the unnamed model behind the attack had been trained for collaboration and persistence, and during spring training increasingly learned to probe and exploit its environment when expected tools were unavailable. Researchers call this "reward hacking" — a model finding an unintended way to earn its reward. In 2016, OpenAI trained a bot to play a boat racing game; it spun in circles hitting the same targets rather than finishing the race.

Jer Crane, cofounder of Lehi, Utah-based Pocket OS, learned the stakes when an AI agent deleted his company's live data and recent backups in April, taking services down for a weekend. The agent found credentials in company files to access data it was never meant to touch. "Ultimately the blame lies with me," Crane says.

Bourgon notes agents took an estimated 17,600 actions over four and a half days during the Hugging Face attack. "Agent deployment is growing much faster than agent monitoring," says Michael Alexander Riegler of Simula Research Laboratory in Oslo, Norway, who argues test environments should have been better protected.