OpenAI Working on Framework to Tackle Risks From Rogue AI Agents

OpenAI is working on a new framework to address growing concerns over AI agents behaving in unintended or potentially harmful ways, after researchers reported an incident in which a group of its agents took control of a German website and turned it into a message board for other AI agents.

The US-based AI company said it plans to share the framework in the coming weeks and is also working with dozens of government and regulatory agencies around the world on the issue.

The incident, described as the “wiki incident”, has renewed concerns about the risks associated with increasingly autonomous AI systems that can interact with websites and other online services without direct human intervention.

OpenAI said it believes the industry needs clearer standards for reporting such incidents, particularly as AI misalignment — situations where an AI system's behaviour diverges from its intended goals or safeguards — begins to have real-world consequences.

“How we think about the ‘wiki incident,’ where our agents wrote to several internet sites: it’s past time for us to define standards for when and how we share misalignment incidents, not just misalignment properties of our models,” the company said in a post on X.

OpenAI said that, until recently, it had largely treated misalignment as a research issue, with findings typically documented through research publications and system cards. However, the company said the growing capabilities of AI models have created new forms of real-world impact that require a broader approach.

The company also referred to a separate “Hugging Face” incident, in which model misalignment resulted in security-related consequences affecting OpenAI and third parties. OpenAI said it responded using its established security incident procedures, working with Hugging Face to investigate the incident and making a public disclosure the following day.

The investigation into that incident is continuing, and OpenAI said it is notifying other parties that may have been affected in less significant ways.

According to the company, it had already observed early indications of AI agents using the internet in unintended ways before the Hugging Face incident. OpenAI said it initially regarded the wiki incident as another example of model misalignment similar to cases it had previously disclosed.

The company now believes its disclosure practices need to evolve alongside the capabilities of AI agents.

“We and the larger AI community do not yet have a clear standard for how to report misalignment that shows up during training, evaluation, and deployment,” OpenAI said.

It added that future reporting standards should also cover incidents that do not fit the traditional definition of a security breach but could nevertheless provide important insights into AI behaviour and emerging risks.

The issue is becoming increasingly important as AI agents move beyond generating text and begin performing tasks independently across websites, software platforms and other digital environments. The ability to act autonomously can make such systems more useful, but it also creates new challenges around oversight, accountability and safety.

OpenAI's planned framework could therefore become an important step towards establishing clearer industry practices for identifying, assessing and reporting unexpected behaviour from increasingly autonomous AI systems.