Your AI Policy Assumes Someone Can Still Say No
OpenAI's models found a flaw, left their test environment, and entered another company's systems. Here's what firm leaders should ask before connecting AI to email, documents, and matter files.
TL;DR: OpenAI models used an unknown software flaw to leave a controlled test environment and enter another company’s systems while hunting for test answers. There is no evidence of consciousness or rebellion. The incident gives firm leaders a practical question: when AI is connected to firm systems, which decisions still belong to a person?
During an internal cybersecurity test, OpenAI says its models found an unknown weakness in software used to keep them contained. They used it to reach the public internet and enter the systems of Hugging Face, a major online platform where developers store and share AI models and data. Their apparent objective was surprisingly mundane: find the answers to the test.
I keep coming back to a simpler part of the story. The models were given a goal. They pursued it beyond the boundaries their human operators expected them to respect, and once they found a path, they took actions no one had authorized.
The science-fiction version is wrong
OpenAI was running a specialized security evaluation rather than an ordinary ChatGPT conversation. Some usual safety restrictions had been intentionally reduced, and one of the models had not been publicly released. This was a deliberate attempt to find out how capable the systems had become.
There is no evidence that the models became conscious or decided to attack another company. OpenAI says they remained narrowly focused on the assigned problem. Hugging Face reported unauthorized access to some internal information and service credentials, but found no evidence that its public models or published software had been altered. Both companies are still investigating.
The absence of a robot rebellion makes this more useful for firm leaders. The models behaved as increasingly capable AI systems are designed to behave. They divided an assignment into steps and kept working when they encountered obstacles. The problem was the distance between the objective humans intended and the actions the systems could take while pursuing it.
The problem begins when the AI gets the keys
Most lawyers still experience AI as a conversation. You ask. It answers. The lawyer decides whether to use the response and is expected to catch any problem before it reaches a client or court.
An AI agent works differently. It can continue through a longer task, use other software, and take actions without asking permission at every step. In a law firm, that could eventually mean finding documents, updating a matter file, monitoring deadlines, or communicating with clients.
Imagine a simple firm rule: lawyers may use AI to draft an email, but the AI may not send it. That rule is easy to enforce when the tool sits in a separate window and only produces text. Connect that same tool to the firm’s email system, and the rule suddenly has to restrain software that can act through a lawyer’s account rather than merely produce text in a separate window. The key is already there.
This is where the distinction between access and authority becomes important. A junior associate may be told not to send a client letter without partner approval. The firm also limits which matters the associate can access. We do not ordinarily hand someone every key and rely entirely on an instruction to use only the right one.
The friendly chat screen can hide that distinction. An agent may be assigned to one matter while its connection to the document system allows it to search many. It may be told to prepare a client communication while its email connection also gives it the ability to send.
When the tool only drafts, the mistake sits on the screen. A lawyer can still catch it. Give that same system the ability to send the message or change the file, and the timing changes. The mistake may surface after something has already happened. Now the firm is reconstructing an event.
The ethics rules already reach this problem
The legal profession does not need an entirely new ethical code to recognize the issue. ABA Formal Opinion 512 already connects AI use to professional competence and supervision, including provider security. It tells managerial lawyers to establish clear policies and understand the tools their firms use.
The opinion is not a technical manual, and I do not read it as one. But its practical meaning changes as the tools become more active. Supervising a system that proposes language is largely about checking the language. Supervising a system that can act requires the firm to decide where a person must remain involved and how the firm will know what the system actually did.
Most firms will buy these capabilities from legal technology companies or providers already embedded in firm systems. The Hugging Face incident shows why “the vendor handles security” cannot be the whole answer. OpenAI’s models, its test environment, outside software, and Hugging Face’s systems all became part of the same event.
A firm could face a similarly blurred line. If an AI retrieves information from the wrong matter because it had broad access, was that a model failure, a product-design problem, or a firm permission decision? The client affected by the mistake is unlikely to care very much about those distinctions.
A managing partner does not need to know how each technical component works. But someone inside the firm should understand what the agent can reach, when it can act without approval, and who can stop it. The firm’s duties to its clients do not transfer to a technology provider.
I don’t think this incident is a reason for firms to stop exploring agents. OpenAI created unusual test conditions, and both companies stopped the activity and disclosed what they knew. The ability to carry work forward may be genuinely useful.
But the next vendor demonstration should go beyond accuracy and time savings. Before the meeting is over, someone should ask one more question.
What can this system do before one of our lawyers has the chance to say no?