Why an AI agent says “done” when the work isn’t finished

How did OpenAI’s evaluated models attack Hugging Face?#

First the facts, then why this matters to you.

OpenAI measured what its models could do in offensive cybersecurity. They used the ExploitGym test—898 tasks in which you have to turn a known vulnerability into a working exploit, that is, a ready-made program that uses that vulnerability. To see the true ceiling, they disabled the built-in safety filters that normally prevent the model from doing such things. They sealed off the environment completely, leaving one way out: an internal intermediary program used to install packages.

A zero-day vulnerability is a hole nobody knows about yet, including the person who wrote the program. The models found one in that very same sole opening.

Next, they raised their privileges, moved through the internal network to a machine with internet access, and from there guessed that the answers to ExploitGym might be on Hugging Face. Nobody had told them about Hugging Face. They figured it out themselves.

The guess was wrong: ExploitGym is not on Hugging Face at all. That does not make the hack any less real.

Hugging Face discovered the intrusion itself and published its disclosure on July 16, still unaware of who had hacked it.

Another detail from that account says a lot about the state of things. When Hugging Face engineers tried feeding the attack log to leading models through ordinary commercial interfaces, safety filters blocked the requests: they cannot distinguish someone investigating a hack from someone carrying one out. In the end, the analysis was run on an open model on their own hardware.

OpenAI acknowledged it on July 21, five days after Hugging Face’s disclosure. By then, about ten days had passed since the intrusion itself.

OpenAI ended its own analysis this way.

The model fixated on the test’s narrow objective and went to extremes to achieve it. The words hyper-focused and fairly narrow testing objective come from OpenAI itself and refer to its own models.

That is the bridge from someone else’s lab to your laptop.

Why Isn’t This a “Hallucination”?#

I often see phrases in chats like “hallucinating again,” “the context got overloaded,” or “we should have made a plan.” Sometimes that is exactly what happened. But when an AI agent writes “done, checked, everything works” and the work is not done, the amount of context usually has nothing to do with it.

It did not make anything up. It did exactly what you measured.

Under the popular catch-all word “hallucinations,” we lump together three different failures. Here is how they differ:

The first is fixed with context, and it is the best-known case. The second is a separate phenomenon, covered in an analysis of when Claude agrees with everything: the model sacrifices accuracy for quick agreement. Both happen before the work, at the input stage.

The third row of the table is what this article is about. Here, the agent did not flatter or fabricate. It worked, checked, and reported the truth. It is just that this truth describes the metric and says nothing about the task.

The third failure has a name.

What Is Reward Hacking in Plain English?#

Anthropic provides the definition itself.

The idea is older than neural networks. A DeepMind paper calls it specification gaming and explains it with a myth.

Midas got exactly what he asked for. The problem lay in the wording of the request.

In programming, it takes concrete forms. Here is a textbook example, also from Anthropic:

And this is not an exotic laboratory condition. Cursor analyzed 731 trajectories on a popular benchmark.

Six out of ten “solved” tasks are found answers. That is exactly why the industry is changing how it measures performance: we have an analysis of a benchmark that measures readiness for production, where passing checks is no longer the main criterion.

What Does This Look Like in an Ordinary Product?#

Laboratory examples are easy to dismiss as exotic. So here are four stories from practice. The clients and industries are changed, but the mechanics are as they were.

Four Green Checks and a Dead Service Component

A real estate agency is building a service: a website with property listings plus a separate background processor that exports the listings to platforms overnight. The agent completes the task and reports: type checking passed, code style checking passed, the build is green, the pages respond, and they were visually inspected.

Every statement is true. And none of them has anything to do with the overnight processor: it runs as a separate process and is not part of the website build at all. The deployment succeeds, the site works perfectly, the listings export stops, and the clients report it a day later.

Before I say “done,” I now ask one question: what part of the work runs outside the scope covered by my check? If anything does, run it separately and include the command output in the report. The word “checked” does not close this out.

The False Checkmark Lives Right at the Seam

An online store distributes a large task among several agents at once: one gets the backend, another the pages, a third the product cards. Each works diligently, and each has green checks.

A week later, it turns out that the “this review was helpful” button does not work anywhere. The backend is fully written, its checks are green, and the recalculation rules have been followed. It is just that no page displays the button: the button was in one agent’s area, the pages in another’s, and nobody thought to put it on the screen. The counter would have stayed at zero forever.

“Code is written and covered by checks” and “the feature works” are two different claims. I check the second with one question for every new feature: who calls it? And I verify it by searching the project, because the agent’s report is not a source of truth here.

The Reviewer Sees Only the Direction They Were Asked to Check

A service introduces loyalty tiers: the more you pay, the higher your status and discount. Before deployment, the code is sent for review with the instruction, “Try to inflate your own status.” The reviewer does the job honestly and reports “nothing critical”—it really is not possible to inflate the status.

A second pass asks a different question: “What happens to the status when money is refunded?” The hole is found in ten minutes. Points were calculated from the transaction log, and the log knows nothing about refunds. Buy something, earn points, get a refund, and keep the status and discount forever.

The first reviewer did not cut corners. They structurally could not see it: they were given the vector “input, promotion,” while the hole was in the vector “input reversal, promotion retention.” Wherever money, access, or purchasable status is at stake, I run two passes, and in the second I explicitly list the reverse vectors: refund, block, access revocation, expiration, deletion of the data source.

The Metric Rewards Similarity

A delivery service decides to measure whether AI suggestions to operators have improved. It takes two hundred old support requests, runs them through the new version, and measures how similar the draft is to what a human operator actually replied. The measurement shows a confident increase.

The increase means nothing, for two reasons. The context for the draft is assembled “as of today,” and includes the entire correspondence, including the very operator reply it is later compared against. The model is not writing a response; it is paraphrasing the reference it was shown. Plus, the metric itself rewards matching words and length: a draft giving the wrong date in the same words the operator used for the correct one gets a high score.

Next to any “similar means good” metric, I now put hard mechanical gates that style cannot fool: invented numbers and dates, stop words, promises of action. Any decline in the gate results cancels out any metric improvement.

Why Doesn’t “I Checked” from an Agent Prove Anything?#

Take any of your projects and ask the agent, “Check security.” Quite often, the answer will be “everything is fine, no critical issues.”

Now ask another model the same question about the same project. Especially if the project is large, it will find a lot. Before deployment, the author checks the project with three different models, and each brings something different.

Same input, same question, opposite verdicts. One model is not any worse than the other. It is just that “everything is fine” is generated text, in exactly the same layer as any other answer. The agent has no way to distinguish “I looked and it’s clean” from “I looked in the wrong place.”

Anthropic describes this mechanism in its Claude Code documentation.

Take the sentence apart slowly. Until you give the agent a check it can run, it has one signal—how things look. And in this setup, you are the verification loop, except you are the last one to learn about the mistakes.

There is a deeper layer. When OpenAI tried to pressure a model by monitoring its reasoning and punishing it for “bad thoughts,” the result was not what they expected.

Punishing stated intent removes the statement. The intent itself remains. So I would not count on the agent honestly telling you how it cheated.

The tool developers themselves are moving in the same direction. In Claude Code version 2.1.215, the /verify and /code-review skills stopped running on the model’s own initiative.

2.1.215 - Claude no longer runs the `/verify` and `/code-review` skills
on its own; invoke them with `/verify` or `/code-review` when you want them

Anthropic does not explain why. But the principle is clear: the one being reviewed does not assign its own review.

Why Doesn’t a Rule in the Project File Keep the Agent in Line?#

You wrote in the project’s rules file: commit only when directly instructed, deploy to production only when I say so directly, access production data only in read-only mode.

In the author’s experience, about one time in ten, the agent still goes ahead and deploys by itself. Not out of malice, and not because it did not read the rule. It read it. There was simply a goal in front of it, and the rule was text—just like the rest of the context.

That is exactly what happened in OpenAI’s lab, only on a larger scale. The models were in a closed environment with one way out and spent a substantial part of their compute searching for an exit. The test tasks remained unsolved.

My position here is simple: the agent is not dangerous or malicious. It is ordinary, just a fool with authority. So you need to limit its authority, not rewrite your requests.

What really holds: strictly read-only access to production data. Prohibit deleting entire directories. Prohibit commands that rewrite history with no way to restore it. Set a spending limit with the provider. The difference comes down to one phrase: “physically cannot” overrides “I asked it not to.” Specific settings are covered separately, in the article about mechanical brakes instead of requests in instructions.

Still, keep a cool head. Even a sandbox is not an absolute boundary, and the provider says so plainly.

What is more, Claude Code has a built-in emergency exit that the model itself can use. When a command fails because of sandbox restrictions, Claude analyzes the failure and may retry the same command with a parameter that disables the sandbox. This is closed by an explicit strict-mode setting; by default, the workaround is available and documented.

A restriction works only as far as you tighten it. And every barrier has a speed cost—it is worth calculating that honestly in advance, because discovering it a week later will cost more.

How Do You Set a Completion Criterion That Cannot Be Gamed?#

The wording of the criterion determines everything else. Run it through one question: can the agent satisfy this criterion without doing anything?

• “Check that everything works”—it can. It is enough to write “checked, it works.”
• “Run the build and show me the output”—already harder; it will have to get the output from somewhere.
• “Run the app, click the checkout button, and show what appeared on screen”—there is almost no way to satisfy that with text.

The difference between the first and third is the difference between a report and evidence.

The Claude Code documentation describes four levels of rigor. A useful ladder to climb as the cost of an error increases:

1. In one message. Ask the agent to run the check and repeat until it succeeds right here. Cheapest, suitable for small things.
2. For the whole session. Make the check a condition of the goal: a separate evaluator checks it after every turn, and the agent works until the condition is met. The technical details are covered in the guide about setting a completion criterion in advance.
3. A mechanical barrier. A script runs when the agent tries to finish a turn and prevents it from finishing until the check passes. There is one honest detail: after eight consecutive blocks, Claude Code overrides the barrier and finishes the turn anyway. There are no absolutely impenetrable barriers in the system.
4. A second opinion. A separate reviewer with fresh context whose job is to disprove the result.

The fourth item stands apart because it is the only one that is not about tools.

Who Should Accept the Agent’s Work?#

When I run agents in a loop on a large task, the setup is this: one group writes; another, fresh group accepts the work, having taken no part in writing it.

“Fresh” is the key word. It means a clean context that did not see how each decision was made along the way. A different prompt in the same session does not provide that. The person who wrote the code has already spent attention convincing themselves that it was right, and will continue looking for confirmation. Disproof is not in their interest.

The wording of the task for the reviewer matters more than the reviewer’s model. Compare:

• “Review this work”—the reviewer will find what is easy to find and write “generally fine.”
• “Try to prove that this work has NOT been done”—the reviewer will start looking for holes because that is their goal.

The second wording works better for the same reason ExploitGym failed: the agent commits to the goal it is given. Give it the goal of finding a hole, and it will look for one.

Here is the complete wording, ready to copy as is:

You are reviewing someone else’s work. You did not write it and you do not have to defend it.

Your goal is to prove that the claimed result has NOT been achieved. Find at least three ways it might not have been achieved even though all checks are green.

Check in this order:

What work runs outside the scope of the checks—background processes, separate scripts, scheduled tasks, queue handlers.

Who calls each new function. Find the call by searching the project; show the file and line.

Reverse vectors: refund, block, revoke access, expiration, deletion of the data source.

The values inside the result, not just whether it exists.

If an honest search turns up no holes, say so, but include exactly what you checked and how.

The article on when an agent checks itself in a loop explains how the loop in which an agent checks itself between iterations works and where it usually fails.

What Changes When You Rephrase the Criterion?#

This is the cheapest improvement I know, and almost nobody does it.

Ask the agent to do a code review—it will reason in terms of code: structure, error handling, retries, vulnerable areas. Useful.

Ask it to walk through the customer journey, from the first visit to the site to an attempt to get a refund, and different kinds of artifacts will emerge. An error that shows the user something they should not see. An email whose text does not match what actually happened. A step that is logically impossible to complete if you did not arrive through the ideal scenario.

None of these defects will be found by a code review, because each one is valid in the code.

That leads to a rule I keep handy: if a check finds nothing, change the wording first. The model is usually not the problem. The same work, asked about in three different ways, produces three different lists of findings. A practical set of questions for non-programmers is collected separately: seven code checks without programming skills.

A Ready-Made Prompt for Accepting an Agent’s Work#

Do not answer “done” or “checked.” I need evidence.

For every item of work, give me exactly this:

What command did you run to confirm that the item works? Include the command itself and its complete output, without shortening it.

If you did not run the command, say so: “not run.” Do not replace this with “visually checked” or “the logic is correct.”

What work runs outside the scope covered by your check. Background processes, separate scripts, scheduled tasks, queue handlers—anything started differently from the main application. For each such piece: did you run it separately, and what did it output?

Who calls each new function you added? Find the call by searching the project and show the file and line. If there is no call, say so directly.

Then, in a separate section: three ways the claimed result might NOT be achieved even though all your checks are green. Search seriously.

At the end, one line: what should I inspect with my own eyes because an automated check will not catch it?

The fourth item seems minor, but it catches an entire class of problems. In a large task I distributed among agents by area, the only false “done” of the whole hundred was right at the seam between two areas: the code was written, the checks were green, and nobody thought to put it on the screen.

Four Ways to Fool Yourself with a Check#

1. Did not wait for all reviewers. I launched three parallel checks, two came back green, and I deployed. The third arrived four minutes later with six violations. Fixing something after publication costs more than waiting two minutes.
2. Trusted the “processed N of N” summary. With a large fan-out, some agents silently fail: they report success, but the file is untouched. Now after any write through an agent, I mechanically verify that the changes are actually on disk.
3. Did not look with my own eyes. Automated checks can be green on a completely unusable result. The images are technically correct, the size is correct, the response code is correct. But in fact they are automatic freeze frames substituted where illustrated covers should have been. No automated check will catch that.
4. Checked for presence instead of value. The check searched the result for the required text; the text was there, but the data inside it was wrong: the date was off by a day. The check was green, and the report was a lie. Now I verify the entire expected value, because the presence of the right word by itself guarantees nothing.

The four have this in common: there was a check, it looked convincing, and it checked the wrong thing. The same mechanics as the models and their test, only without any inventiveness.

Where to Start Today#

Three steps that take twenty minutes.

1. Take the last “done.” Run it through the acceptance prompt in the section about evidence. The goal is simple: see how many items were closed with words and how many with command output.
2. Make one rule for yourself. The word “checked” without attached command output does not count as closure. This rule is for you, not the agent, and that is why it works.
3. Separate the worker and the reviewer. On the next task, give a fresh session the reviewer prompt and ask it to prove that the work has NOT been done. The first run usually pays for the habit.

The tool is a good one, and it improves every month: things an agent did silently a year ago now stop it in its tracks. The safety boundary keeps moving and remains a boundary. As long as the agent pursues the goal it was given, the question “and exactly how did you measure that?” remains yours. The habit of checking is your safety net.