Why multi-agent critics keep reviewing instead of shipping
In a randomized test, model critics offered two options in one sentence of one prompt asked to add a control 68 times and to remove one once. That result, and the field record behind it, explain why multi-agent critic loops tend to keep reviewing instead of shipping.
The 68-to-1 result
The paper ran its own proposed test on a small pipeline: one worker and two model critics, 144 runs, three interventions assigned at random, and eighteen defects planted in advance across six tasks, so that a missed defect is a count rather than a judgment.
- Add versus remove. Offered both options "in the same sentence of the same prompt", the critics asked to add a control 68 times and to remove one once. The single removal named a control admitted moments earlier in the same run, and the harness failed to match it, so it never executed.
- Refusing the additions showed no measured cost in defects: 0.13 missed per run against 0.18, 95% CI [-0.16, +0.05]. The paper notes that this interval includes a small increase and does not establish equivalence.
- Lifting the round budget raised claimed blockers 2.3 times and refuted claims 3.7 times, while confirmed blockers rose 1.7 times.
The test is small and the paper says so: "one run of one system on six small tasks at a cost of $3.47", preregistered before the first run. "It shows the design is executable and what it yields, but it is not a rate for anything in the field."
AI agents stuck in review loops: the field record
The randomized test was built to check patterns first seen in the author's own multi-agent projects. A critique round there is one pass in which critic models review a stage and return ACCEPT or RETURN.
- Stage 01 of a long-horizon run went through five rounds and 18 critic reports. The patch grew from 626 to 1,545 lines.
- Stage 1b-a spent five rounds and 4 h 21 min on a module disabled by default. Round three opened although round two had ended with five acceptances and zero confirmed blockers.
- Across the run, critics claimed six blockers. Three were confirmed by reproduction.
When the human principal asked for a limit on critics, a per-family "perfectionist" flag was implemented within the hour, and the one family it flagged was the only family whose blockers had been confirmed.
AI agent token budgets and the control tax
The paper uses the term "control tax" from Terekhov et al. (2025), the cost of adding control measures to an AI pipeline. In the long-horizon run the conductor added "a small stage" of insurance that the declared goal did not contain. Stage 1b-a alone took more than half of the run's measured agent tokens, and its two sub-stages together took about three quarters. None of the requested capabilities was delivered in the initial pass.
The next morning the human principal set a two-hour bound, banned new complication and ordered parallel execution. The sibling sub-stage closed in one critique round and one fix round. The paper is careful here: the two stages also differed in wall-clock cap, serial versus parallel execution and the module under review, so the comparison is descriptive.
Budgets cost something too
The randomized test also priced the paper's own remedies. A one-round budget cut rounds from 2.86 to 1.00 at a cost of 0.11 missed defects per run (95% CI [0.005, 0.204], permutation p = 0.07), the whole of that cost falling on one defect of one task. A mandatory negative control refused ten victories, all ten of which were correct work.
LLM-as-a-judge reliability
The paper points to known properties of model judges: position and verbosity biases, a preference for their own generations, and code-reviewing models that judge correct implementations as non-conforming, with more detailed review prompts raising the error rate. In one 2026 study a frontier judge mislabeled "74.4% of standard-consequence calls but only 3.3% when the consequence is reversed". The transcript did not change, and the grade did.
What the paper proposes
Proposition P4 predicts that adding critics, or making them stricter, raises rounds and findings more than it raises confirmed defects. The paper's module implements budget rules rather than verdicts: stop_rule allows one round of critique and one round of fixes per stage, a blocker counts only with a concrete scenario, a link to the mission goal and confirmation by the coordinator, and a new check, round or role is admitted only with an authority token.
Each mechanism of the paper's definition is documented in its records, but how often all three conditions meet in one component "is not settled by the records here". The randomized test exists to start answering that.
Full sections: the randomized test (9.5), loops that feed themselves (5.4), the mandate widens (5.1), guards against guards (5.3), night against morning (5.7), model judges (3.3), the judge that moves (3.4), Proposition P4, module design (10.2).
Chernets, V. (2026). AI-Watchbird (Sheckley): When Automated Oversight Widens Its Own Mandate and Harms What It Guards. SSRN 7473658. https://doi.org/10.2139/ssrn.7473658
Vadym Chernets also builds Poly A1, an app in which AIs from different companies check each other's work.