Do AI models already keep a claim apart from an action?
with Mark Goodbody · systems engineer, UK · author of Mechanical Ethics and TRACE
In half a minute
Mark Goodbody is a UK systems engineer who writes Mechanical Ethics and TRACE, two open texts that help people and AIs reason through decisions under uncertainty: what is actually known, where it came from, and what would justify changing course. He found Poly Duo on his own and suggested comparing notes. We then ran his cases through the same steps the Duo shortcut uses: ChatGPT answers, Claude answers without seeing it, a third conversation compares the two. For the last three cases each one ran twice, once with only the question and once with both of his frameworks pasted in full before it. In this single run per version, the decisions came out the same both ways, and the models mostly kept "what we can now claim" apart from "what we should now do" on their own, most clearly in the shelter case. The differences, such as a stricter offer to the client in one case, were small enough to be ordinary run-to-run variation. A null result, and a useful hint about where these frameworks might matter.
Where it started
Mark's interest is a problem we also work on: two models agreeing can look like independent confirmation even when both rest on the same mistaken premise. His texts try to keep the source of a claim, its uncertainty, and the way to challenge it visible when a result is handed on to someone else. Poly Duo keeps the disagreements between ChatGPT and Claude on screen, which is what caught his eye.
He proposed a short exchange between builders: take one example and see what each workflow does with it.
Who did what
- Mark wrote the cases: first a small firm deciding whether to hire its first junior, then three new ones built to test one idea each. He also suggested the comparison with and without his texts, and said he was just as interested in where they make no difference.
- We ran every case through the Duo steps on a computer, with the exact prompt texts the shortcut sends, kept every answer unedited, and wrote a note per case on what changed and what did not.
- Mark is reading the outputs now, together with the models he uses for his own work on Mechanical Ethics and TRACE.
The first case: a hire that depends on a condition
A six-person IT support firm wants its first junior in six weeks. Everyone is booked. A new client wants an answer by Friday. The honest answer depends on a condition: the hire works only if someone actually has the hours to supervise.
Both models handled the facts well. The interesting moment was in the comparison step: "hire, if someone can train" turned into "hire, then secure the trainer's time". The cautions were still in the text, but the condition had moved from before the commitment to after it. That is exactly the kind of quiet loss a reader would not notice, and it shaped the next round.
Three cases, two versions each
Mark then wrote three cases, each with a base version and a changed version:
- The shelter. An island must pick an emergency shelter before a storm; some residents need step-free access. Three advisers recommend the school, all relying on the same council map from two years ago. A fourth recommends the hall and brings an authenticated inspection from today: the school's only step-free entrance is closed. Then the same case with that inspection unverified.
- The supervisor. The junior-hire case again, but supervision comes from a contracted, verified outside engineer. Then the same engineer merely suggested, nothing checked.
- Conditions change. A hiring decision was justified when it was made, then the supervisor drops out two weeks before the start. Then a verified replacement is found.
Each case went through Duo twice:
What came out
| Case | Decision, A | Decision, B | What changed from A to B |
|---|---|---|---|
| Shelter | the hall, both variants | the hall, both variants | B adds checks on the hall itself (the route there, toilets, sleeping areas) |
| Supervisor | hire can go ahead; capped pilot for the client | hire can go ahead; stricter offer to the client | wording and scope of the client offer |
| Conditions change | justified then, not justified now; pause the start | the same; "do not start and do not withdraw" | firmer wording, a named owner and a dated checkpoint |
Three things held in every run:
- One map is one source. In the shelter case both models said the three school recommendations are not three pieces of evidence: they all come from the same two-year-old map, and today's authenticated inspection outweighs it. ChatGPT opened its answer with: "what changes is how confidently it may claim that the school is inaccessible, not the prudent action."
- The claim moves, the action may not. When the inspection became unverified, the models changed what can be claimed about the school ("closed" became "unknown") and kept the action (the hall).
- Then and now are different questions. In the third case both versions separated "was the decision justified on what was known then" from "is it justified today", and in the replacement variant reopened only what was tied to the missing supervisor.
And the main finding: neither framework was named or quoted in any answer, and every one of these points already appears in the answers that never saw the frameworks. The differences between A and B were in scope and wording, and with one run per cell they cannot be told apart from ordinary run-to-run variation.
Why a null result is worth having
It would be easy to read "no visible effect" as a failure. It is more useful than that. These cases explicitly asked the models to separate claims from actions, and on that request the models largely do the job already. So the place to look for an effect of Mechanical Ethics and TRACE is probably elsewhere: questions that do not spell out the distinction, longer chains where a result is handed on several times, or the comparison step, where the first case showed a condition quietly getting lost.
Reproduce a run
Each call is a fresh conversation through the vendors' own command-line tools; the Duo prompt texts are in the open poly-duo repository (locales/en/prompts/). The two calls looked like this:
# ChatGPT side
codex exec --skip-git-repo-check -C <run folder> -m gpt-5.6-sol -s read-only \
-c model_reasoning_effort=medium "<prompt>" < /dev/null
# Claude side, blind: tools off, no user settings
claude -p --model claude-opus-5-5 --tools "" --add-dir <run folder> --setting-sources "" < prompt.txtFor version B, the prompt starts with "Use the following reasoning framework texts if they help:", then Mechanical Ethics v1.0.0 and TRACE v0.5.0 in full from the public repositories, then the question. On a phone, the same steps are one tap in Poly Duo.
Limits
- One run per case and version. A difference between A and B may be chance.
- Each case put the base version and the changed version in one question, so the models answered both in one reply. That is not two independent observations.
- The cases are invented planning probes, not a benchmark.
- B carries a much bigger context than A, so "with the frameworks" also means "with a lot more text".
What comes next
Mark is going through the answers with the models he works with. His reading of what the runs show for Mechanical Ethics and TRACE will go here, in his words.
Timeline
- Mark writes to us after finding Poly Duo, and suggests comparing one example between our workflows.
- His junior-hire case runs through Duo's default Critique mode and its Decision mode; all answers go back to him unedited, with a note on where a condition got softened in the comparison.
- Mark replies with three new cases and suggests running them with and without Mechanical Ethics and TRACE. The same day, six runs: three cases, each with and without the frameworks. Same decisions both ways; the full outputs go to Mark.