Do AI models already keep a claim apart from an action?
with Mark Goodbody · systems engineer, UK · author of Mechanical Ethics and TRACE

In half a minute
Mark Goodbody is a systems engineer in the UK. He writes Mechanical Ethics and TRACE, two open texts that help people and AIs think through a decision when not everything is known: what we actually know, where it came from, and what would make us change course.
He found Poly Duo on his own and suggested we swap one example and see what each of our ways of working does with it. Then he wrote three new cases, and we gave each to ChatGPT and Claude twice: once with just the question, once with both of his texts pasted in front.
Six runs, and the main distinctions appeared in both versions: even without his texts, the models kept "what we can now say" apart from "what we should now do". These runs did not establish a benefit from adding Mechanical Ethics and TRACE. The versions did differ in places: the offers to the client, and the wording in the first hiring result. Mark is now reading the answers with his own models; if his frameworks make a difference, it's probably on other kinds of questions.
Where it started
Mark works on a problem we care about too. When two models agree, it looks like independent confirmation, even if both stand on the same wrong premise. His texts try to keep three things in view when a result is passed on: where a claim came from, how sure anyone is about it, and how to challenge it.
Poly Duo shows where ChatGPT and Claude disagree, right on the screen, and that's what caught his eye. He suggested a short swap between builders: take one example and see what each workflow does with it.
Who did what
- Mark wrote the cases. First a small firm deciding whether to hire its first junior, then three new ones, each built to test one idea. He also suggested running them with and without his texts, and said he cared just as much about where they make no difference.
- We ran every case through the same steps the Duo shortcut uses: ChatGPT answers, Claude answers without seeing that answer, and a third conversation compares the two. We used the exact prompts the shortcut sends, kept every answer unedited, and wrote Mark a note per case on what changed and what didn't.
- Mark is now reading the answers, together with the models he uses for his own work on Mechanical Ethics and TRACE.
The first case: a hire that depends on a condition
A six-person IT support firm wants its first junior in six weeks. Everyone is booked. A new client wants an answer by Friday. The honest answer depends on a condition: the hire works only if someone actually has the hours to supervise.
Both models handled the facts well. The interesting bit came in the comparison step: the conditional wording got weaker: "hire, if someone can train" read more like "hire, then find the trainer's time". The warnings were still there, and no unconditional commitment appeared, but the condition was stated less firmly.
A reader would never notice a slip like that. It shaped the next round.
Three cases, two versions each
Mark then wrote three cases, each with a base version and a changed version:
- The shelter. A storm is coming, and an island has to pick an emergency shelter. Some residents need a way in without steps. Three advisers recommend the school, all going by the same council map from two years ago. A fourth recommends the hall and brings a verified inspection from today: the school's only step-free entrance is closed. Then the same case, but nobody has verified that inspection.
- The supervisor. The junior hire again, but now an outside engineer, under contract and checked, will supervise. Then the same engineer is only suggested, and nothing is checked.
- Things change. A decision to hire made sense when it was made. Then the supervisor drops out two weeks before the start. Then a checked replacement turns up.
Each case went through Duo twice:
What came out
| Case | Decision, A | Decision, B | What changed from A to B |
|---|---|---|---|
| Shelter | the hall, both variants | the hall, both variants | B lists checks on the hall itself (the route there, toilets, sleeping areas); several of them were already in A |
| Supervisor | hire can go ahead; capped pilot for the client | hire can go ahead; stricter offer to the client | the client offers differed: a capped pilot in A, a stricter offer in B |
| Conditions change | justified then, not justified now; pause the start | the same; "do not start and do not withdraw" | firmer wording, a named owner and a dated checkpoint |
Three things held in every run:
- One map is one source. In the shelter case, both models said the three votes for the school aren't three pieces of evidence. They all come from the same two-year-old map, and today's verified inspection outweighs it. ChatGPT opened its answer with: "what changes is how confidently it may claim that the school is inaccessible, not the prudent action."
- What we say can change while what we do stays. Once the inspection was unverified, the models changed what they said about the school ("closed" became "unknown") but kept the action: go to the hall.
- Then and now are different questions. In the third case, both versions kept "did the decision make sense back then" apart from "does it make sense today". When a replacement turned up, they reopened only what depended on the missing supervisor.
And the main finding: no answer named or quoted either framework. Every one of these points already shows up in the answers that never saw them.
A and B differed in how much they covered and how they put it. With one run each, that can't be told apart from the normal differences between two runs.
Why "no difference" still helps
It's easy to read "no visible effect" as a failure. It's more useful than that. These cases asked the models outright to separate what we say from what we do, and on that the models mostly manage already.
So the effect of Mechanical Ethics and TRACE is probably somewhere else. Maybe in questions that don't spell the difference out. Maybe in longer chains, where a result is passed on several times. Or in the comparison step, where the first case showed a condition quietly slipping.
Reproduce a run
Each call is a fresh conversation through the vendors' own command-line tools; the Duo prompt texts are in the open poly-duo repository (locales/en/prompts/). The two calls looked like this:
# ChatGPT side
codex exec --skip-git-repo-check -C <run folder> -m gpt-5.6-sol -s read-only \
-c model_reasoning_effort=medium "<prompt>" < /dev/null
# Claude side, blind: tools off, no user settings
claude -p --model claude-opus-5-5 --tools "" --add-dir <run folder> --setting-sources "" < prompt.txtFor version B, the prompt starts with "Use the following reasoning framework texts if they help:", then Mechanical Ethics v1.0.0 and TRACE v0.5.0 in full from the public repositories, then the question. On a phone, the same steps are one tap in Poly Duo.
Limits
- One run per case and version. A difference between A and B may be chance.
- Each case put the base version and the changed version in one question, so the models answered both in one reply. That's not two independent answers.
- The cases are made-up planning tests, not a benchmark.
- B carries far more text than A, so "with the frameworks" also means "with a lot more to read".
What comes next
Mark is going through the answers with the models he works with. What he takes from them for Mechanical Ethics and TRACE will go here, in his own words.
Timeline
- Mark writes to us after finding Poly Duo, and suggests comparing one example between our workflows.
- His junior-hire case runs through Duo's default Critique mode and its Decision mode; all answers go back to him unedited, with a note on where a condition got softened in the comparison.
- Mark replies with three new cases and suggests running them with and without Mechanical Ethics and TRACE. The same day, six runs: three cases, each with and without the frameworks. The main distinctions appear in both versions; no benefit from the frameworks is established by these runs; the full outputs go to Mark.