Does FP8 change a model's answers? One flip in 120

with Anagha Langhe · self-hosted LLM on one NVIDIA L4

In half a minute

Anagha Langhe runs a language model herself, on a single NVIDIA L4 card, and wanted to know if switching it to FP8 costs accuracy. On 120 questions the two builds scored 72 and 71. That alone says little, so we counted the flips: one question changed, and it went bf16's way. 49 answers were identical word for word, and when both builds were wrong they gave the same wrong answer 42 times out of 48. Anagha ran an open comparison script on her own two runs, got exactly the same figures, and drew her own conclusion: the mistakes come from retrieval, not from quantization.

The question

Running a model in FP8 instead of bf16 roughly halves the memory it needs, which is the whole reason to do it on one GPU. The worry is simple: does the smaller format quietly make answers worse?

Anagha measured it directly. She asked the same 120 questions to both builds and wrote it up on LinkedIn: 72 correct in bf16, 71 in FP8, an exact McNemar test with p = 1.0. In the same post she shared another number from her setup, the prefix-cache hit rate going from 12% to 26%.

72 against 71 looks like "no difference". With 120 questions, though, a test like this can only rule out a big drop. A loss of three or four points would probably still not show up as significant. So the interesting question is not the p-value but what actually happened to each question.

Who did what

Anagha built the setup, ran both builds on the same 120 questions, graded the answers, and later ran the comparison on her own files.
We (Vadym Chernets) asked her how many questions flipped each way, then added a small compare mode to an open script we keep for checking model agreement, and computed the interval and the planning numbers from her summary.
Anagha's run of that script on her full data came back with the same numbers, plus one we could not see from the summary: how often the two builds were wrong in the same way.

What the numbers look like

Each of the 120 questions lands in one of four boxes. Only one sits off the diagonal. Most of the shared mistakes are the very same answer.

1 of 120questions flipped between the two builds, in bf16's favour

49answers came back word for word identical

42 of 48shared mistakes were the same wrong answer

Measure	Value
Accuracy, bf16	72 / 120
Accuracy, FP8	71 / 120
Flips: only bf16 right / only FP8 right	1 / 0
Exact McNemar test	p = 1.0
Difference bf16 − FP8	+0.8 points, 95% interval about −2.3 to +4.6
Identical answers, word for word	49 of 120
Both wrong, with the same wrong answer	42 of 48

How to read it. One flip is fully consistent with FP8 costing nothing. It is not proof that it costs nothing: a drop of a couple of points still fits inside the interval. With fewer than six flips an exact McNemar test cannot reach p < 0.05 whichever way they go, so here the useful numbers are the flips themselves and the interval. To see a one-point drop you would need roughly 800 questions, even if every flip went the same way.

The 42 out of 48 is the part that surprised us. If the FP8 build were simply noisier, its mistakes would scatter. Instead the two builds stumble on the same questions and land on the same wrong answer. In Anagha's words, that "fits my read that the errors come from retrieval, not quantization": when the retrieved context is bad, both builds read the same bad context and say the same thing.

Try it on your own runs

If you have two runs on the same questions as JSONL (a question id, the answer, and either a correct field or a gold file), the comparison is one command. It is part of the open repository memory-agreement-check, MIT licence, no dependencies:

git clone https://github.com/vadimchernets/memory-agreement-check
cd memory-agreement-check
python -m memcheck compare bf16.jsonl fp8.jsonl --label-a bf16 --label-b FP8

One line per question looks like this:

{"qid": "q001", "answer": "Paris", "correct": true, "category": "factual"}

It prints the 2×2 table, the exact McNemar test, the interval for the accuracy difference, how many answers are identical, how often the two runs agree and are both wrong, and a split by category if your questions have one. --md report.md saves the report.

What this shows, and what it does not

120 questions is enough to see that FP8 did not wreck this model. It is not enough to promise there is no drop of one or two points.
The test treats questions as independent. If many of them come from one document or one template, the real sample is smaller than 120.
"Errors come from retrieval" is Anagha's reading of her own system. The 42 of 48 supports it; checking it directly would mean answering the same questions once with a good context and once with a bad one.

What comes next

Anagha agreed to let her FP8 vs bf16 check go into the repository as a worked example, credited to her, so the next person comparing two builds has a real case to start from, not only a synthetic one.

Timeline

11 November 2026Anagha posts her FP8 check on LinkedIn: 71 vs 72 of 120, McNemar p = 1.0. We comment that the flip counts say more than the p-value. She writes back, and we ask how many questions flipped each way.
11 November 2026Her answer: just one, bf16 right and FP8 wrong, none the other way, and 49 of the 120 answers word for word identical. Overnight the open script gets its compare mode, and we send her the interval and the planning numbers computed from her summary.
12 November 2026Anagha runs it on her own two runs: the numbers match exactly, and when both builds were wrong they gave the same wrong answer on 42 of 48. She agrees to the worked example.

← All collaborations · Propose a collaboration