Does FP8 change a model's answers? One flip in 120
with Anagha Langhe · self-hosted LLM on one NVIDIA L4 · DocumentSyncAI

In half a minute
Anagha Langhe runs her own language model on a single graphics card, an NVIDIA L4. She wanted to know if storing the model's numbers in a lighter format makes it answer worse. That format, FP8, cuts weight memory by about 43% (14.3 to 8.2 GiB) compared with the usual bf16, so it's tempting.
She asked both builds the same 120 questions: 72 right for bf16, 71 for FP8. Only one answer flipped. And when both were wrong, they gave the same wrong answer 42 times out of 48.
Anagha ran our open comparison script on her own data and got the same numbers. Her conclusion: the mistakes come from retrieval, the step where her system looks up the documents it answers from, and not from the lighter format.
The question
Anagha's system answers questions about documents. It first looks up the right passages, then the model answers from them. The code is open: DocumentSyncAI.
FP8 cut weight memory by about 43% (14.3 to 8.2 GiB), not quite half, since the embedding and output layers stay in bf16. On a single GPU, that's the whole point. The worry is just as simple: does the smaller format quietly make the answers worse?
Anagha measured it. She put the same 120 questions to both builds and posted the result on LinkedIn: 72 right in bf16, 71 in FP8, and an exact McNemar test with p = 1.0 (a test of whether a difference is more than luck; 1.0 means no sign of one). In the same post she shared one more number from her setup: the prefix-cache hit rate, how often the server could reuse work it had already done, went from 12% to 26%.
72 against 71 looks like "no difference". But with 120 questions, a test like this only catches a big drop. Losing three or four points would probably still not show up. So what matters is not the p-value but what happened to each question.
Who did what
- Anagha built the setup, ran both builds on the same 120 questions, graded the answers, and later ran the comparison on her own files.
- We (Vadym Chernets) asked her how many questions flipped each way. Then we added a small
comparemode to an open script we keep for checking whether models agree, and worked out the interval and the planning numbers from her summary. - When Anagha ran that script on her full data, she got the same numbers, plus one we couldn't see from the summary: how often the two builds were wrong in the same way.
What the numbers look like
| Measure | Value |
|---|---|
| Accuracy, bf16 | 72 / 120 |
| Accuracy, FP8 | 71 / 120 |
| Flips: only bf16 right / only FP8 right | 1 / 0 |
| Exact McNemar test | p = 1.0 |
| Difference bf16 − FP8 | +0.8 points, 95% interval about −2.3 to +4.6 |
| Identical answers, word for word | 49 of 120 |
| Both wrong, with the same wrong answer | 42 of 48 |
How to read this. One flip fits FP8 costing nothing at all. It doesn't prove it, though: a drop of a couple of points still fits inside the interval, the range where the real difference most likely lies.
With fewer than six flips, an exact McNemar test can't reach p < 0.05, whichever way they go. So here the flips and the interval tell you more than the p-value.
And to see a one-point drop, you'd need about 800 questions, even if every flip went the same way.
The 42 out of 48 surprised us. If FP8 just made the model noisier, its mistakes would scatter. Instead, both builds trip on the same questions and land on the same wrong answer.
Anagha put it this way: it "fits my read that the errors come from retrieval, not quantization". When the system pulls up the wrong passages, both builds read the same wrong text and say the same thing. Both builds also saw exactly the same retrieved passages for every question, which is why the shared mistakes line up so well.
Try it on your own runs
Got two runs on the same questions as JSONL? Each line needs a question id, the answer, and either a correct field or a gold file. Then the comparison is one command. It's part of the open repository memory-agreement-check, MIT licence, no dependencies:
git clone https://github.com/vadimchernets/memory-agreement-check
cd memory-agreement-check
python -m memcheck compare bf16.jsonl fp8.jsonl --label-a bf16 --label-b FP8One line per question looks like this:
{"qid": "q001", "answer": "Paris", "correct": true, "category": "factual"}It prints the 2×2 table, the exact McNemar test and the interval for the accuracy difference. It also counts identical answers and how often both runs are wrong with the same answer, and splits it all by category if your questions have one. --md report.md saves the report.
What this shows, and what it does not
- 120 questions is enough to see that FP8 didn't wreck this model. It's not enough to promise there's no drop of one or two points.
- The test treats every question as independent. If many come from one document or one template, the real sample is smaller than 120.
- "Errors come from retrieval" is Anagha's reading of her own system. The 42 of 48 backs it up. To check it head-on, you'd answer the same questions once with the right passages and once with wrong ones.
What comes next
Anagha agreed to let her FP8 vs bf16 check go into the repository as a worked example, with her name on it. The next person comparing two builds will have a real case to start from, not just a made-up one.
Timeline
- Anagha posts her FP8 check on LinkedIn: 71 vs 72 of 120, McNemar p = 1.0. We comment that the flip counts say more than the p-value. She writes back, and we ask how many questions flipped each way.
-
Her answer: just one, bf16 right and FP8 wrong, none the other way, and 49 of the 120 answers word for word identical. Overnight the open script gets its
comparemode, and we send her the interval and the planning numbers computed from her summary. - Anagha runs it on her own two runs: the numbers match exactly, and when both builds were wrong they gave the same wrong answer on 42 of 48. She agrees to the worked example.