When an agent's memory is squeezed into short codes, do two AIs agree on the same wrong answer?

with Mohan Bheemadolu · AI agent developer, Redmond · co-author of FlyHash for LLM agent memory

In half a minute

Mohan Bheemadolu and Dr. Chrisogonas O. Odhiambo wrote FlyHash: a way to store an AI agent's memories as very short binary codes, modelled on how a fruit fly's nose tells smells apart. At the same code size it finds the right memory about 2.5 times as often as the usual SimHash. That raises a question neither of us could answer alone. Agents are often checked by a second AI, but both AIs read the same retrieved memory. If a smaller memory misses the fact more often, do the two checkers start agreeing on the same wrong answer? A first run with ordinary memory suggests it can: when memory missed the evidence, 45% of the answers both AIs agreed on were wrong, against 3% when it held the evidence. That was one retriever and one seed, so it shows an association, not a cause. Mohan's plan is to measure FlyHash at four code sizes on the same benchmark and send the retrieved memories; the agreement check then runs on them.

The paper

An agent that remembers past conversations has to store them somewhere and find the right piece quickly. FlyHash borrows from the fruit fly's sense of smell: it turns the embedding of each memory into a short, sparse code in which only a few "active bits" are switched on. Short codes mean a small memory.

The results in the paper are refreshingly plain. At equal code size FlyHash retrieves the right memory about 2.5 times as often as SimHash (0.27 against 0.11). When memory is not tight, a standard HNSW index still beats both, and the authors say so. Papers that state where their method loses are rarer than they should be, and that is what made us write to them.

The code is open: github.com/mohanji/flyhash-agent-memory.

How the conversation went

It started in the comments under the paper on Hugging Face. We asked how the codes cope with questions that are paraphrased rather than templated. Mohan answered that the tests so far used templated phrasing and asked which dataset we would suggest. We suggested two built from long, real multi-session dialogues: LoCoMo and LongMemEval. He decided to start with LoCoMo and follow with LongMemEval.

Then we asked the question that ties his work to ours: will the shortest codes lose more on questions about time ("when did she move?") than on single facts?

Why memory matters for two AIs that check each other

Two models from different companies agreeing on an answer is often taken as a good sign. But in an agent, both models usually read the same retrieved memory. If retrieval missed the fact, they are not independent anymore: they can agree, confidently, on the same wrong answer.

To see how often that happens, we put together a small open check on LoCoMo, memory-agreement-check, and ran it once with plain keyword memory (BM25, the top 10 dialogue turns). GPT and Claude answered separately, and Kimi graded them blind.

2 of 63agreed answers were wrong when memory held all the evidence (3%)

15 of 33agreed answers were wrong when memory missed some evidence (45%)

40% vs 20%both wrong, observed, against what independent errors would give

The two models' errors were strongly correlated (φ = 0.81). That is one retriever and one seed, so it shows an association, not a cause. But it gives FlyHash a precise question to answer.

The experiment we are running together

Mohan offered to supply the retrieval side; we take it from there.

Mohan's FlyHash builds the memory; the same check as in the first run measures how often two AIs agree on a wrong answer because of it.

Mohan measures retrieval first: recall@k for each LoCoMo category (single-hop, multi-hop, temporal) at several FlyHash and SimHash code sizes, 16, 32, 64 and 128 active bits, with the top 10 memories per question, matching the first run.
We suggested adding the top 5 as well: fewer memories means more misses, which should show how quickly the share of confidently wrong agreed answers grows.
He sends the retrieved contexts as JSONL, one line per question.
We run the agreement check on each file and send him the results.

The question we will read off at the end: as the codes get shorter, does the share of answers both AIs agree on and get wrong go up, and does it go up fastest for questions about time? If it does, a smaller memory has a hidden cost that a second-AI check will not catch. If it does not, FlyHash looks even better than the paper says.

The code

The check takes any retriever's output. One line per question:

{"qid": "conv-26/0", "memories": [{"text": "[D1:3 | 1:56 pm on 8 May, 2023] Caroline: I went to a LGBTQ support group yesterday...", "dia_id": "D1:3", "score": 0.82}]}

The repository already has an example that builds these contexts with the FlyHash and SimHash classes from Mohan's code:

git clone https://github.com/mohanji/flyhash-agent-memory
pip install numpy sentence-transformers
python examples/flyhash_contexts.py --flyhash-src flyhash-agent-memory/src \
  --method flyhash --active-bits 32 --out flyhash-k32.jsonl
python -m memcheck run --backend cli --contexts flyhash-k32.jsonl --memory-name flyhash-k32

Running the same questions at several code sizes, or against --method dense as a reference, gives the comparison.

What the first numbers can and cannot say

They come from one retriever (BM25), one pair of models, one judge and one seed, on 200 questions. Other retrievers and models can move every number.
Questions whose evidence BM25 misses are different from the others (multi-hop and open-domain ones are missed far more often), so the comparison is also made within each category.
A causal test would answer the same questions once with and once without the evidence turns. FlyHash at several code sizes is a natural step toward that: same questions, same models, only the memory changes.

Timeline

4 November 2026FlyHash is published on arXiv (2611.05791) by Mohan Bheemadolu and Chrisogonas O. Odhiambo.
10 November 2026The paper appears on Hugging Face Papers. In the comments we ask about paraphrased questions; Mohan asks which dataset to use; we suggest LoCoMo and LongMemEval, and he picks LoCoMo first.
11 November 2026We ask whether short codes lose more on questions about time. The open check runs for the first time on 200 LoCoMo questions, with an example for FlyHash, and we send Mohan the numbers.
12 November 2026Mohan sets out the plan: retrieval recall per category at 16, 32, 64 and 128 bits with k = 10, contexts as JSONL. We agree and add k = 5.

← All collaborations · Propose a collaboration