When an agent's memory is squeezed into sparse codes, do two AIs agree on the same wrong answer?
with Mohan Bheemadolu · AI agent developer · co-author of FlyHash for LLM agent memory

In half a minute
Mohan Bheemadolu builds memory for AI agents. With Dr. Chrisogonas O. Odhiambo he wrote FlyHash, which stores each memory as a sparse code, a long vector with only a few "active bits" switched on, the way a fruit fly's nose tells smells apart. At the same number of active bits (32), on the authors' synthetic benchmark, it ranks the right memories about 2.5 times better than the usual method (mAP 0.27 vs 0.11). At equal storage, the usual method with about four times the bits matches or beats it, and the gain is smaller on real conversations.
Our question: an AI's answer is often checked by a second AI, but both read the same memory. If that memory misses the fact, do the two start agreeing on the same wrong answer?
In our first run, with plain keyword memory, it looked that way: when the memory missed the fact, 45% of the answers both AIs agreed on were wrong, against 3% when it had it. Next, Mohan runs FlyHash at four code sizes, and we check whether codes with fewer active bits make this worse.
The paper
An agent that remembers past conversations has to keep them somewhere and find the right piece fast. FlyHash borrows from the fruit fly's sense of smell. It stores each memory as a sparse code: a long vector in which only a few "active bits" are switched on. The full code is still long (2,000 units), so it is not smaller in storage than a SimHash code with the same number of active bits.
The paper is refreshingly plain. Compared with SimHash, the usual way of making such codes, at the same number of active bits (32), on the authors' synthetic labeled benchmark, mean average precision (mAP) is 0.27 for FlyHash versus 0.11 for SimHash. This is not a storage comparison. A FlyHash code is sparse but long (2,000 units), so it takes more storage than a SimHash code with the same active bits. At equal storage, SimHash with about four times the bits matches or beats FlyHash. On real LoCoMo conversations the advantage at matched active bits is smaller (recall@10 at 32 active bits: about 0.32 vs 0.17). And it says plainly where FlyHash loses: when memory isn't tight, a standard HNSW index, the ordinary full-size kind of search, still beats both.
Exact dense retrieval on the unhashed embeddings stays ahead of every hashed code in these tests. At 128 active bits FlyHash is about 0.04 to 0.05 below dense recall@10. SimHash with 1,024 bits keeps about 97% of dense recall at recall@50. Hashing buys compact codes, not better quality.
The code is open: github.com/mohanji/flyhash-agent-memory.
How the conversation went
It started in the comments under the paper on Hugging Face. We asked how the codes cope when a question is put in new words rather than a fixed template. Mohan said the tests so far used templates and asked which dataset we'd suggest.
We suggested two built from long, real conversations over many sessions: LoCoMo and LongMemEval. He chose to start with LoCoMo and do LongMemEval after.
Then we asked what we most wanted to know about his codes: will codes with the fewest active bits lose more on questions about time ("when did she move?") than on single facts?
Answer from the first measurement: on LoCoMo (MiniLM embeddings, mean-centred, recall@10), at 128 active bits FlyHash keeps about 0.88 of exact dense recall on multi-hop questions, 0.90 on temporal, 0.89 on open-domain and 0.91 on single-hop. The differences from single-hop are small (multi-hop −0.03, temporal −0.01), and their 95% bootstrap intervals include zero at 16, 32, 64 and 128 active bits. So there is no evidence that temporal questions lose more to hashing than single-hop ones. Multi-hop has the lowest absolute recall for every method, dense included, which points to task difficulty, not hashing.
Why memory matters for two AIs that check each other
When two models from different companies agree, people usually take it as a good sign. But in an agent, both models tend to read the same memory. If that memory missed the fact, they're no longer independent. They can agree, confidently, on the same wrong answer.
To see how often that happens, we built a small open check on LoCoMo, memory-agreement-check, and ran it once with plain keyword memory (BM25, ordinary keyword search, pulling the top 10 lines of dialogue). GPT and Claude answered separately, and Kimi graded them blind.
The two models' mistakes almost always came together (a correlation of φ = 0.81). This is one kind of memory and one random seed, so it shows a link, not a cause. But it gives FlyHash a sharp question to answer.
The experiment we are running together
Mohan offered to supply the retrieval side; we take it from there.
- Mohan first measures how often the right memory lands among the top results (recall@k). He splits it by type of LoCoMo question: single facts (single-hop), facts that need joining up (multi-hop), and questions about time (temporal). He runs FlyHash and SimHash at 16, 32, 64 and 128 active bits, with the top 10 memories per question, like the first run.
- We suggested adding the top 5 as well: fewer memories means more misses, which should show how quickly the share of confidently wrong agreed answers grows.
- He sends the retrieved contexts as JSONL, one line per question.
- We run the agreement check on each file and send him the results.
The question we'll answer at the end: with fewer active bits, do both AIs agree on wrong answers more often, and does it grow fastest for questions about time?
If yes, a more compressed code has a hidden cost that a second AI won't catch. If no, FlyHash looks even better than the paper says.
The code
The check takes any retriever's output. One line per question:
{"qid": "conv-26/0", "memories": [{"text": "[D1:3 | 1:56 pm on 8 May, 2023] Caroline: I went to a LGBTQ support group yesterday...", "dia_id": "D1:3", "score": 0.82}]}The repository already has an example that builds these contexts with the FlyHash and SimHash classes from Mohan's code:
git clone https://github.com/mohanji/flyhash-agent-memory
pip install numpy sentence-transformers
python examples/flyhash_contexts.py --flyhash-src flyhash-agent-memory/src \
--method flyhash --active-bits 32 --out flyhash-k32.jsonl
python -m memcheck run --backend cli --contexts flyhash-k32.jsonl --memory-name flyhash-k32Running the same questions at several code sizes, or against --method dense as a reference, gives the comparison.
What the first numbers can and cannot say
- They come from one retriever (BM25), one pair of models, one judge and one seed, on 200 questions. Other retrievers and models can move every number.
- The questions BM25 misses aren't like the rest: multi-hop and open-domain ones get missed far more often. So we also compare within each type.
- To prove cause, you'd answer the same questions once with the evidence and once without. FlyHash at several code sizes is a natural step there: same questions, same models, only the memory changes.
- The FlyHash numbers measure retrieval, meaning whether the annotated evidence turns are found. They do not measure whether a language model then answers correctly, which is what the agreement check studies.
Timeline
- FlyHash is published on arXiv (2611.05791) by Mohan Bheemadolu and Chrisogonas O. Odhiambo.
- The paper appears on Hugging Face Papers. In the comments we ask about paraphrased questions; Mohan asks which dataset to use; we suggest LoCoMo and LongMemEval, and he picks LoCoMo first.
- We ask whether codes with fewer active bits lose more on questions about time. The open check runs for the first time on 200 LoCoMo questions, with an example for FlyHash, and we send Mohan the numbers.
- Mohan sets out the plan: retrieval recall per category at 16, 32, 64 and 128 bits with k = 10, contexts as JSONL. We agree and add k = 5.