Two ways to remember
An agent that talks to a user for months has two ways to find an old fact. It can build memory in advance — summaries, extracted facts, structured records — and search over them. Or it can keep the history almost as is and, at question time, pull out the right fragments with ordinary search — RAG.
Memory promises convenience: the facts are collected and organized. But something can be lost or distorted along the way. The authors are the first to measure this directly, across several benchmarks and models.
Benefit and tax
The baseline is simple RAG over the raw history. If a memory system answers correctly where RAG was wrong, that is the memory benefit. If it is wrong where RAG got it right, that is the memory tax. The net value of memory is the difference, and in many settings it is negative.
Authors' data. The picture is not the same everywhere: on MINT-Eval the PlugMem system scores 33.3% against 33.1% for RAG.
Where the tax comes from
The authors split the losses into two kinds. The first is fidelity: whether the fact made it into memory when the memory was built. The second is accessibility: the fact was kept, but cannot be found at question time.
- Loss at construction. Compressing and extracting facts drops some details. Labeling 500 random turns confirmed that many systems have such losses.
- Fragmentation. One fact is split across several records, and none of them resembles the question as a whole.
- Blurring and paraphrasing. The fact is mixed with unrelated content or retold in other words, and similarity search misses it.
How to get the losses back
A simple fix: search memory and the original history at the same time, and rank the candidates together. For SimpleMem this raised accuracy from 73.6 to 82.8% and cut the tax share from 12.0 to 5.2%. This is a check in one setting, not a guarantee for every system.
What it means in practice
Corporate agents should keep the source documents and correspondence searchable, even if memory is built on top. And any new memory scheme should be tested on your own data against a RAG baseline. Limitation of the study: the benchmarks are offline and English-only; live multilingual scenarios have not been studied yet.