Every agent framework now ships long-term memory. Store a fact, retrieve it in a later session, congratulations, your agent remembers the user. The benchmarks look great. Mem0 v0.8.2 posts 91.6 on LoCoMo and 93.4 on LongMemEval.
Then independent production testing across 50,000 sessions returned 49.0% effective accuracy after 30 days, once stale data and entity contradictions were in the mix.
That is not a small regression. That is the system being wrong about half the time, confidently, using information it stored itself.
Benchmarks Test Recall. Production Tests Revision.
LoCoMo has 1,540 questions across roughly 35 sessions. It asks whether the agent can find the right fact. Single-hop, multi-hop, open-domain, temporal recall. All of it is retrieval.
None of it is invalidation.
A real user changes jobs. Cancels a plan. Moves cities. Changes their mind about the thing they told you in March. The memory you wrote is still perfectly retrievable, still highly ranked, and now wrong. Retrieval quality made it worse, not better, because the confidently stale fact outranks the vague recent one.
Nobody scores this. There is no widely adopted public benchmark that measures dropping stale facts, merging duplicates, or reconciling contradictions. So teams optimise the thing they can measure and ship the thing they can't.
The Harder Failure Is Behavioural, Not Factual
The STALE work this year found something worse than wrong facts. Systems can resolve the user's current state correctly and still fail to change how they act. One write-side prototype hit 91% accuracy on state resolution and 32% on implicit policy adaptation.
Read that again. The agent knows the user is vegetarian now. It still recommends the steakhouse, because the recommendation behaviour was shaped by older context that never got revisited.
Your memory layer can be factually current and behaviourally stale at the same time. If your evals only check "does it know X", you will never see this.
What We Actually Do About It
We treat agent memory as mutable state with a lifecycle, not an append-only log. In practice that means:
- Every memory gets a write timestamp and a source. If you can't tell where a fact came from or when, you can't argue with it later.
- Contradiction is a write-path event, not a read-path problem. When a new fact conflicts with a stored one, resolve it on write. Retrieval time is too late and too expensive.
- Facts get types with different decay rates. A user's name is durable. Their current project is not. Treating both as permanent is the root of most staleness.
- Evals include a change. Our test sets flip a fact mid-conversation and check both what the agent says and what it recommends afterwards. Recall-only evals are theatre.
- Memory is inspectable by the user. The cheapest invalidation mechanism ever built is a screen that says "here's what I remember about you" with a delete button.
None of this is exotic. It is ordinary data modelling applied to a layer most teams treat as magic because it came bundled with the framework.
The Uncomfortable Version
If you shipped agent memory in the last year and haven't measured accuracy at day 30, you don't know what your accuracy is. You know what your accuracy was on day one, which is the number the vendor benchmark measures and the number your users stopped experiencing weeks ago.
Go pull twenty real sessions from a month ago. Check what the agent believes. Most teams find at least one memory that is quietly, confidently wrong and has been shaping responses ever since.
We're here to help founders and teams design and build digital products that are built to scale with you, not slow you down. If you're looking to build something, get in contact with us today!