Memory deduplication
Also called: dedup, near-duplicate merging, memory consolidation
What is memory deduplication?
Memory deduplication detects when a new memory says substantially the same thing as one already stored, and merges instead of adding another copy. Agents rediscover the same facts constantly, so without it a corpus fills with near-identical entries and retrieval quality falls as it grows.
Why duplicates appear so fast.
It is not a bug in the agent. It is the normal consequence of many sessions doing similar work.
Independent sessions rediscover the same things
Two runs a week apart hit the same constraint and both write it down, phrased differently. Neither did anything wrong, and now there are two entries that mean one thing.
Near-identical is the hard case
Exact duplicates are trivial. The real work is recognising that two differently worded notes describe one fact, which needs semantic comparison rather than string equality.
Merging has to preserve the useful difference
Sometimes the second write adds a detail the first lacked. Collapsing to whichever arrived first throws that away, so the merge has to be additive rather than a discard.
Why it decides whether memory stays useful.
An unmanaged memory store degrades with use, which is the opposite of the promise.
Retrieval degrades with noise
Forty phrasings of one fact crowd out the other things a query should have surfaced.
Context space is finite
Injecting five paraphrases of the same note wastes the space five distinct facts could have used.
Humans have to read it too
A browsable corpus stops being browsable when most of what is in it is repetition.
Commonly confused with.
Three things it is not.
Memory deduplication questions
What is memory deduplication?
Detecting that a new memory says substantially the same thing as an existing one and merging rather than storing both, so a corpus does not fill with near-identical entries.
Why do agents write duplicates?
Because independent sessions rediscover the same constraints. Neither run knows the other happened, so both record what they learned, in different words.
Is anything lost when memories merge?
It should not be. A later write often adds a detail the earlier one lacked, so a good merge is additive rather than keeping whichever arrived first.
How is it different from importance scoring?
Scoring decides how much a memory matters now. Dedup decides whether it is a distinct memory at all. Both are lifecycle management, answering different questions.
Why can a uniqueness constraint not do this?
Because it only catches identical text. Agents almost never phrase the same conclusion identically, so the duplicates that matter are semantic rather than literal.
A corpus that improves as it grows.
Virex Memory consolidates what your agents rediscover instead of stacking it up.