PrismMem: Isolating Cross-Representation Interference in Long-Term Agent Memory

Long-term memory systems often maintain multiple representations of the same interaction history, but interactions among these representations during memory construction and retrieval can distort which evidence is preserved and later selected. Existing systems may pass abstractions into downstream memory construction or rank heterogeneous records in a shared pool, allowing information lost during abstraction to propagate and relevant records to be displaced. We formulate these two failure modes as cross-representation interference: construction-time abstraction leakage and retrieval-time cross-facet suppression. We introduce PrismMem, a long-term agent memory system that decouples semantic organization, factual construction, and retrieval. PrismMem uses abstractions to delimit semantic scopes, constructs durable-state and situated-content facets directly from source interactions, and retains provenance for evidence recovery. At inference time, it retrieves each representation through its own isolated channel, composes channel-specific selections, and recovers linked source evidence for answer generation. Across evaluations over five independent runs on LoCoMo, LongMemEval-S, PersonaMem-32K, and BEAM-1M, PrismMem achieves the highest overall score on each benchmark among the evaluated memory systems. Matched controlled experiments further show that source-grounded construction reduces abstraction leakage, while channel-isolated retrieval mitigates cross-facet suppression and preserves answer quality under fixed budgets.