acceptodds
Under review as a conference paper at ICLR 2027

PrismMem: Isolating Cross-Representation Interference in Long-Term Agent Memory

Abstract

Long-term memory systems often maintain multiple representations of the same interaction history, but interactions among these representations during memory construction and retrieval can distort which evidence is preserved and later selected. Existing systems may pass abstractions into downstream memory construction or rank heterogeneous records in a shared pool, allowing information lost during abstraction to propagate and relevant records to be displaced. We formulate these two failure modes as cross-representation interference: construction-time abstraction leakage and retrieval-time cross-facet suppression. We introduce PrismMem, a long-term agent memory system that decouples semantic organization, factual construction, and retrieval. PrismMem uses abstractions to delimit semantic scopes, constructs durable-state and situated-content facets directly from source interactions, and retains provenance for evidence recovery. At inference time, it retrieves each representation through its own isolated channel, composes channel-specific selections, and recovers linked source evidence for answer generation. Across evaluations over five independent runs on LoCoMo, LongMemEval-S, PersonaMem-32K, and BEAM-1M, PrismMem achieves the highest overall score on each benchmark among the evaluated memory systems. Matched controlled experiments further show that source-grounded construction reduces abstraction leakage, while channel-isolated retrieval mitigates cross-facet suppression and preserves answer quality under fixed budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-term memory, multi-representation memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VrCZOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies interference among multiple representations in long-term agent memory systems. It identifies two failure modes. Abstraction leakage occurs when omissions in an abstraction propagate to downstream memories. Cross-facet suppression occurs when heterogeneous records compete under a shared ranking budget. PrismMem addresses these issues in two ways. It constructs factual facets directly from source interactions, and it retrieves different representations in separate channels before restoring source evidence. Controlled experiments support both failure modes. PrismMem achieves the highest overall score among the evaluated memory systems on four long-term memory benchmarks.

Critical strengths
  1. The paper tests its motivating failure modes rather than only using them as intuition. The abstraction analysis includes matched placebo conditions. The suppression analysis uses controlled distractor composition, dose-response experiments, and equal-budget evidence reinsertion.

  2. The evaluation is broad. It covers four benchmarks and eight recent memory systems, with a shared answer model and five runs. PrismMem obtains the highest overall score on all four benchmarks.

Critical weaknesses
  1. The benchmark gains are not clearly attributable to the proposed isolation principles. PrismMem combines hybrid lexical and dense retrieval, reciprocal rank fusion, a cross-encoder reranker, and relatively large channel-specific retrieval capacities. The competing systems use substantially different retrieval pipelines and answer-context sizes. The comparison therefore does not show whether the gains come from representation isolation or from stronger retrieval engineering. In the ablation, removing the central representation-isolated retrieval component reduces LoCoMo accuracy by only 1.52 points. In addition, the hyperparameter sweeps use the same 1,540 LoCoMo questions as the main evaluation rather than a held-out development set.

  2. The proposed failure modes have fairly simple alternative explanations, and simple baselines are missing. Abstraction leakage is expected once an answer-critical fact is removed from the representation used for downstream extraction. Cross-facet suppression is closely related to score calibration and quota allocation across heterogeneous retrieval sources. The paper compares isolated retrieval mainly against shared-pool ranking. It does not compare against simple alternatives such as per-source normalization, quotas, or reranking after pool fusion. It is therefore unclear how much of the benefit requires the proposed architecture rather than standard retrieval controls.

Questions for authors
  1. How do the baselines perform with the same hybrid retrieval, reranker, and answer-context budget as PrismMem?

  2. How does isolated retrieval compare with simpler alternatives such as per-source normalization, quotas, or reranking over the fused candidate pool?

  3. Can the hyperparameters be selected on held-out data, and do the main results remain similar?

  4. On LongMemEval-S, PrismMem's evidence recall is lower than that of MemGAS and EverMemOS at all three cutoffs. Why does it still achieve the best answer accuracy on this benchmark?

Reviewer_4QZROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PrismMem studies information loss propagated through intermediate abstractions and displacement of relevant records during retrieval across heterogeneous memory representations. It constructs durable and situated memories directly from source interactions, retrieves three representation types through separate channels, and appends linked source evidence after selection. The paper reports the highest mean overall scores among the evaluated memory systems on four benchmarks over five answer-generation and evaluation runs. Natural audits and controlled interventions examine the proposed failure modes and their mitigation.

Critical strengths

The paper turns two plausible memory failure modes into testable interventions, including matched deletion controls and retrieval comparisons with equal record and token budgets. Holding candidate facts fixed while changing their representations is a useful design for separating representation effects from differences in factual content. The four-benchmark comparison uses a common answer model and reports variation across five runs, providing useful evidence that the complete system is competitive across different memory tasks.

Critical weaknesses
  1. Table 4(c) reports QA gains on an omission-selected, pipeline-eligible subset of LoCoMo (Appendix C.1). The reported sample breakdown does not identify queries unaffected by omission, so it does not substantiate the paper's dilution explanation or establish the average construction benefit across the full benchmark.
  2. Tables 2 and 5 label the suppression experiments as Qwen3-32B-Instruct results, whereas Appendix D.3.3 assigns both representation rendering and QA reading to GLM-5.2. Qwen's role remains unspecified, leaving the model configuration of these mechanism experiments ambiguous.
  3. Section 4.3 restores original evidence after retrieval, but Table 6 contains no ablation that independently removes this step. The paper therefore does not quantify its independent contribution, limiting attribution of the full-system gains to representation design versus the inclusion of source text.
  4. Table 6 omits reranking candidate counts and answer-input token counts for the full system and w/o RIR, while Appendix A.5 specifies no final token cap. Without these figures, the reported ablation does not establish a benefit from channel isolation under matched resources in the main configuration; Table 5 controls budgets only in a separate experiment.
  5. Table 3's BEAM-1M lead over Memora is only 0.47 points, but no paired confidence interval for this difference is reported. The answer-generation and evaluation repetitions described in Section 5 also do not specify whether memories were rebuilt, leaving the stability of this small lead under data and construction variability unresolved.
Questions for authors
  1. Do the saved configurations for the full-system and w/o RIR rows in Table 6 use the same total reranking candidate budget and evidence-rehydration policy? Please report each variant's candidate budget, output-record limit, and mean answer-input token count.
  2. Has a LoCoMo comparison been completed that toggles only evidence rehydration while keeping the memory store and retrieved record IDs fixed? If so, report the QA scores and mean answer-input token counts for the two conditions.
Reviewer_fPQ7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PrismMem to address information loss during cascaded memory construction and competition among heterogeneous memories during retrieval. It constructs complementary memories directly from source interactions, retrieves them through separate channels, and restores linked source evidence for answering. Experiments on four long-term memory benchmarks report the highest overall mean scores among evaluated systems, supported by controlled mechanism analyses and component ablations.

Critical strengths
  1. The experiments are fairly thorough, covering four benchmarks and eight competing memory systems. PrismMem achieves the highest overall mean score on all four, with five-run results and ablations providing useful support for the improvements.

  2. I appreciate the effort to investigate why the system works. In particular, changing the memory representation while keeping the underlying facts and retrieval budget fixed provides useful evidence for the cross-facet suppression argument.

Critical weaknesses
  1. My main concern is that the method itself feels relatively incremental. Extracting memories directly from the source, retrieving different memory types separately, and combining them with supporting evidence are reasonable design choices, but the technical novelty beyond this combination remains limited.

  2. I am not fully convinced that separate retrieval channels are needed to address the ranking problem identified here. Comparisons with score calibration or representation-aware rank fusion would help clarify whether the gains require isolation or could be achieved through a simpler adjustment to shared-pool ranking.

Questions for authors
  1. Could you clarify what you see as the main methodological advance over existing multi-representation memory systems? The empirical analysis is useful, but I would appreciate a clearer distinction between the new design elements and the combination of existing techniques.

  2. Have you considered comparing isolated retrieval with a shared pool using score calibration or representation-aware rank fusion under the same budget? Such a comparison would help clarify when separate channels are particularly beneficial.

1 comment

Thank you for the detailed response. The additional comparison with shared-pool alternatives (per-source z-score and min-max calibration, and representation-aware RRF) addresses my question on channel isolation well, and the consistent gains across all four benchmarks show that the isolation is not easily replaced by simpler ranking adjustments. Please include this comparison in the final version and clarify that all variants use the same record budget and reranker. Regarding novelty, the response helps clarify the two design elements, but I still feel the methodological advance over existing multi-representation memory systems is not clearly demonstrated, as the method largely builds on existing techniques. Given the solid experimental evidence, I am raising my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.