- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a retrieval framework for LLM agents, introducing a principle for managing memory-system complexity: "pay for on demand." Based on this principle, the authors derive three guidelines for memory writing, maintenance, and retrieval, respectively. They evaluate their proposed method, Omega, on LoCoMo, LongMemEval-S, and the held-out PersonaMem benchmark. The results show that Omega matches or outperforms strong baselines while reducing LLM calls by approximately 80%.
Critical strengths
-
Practical motivation and reasonable design principles. The paper addresses a practical and important problem, and its guiding principle is intuitive and well-motivated. The overall analytical framework is sensible. In particular, the controlled experiments that isolate the writing, maintenance, and retrieval components provide a reasonable way to quantify their individual contributions.
-
Insight into retrieval granularity. The authors observe that long memories suffer from signal dilution during cross-encoder reranking. This observation highlights the importance of separating document representation from the retrieval unit, a concept closely related to passage-level retrieval and document expansion in traditional information retrieval (IR).
-
Extensive empirical evaluation. The paper provides comprehensive experimental evaluations and cost analyses, offering useful evidence of the framework's effectiveness and efficiency.
Critical weaknesses
-
Limited technical novelty. Although the overarching principle is reasonable, it remains largely heuristic. The three proposed guidelines—compact type sets, key–payload separation, and recall-before-compression—have clear precedents in IR and retrieval-augmented generation (RAG). For instance, document expansion and doc2query have long explored generating short queries to improve the retrievability of longer documents. Similarly, late fusion and reciprocal rank fusion (RRF) are well-established IR techniques, while query rewriting has also been extensively studied.
-
Primarily an integration of existing techniques. The main contribution lies in integrating established techniques into a unified memory lifecycle and using controlled experiments to attribute performance gains to individual components, rather than introducing fundamentally new retrieval or memory mechanisms.
-
Venue fit. Given its emphasis on retrieval system design, empirical evaluation, and efficiency rather than methodological novelty, I believe this work would be a better fit for an information retrieval venue such as SIGIR.
Questions for authors
NA
## Response to Q1: When Compact Context Is Insufficient (P-Evol & U-Reas Analysis)
We thank the reviewer for this highly insightful observation. We agree that the overall average score should not obscure specific weaknesses. To address your core question-is the needed memory missing from the selected context, or is it present but not used correctly?—we conducted a manual diagnostic audit on a sampled subset of errors.
Diagnostic Evidence: Retrieval Omission in the Sampled Cases
We examined 20 diagnostic cases selected from the P-Evol and U-Reas error pools, tracing each from the source history to the constructed memory pool and the final selected context. As summarized below, retrieval omission was a prominent pattern in this sample.
In this sample of 20 cases, 16 exhibited retrieval omission where the required historical evidence was already successfully extracted and stored in the memory pool, but missed by the compact selector. Manual tracing of the source history confirmed this presence. Among these 16 omission cases, 11 involved cross-activity history. The fact that Full Projection successfully corrected 15 of these 16 cases provides supplementary support that the deficit in these specific instances lies in the selection stage rather than initial construction.
The Cross-Activity Boundary Challenge
The most frequent pattern in this diagnostic sample (11 of the 20 diagnostic cases) is cross-activity retrieval omission. This occurs when the critical supporting evidence for a queried preference is embedded in the history of a different, seemingly unrelated activity. Compact selectors, which rely on surface-level semantic similarity to the current query, can struggle to bridge the logical dependencies across these activity boundaries (e.g., a query about current shopping preferences failing to retrieve a "gift-induced change in quality expectations" from a prior, distinct activity).
Conversely, for the remaining 4 cases, the correct context was retrieved, but the model still produced an incorrect answer (answer errors). Full Projection only corrected 1 of these 4 cases, suggesting that for this subset, the issue is not a retrieval deficit.
This diagnostic explicitly identifies "improving cross-activity semantic bridging in the selector" as a concrete, high-priority direction for future system iterations.
Manuscript Revisions To ensure full transparency, we have added a dedicated diagnostic analysis in Section 4.5 and Appendix H.1 (including a summary table of these 20 cases), explicitly acknowledging the P-Evol/U-Reas deficits and documenting cross-activity evidence omissions alongside evidence-present answer errors in the diagnostic collection. Detailed records of all 20 diagnostic cases are provided in our anonymous supplementary artifact.