- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how to diagnose failures in long-context memory systems, where errors may arise during memory writing or retrieval. It proposes a four-condition evaluation protocol using truncated full context (TFC), oracle evidence (OE), complete stored memory (CSM), and retrieved memory (RM). The authors define the OE-to-CSM performance gap as the write-side gap and the CSM-to-RM gap as the retrieval-side gap. They also introduce Expected Predictive Compression (EPC), which generates likely future questions, identifies supporting evidence, and selects evidence under a fixed memory budget using a utility function. The method is evaluated on LongMemEval and LoCoMo with multiple readers and memory-system baselines. The paper reports that EPC improves CSM performance over LLM-based summarization, and includes component ablations, budget analyses, and bootstrap comparisons.
Critical strengths
The paper proposes a useful diagnostic decomposition for analyzing failures in long-conversation memory systems. By evaluating the same reader under different evidence and memory conditions, the protocol aims to distinguish losses associated with memory writing from those associated with retrieval. This perspective could make end-to-end memory evaluation more actionable by helping identify which stage should be improved.
Critical weaknesses
-
The attribution in the proposed diagnostic protocol is not sufficiently rigorous. Differences in performance between evaluation conditions may reflect not only information lost during writing or retrieval, but also differences in context length, the proportion of irrelevant content, formatting, and contextual cues. Consequently, the measured gaps do not necessarily isolate write-side or retrieval-side information loss, which limits the strength of the resulting diagnosis.
-
The key evidence-selection procedure in EPC is underspecified for reproduction. Equation (5) defines utility in terms of coverage, specificity, and redundancy, but the paper does not sufficiently specify how these scores are computed and updated, how overlapping evidence is merged or penalized, or how evidence is handled when it exceeds the token budget. The prompts alone do not make these algorithmic choices clear, making it difficult to reproduce the selection procedure faithfully.
Questions for authors
The cost-matched comparison appears to match the number of API calls. Could the authors also report and clarify the per-example input/output token counts, latency, and API costs for EPC and the two-pass summarization baseline? This would help establish what “cost-matched” means in practice and assess the method’s resource overhead.