acceptodds
Under review as a conference paper at ICLR 2027

WhenLoss: Diagnosing LLM Memory Beyond Answer Accuracy

Abstract

Conversation histories average 121K tokens in our LongMemEval setting, yet the default persistent store holds 5K tokens and must be written before future questions arrive. A wrong answer can reflect omitted evidence, missed retrieval, or limits of the reader, the answering model. We present WhenLoss, a controlled diagnostic protocol that compares truncated history, oracle evidence, complete stored memory, and retrieved memory while fixing the reader. These comparisons expose performance changes at intermediate interfaces; an analytic counterexample shows why their score gaps do not identify information loss. Across 500 LongMemEval questions, four of six baselines have write-side gaps exceeding retrieval-side gaps by more than 0.02 for each of three readers. This observation motivates inspecting stored evidence and improving its selection. Expected Predictive Compression (EPC) uses prospective questions to select supporting evidence before the actual question arrives. At a 5K-token write budget, EPC raises complete-memory Contains Match from 0.44 to 0.49 and retrieved-memory Contains Match from 0.38 to 0.42 over LLM summarization, averaged across three readers. Its complete-memory advantage increases to 0.11 at 2K tokens. These gains support the writing intervention under the tested budgets; evidence-retention checks and cost comparisons further characterize its benefits and requirements.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, persistent memory, diagnostic evaluation, evidence retention, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7n3wOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to diagnose failures in long-context memory systems, where errors may arise during memory writing or retrieval. It proposes a four-condition evaluation protocol using truncated full context (TFC), oracle evidence (OE), complete stored memory (CSM), and retrieved memory (RM). The authors define the OE-to-CSM performance gap as the write-side gap and the CSM-to-RM gap as the retrieval-side gap. They also introduce Expected Predictive Compression (EPC), which generates likely future questions, identifies supporting evidence, and selects evidence under a fixed memory budget using a utility function. The method is evaluated on LongMemEval and LoCoMo with multiple readers and memory-system baselines. The paper reports that EPC improves CSM performance over LLM-based summarization, and includes component ablations, budget analyses, and bootstrap comparisons.

Critical strengths

The paper proposes a useful diagnostic decomposition for analyzing failures in long-conversation memory systems. By evaluating the same reader under different evidence and memory conditions, the protocol aims to distinguish losses associated with memory writing from those associated with retrieval. This perspective could make end-to-end memory evaluation more actionable by helping identify which stage should be improved.

Critical weaknesses
  1. The attribution in the proposed diagnostic protocol is not sufficiently rigorous. Differences in performance between evaluation conditions may reflect not only information lost during writing or retrieval, but also differences in context length, the proportion of irrelevant content, formatting, and contextual cues. Consequently, the measured gaps do not necessarily isolate write-side or retrieval-side information loss, which limits the strength of the resulting diagnosis.

  2. The key evidence-selection procedure in EPC is underspecified for reproduction. Equation (5) defines utility in terms of coverage, specificity, and redundancy, but the paper does not sufficiently specify how these scores are computed and updated, how overlapping evidence is merged or penalized, or how evidence is handled when it exceeds the token budget. The prompts alone do not make these algorithmic choices clear, making it difficult to reproduce the selection procedure faithfully.

Questions for authors

The cost-matched comparison appears to match the number of API calls. Could the authors also report and clarify the per-example input/output token counts, latency, and API costs for EPC and the two-pass summarization baseline? This would help establish what “cost-matched” means in practice and assess the method’s resource overhead.

Reviewer_JnVcOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how long-context memory systems lose useful information. It compares four input conditions to measure performance gaps at the write and retrieval stages in the extensive evaluation.

Critical strengths

S1. The diagnostic protocol addresses a practical problem: end-to-end accuracy cannot effectively guide which stage to improve, and this paper's proposed comparisons provide a simple way to study how to locate the problems in memory system. With this method, they successfully identify the bottleneck.

S2. The design of EPC and self-questioning is interesting, and it effectively improves the output quality as well as reducing the write-side gaps.

Critical weaknesses
  1. The gaps do not fully separate the cases between missing information and difficulty in using stored information. Complete stored memory may contain the needed evidence, but the reader may miss it among other entries, while oracle evidence removes those distractions. This can make the write-side gap look larger even when the evidence was preserved. A stronger check is needed before using the gap sizes to decide which stage needs more effort.

  2. The main evaluation relies on substring matching. This can mark a correct paraphrase as wrong, or accept an answer that contains the gold text but adds a wrong claim. Token F1 provides some support, but it also measures word overlap. Since EPC explicitly preserves exact names, dates, and numbers (while other baselines like summarization do not), part of its advantage could result from better matching the scoring rule. A small human check or an evaluation that judges answer meaning would strengthen the result.

Questions for authors

Does EPC keep its advantage when answers are judged by meaning rather than word overlap? It would also help to inspect some oracle-evidence failures. The average oracle-evidence CM score is only 0.53, so understanding these failures matters for interpreting both diagnostic gaps.

Reviewer_AY9POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a four-condition diagnostic protocol that feeds a fixed reader truncated full context, oracle evidence, complete stored memory, and retrieved memory, splitting end-to-end loss into a write-side gap and a retrieval-side gap. On LongMemEval, write-side gaps exceed retrieval-side gaps for most of six baselines, which motivates EPC, a write-time method that generates likely future questions and keeps compact supporting evidence under a token budget. EPC reaches the highest stored-memory score of 0.49 against 0.44 for the LLM summary baseline, with the lowest write-side gap on both LongMemEval and LoCoMo.

Critical strengths
  • The additive split into write-side and retrieval-side gaps gives memory research a shared way to locate loss, and a degradation test shows each gap responds selectively to its own stage.
  • A cost-matched two-pass baseline rules out the extra LLM call as the source of EPC's gain, over which EPC still gains 0.03.
Critical weaknesses
  • The protocol assumes oracle evidence is the right ceiling, but readers reach only 0.53 with gold turns, so the write-dominant conclusion rests on a weak reference, and what memory should be compared against remains open.
  • The paper calls its gaps operational yet uses the write-side gap to decide where to intervene, but EPC's question-keyed entries also change what the retriever finds, so whether the diagnosis predicts which intervention helps remains open.
Questions for authors

Clarification

  • Readers reach only 0.53 even with gold turns, so is the missing 0.47 a memory problem or a reading problem? If the ceiling itself is low, what does a small write-side gap say about a memory?
  • The gaps are operational rather than causal, so what licenses choosing a write-side intervention from them? If EPC's question-keyed entries change what the retriever finds, did writing improve, or did it shift the burden onto retrieval?

Future work

  • EPC gains only 0.04 when its guessed questions miss and 0.02 on inferential ones, so is the problem guessing the wrong questions, or deciding once what to keep before any question arrives?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.