- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies if LLMs can predict facts that will be lost during history compaction and whether the predictions can help choose facts to explicitly preserve in the compaction summary. Fact protection experiments were conducted on 3 open-weights models and show that LLM generated predictions do not provide a clear preservation gain over other methods such as random choice or a simple repetition rule. The authors also find that explicit requests to preserve important facts can improve overall recall but lead to reduced recall of other facts. More realistic settings, LoCoMo and 12 Multi-Session Chat dialogues, however do not clearly support the same findings, which were established in the synthetic settings.
Critical strengths
- The paper includes appropriate controls such as tracking the same set of unprotected facts across different settings and include comparisons to random and repetition baselines.
Critical weaknesses
- Clarity: The paper is difficult to follow. Nearly all claims in the paper are heavily qualified and includes many key terms that are not clearly defined (depth_frac, outcome rate, interference cue, common-valid) or rely on inconsistent naming (repeat cue, repetition rule, mention count all refer to the same thing).
- Significance: The main findings of the paper are inconclusive. The main fact protection experiment is only conducted on three small open-weight models. LLM self-prediction is a weak policy and the authors do not identify any selection policy that is reliably best. Further, the key finding that overall recall hides the reduced recall of unprotected facts does not hold in more natural/realistic settings. The paper does not make clear why its specific settings and experiments matter and findings do not provide a clear takeaway for the reader.
Questions for authors
- What is the main takeaway a reader should draw from this paper? It appears the main goal is to measure if prediction is a useful policy for improving fact recall under constrained compaction settings, but this is not communicated clearly.
- Why should the reduced recall of unprotected facts not be taken as an expected resulted of the limited budgets that models are evaluated under? The compaction settings are capped to 3-9 facts, which seems detached from realistic agent compactions which typically span thousands of tokens.