- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Gated Context Memory (GCM) for detecting when compressed context lowers answer quality relative to raw context. A prompt-conditioned writer extracts intermediate LLM states into latent memory, and a LoRA-adapted reader learns from task, distillation, and reconstruction losses. The reader’s first-token confidence determines whether to continue with memory or fall back to raw context. Experiments cover BFCL, HotpotQA, NarrativeQA, and LongBench transfer, with additional signal comparisons and ablations. Gating often recovers raw-context performance, though detection quality and fallback rates vary substantially across settings.
Critical strengths
-
First-token confidence is a useful and inexpensive signal once memory has been constructed. This provides a concrete example where the signal supports selective use of compressed memory.
-
The ablations and negative results help establish the method’s limits. In particular, near-chance detection on NarrativeQA and Qwen3.5-9B LB-2WikiMQA makes clear that improved task scores do not necessarily imply successful failure detection.
Critical weaknesses
-
The comparisons do not isolate the benefit of compression. LongBench gives GCM 128S states but textual baselines only 128 tokens. GCM also has an adapted reader, while the raw branch does not, and the first-line scorer can include reasoning markers. Budget, task adaptation, and output format could therefore explain part of the gains.
-
The practical efficiency benefit remains unclear. Table 15 reports higher encode-plus-read latency than raw reading in all four profiles, before fallback. Memory is rebuilt for each prompt, and HotpotQA fallback reaches 60–80% in Table 3 and over 90% in Table 8. The reader savings do not yet establish a lower-cost system.
Questions for authors
Do the gains and detection AUROC persist when scoring the final answer?
Thank you for your response. The fix of the intro plus the fallback ratios and random control are helpful. My main concern is still the missing raw context + LoRA baseline as your SQuAD v2 SFT reference (93.1 vs. about 64.8 for gated GCM) suggests that somehow task adaptation can explain much of the gain?
Since most of my points were addressed, I’m raising my score to 2.