- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces IncreMem, a query-independent long-term memory compression mechanism designed for decoder-only transformer LLMs. Instead of assigning uniform token lengths, IncreMem allocates gist tokens to text spans based on approximately equal estimated information gain. The gain signal is derived from a retrieval head's attention weights and approximated via an auxiliary estimator, while gist-token projections are trained with a frozen backbone. A conditional rate–distortion analysis provides theoretical grounding for this allocation rule. Evaluations across Qwen3 and Llama-3.1 on LongBench-v2, LongMemEval-S, BEAM, and NIAH demonstrate consistent accuracy gains over uniform gist compression and KV eviction, alongside practical reductions in latency and memory usage.
Critical strengths
- The paper develops a clear and useful approach to allocating latent memory capacity according to content before future queries are known.
- The information-theoretic perspective is distinctive: framing memory allocation with information gain provides a principled and intuitive motivation for moving beyond fixed-length compression.
- Improvements are broadly consistent across three backbones, multiple benchmarks, and compression ratios from 2× to 16×.
Critical weaknesses
- Mismatch between theory and practice. The optimality result assumes that expected span distortion is a common convex function of gain alone. The attention-derived gain is a useful empirical proxy, but the experiments do not establish that it satisfies this assumption.
- History awareness issue. The gain estimator operates on 24K-token windows with a 12K stride, so it cannot directly recognize repetitions beyond its available window. Consequently, memory allocation may treat previously stored information as novel, even though the compressor can access earlier gist memory.
Questions for authors
- How does the estimator’s 24K-token window affect detection of information repeated farther back in the history, and how are predictions combined across overlapping windows?
- Could the authors discuss or, if feasible, compare IncreMem with system-level context compression methods such as ACON [1] and context folding [2]? Can they interact with the proposed method?
References
- Kang et al. ACON: Optimizing Context Compression for Long-horizon LLM Agents. ICML 2026.
- Sun et al. Scaling Long-Horizon Agent via Context Folding. ICML 2026.
2 comments
Thank you for the rebuttal. I maintain my possitive assessment. I also recommend the authors to include a discussion regarding ACON and ContextFold in the main text. Good luck!
Reply to Reviewer 6BRk
We sincerely thank Reviewer 6BRk for the positive assessment and for recognizing the clarity of our approach, its information-theoretic motivation, and the consistency of our improvements. Our responses below directly validate our modeling assumption, examine the history awareness of the gain estimator, and relate IncreMem to system-level context compression.
W1. Validation of the Modeling Assumption.
We directly measure the span distortion dj (Eq. 10) against the span gain to validate Assumption A.2, while controlling for span length. We place 50 probe spans of 2–24 tokens in each of 100 examples sampled from benchmarks, with the preceding context compressed at ×8. For each probe, we compress the span into one memory unit and take dj as the increase this causes in the KL divergence from the uncompressed model, summed over the following 2,048 tokens. The results on Qwen3-8B are as follows.
The span distortion is convex in the gain. The distortion per unit gain increases with the gain, as Assumption A.2 posits. Fitting E[dj]=cVjp over all probes gives p=1.27 [1.08, 1.45] (95% CI, bootstrapped over examples), and p=1.19 and 1.33 on Qwen3-4B and Llama-3.1-8B.
The distortion follows the gain rather than the length. As gain accumulates over tokens, longer spans tend to carry more of it, so we write Vj=ℓjρj, where the mean token gain ρj varies independently of length (r=0.08). Fitting E[dj]=cℓjβℓρjβρ gives βℓ=1.29 [1.12, 1.47] and βρ=1.07 [0.38, 1.81]: at equal length, spans with denser gain incur more distortion, at the rate their gain predicts.
Overall, the span distortion is a convex function of the gain rather than of the length, so the attention-derived gain satisfies the assumption on which the optimality result rests.
W2 & Q1. Estimator Window and Distant Repetitions.
The window moderately affects the memory units a distant repetition receives. Following the NIAH setup (Section 5.2), we insert a 64-token passage into 30 documents of 128K tokens and repeat it D tokens later. We then count the memory units allocated to the repetition at ×8 on Qwen3-8B, with the gain computed within the estimator’s local window or over the full history. The first occurrence, as new content, receives 8.1 units. As shown below, the local window recognizes nearby repetitions as the full history does, and treats distant ones close to new content, spending at most 0.5 extra units per 64 tokens, while their earlier occurrences remain accessible in the gist memory. Additionally, each window scores only the tokens that follow its overlap with the previous window, using the overlap as context, so every token receives a single estimate.
Q2. Relation to System-level Context Compression.