- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes dynamic compression for recurrent models, allowing them to selectively revisit past tokens and update their recurrent state after future task requirements are revealed. The authors introduce a re-scanning mechanism based on Gated DeltaNet and a self-supervised strategy to learn re-scan decisions from write-strength signals. Experiments on a synthetic continual function-reuse benchmark show that dynamic compression can achieve better accuracy with substantially smaller recurrent states compared with single-pass compression.
Critical strengths
1.Novel idea and clear motivation. The paper identifies the limitation of irreversible compression in recurrent models and proposes dynamic re-scanning as an interesting way to trade computation for memory.
2.Well-designed controlled evaluation. The synthetic function-reuse task effectively isolates the memory compression challenge and demonstrates the benefit of selective re-scanning.
3.Self-supervised re-scan learning. The proposed use of write-strength signals to derive re-scan policies avoids relying solely on oracle supervision.
Critical weaknesses
-
Limited evaluation on synthetic tasks. The experiments are conducted only on a synthetic function-reuse benchmark, making it unclear whether the proposed method generalizes to realistic long-context modeling scenarios. Evaluation on mature recurrent LMs (e.g., Mamba-family models) and standard long-context benchmarks would be necessary to validate the broader claims.
-
Insufficient comparison with existing memory mechanisms. The paper only compares against variants of its own approach and does not include existing retrieval-based or memory-augmented methods. Therefore, it is unclear whether selective re-scanning provides advantages over simpler alternatives.
-
Incomplete analysis of memory efficiency. Although the method reduces the recurrent state size, it still requires retaining the full raw context for re-scanning. Thus, the claimed memory benefit may not translate to reduced total memory usage in practical long-context settings.
-
Lack of computation and latency analysis. Dynamic compression introduces additional recurrent passes during inference, but the paper does not quantify the resulting computational overhead or latency. A more complete analysis of the computation–memory tradeoff is needed.
Questions for authors
-
The evaluation is currently limited to a synthetic function-reuse task. Do the authors have experiments on realistic long-context benchmarks or pretrained recurrent language models to demonstrate the generality of dynamic compression? In addition, could the authors compare against existing memory/retrieval-based approaches to clarify whether selective re-scanning provides advantages over the alternatives?
-
The method retains the full raw context to enable re-scanning. How should the reported state-size reduction be interpreted when accounting for the memory cost of storing the raw context?
-
Dynamic re-scanning introduces additional recurrent computation during inference. Could the authors provide latency or FLOPs comparisons under matched accuracy/memory settings?