acceptodds
Withdrawn from ICLR 2027

Dynamic Compression in Recurrent Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recurrent models process long contexts efficiently by compressing their history into a fixed-size state, but modern architectures typically do so in a single causal pass over the sequence. Each input must therefore be compressed before the model knows how it will later be used, forcing a limited state to compromise across possible future demands. We introduce dynamic compression, which allows a recurrent model to selectively revisit past tokens and revise its fixed-size state through additional recurrent updates. The model need not preserve every part of the history at uniformly high fidelity in its recurrent state, because lower-fidelity information can be revisited from the retained raw sequence when it becomes relevant. We study this in a controlled setting where the model first learns multiple functions in-context and, later in the same sequence, encounters a series of few-shot tasks that each require it to identify and reuse one of those functions. A single-pass model must preserve every function at sufficient fidelity for any future task, whereas selective re-scanning allows the model to revisit and refine only the function currently needed. We find that dynamic compression substantially reduces the recurrent state required for accurate reuse and scales more favorably as the number of stored functions grows. These results demonstrate a computation–memory tradeoff in which recurrent models can spend more computation revisiting their history to make more effective use of a fixed-size state.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
RNNs, compression

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Lnc2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes dynamic compression for recurrent models, allowing them to selectively revisit past tokens and update their recurrent state after future task requirements are revealed. The authors introduce a re-scanning mechanism based on Gated DeltaNet and a self-supervised strategy to learn re-scan decisions from write-strength signals. Experiments on a synthetic continual function-reuse benchmark show that dynamic compression can achieve better accuracy with substantially smaller recurrent states compared with single-pass compression.

Critical strengths

1.Novel idea and clear motivation. The paper identifies the limitation of irreversible compression in recurrent models and proposes dynamic re-scanning as an interesting way to trade computation for memory.

2.Well-designed controlled evaluation. The synthetic function-reuse task effectively isolates the memory compression challenge and demonstrates the benefit of selective re-scanning.

3.Self-supervised re-scan learning. The proposed use of write-strength signals to derive re-scan policies avoids relying solely on oracle supervision.

Critical weaknesses
  1. Limited evaluation on synthetic tasks. The experiments are conducted only on a synthetic function-reuse benchmark, making it unclear whether the proposed method generalizes to realistic long-context modeling scenarios. Evaluation on mature recurrent LMs (e.g., Mamba-family models) and standard long-context benchmarks would be necessary to validate the broader claims.

  2. Insufficient comparison with existing memory mechanisms. The paper only compares against variants of its own approach and does not include existing retrieval-based or memory-augmented methods. Therefore, it is unclear whether selective re-scanning provides advantages over simpler alternatives.

  3. Incomplete analysis of memory efficiency. Although the method reduces the recurrent state size, it still requires retaining the full raw context for re-scanning. Thus, the claimed memory benefit may not translate to reduced total memory usage in practical long-context settings.

  4. Lack of computation and latency analysis. Dynamic compression introduces additional recurrent passes during inference, but the paper does not quantify the resulting computational overhead or latency. A more complete analysis of the computation–memory tradeoff is needed.

Questions for authors
  1. The evaluation is currently limited to a synthetic function-reuse task. Do the authors have experiments on realistic long-context benchmarks or pretrained recurrent language models to demonstrate the generality of dynamic compression? In addition, could the authors compare against existing memory/retrieval-based approaches to clarify whether selective re-scanning provides advantages over the alternatives?

  2. The method retains the full raw context to enable re-scanning. How should the reported state-size reduction be interpreted when accounting for the memory cost of storing the raw context?

  3. Dynamic re-scanning introduces additional recurrent computation during inference. Could the authors provide latency or FLOPs comparisons under matched accuracy/memory settings?

Reviewer_hKRsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper departs from the context-window-only and single-hidden-state-only paradigms. It proposes to allow RNNs to revisit past tokens from the context to retrieve information, rather than forcing the RNNs to compress all the information that might be needed in the future. In doing so, it allows the model to keep in its hidden state only the information that will allow it to retrieve what is needed later, rather than forcing it to keep everything at full fidelity without knowing what it will need. Experiments on a toy dataset show that the proposed approach performs better than standard RNNs on an in-context learning task.

Critical strengths
  • Important axis of research
  • The paper is easy to follow
  • The toy task is well described
Critical weaknesses
  • There is no direct comparison with any other models
  • No real dataset is used
  • The current model implementation is limited to the synthetic dataset used and does not generalize to other tasks
  • I think that the toy task does not really demonstrate the importance of the model. The functions to learn are always placed at the start of the sequence, the sequence contains very little noise, and each sequence only uses one function seen in the past.
  • The authors state that this codebook is specific to this synthetic dataset, and I think that it is not actually self-supervised, as designing it requires a lot of information about the task itself. It looks more like a hand-crafted, specific approach designed for this toy dataset than a self-supervised one.

While I think this paper deals with an important subject, the current version is too limited for me to recommend acceptance. The implementation needs to be further improved to work on datasets beyond a toy case. Experiments need to be conducted beyond the toy dataset, with more comparisons and analyses against state-of-the-art and other methods (depending on how the authors propose to adapt this model to real-world usage, it could also be compared to RAG approaches, for example). Overall, I think this work needs substantial further improvement before being considered for acceptance.

Questions for authors
  • Is the approach really self-supervised (Step 2, page 5 of the paper)? To create and use the codebook, you already need to know how many functions to learn, and where they are placed and declared.
Reviewer_DMkwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes dynamic compression to help recurrent models reuse past information with a limited recurrent state (like a retrieval system + RNN). Specifically, the model selectively re-reads relevant past inputs to update its state. Experiments on synthetic function-reuse tasks show improved accuracy with smaller recurrent states compared to single-pass processing.

Critical strengths

The simplicity of the method could make it easier to scale.

Critical weaknesses

Unclear incremental contribution. Revisiting past context to address recurrent-state limitations is already explored in Just Read Twice and retrieval-augmented recurrent models. It is unclear what new insight the proposed method provides beyond demonstrating the benefits of selective re-reading in a controlled synthetic setting.

Limited empirical validation. The method is simple and potentially scalable, but evaluation is restricted to synthetic linear-function tasks, with the main scaling results relying on oracle re-scan supervision. Experiments on realistic sequence tasks, such as language modeling or biological sequences, would help establish its practical value. If the authors can show more scaling effects on realistic dataset, it will be helpful.

Questions for authors

Could the authors clarify the practical advantages of the proposed approach over attention–RNN hybrids and provide a direct empirical comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.