acceptodds
Withdrawn from ICLR 2027

Compressing History into Memory: Distilling Transformers into Recurrent Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformers are AI's workhorse but their computational cost becomes prohibitive when processing long sequences. We target long-horizon streaming vision and robotics applications, where it is particularly impractical to store and maintain a history of observations. Recurrent Transformers address this limitation by maintaining fixed-size memory but their performance lags behind that of transformers operating over the full observation history. We argue that this gap does not stem from architectural limitations, but from differences in how these models learn to compress past information. Without access to an observation history, recurrent models must explicitly decide what to retain in memory at each step, a significantly harder learning problem. In this work, we propose a distillation approach that transfers the compression strategy of a classical full-history transformer to a recurrent variant. We enable this by designing a teacher model that explicitly compresses its observation history into a fixed-size bottleneck representation and directly supervise the student's memory with this bottleneck representation, effectively aligning the two compression mechanisms. We show that this approach allows to train a recurrent latent robotic memory with linear-time complexity on the Mem-RPE task while substantially narrowing the performance gap to full-history transformers. We additionally validate the same principle on streaming visual question answering (VQA) and observe improved recurrent predictions thanks to memory distillation.

Primary area:
learning on time series and dynamical systems
Keywords:
recurrent transformers, distillation, robot memory, pose estimation, streamingVQA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KEDEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Chimera, a distillation framework that trains a Latent Bottleneck History Transformer (LBHT) as a teacher to explicitly compress observation histories into fixed-size latent representations, and then distills these representations into a recurrent transformer student. The proposed approach is evaluated mainly on memory-based relative pose estimation (Mem-RPE), where the student recurrent model is shown to narrow the gap with the full-history teacher under long sequence settings. The authors additionally validate the approach on streaming VQA to demonstrate broader applicability of memory distillation.

Critical strengths
  1. The proposed distillation framework is effective. The use of a latent bottleneck history transformer as a teacher and direct memory representation alignment between teacher and student is a clean approach. The experiments show that this strategy can significantly improve recurrent transformer performance compared with existing recurrent baselines while maintaining fixed-size memory.

  2. The experimental evaluation targets a meaningful long-horizon setting. The Mem-RPE task is appropriate for studying persistent memory, where retaining information from distant observations is essential. The evaluation on longer sequences provides evidence that the proposed method improves recurrent models in scenarios where standard recurrent approaches struggle.

Critical weaknesses
  1. The evaluation scope is relatively narrow compared with the claimed general applicability. Although the paper includes Mem-RPE and streaming VQA experiments, the validation is still limited to a small number of tasks. More diverse long-context tasks would be necessary to demonstrate that the proposed approach is broadly useful for recurrent transformers.

  2. Limited generality of the proposed approach. The learned compression mechanism is optimized with task-specific objectives, making it unclear whether the method can generalize across different tasks. If a separate teacher and distillation process are needed for each task, the training cost may limit practical applicability.

  3. Insufficient analysis of memory size and compression trade-offs. The proposed method relies on a fixed-size bottleneck representation, but the paper does not provide comprehensive ablations on different memory sizes, numbers of bottleneck tokens, or compression ratios.

Questions for authors
  1. Can the authors provide evaluations on more diverse long-context tasks to demonstrate that the proposed approach generalizes beyond Mem-RPE and streaming VQA?

  2. Can the authors demonstrate that the learned memory compression mechanism transfers across different tasks without retraining a separate teacher?

  3. How does performance change with different bottleneck sizes or memory budgets?

3 comments

We are surprised that the scope was perceived negatively. This is a paper about streaming computer vision and our work goes beyond the scope of a large body of computer vision work, most of which target single tasks.

Generality. The compression is query-independent by design: neither the teacher bottleneck nor the student memory sees the query. The same recipe (N=4 segments, L1 matching, Eq. 4) was applied unchanged to both applications.

However, a teacher is trained per task, as in most distillation work. This is a one-time training cost, and the teacher is never deployed. Given the gains we obtain, we are surprised that the advantage of this procedure is not acknowledged. Focusing on a unified model for computer vision was not the claim of this work and we are not aware of work that targets this.

Memory size. Table 7 ablates the number of bottleneck tokens and the teacher depth for streaming VQA: 8, 16 and 32 tokens give 71.9, 72.2 and 72.4% on LLaVA-Video val, and larger bottlenecks bring no further gain. For Mem-RPE we deliberately kept the memory configuration of Kinaema (20 × 3072), whose paper ablates the number and dimension of memory embeddings for the same architecture. This isolates the effect of distillation at an identical memory budget.

Thank you for the clarification. I apologize for my earlier concern regarding generality. My research background is primarily in long-context and recurrent LLMs, where a common expectation is that a single model or method should generalize across a wide range of downstream tasks. This prior may have influenced my assessment of the paper's scope.

Regarding the memory-size analysis, the additional clarification about Table 7 and the rationale for adopting Kinaema's memory configuration for Mem-RPE also addresses my concern. I appreciate the authors' responses and will raise my score to 3 accordingly.

Thank you for having kept an open mind about our work, engaged in the discussion and for having raised the score! This is very much appreciated.

Reviewer_4FNyOpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how to improve recurrent Transformers’ ability to retain long-horizon visual histories. The authors first train a Latent Bottleneck History Transformer that accesses the full observation history available at each time step and compresses it into a query-independent, fixed-size representation. They then use the teacher’s bottleneck representations to supervise the recurrent student’s memory through segment-wise L1 matching, jointly with the downstream task objective. Experiments on Mem-RPE show that the proposed approach outperforms existing methods.

Critical strengths
  • The paper proposes a novel distillation approach that transfers knowledge from a teacher model while preserving performance.
  • The experimental results show that Chimera outperforms the baselines included in the comparison.
Critical weaknesses
  • In Figure 1, the statement “with O(T2)O(T^2) at each step tt” is inaccurate for causal Transformers, as opposed to the specific teacher architecture used in this paper. With KV caching, each step has O(T)O(T) attention complexity, since the current query attends to the cached keys and values. Consequently, the discussion of quadratic complexity does not provide a meaningful comparison against cached causal Transformers. The timing analyses throughout the paper share this issue.
  • The evaluation does not adequately compare against more recent linear-attention and recurrent sequence models, such as Mamba, RWKV, GDN, and TTT.
  • The downstream task is too simple, involving sequences of only 800 tokens. At this sequence length, a causal Transformer with a sufficiently optimized implementation may well outperform Chimera in both predictive performance and latency. The experiments also do not establish that the proposed method has meaningful long-term memory capabilities.
Questions for authors

I recommend training and evaluating the method on long-context modeling tasks, such as language modeling, with sequence lengths of at least 16K–32K tokens. The evaluation should compare against recent SSMs and TTT-based methods on the same tasks, using thoroughly optimized implementations and infrastructure.

5 comments
AuthorsOpenReview ↗

800 images and not tokens -> 51,200 tokens

We thank the reviewer for the feedback, here are the answers to the points raised, and which concerns information we think the reviewer has overlooked.

  1. KV caching does not give O(1). As the reviewer writes, a cached causal transformer costs O(T) per step, and its cache grows linearly without bound (and the observations need to be stored!). Chimera costs O(1) per step with constant memory and does not need to hold observations: compute is about 7 ms per step at any sequence length (Fig. 8). The paper already discusses KV caching in Sections 2 and 3. The O(T2)O(T^2) in Fig. 1 refers to the non-causal history transformer we actually use and time, where it is accurate. We agree the caption should say so explicitly.

  2. Causal transformers were evaluated, and perform much worse. Table 3 (right) compares causal and non-causal teachers on RPE-val: 40 vs. 76 at 2m/90° for T=200, and 27 vs. 38 for T=800. The cached causal transformer is therefore not a strong alternative on this task, independently of latency. Causality does therefore not increase prediction quality and decreases computational complexity only, which is (almost) irrelevant since it concerns the teacher only. The recurrent student can in principle very well represent latent representations collected by a non-causal teacher (neither causal nor non-causal teacher tokens contain information beyond the current context).

  3. The sequences are 800 images, not 800 tokens. Each observation is an image encoded into 64 patch tokens, i.e. 51,200 visual tokens per sequence. Evaluation is at 4x the training length (200 →\rightarrow 800). Comparable recurrent transformers train on far shorter sequences: MooG on 8 steps, TRecViT on 64, Kinaema on 100. Streaming VQA uses videos of up to 1080 frames.

  4. Long-term memory is evaluated directly. Figs. 4 and 5 fix the memory after 800 frames and query observations from the whole past, including the start of the sequence. Chimera stays close to the teacher at all query ages and is more stable than the teacher beyond its training length.

  5. Recurrent and state-space baselines are included. Table 1 compares against LRU (a state-space model), xLSTM, GRU, EMA and MooG; all are far below Chimera.

  6. Language modeling at 16K–32K tokens is outside the scope. The paper targets streaming vision and robotics, as stated in the title, abstract and introduction. There the challenge is high-dimensional observations with a query unknown at compression time, not next-token prediction. Requesting a language-modeling benchmark with optimized SSM/TTT infrastructure asks for a different paper.

Thank you for the authors' response. I accept point 1 and thank the authors for clarifying points 2, 3, 4, and 6. I apologize for some of my previous misunderstandings.

However, for point 2, could you provide a more detailed analysis or relevant experiments showing why the causal transformer performs much worse than the non-causal one, instead of just giving a conclusion? In 2026, this is not a common finding and is rather interesting—intuitively, there shouldn't be such a huge gap between them, particularly for a temporal modeling task. Meanwhile, I hope to see a fair comparison of the causal transformer against the other baselines in table 1, along with real inference speed measurements in practical scenarios (such as with KV cache & flash attention & all other infra optimization), rather than only a simple comparison of teacher performance.

For point 5, I still think the baselines are too old and more recent ones should be compared.

Given the above, I am willing to raise my score to 2 for now.

We thank the reviewer for the careful reading of our response, for the open discussion, and for raising the score!

  1. Why causal attention is standard in LLMs. While in 2026 causality is standard in language models, even there causal masking was not adopted because it yields better representations of a sequence, but because next-token prediction requires it for efficiency: with teacher forcing, one parallel forward pass supervises all T positions at once, and at inference the same mask enables KV caching. Neither argument applies here. Our models do not predict the next observation; they are supervised by a pose or an answer computed from the whole observed prefix. Where no autoregressive factorization is needed, bidirectional attention is the usual choice: ViTs, BERT-style encoders, and 3D reconstruction models such as DUSt3R, MUSt3R and VGGT all use it. Work that removes the causal mask from decoder-only LLMs to obtain better embeddings (e.g. LLM2Vec) points in the same direction.

  2. Why the gap is large on Mem-RPE. The task asks for a pose in the agent's current, moving coordinate frame. Every past observation therefore has to be re-expressed relative to the newest pose, using the odometry that arrived after it. A non-causal model can do this in every layer, for every token. In a causal model the representation of an observation at time τ is fixed once and can never integrate what was sensed after τ, such as later odometry or a later view of the same place. All re-referencing must then happen inside the 20 bottleneck tokens, which is a much harder compression problem. This is consistent with Table 3 (right): the gap is largest in the fine-grained regime (40 vs. 76 at 2m/90°, T=200) and present under both training schedules.

  3. Complexity of a a causal transformer. Any cached causal transformer still costs O(T) per step, optimizations would be constant factors. Additionally, the observation history needs to be stored. We have updated the paper with new Figure 8, which shows the relationship between a causal model (with KV-cache!) and Chimera: Chimera has constant compute time, the causal model with KV-cache scales linearly. We report two causal models: w/ and w/o bottlenecking layers.

  4. "In Table 1, with optimized inference. The rebuttal period does not allow us to retrain a new causal teacher with the requested training regime, which would take more than one month for this large-scale task. However, we politely object to the request: Table 1 lists comparable (streaming) variants and the teacher is given “for information only”, being non-comparable. The non-causal teacher is bounded in accuracy by the results of Table 3, below the non-causal teacher that Table 1 already lists.

  5. Age of the baselines. The baselines in Table 1 include xLSTM and MooG (both 2024) and Kinaema (2025), the current state of the art on this benchmark. Our contribution is a training method and not an architecture: the controlled comparison is the same recurrent model with and without distillation.

Reviewer_p43nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper distills the knowledge of context dependent Transformer into RNN. The strategy they follow is highly similar to https://arxiv.org/abs/2606.06479, but they extend its scope much more to real world applications.

Critical strengths

It is tackling the important problem of drawing TF's strength to RNN, an essential arch for length generalization. It also tackles applications to the real world.

Critical weaknesses

The most important and possibly sole weakness of this paper is comparison to Isola's SMT. In my understanding, their core strength borrows much from Dagger Memory training, that distills the student's own recurrent traj: Their complete method is SMT → DMT. The strategy itself has already been established & validated in Kumar/Isola. The distill over sequences does not establish an entirely new principle relative to the complete Kumar-Isola method because they use DMT as well.

Indeed, there can be a chance for a difference in architecural choice (bottleneck, co-trained/frozen teacher)-- but this I believe is a difference the authors are more responsible to clarify both emprically and theoretically that makes a meaningful difference in performance. In my understanding, the problem of Kumar/Isola remains in its optimization and in the capturing of the hard rules, along with its distillation to the RNN capable of generalization over super long horizon. Is there anything in this work that differentiate itself from Kumar/Isola in this direction? Any other direction would help me value this paper better as well, if it makes significant improvement upon Kumar/Isola.

To that end,I humbly posit that authors need to significantly revise the paper to equationally distinguish the complete strategy of Kumar & Isola from their work, and make the essential algorithmic difference from their work to be the focal point of the novelty. That given said, the paper must justify that the subtle difference from their work from SMT+DMT makes substantial gain in performance.

Questions for authors

Please see the last part of my Critical Weakness.

4 comments

We thank the reviewer for the feedback. We are a bit concerned that the reviewer “overfit” its assessment to a single work, which should be concurrent to ours, and is also non-peer-reviewed. Under ICLR policy this is contemporaneous work. We have nevertheless cited it and stated the difference in Section 2.

Additionally, we here further detail the differences, which we think are major and significant.

  1. The topic and experimental evidence differs in scale. Kumar & Isola evaluate on synthetic tasks, character-level TinyStories, and MNIST/Sketchy pixel sequences, with BPTT as the only baseline. We evaluate on real computer vision tasks with large-scale models (map-free pose estimation and streaming VQA). We process visual streams of 800 frames (4x the training length), against six recurrent baselines, and on streaming VQA with a 7B video-LLM. We believe that our work has proven its applicability to real computer vision and robotics tasks.

  2. The training procedures differ. SMT(Supervised Memory Training) trains the RNN on one-step transitions between teacher memories, (mt,xt+1)→mt+1(m_t, x_{t+1}) \rightarrow m_{t+1}, without ever unrolling it. DMT (DAgger Memory Training) is described by its authors as "a lightweight fine-tuning phase following SMT", run with a frozen encoder and a small learning rate. In SMT, the RNN itself never receives gradients through time; for dependencies beyond the truncated future length, Kumar & Isola note that credit must be "amortized into gradient optimization steps", requiring at least T such steps. Our Chimera has no one-step phase. The student is unrolled on its own memory from the first iteration, over up to 200 steps, with backpropagation through time over each segment. It is supervised at only N=4 segment endpoints, jointly with the downstream task loss, so the losses are applied directly to the memory produced by the student's own long rollout.

  3. Shortcomings of SMT. SMT does not apply to our student as is: Kumar and Isola report that "SMT -> DMT is unable to train GRU RNNs" because of memory-space collapse. In contrast, our architecture leverages a GRU gating block for stability, and this although we apply it to realistic computer vision and robotics applications.

  4. The requested "substantial gain" is reported. Table 1 (with vs. without distillation), Table 5 (distillation weight, including zero) and Table 6 (number of segments, backpropagation horizon) isolate the contribution of each design choice for Mem-RPE. This is also confirmed on Streaming-VQA in Table 4 (with vs. without distillation).

Dear reviewer,

May we kindly ask to take into account our answers to your concern?

Thank you, The authors

Reviewer_p43nOpenReview ↗

Thank you very much for the response

I thank the author for the response. However, I want to keep my score as is for the following reasons:

The topic and experimental evidence differs in scale.

With all due respect, I do want to point out that there are some reviewer that take greater emphasis on the innovation in the mathematical model itself while there are some reviewers that more greatly values the difference in the scale that has been achieved. I feel that the latter has a tendency to be taken lightly; since the abundancy of the computational resource varies across researchers, it is my belief that the latter perspective deserves more attention. While I respect the former myself, I believe that the identification of "what was necessary for the scaling" in terms of mathematical model is warranted, either through ablations or through theory. If indeed scalability is important, what is the specific delta of this research that makes Chimera to scale where Isola et al fails to do so? This reviewer is merely using Isola's work as a reference point to isolate what is truly needed and what is not, to be inclued into a mathematical model itself.

The training procedures differ.

The authors seem to imply a narrow but defensible claim taht “Chimera's student-learning procedure dispenses with one-step pretraining and uses segment-level trajectory supervision, including task gradients, throughout training.” But I was rather making a more general comment that “Supervising student-generated recurrent trajectories with teacher memories was already introduced in the cited paper, so that the identification of merit/demerits of their variations warrants investigation and report”.

Shortcomings of SMT

Thank you for raising this point, as this is closer to what I was wishing to be replied about. Indeed, there is a concrete difference that could avoid one route to collapse: Chimera trains its teacher independently and freezes it before student training, whereas SMT jointly optimizes the teacher’s memory representations and the recurrent updater. But at the same time, this is a discussion about "SMT -> DMT" route only. I also believe that DMT can be applied from the get-go as well?
Also, I believe that the paper does not establish that this difference is what resolves the reported GRU failure. There is also an architectural mismatch: the GRU network that failed in Kumar–Isola is different from Chimera’s transformer followed by GRU gating. Or is the architectural design the central novelty to be claimed here?

The requested "substantial gain" is reported.

My question solely concerned the benefit exclusively attributable to their departures from SMT+DMT / DMT.

All in all, my single concern has consistently been about the delta in the distillation procedure from a cited work that actually makes a difference in what the authors value, which I believe is scalability? As of now it is still unclear what is the important lever that must be/ must not be pulled in comparison an idea that had been publisized for 3~4months before the submission, as far as I am aware.

We strongly disagree with this view of our result and the comparison.

  • It is not our role to justify failure modes of existing work, in particular non-peer reviewed concurrent work.
  • The pipeline in Kumar+Isola has not been shown to work in the realistic settings in which we work.
  • The fact that DMT of Kumar+Isola “could be applied from the get-go as well” is a conjecture by the reviewer, and has not been tested in Kumar+Isola. DMT is their post-processing step.

We disengage from this discussion as we think that nothing can be gained. We believe that the discussed concurrent pre-print does not yet have the final answer on this topic, should not shut down the scientific discussion, and that our work does provide new insights on how to supervise recurrent memory, in particular on realistic computer vision and robotics problems.

We will withdraw this submission.

Comments on the paper

2 comments

We thank the three reviewers for the efforts they put into reviewing our work and for their service upkeeping peer-review for the ML community. However, we were very surprised about the harsh and negative reviews for this paper and we do think that the ratings might be linked to the current state of peer-review and the perceived pressure to this conference edition which became famous for submission IDs beyond 60k. We politely and kindly ask the reviewers to re-assess this paper taking into account our answers below. We are happy and grateful that reviewer 4FNy already started to reconsider their assessment of our work.

Thank you!

AuthorsOpenReview ↗

Withdrawal statement

We thank the reviewers for having engaged with us, and in particular reviewers 4FNy and KEDE for having kept an open mind and for having raised their scores.

We nevertheless withdraw our work from this venue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.