acceptodds
Under review as a conference paper at ICLR 2027

Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models

Abstract

Recurrent LLM architectures have emerged as a promising approach for improving reasoning, as they enable multi-step computation in the embedding space without generating intermediate tokens. Models such as Ouro perform reasoning by iteratively updating internal representations while retaining a standard KV cache across iterations, causing memory consumption to grow linearly with reasoning depth. Consequently, increasing the number of reasoning iterations can lead to prohibitive memory usage, limiting the practical scalability of such architectures. In this work, we propose MELT, a novel architecture that decouples reasoning depth from memory consumption. Instead of using a standard KV cache per layer and loop, MELT maintains a single KV cache per layer shared across reasoning loops, while retaining per-loop KV caches for a small sliding window. This cache is updated over time via a learnable gating mechanism. To enable stable and efficient training under this architecture, we propose to train MELT using chunk-wise training to align training and inference. Empirically, we show that MELT models fine-tuned from pretrained Ouro parameters maintain the original model performance and outperform standard LLMs of comparable size, while maintaining a memory footprint comparable to those models and dramatically smaller than Ouro's. Overall, MELT achieves constant-memory iterative reasoning without sacrificing LoopLM performance, using only a lightweight post-training procedure.

Primary area:
generative models
Keywords:
Looped transformers, LLM, memory efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_n28KOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces MELT, a framework designed to address the KV-cache memory challenges in looped LMs. The approach is mainly based on the Ouro LM. The problem statement correctly identifies that although looped LMs offer parameter efficiency, their KV-cache size scales linearly with the number of loops. The objective is to reduce the KV-cache size, and to achieve this, the authors introduce architectural changes where tokens outside a sliding window use a shared KV representation across recurrent iterations, updated through a learnable gating mechanism. To adapt the pretrained Ouro model to this new architecture, they propose chunk-wise training and knowledge distillation across recurrent steps.

Critical strengths
  1. In my opinion, the greatest strength of this paper is identifying a timely and relevant problem: a concrete architectural inefficiency in this increasingly popular class of models, and proposing a solution to address it.

  2. The ~3–4× reduction in memory usage is quite a substantial empirical improvement.

  3. The proposed pipeline, including the architectural design and adaptation procedure, is novel and reasonable.

Critical weaknesses
  1. I am not entirely sure whether to consider this a weakness, but when I initially read the paper, I thought the authors aimed to generally address the issue of KV-cache memory growth per iteration. However, it turns out that there is a sliding window (W=50) within which the KV-cache remains similar to Ouro, and compression is only applied outside this window, to older tokens. I think this distinction should be stated more clearly in the paper.

  2. Regarding the sliding window, it is not sufficiently ablated, and the choice of W=50 seems somewhat arbitrary, especially since it appears to be an important component of the proposed approach.

  3. Although MELT is motivated as a solution to the KV-cache memory challenges of looped LMs, in practice, its architectural design and evaluation are essentially limited to Ouro. Whether it is applicable to other architectures, such as RLMs, LoopFormer, Recurrent Depth Transformers, PARCAE, or other recurrent architectures, is not sufficiently discussed.

  4. As shown in Table 10 of Appendix D, although the memory savings are substantial, they do not translate into decoding throughput improvements. In fact, MELT reduces the decoding throughput from 13.5 to 8.5 tokens/s.

  5. It seems that even within the same architecture, the performance gains are not always consistent. As the paper moves from the smaller Ouro model to the larger one in Table 2, the results on both mathematical and non-mathematical benchmarks are affected.

Questions for authors
  1. In Tables 1 and 2, there is little discussion of why the performance changes, particularly the improvements on HumanEval+. How does MELT achieve such substantial performance gains on this benchmark?

  2. How do the authors envision extending MELT to other looped models? Is there a specific reason why the experiments focus exclusively on Ouro?

5 comments
AuthorsOpenReview ↗

Authors response (1/2)

We thank the reviewer for their feedback and suggestions, which we think helped strengthen the paper. We believe the revisions address their concerns. We have submitted an updated version of the paper, with additions highlighted in blue so they are easy to find.

W1

Without the sliding window, MELT’s KV cache size scales as O(N×L)O(N × L). Adding the window changes this to O(N×(L−W)+N×T×W)O(N × (L − W ) + N × T × W). Here, W≪LW ≪ L is the window size, NN is the number of layers, TT is the number of iterations and LL is the number of tokens. Given that in Ouro KV size grows like O(N×L×T)O(N × L × T) and that W≪LW ≪ L and that we focus on long reasoning scenarios, we think that the claim that we "reduce memory growth by iteration" is fair. For example, with a 32K-token completion, four iterations, and a 50-token window, MELT’s KV cache is approximately 4× smaller than Ouro’s.

We have now clarified the explanation of the KV size scaling for Ouro and MELT in Section 3.2. Hopefully this avoids further confusion.

W2

We agree that the local window decision is underexplained in the paper, so we expanded our results in Appendix C.2, which we now call "Hyperparameter ablations". Results are also shown in the table below:

Window sizeMATH-500AIME 24AIME 25AIME 26Math avg.GPQAMMLU ReduxHumanEvalNon-math avg.
085.2034.5826.6732.2944.6940.8575.4071.9562.73
191.6041.8832.9241.0451.8640.8576.7471.3462.98
593.2045.2134.3844.1754.2438.3976.6081.1065.36
1092.2049.3833.7847.0855.6143.0877.0081.1067.06
5093.4050.4235.8345.0056.1642.8677.1984.7668.27
10094.0049.1738.7542.9256.2141.9676.4984.1567.53
50094.2050.6237.9245.6257.0941.9675.9882.9366.96
1,00093.4046.6736.6747.0855.9541.0776.5683.5467.06
5,00093.0047.7138.7546.6756.5340.8575.2884.1566.76
10,00093.8050.2133.7543.5455.3237.9575.3085.3766.20

We observe that for low window values performance is worse but it stabilizes at around 10-100 tokens and does not increase much after that. We chose 50 since it provides a good tradeoff between accuracy and memory. However, in this range the exact window size does not have a significant impact on memory usage or accuracy: while having a small window is clearly much better than not having one, in the heavy reasoning setup that we study, a difference of a few tens of tokens has a negligible impact on memory.

In Appendix C.2 we have also added an equivalent table (Table 11) for a model trained with a chunk size of 100, that the reviewer might appreciate.

W3

MELT could be applied directly to any looped transformer architecture that repeats layers and has separate KV cache for each loop. This includes most Recurrent Depth Transformers (LoopFormer, Huggin, PARCAE, Nanbeige...). For RLMs, if the reviewer refers to https://arxiv.org/abs/2512.24601v1, here the recursion occurs through repeated model calls rather than repeated layers within the model, so MELT does not directly apply.

Given our experiments, we propose to first pre-train the models without MELT, which is faster, and then do our post-training with the chunk-wise distillation loss. Training with MELT-like KV-cache from scratch would take more time. As an example, say we wanted to apply MELT to Nanbeige4.2-3B, which keeps separate KV caches for each of the two loops. We would take the weights of the model and initialize the gates with random values. Then, we would train on the same reasoning dataset with our distillation loss to get the MELT version of Nanbeige4.2-3B. We have also added a comment about this in Section 3.2.

W4

Table 10 reports the decoding time for batch size 1, which is expected to be higher with MELT since it adds the gating operations with respect to Ouro. However, because of the memory savings the maximum batch size increases much more, so with maximum batch size the decoding time throughput would be much faster for MELT than for Ouro. We have clarified this in the caption of the table.

W5

From our point of view, the results for MELT-3.1B are very similar to those of MELT-1.6B. At both scales, the majority of accuracy differences between MELT and Ouro are within the reported standard errors (originally Table 2 did not contain standard errors, we have now added them). MELT slightly outperforms Ouro on non-math benchmarks on average and slightly underperforms on math benchmarks on average. We think that this consistent pattern across both model sizes strengthens our results.

AuthorsOpenReview ↗

Authors response (2/2)

Q1

This is a nice observation from the reviewer, which we had not noticed. A brief analysis of the responses of MELT and Ouro suggests that MELT responses on HumanEval+ are shorter and contain less repetitive reasoning . The number of responses exceeding 30k tokens decreases from 11 to 4 for MELT-1.6B and from 7 to 3 for MELT-3.1B, compared to their respective Ouro-Thinking baselines.

One possible explanation is that the finetuning data encourages shorter responses. To test whether the improvement comes from the data or the model, we are working on a data ablation that finetunes Ouro-Thinking under the same conditions and with the same data.

Q2

We chose Ouro because it is state of the art at its size, publicly available, and offers two model sizes for evaluation. For how to apply MELT to other models, see our response to W3.

I thank the authors for their answers to my concerns and running the experiments. I know it's a busy time so if the authors found some of my suggestions made them run too many unnecessary experiments, I apologize for it. Regarding the weaknesses and concerns: I feel like the authors have provided acceptable responses to W1 and W2. Regarding W3, I am confused on the authors' response on how they think the approach could be extended to other models; and if it's possible to show some results for it. For W4, I think at the moment that table is Table 12 in your new draft? I still don't an update in caption, and I am not quite sure what do the authors mean about the throughput and tokens/s. Is your claim about how your method increases throughput but the decoding per token gets slower? For W5 I think the response is acceptable.

Thanks for the answering the questions. I am leaning toward increasing my score and will wait to see if the authors response to my comments here.

Reviewer_uuNcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a lightweight adaptive method to allow looped transformer models to reuse a single copy of KV/MLA cache from the prefixes, hence saving memory footprint during decoding.

The authors first demonstrated that the off-the-shelf OURO-thinking checkpoints exhibit severe degeneration when doing rollouts under KV sharing (using just the hidden state from the last loop). Then they proposed an layer-wise gated recurrent memory module which aggregates the hidden states of the same layer through the different loops, and demonstrated that under a light-weight chunk-based sequential training schema, one can adapt the OURO-thinking checkpoints to work with recurrent memory module on math and reasoning tasks, with comparable performance with respect to the OURO-thinking baselines.

I find the paper's results of interest to the looped transformers community in general. However the current draft is not fully self-contained and lacks important analysis for several proposed design choices, thus I do not find the paper in its current form ready for acceptance.

Critical strengths

The idea of saving KV cache during the inference of looped transformers is a natural and well-motivated line of research. It was known from the OURO paper that cache sharing was a potentially valid path, but it only worked in limited settings which does not support KV sharing during prefilling. The adaption training proposed in this paper was surprisingly lightweight and effective. The benchmark evaluation for the trained model appears comprehensive.

Critical weaknesses
  1. There are train-test divergence due to the temporal recurrent nature of the memory. Namely the current design employs a local window of size 50 using OURO style full cache vs the training chunk size of 500. While exact train-test match for temporal recurrence is hard to attain if one wants temporal parallel training, I believe understanding / analyzing the tradeoff for different configurations shall be a critical part of such studies.

  2. (Minor issue) The initial state for the recurrent gate is not described anywhere in the paper, that equation (1) is not self-contained when h−1(l)h_{-1}^{(l)} is not well-defined.

Questions for authors
  1. As stated in the weakness 2, can the authors provide a self-contained definition of the recurrent memory module? i.e. what is h−1(l)h_{-1}^{(l)}?

  2. I wonder what is the relationship between downstream performance and the local cache window size. In particular: (a) What is the underlying principle for selecting 50 as the current window size? (b) For the reasoning tasks evaluated. Does different task favor different local window size? (c) For prefilling-dominant tasks (e.g. tasks with input length over a few thousand tokens, such as code completion), does sequential chunked prefilling provide an edge over fully parallel prefilling? An inference only grid search over different configurations and sources would be appreciated.

  3. I wonder if the authors can provide some insights about the gate behaviors. Do they appear to be input dependent or is the bias primarily governing the gate scores? I also wonder if the projection matrices exhibit any low rank structures. Understanding the nature of such projection gates can better justify the non-negligible added parameters. Inference-only analysis should be sufficient to answer this question.

  4. The authors demonstrated the rollout degeneration behavior under for the OURO-thinking checkpoints when only retaining the last-loop KV. However in the original OURO paper, the cache reuse was demonstrated on the base model (not tested in this paper), with reasonable off-the-shelf generalization. I wonder if the authors can show whether the degeneracy is a phenomenon that exists from OURO-base, or does it only emerge during reasoning post training.

I will raise my score if the questions above are properly addressed.

3 comments
AuthorsOpenReview ↗

Authors response (1/2)

We thank the reviewer for their feedback and suggestions, which we think helped strengthen the paper. We believe the revisions address their concerns. We have submitted an updated version of the paper, with additions highlighted in blue so they are easy to find.

Q1

We fully agree that this was unclear in the paper. We have added the following clarification in Section 3.2

"The gated update applies only for t≥2t \geq 2, with h1(l)=x1(l)h^{(l)}_1 = x^{(l)}_1 at the first loop iteration."

Q2

(a) We agree that the local window decision is underexplained in the paper, so we expanded our results in Appendix C.2, which we now call "Hyperparameter ablations". Results are also shown in the table below:

Window sizeMATH-500AIME 24AIME 25AIME 26Math avg.GPQAMMLU ReduxHumanEvalNon-math avg.
085.2034.5826.6732.2944.6940.8575.4071.9562.73
191.6041.8832.9241.0451.8640.8576.7471.3462.98
593.2045.2134.3844.1754.2438.3976.6081.1065.36
1092.2049.3833.7847.0855.6143.0877.0081.1067.06
5093.4050.4235.8345.0056.1642.8677.1984.7668.27
10094.0049.1738.7542.9256.2141.9676.4984.1567.53
50094.2050.6237.9245.6257.0941.9675.9882.9366.96
1,00093.4046.6736.6747.0855.9541.0776.5683.5467.06
5,00093.0047.7138.7546.6756.5340.8575.2884.1566.76
10,00093.8050.2133.7543.5455.3237.9575.3085.3766.20

We observe that performance generally improves as the window size increases up to 50 tokens. Beyond 50, accuracy fluctuates without consistent gains, so we select 50 tokens to balance accuracy and window size. However, in this range the exact window size does not have a significant impact on memory usage or accuracy: while having a small window is clearly much better than not having one, in the heavy reasoning setup that we study, a difference of a few tens of tokens has a negligible impact on memory.

(b) The new expanded table above also includes diverse benchmarks (both maths and non-maths). We don't see a clear difference worth mentioning in how window size affects different benchmarks. Both math and non-math benchmarks show similar behavior.

(c) All our inference modes handle prompt tokens in the same way: the prompt is processed in parallel with gating (as a single chunk). After prefilling, only the final gated KV cache is retained, and all loops share this prompt cache when generating subsequent tokens. We do this as it provides a very similar setup to the one during chunk-wise training, where past chunks are treated in the same way. We think that fully parallel prefilling would introduce additional differences between the training and inference models, which would not benefit performance.

(From weakness 1) The reviewer also suggests in weaknesses that ablating the training chunk size would be useful. While ablating the chunk size is very expensive (and it gets even more expensive in time as chunk size decreases), we performed a new run reducing the chunk size from 500 to 100. For that run, we also report accuracies for all the window sizes, which can be found in Table 11 in the updated paper. We observe no gain from decreasing the chunk size to 100, and since a larger chunk size is much faster to train, we stay with the previous setup of 500.

Q3

We agree the proposed gate analysis is relevant. We have added the following analysis in Section 4.5.1.

"We further analyzed the learned gates in MELT-1.6B across four loops. Loop 1 initializes the latent state without gating, while mean activations across tasks range from 0.225–0.312, 0.160–0.254, and 0.153–0.225 for loops 2–4. Within tasks, gate activations have average standard deviations of 0.162, 0.128, and 0.119, respectively, indicating input-dependent behavior. Moreover, the projection matrices have stable ranks of 3–15, suggesting parameter-efficient low-rank representations as promising future work."

AuthorsOpenReview ↗

Authors response (2/2)

Q4

We agree that it makes sense to look at this, as it is possible that this degeneracy arises during post-training. We have now improved Table 6 in Appendix B to include results for both Ouro-1.4B and Ouro-Thinking-1.4B. Additionally, we have added results with a window size of 50, matching MELT inference. We add a copy of that table below.

ModelKV settingAIME24 (pass@1)AIME25 (pass@1)AIME26 (pass@1)MATH-500 (accuracy)
Ouro-1.4B-BaseFull KV33.326.720.083.4
Ouro-1.4B-BaseLast, window size 00.00.00.023.2
Ouro-1.4B-BaseLast, window size 506.73.33.358.0
Ouro-1.4B-ThinkingFull KV46.743.343.394.4
Ouro-1.4B-ThinkingLast, window size 00.00.00.022.2
Ouro-1.4B-ThinkingLast, window size 503.30.03.361.2

As the table shows, the exact same degeneracy that we observed in Ouro-Thinking-1.4B happens in Ouro-1.4B. Looking at completions, we observe the same phenomenon as in Figure 3: generations begin coherently but eventually degenerate during extended reasoning. This strengthens our original hypothesis: that the reason for the discrepancy of the results in Appendix C.1 of the Ouro paper is that their aggressive few-shot evaluation setup constrains the generation format and reduces the likelihood of the model drifting during decoding. However, during zero-shot long generations this drift is enormous.

Reviewer_uuNcOpenReview ↗

Response to Submission11410

I would like to thanks the authors for the comprehensive response to my questions. My main concerns has been cleared and as promised I will raise my score.

Reviewer_nFbeOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the growth of KV-cache memory with recurrent depth in looped Transformers. It proposes Memory-Efficient Looped Transformer (MELT), adapted from pretrained Ouro models. MELT maintains one latent memory state per token and layer, updates it across loops using an element-wise gate, and projects that state into keys and values shared by subsequent tokens. A sliding window retains loop-specific caches for the most recent 50 generated tokens. The authors adapt Ouro using chunk-wise training and distillation supervision at every loop. Experiments at approximately 1.6B and 3.1B parameters report broadly comparable reasoning performance to the original Ouro checkpoints. For the smaller model, the reported KV-cache memory at 32k generation length is approximately four times smaller than Ouro's, with slower batch-1 decoding and substantially slower training.

Critical strengths

1. A useful reduction in the memory cost of looped reasoning.

The paper tackles a concrete limitation of Ouro and reports substantially lower KV-cache memory while preserving much of its benchmark performance. Matching the original model at lower memory is meaningful even without a consistent accuracy improvement. Applying the adaptation to two model sizes also provides evidence beyond a single checkpoint.

2. The ablations identify relevant architectural and training choices.

The comparisons with mean aggregation, fixed EMA, last-loop sharing, and scalar gating help assess the value of the learned element-wise gate. The cumulative training-component ablations also show that aligning training with the changed cache dynamics matters. These results are useful, although the absence of chunk-size and window-size sensitivity limits the conclusions.

Critical weaknesses

1. The additional contribution over simpler cache sharing is insufficiently established.

Table 6 reports zero accuracy for every untrained first- and last-loop sharing variant, including on MATH-500. Appendix B explicitly acknowledges that this conflicts with Zhu et al.'s reported last-loop-sharing performance. Its explanation based on prompt format and accumulated cache mismatch is plausible, but no matched reproduction or controlled change of evaluation settings establishes why the discrepancy occurs. This is consequential because successful untrained sharing would be a strong low-memory baseline.

Furthermore, the trained “Last” variant in Table 4 performs close to MELT: its reported four-task average is 63.3 versus 64.2, with identical MATH-500 accuracy and better AMC23 pass@1. Section 4.5.1 indicates that this variant keeps the other training and architectural components unchanged. Broader benchmark results, variability, memory, and throughput for this simpler alternative are needed to establish the additional benefit of learned gating, particularly given its approximately 0.2B extra parameters in MELT-1.6B.

The headline comparison also lacks an unchanged Ouro model fine-tuned on the same data, limiting attribution of accuracy improvements to the architectural change rather than additional training. Together, these issues leave the significance of the proposed gating mechanism insufficiently demonstrated.

2. The central adaptation procedure is underspecified, and its quality–cost trade-off is incompletely characterized.

Section 3.3 describes supervision at all loops, and Appendix A suggests alignment of intermediate representations, but the paper does not specify the precise distillation targets, supervised layers, loss function, normalization, or weighting relative to the language-model objective. It also does not explain whether gradients propagate through cached states across chunk boundaries. These details are necessary to reproduce the method and interpret the training ablations.

Chunk-wise training is central to handling the altered cache dependencies, yet only one chunk size is reported. The fully parallel comparison is part of a cumulative ablation after removing distillation, so it does not independently isolate the effect of chunking under the full training objective. No sweep characterizes quality against chunk size, throughput, or training memory, and the 50-token window is not ablated. This matters because Appendix D reports about a tenfold training slowdown and slower batch-1 decoding than Ouro. In addition, the lightweight-adaptation claim compares 32K MELT samples with 16.4M Ouro tokens, while Table 7 reports 320M MELT training tokens. The claimed adaptation time below 5% of Ouro's final training stage therefore needs consistent token and GPU-hour accounting. The adaptation may still be economical relative to total pretraining, but the specific comparison is not established by the numbers as written.

Questions for authors

These questions concern clarification and evidence supporting the paper's current claims. Training a MELT model from scratch would be a separate extension.

  1. Reproducibility of Ouro last-loop sharing: Can you reconcile Table 6 with Zhu et al.'s reported success of last-loop KV-cache sharing under a matched reproduction? Which differences in checkpoint, prompting, decoding, cache implementation, or evaluation account for the complete failure, particularly on MATH-500?
  2. Random-gate discrepancy and the local window: Why does untrained MELT with random gates retain substantial performance in Table 5, while all sharing variants score zero in Table 6? Do these variants use the same prompt-cache handling and the same 50-token loop-specific window? What evidence separates the effects of gating, the local window, and adaptation training?
  3. Strength of the trained “Last” baseline: Can you provide the trained “Last” variant's results on the remaining headline benchmarks, together with its parameter count, memory, and decode throughput? Do matched additional-training results for unchanged Ouro alter the interpretation of MELT's quality gains?
  4. Training specification and sensitivity: What is the exact distillation objective, including target representations, layers, loop weighting, and the SFT loss weight? Are cached states detached at chunk boundaries? What evidence supports the choice of 500-token chunks and a 50-token window, and does chunk-wise training remain essential when distillation is retained?
  5. Scope and accounting of efficiency: Can you reconcile the adaptation-budget units and the contradictory maximum-batch entries in Tables 3 and 10? Does Figure 1 compare KV memory alone or total model-plus-cache memory? What evidence supports quality as loop depth increases beyond the four-loop configuration, and how does the remaining window memory scale with depth?
4 comments
AuthorsOpenReview ↗

Authors rebuttal (1/2)

We thank the reviewer for their feedback and suggestions, which we think helped strengthen the paper. We believe the revisions address their concerns. We have submitted an updated version of the paper, with additions highlighted in blue so they are easy to find.

Q1

First, we want to say that we did manage to reproduce Ouro’s last-loop GSM8K results using lm-eval-harness. However, Zhu et al. do not specify the MATH-500 evaluation harness, so we cannot exactly replicate their setup or know the exact implementation and prompting details.

We have expanded Table 6 with more untrained last-loop sharing experiments, now with both Ouro-1.4B and Ouro-1.4B-Thinking. Importantly, we see that the collapse in performance we observed is not exclusive to Ouro-1.4B-Thinking; the same thing happens with Ouro-1.4B. We have also added experiments for sliding-window inference (see next point). Based on the reported setup, n-shot prompting remains our leading hypothesis for the discrepancy between Ouro's and our MATH-500 results. However, without the exact MATH-500 evaluation configuration, we cannot conclusively attribute the discrepancy to prompting or rule out differences in decoding or cache implementation.

Q2

The reason is that the experimental setups were different: Table 5 used a 50-token sliding window, while Table 6 used no window at all. We have now expanded Table 6 with 50-token window experiments. We see similar behavior to Table 5: the window allows some performance to be retained on both AIME and MATH benchmarks.

These added experiments isolate the effect of the local window within each sharing variant, without adaptation training. They show that the window contributes to retaining performance, although training is needed to achieve performance at Ouro level. The effect of gating is isolated separately in Table 5.

Q3

We have now expanded Table 4 and Table 12 (Table 10 in the first version) to include all benchmarks for Last as well as an equivalent inference efficiency analysis. Results show that Last remains slightly below MELT on average (Table 4) and that it is slightly more efficient than MELT (Table 12), due to the extra gate computations of MELT with respect to Last. The number of parameters of Last is 1.4B, 0.2B less than MELT.

We agree that it is a tradeoff. In the paper, we wanted to prioritize accuracy, so the performance increase on average was more important to us than the slight decrease in inference efficiency. However, in specific scenarios where even small inference efficiency gains are very important, one might decide to use Last. Therefore, we have made this tradeoff more clear in Section 4.4 of the paper. We also plan to make Last available when we release our model weights.

Q5

We have clarified the training budgets for MELT and Ouro (the correct number were the ones in the table, so training took 320M tokens) and the maximum batches (12 for MELT, 3 for Ouro).

We have also change the caption of Figure 1 to make clear that the x axis is KV memory.

We have no experiments with more than 4 loops, as the Ouro models are trained for 4 loops. The window memory grows linearly with loop depth: total KV-cache memory scales as O(N × (L − W ) + N × T × W ), where N is the number of layers, L the sequence length, W the window size, and T the loop count. However, in our long-reasoning setting, W=50 ≪ L, so the memory increase per additional loop is very small.

AuthorsOpenReview ↗

Authors rebuttal (2/2)

Q4

We agree that our KD implementation was a bit unclear in the paper, so we clarified Section 3.3. Now, we explain that we minimize the token-averaged Jensen–Shannon divergence between the frozen Ouro teacher’s final-loop next-token distribution and MELT’s next-token distribution. MELT’s logits are computed as an exit-probability-weighted sum of the logits produced after the full transformer stack at each of its four loops. This weights are token-dependent and sum to one, with the final loop receiving the remaining exit probability. Thus, all loops receive supervision through the aggregated logits, without separate losses on intermediate layers or hidden representations.

Cached states are not detached at chunk boundaries. Gradients propagate across chunks. This has also been clarified in the paper.

About the choice of chunk size (500) and inference window (50), we fully agree that these were underexplained in the paper, so we expanded our results in Appendix C.2, which we now call "Hyperparameter ablations". See new Tables 10 (MELT-1.6B, chunk size 500) and 11 (chunk size 100) for the results.

Regarding window size, looking at the first table we observe that performance generally improves as the window size increases up to 50 tokens. Beyond 50, accuracy fluctuates without consistent gains, so we select 50 tokens to balance accuracy and window size. However, in this range the exact window size does not have a significant impact on memory usage or accuracy: while having a small window is clearly much better than not having one, in the heavy reasoning setup that we study, a difference of a few tens of tokens has a negligible impact on memory.

About ablating the chunk size, while ablating the chunk size is very expensive (and it gets even more expensive in time as chunk size decreases), we performed a new run reducing the chunk size from 500 to 100. In the second table, we also report accuracies for all the window sizes as well. We observe no gain from decreasing the chunk size to 100, and since a larger chunk size is much faster to train, we stay with the previous setup of 500.

About removing chunk-wise from our training, Table 4 shows that fully parallel SFT substantially degrades performance, even relative to the untrained initialization. We therefore expect chunk-wise training to remain important when distillation is retained, as training and inference work too differently in that case for the training signal to be useful.

Remaining from Weakness 1

The reviewer also suggests fine-tuning Ouro on the same data. We agree that this is a very useful ablation to check whether MELT benefits unfairly from better data. We have added Appendix F, where we show results (Table 13) for Ouro-1.4B-Thinking and Qwen3-1.7B fine-tuned on the same data as MELT. Results show that Ouro’s performance is maintained after fine-tuning, while Qwen3’s performance decreases slightly, possibly because its original training data was better. These results suggest that MELT’s performance is not simply due to the additional training data.

Thank you for the work you have put in the rebuttal. The clarified distillation objective, sliding-window and chunk size analysis as well as the data control ablations address my concerns. I now view the results and presentation in the paper more positive and I will increase my score from 1 to 2.

The experiments establish adaptation of pretrained Ouro models, but leave open whether the MELT is an effective architecture throughout pre-training. Even a small scale pretraining study would strengthen my confidence. However, I recognize that this would extend beyond the current post-trainign scope. Moreover, while adaptation to other looped model families seems plausible, the paper does not provide empirical evidence for this. Extending to other model families would establish MELT as general post-training method for looped transformers. Given the scores of the other reviewers I regard the overall rating as fair.

AuthorsOpenReview ↗

Authors response

We thank the reviewer for acknowledging that we adressed their concerns and for raising the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewers for their constructive feedback and for recognizing MELT’s timely motivation, substantial 3–4× KV-cache reduction, and broadly preserved performance at two model scales. We also thank the reviewers for their feedback and proposed experiments. Here, we summarize the main concerns that reviewers acknowledged as resolved and explain how we addressed them:

  • Window- and chunk-size ablations (nFbe, uuNc, n28K). This issue was raised by all three reviewers, and we agree it was necessary. Therefore, we performed window-size sweeps for both MELT-1.6B and a 100-token chunk training run, which we report in Appendix C.2, Tables 10 and 11, supporting the chosen configuration.
  • Untrained cache-sharing experiments (nFbe, uuNc). We clarified the differences between the experimental setups in the Ouro paper and Tables 5 and 6, and expanded Table 6 with untrained Last-sharing evaluations on Ouro-1.4B and experiments with a 50-token window
  • Expanding MELT to other looped model families (n28K). We clarified how MELT can be appiled to other model families.
  • Expanded analyses. We added accuracy and efficiency results for the trained Last-sharing ablation (nFbe), an analysis of the learned gates (uuNc), and same-data SFT controls using Ouro-Thinking-1.4B (nFbe, n28K).
  • Clarifications throughout the paper (nFbe, uuNc, n28K). The revised version includes clarifications of the training objective, gradient flow, memory scaling and budget accounting, among others. All changes are marked in blue.

We also thank all three reviewers for their engagement throughout the discussion and for raising their scores following our responses and revisions. Their feedback helped us strengthen the paper, and we appreciate their acknowledgment of the concerns addressed. We believe the revised manuscript provides a more thorough and transparent account of MELT’s benefits and limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.