- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces MELT, a framework designed to address the KV-cache memory challenges in looped LMs. The approach is mainly based on the Ouro LM. The problem statement correctly identifies that although looped LMs offer parameter efficiency, their KV-cache size scales linearly with the number of loops. The objective is to reduce the KV-cache size, and to achieve this, the authors introduce architectural changes where tokens outside a sliding window use a shared KV representation across recurrent iterations, updated through a learnable gating mechanism. To adapt the pretrained Ouro model to this new architecture, they propose chunk-wise training and knowledge distillation across recurrent steps.
Critical strengths
-
In my opinion, the greatest strength of this paper is identifying a timely and relevant problem: a concrete architectural inefficiency in this increasingly popular class of models, and proposing a solution to address it.
-
The ~3–4× reduction in memory usage is quite a substantial empirical improvement.
-
The proposed pipeline, including the architectural design and adaptation procedure, is novel and reasonable.
Critical weaknesses
-
I am not entirely sure whether to consider this a weakness, but when I initially read the paper, I thought the authors aimed to generally address the issue of KV-cache memory growth per iteration. However, it turns out that there is a sliding window (W=50) within which the KV-cache remains similar to Ouro, and compression is only applied outside this window, to older tokens. I think this distinction should be stated more clearly in the paper.
-
Regarding the sliding window, it is not sufficiently ablated, and the choice of W=50 seems somewhat arbitrary, especially since it appears to be an important component of the proposed approach.
-
Although MELT is motivated as a solution to the KV-cache memory challenges of looped LMs, in practice, its architectural design and evaluation are essentially limited to Ouro. Whether it is applicable to other architectures, such as RLMs, LoopFormer, Recurrent Depth Transformers, PARCAE, or other recurrent architectures, is not sufficiently discussed.
-
As shown in Table 10 of Appendix D, although the memory savings are substantial, they do not translate into decoding throughput improvements. In fact, MELT reduces the decoding throughput from 13.5 to 8.5 tokens/s.
-
It seems that even within the same architecture, the performance gains are not always consistent. As the paper moves from the smaller Ouro model to the larger one in Table 2, the results on both mathematical and non-mathematical benchmarks are affected.
Questions for authors
-
In Tables 1 and 2, there is little discussion of why the performance changes, particularly the improvements on HumanEval+. How does MELT achieve such substantial performance gains on this benchmark?
-
How do the authors envision extending MELT to other looped models? Is there a specific reason why the experiments focus exclusively on Ouro?
5 comments
Authors response (2/2)
Q1
This is a nice observation from the reviewer, which we had not noticed. A brief analysis of the responses of MELT and Ouro suggests that MELT responses on HumanEval+ are shorter and contain less repetitive reasoning . The number of responses exceeding 30k tokens decreases from 11 to 4 for MELT-1.6B and from 7 to 3 for MELT-3.1B, compared to their respective Ouro-Thinking baselines.
One possible explanation is that the finetuning data encourages shorter responses. To test whether the improvement comes from the data or the model, we are working on a data ablation that finetunes Ouro-Thinking under the same conditions and with the same data.
Q2
We chose Ouro because it is state of the art at its size, publicly available, and offers two model sizes for evaluation. For how to apply MELT to other models, see our response to W3.
I thank the authors for their answers to my concerns and running the experiments. I know it's a busy time so if the authors found some of my suggestions made them run too many unnecessary experiments, I apologize for it. Regarding the weaknesses and concerns: I feel like the authors have provided acceptable responses to W1 and W2. Regarding W3, I am confused on the authors' response on how they think the approach could be extended to other models; and if it's possible to show some results for it. For W4, I think at the moment that table is Table 12 in your new draft? I still don't an update in caption, and I am not quite sure what do the authors mean about the throughput and tokens/s. Is your claim about how your method increases throughput but the decoding per token gets slower? For W5 I think the response is acceptable.
Thanks for the answering the questions. I am leaning toward increasing my score and will wait to see if the authors response to my comments here.
Authors response (1/2)
We thank the reviewer for their feedback and suggestions, which we think helped strengthen the paper. We believe the revisions address their concerns. We have submitted an updated version of the paper, with additions highlighted in blue so they are easy to find.
W1
Without the sliding window, MELT’s KV cache size scales as O(N×L). Adding the window changes this to O(N×(L−W)+N×T×W). Here, W≪L is the window size, N is the number of layers, T is the number of iterations and L is the number of tokens. Given that in Ouro KV size grows like O(N×L×T) and that W≪L and that we focus on long reasoning scenarios, we think that the claim that we "reduce memory growth by iteration" is fair. For example, with a 32K-token completion, four iterations, and a 50-token window, MELT’s KV cache is approximately 4× smaller than Ouro’s.
We have now clarified the explanation of the KV size scaling for Ouro and MELT in Section 3.2. Hopefully this avoids further confusion.
W2
We agree that the local window decision is underexplained in the paper, so we expanded our results in Appendix C.2, which we now call "Hyperparameter ablations". Results are also shown in the table below:
We observe that for low window values performance is worse but it stabilizes at around 10-100 tokens and does not increase much after that. We chose 50 since it provides a good tradeoff between accuracy and memory. However, in this range the exact window size does not have a significant impact on memory usage or accuracy: while having a small window is clearly much better than not having one, in the heavy reasoning setup that we study, a difference of a few tens of tokens has a negligible impact on memory.
In Appendix C.2 we have also added an equivalent table (Table 11) for a model trained with a chunk size of 100, that the reviewer might appreciate.
W3
MELT could be applied directly to any looped transformer architecture that repeats layers and has separate KV cache for each loop. This includes most Recurrent Depth Transformers (LoopFormer, Huggin, PARCAE, Nanbeige...). For RLMs, if the reviewer refers to https://arxiv.org/abs/2512.24601v1, here the recursion occurs through repeated model calls rather than repeated layers within the model, so MELT does not directly apply.
Given our experiments, we propose to first pre-train the models without MELT, which is faster, and then do our post-training with the chunk-wise distillation loss. Training with MELT-like KV-cache from scratch would take more time. As an example, say we wanted to apply MELT to Nanbeige4.2-3B, which keeps separate KV caches for each of the two loops. We would take the weights of the model and initialize the gates with random values. Then, we would train on the same reasoning dataset with our distillation loss to get the MELT version of Nanbeige4.2-3B. We have also added a comment about this in Section 3.2.
W4
Table 10 reports the decoding time for batch size 1, which is expected to be higher with MELT since it adds the gating operations with respect to Ouro. However, because of the memory savings the maximum batch size increases much more, so with maximum batch size the decoding time throughput would be much faster for MELT than for Ouro. We have clarified this in the caption of the table.
W5
From our point of view, the results for MELT-3.1B are very similar to those of MELT-1.6B. At both scales, the majority of accuracy differences between MELT and Ouro are within the reported standard errors (originally Table 2 did not contain standard errors, we have now added them). MELT slightly outperforms Ouro on non-math benchmarks on average and slightly underperforms on math benchmarks on average. We think that this consistent pattern across both model sizes strengthens our results.