acceptodds
Under review as a conference paper at ICLR 2027

Enabling dynamic computation in Looped LMs

Abstract

Looped LMs are parameter efficient and promise dynamic computation (saving memory and FLOPs on easy tokens). However, state-of-the-art open Looped LMs trained with this dynamic computation capability (Ouro models) do not realize it in practice as each loop iteration (depth) requires its own level of KV-cache, necessitating all loop computations. Moreover, Ouro's early-exit prior is enforced on each token equally, which results in static lower-depth like processing of all tokens regardless of difficulty. In this work, we propose a simple “best-available” KV caching strategy that works out-of-the-box, creating a new frontier in the performance vs depth space. Our approach enables up to 30% reduction in FLOPs and KV memory while retaining full-depth performance, showing the true flexibility of Looped LMs. Furthermore, training looped LMs with awareness about this KV caching strategy improves performance and efficiency. Finally, we apply a small but effective fix to the early-exit prior enforcement objective that makes tokens exit at truly heterogeneous depths based on effort. Our findings are validated on Ouro models as well as smaller looped LMs pre-trained from scratch.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Model, LLM, LLM efficiency, Dynamic Computation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A5TBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In looped language models such as Ouro, later tokens attend to earlier tokens' KV entries at every depth, so per-token early exit saves neither compute nor memory. The paper proposes Ragged KV, in which deeper iterations read an exited token's deepest available KV entry. Implemented through vLLM page aliasing, it is applied without training to four Ouro checkpoints, mainly with full-depth prefill (Dense-Ragged). On Ouro-2.6B-Thinking it matches four-iteration accuracy at an average decode depth of 2.83. The paper also argues that Ouro's per-token exit objective makes nearly all tokens exit at one depth, and proposes a batch-level ("global") prior and ragged-aware training, evaluated by perplexity on a 160M looped model.

Critical strengths
  1. Ragged KV recovers much of the accuracy that truncation loses at low depth, without any training. Between two and three decode iterations it keeps far more accuracy than simply cutting depth on both Ouro sizes, and a traced run shows real KV memory savings. That makes it a cheap, drop-in option for serving looped models, not just a FLOP count.

  2. The diagnosis of Ouro's exit objective is useful in its own right. Showing that a per-token KL pulls every token toward one exit distribution explains why a training-free cache change works on Ouro at all, and shows what a better exit objective must fix.

Critical weaknesses
  1. The efficiency claim rests on a weak reference point and a mechanism close to established practice. The headline "up to 30% reduction in FLOPs and KV memory" is relative to four fixed iterations, yet three already nearly match four on both thinking models, and the claimed operating point (decode depth 2.83) sits barely below three. Counting prefill, Dense-Ragged falls a few points below a mix of three and four fixed iterations on non-thinking BBH (Fig. 16), and memory and speed are measured at other operating points, so no reported configuration shows full-depth accuracy with measured savings. Reusing an exited token's deepest state is established (Elbayad et al., 2020; Schuster et al., 2022; Geiping et al., 2025), and Ouro's own last-step KV reuse, which its authors report near full-cache accuracy, is dismissed without per-benchmark results. Efficiency is the main selling point, and against sensible baselines the advantage is much narrower than claimed.

  2. The case for the global prior and ragged-aware training is too thin for its general claims. Conclusions about training objectives rest on one 160M model, perplexity only, where the deciding differences are hundredths of a point and no repeats or seed variance are reported. The prior is not even reached: global's average exit depth stays about one iteration above the prior's mean, so the effects cannot be credited to it. Ragged-aware training brings global below its own four-iteration perplexity, but that baseline is already worse than local's, and at similar depth global only roughly matches local. Global does produce far more large depth jumps between adjacent tokens, yet difficulty alignment rests on a few qualitative samples, and in the illustrating trace the commas and semicolons cited as easy nearly all exit at full depth. The conclusion's "true dynamic exits with difficulty awareness" is therefore not yet supported.

Questions for authors
  1. On all four models, how does Dense-Ragged compare in overall depth with a request-level mixture of three and four fixed iterations?
  2. Could you add Ouro's last-step KV reuse as a baseline in Tables 1 and 2, and explain how Ragged KV differs from the deepest-state reuse in Elbayad et al., Schuster et al. and Geiping et al. (Remark 6.1)?
  3. For the 160M runs, can you report several seeds, the measured KL to the prior, and a difficulty measure such as the correlation between exit depth and full-depth token NLL, comparing local and global at matched depth?
  4. Your four-iteration MATH500 score for Ouro-1.4B is well below the 82.40 that Ouro reports, and on this model both Ragged modes trail full depth on MMLU-Pro and BBH. Do these gaps persist under Ouro's evaluation protocol?
4 comments

Thanks for your deliberate review and for highlighting the intended strengths of our paper. We apologize for the delay in sending our rebuttal.

Before we address your concerns, we want to underline: Ragged KV performing close to fixed@4 out-of-the-box with Ouro is a welcome by-product of the way it is trained. People can use it today without any training for better inference efficiency. Our contribution is finding this out-of-the-box ability. We do not claim that Ouro models are most performant in this mode. That would require correctly training the model and making it ragged KV aware, which we discussed in section 4 and 5.

Now about your concerns:


Reproducing Ouro's reported numbers

All our reported numbers follow the protocol described in the Ouro paper (Tables 16, 17), as much as possible. This is mentioned in our appendix section D, Tables 4,5,6. First and most importantly, Ouro reports a single number performance, while we measured p@1 and p@4 with up to 16 roll-outs for high reliability. Second, for many benchmarks (especially logprob eval cases), Ouro uses the standard lm-eval-harness which we follow down to the number of CoT shots, temperature, top_p, etc; but in many generation cases, Ouro uses an "in-house harness" which is not publicly accessible. The Ouro paper also does not mention the LLM used as judge (therefore we used the best open-sourced model that we could run: Qwen 3.8-27B, judge details in D.3). We used the standard lm-eval-harness to generate responses and evaluated all answers with this judge. With this setup, we are unable to reproduce some numbers (like MATH500) reported in the Ouro paper. To maintain fairness, all inference-modes (fixed/ragged) use the same evaluation setup in our study.

Ouro's own last-step KV reuse, which its authors report near full-cache accuracy, is dismissed

This was the first baseline we tested. Unfortunately, using a single last-step KV cache drops performance to near zero in our reproduction attempts. In Table 17, we reported BBH performance using a single shared KV cache under ragged decode at various thresholds. We tested both: the last-step KV (like Ouro), as well as the exit-step KV. In both cases, the performance dropped to near zero. Our finding is consistent with Vendrell et. al. (https://arxiv.org/abs/2605.07721) Appendix B, Table 6, who find a similar performance drop to zero with shared KV. It is also noteworthy that using the last-step KV requires computing all 4 loops for each token (so it only saves KV memory, no FLOPs). Since this setting fails so severely, we did not run all benchmarks in this setting.


Clarification regarding Dense-Ragged and avg. exit depth

There is a stark difference between the thinking and non-thinking models when it comes to overall avg. exit depths for dense-ragged modes. It is because non-thinking models are evaluated in few-shot CoT setting. Prompt tokens constitute majority of all processed tokens in non-thinking models, but are a tiny fraction in thinking models (see table below). We will add the avg. overall depth for dense-ragged in the tables (like Tables 14-16) to make this clear.

ModelInference modeavg. decode depthavg. overall depth% prompt tokens
Ouro-1.4Bdense @ 0.3 / 0.52.39 / 3.053.76 / 3.8685.0%
Ouro-2.6Bdense @ 0.3 / 0.52.21 / 3.033.64 / 3.8179.8%
Ouro-1.4B-Thinkingdense @ 0.3 / 0.45 / 0.52.30 / 2.50 / 2.922.33 / 2.52 / 2.9401.6%
Ouro-2.6B-Thinkingdense @ 0.3 / 0.45 / 0.52.24 / 2.45 / 2.832.28 / 2.49 / 2.8502.2%

Since the savings vs performance tradeoff for dense-ragged modes in non-thinking models seems not worth it, we will revise the paper to recommend using fixed-depth when possible in non-thinking models. ETE-ragged mode may still be preferred when inference efficiency is critical, especially if the desired avg. depth is between 2 and 3, since ETE-ragged comfortably sits much higher than fixed@2 performance (even better than request-level mixture as discussed later in this rebuttal).

We posit that each few-shot example adds (potentially surprising) valuable information for the non-thinking model to perform In-Context Learning, and hence benefits from full depth prompt processing. This is not the case with thinking models, where most of the work is done in decode, and overall avg. depth is pretty much the same as avg. decode depth in dense-ragged mode. We recommend all ragged modes for thinking models.

how does Dense-Ragged compare in overall depth with a request-level mixture of three and four fixed iterations?

Request-level mixture is a great suggestion to compare Ragged with fixed-depth modes at matched avg. depths. Assuming the difficulty of queries is uniformly distributed, we simulate the mixture by randomly routing queries to depth {2, 3, 4} to achieve the desired avg. depth. We average the simulated performance over 200 draws for reliability. We report numbers at τ=0.3,0.45,0.5\tau=0.3, 0.45, 0.5 in order separated by / (0.45 was not calculated for non-thinking models).

ModelMatched depthDense-Ragged p@1 (%)Mixture p@1 (mean ± SD, %)Δ (pp)
Ouro-1.4B3.625 / 3.78459.54 / 62.8862.94±0.09 / 63.49±0.08−3.40 / −0.61
Ouro-2.6B3.550 / 3.75769.68 / 72.2771.90±0.12 / 72.13±0.10−2.23 / +0.14
Ouro-1.4B-Thinking2.364 / 2.586 / 2.94949.63 / 50.28 / 52.2743.17±0.72 / 46.39±0.66 / 50.81±0.30+6.46 / +3.90 / +1.46
Ouro-2.6B-Thinking2.268 / 2.493 / 2.85558.38 / 59.34 / 62.2652.48±0.55 / 55.55±0.62 / 59.99±0.46+5.89 / +3.79 / +2.27

Like mentioned earlier, Dense-Ragged doesn't offer good value for non-thinking models and we would recommend using fixed-depth when possible. But for thinking models, Ragged provides real performance gains compared to fixed-depth mixtures. For efficiency critical cases in 1.4B, ETE-ragged still provides better value than fixed-depth mixtures. 2.6B has a known prefix instability issue causing catastrophic performance drops in ETE ragged (see Figure 21 in the draft). We will update the paper with these results to further strengthen our case.

ModelMatched depthETE-Ragged p@1 (%)Mixture p@1 (mean ± SD, %)Δ (pp)
Ouro-1.4B2.626 / 3.22158.01 / 62.4551.41±0.16 / 61.24±0.07+6.60 / +1.21
Ouro-2.6B†3.100 / 3.67752.72 / 52.9370.67±0.09 / 71.92±0.11−17.95 / −18.99
Ouro-1.4B-Thinking2.329 / 2.536 / 2.90946.67 / 50.19 / 52.0542.85±0.71 / 45.87±0.68 / 50.48±0.37+3.82 / +4.32 / +1.57
Ouro-2.6B-Thinking2.259 / 2.564 / 2.96858.46 / 60.57 / 61.9852.63±0.55 / 56.74±0.60 / 61.30±0.30+5.82 / +3.83 / +0.68

"up to 30% reduction in FLOPs and KV memory" is relative to four fixed iterations, yet three already nearly match four on both thinking models

We agree: fixed@3 performance follows fixed@4 closely in Ouro thinking models. But our headline is calculated relative to Ouro's officially recommended inference setting of fixed@4. And note that we never claimed out-of-the-box ragged performance should beat fixed@n. It just happens to be the case sometimes and we only report it for reference.


Related Work

Thanks for highlighting these related works, some of which we had missed earlier. In our literature survey, we had found AdaPonderLM the closest to Ragged KV, but Elbayad et al. and other subsequent works already proposed similar caching mechanisms. We will re-write portions of the paper to remove any claims of Ragged KV being a novel caching mechanism proposal, and properly cite these works. Note that, our central contributions remain original and orthogonal to this.

  1. Showing that Ragged KV works out-of-the-box for Ouro, enabling true inference efficiency without any re-training.
  2. Fixing the training objective to make it ragged KV aware and enabling truly dynamic exits.

Ouro 160M experiments: local vs global

The prior is not even reached ... can you report the measured KL to the prior

The prior is only our expectation of what ideal token-routing should look like. What exit distribution is actually achieved depends on the tension between LM loss and early-exit regularization, controlled by β\beta. With a higher β\beta, we were easily able to reach the prior, but training collapses to poor performance (Figure 7). Even the pre-trained Ouro models trained with trillions of tokens never reach their intended uniform exit distribution (in fact, they hardly ever exit at depth 1, see Figure 2). But we find that global is much more effective in reducing the average exit-distribution's KL-to-prior than local. The table below shows numbers for the 20B checkpoints. A more detailed analysis can be seen in our response to Rh5J, and the corresponding training evolution Figure 22 added to the draft.

Priorlocal: KLglobal: KL
Poisson0.77190.4504
Geometric0.91050.5472
Uniform0.72520.3754

comparing global and local at matched depths

We followed Rh5J's suggestion and swept τ\tau to get depth-match performance numbers. In summary, we found that at matched avg. depth, global beats local under geometric (depth: 3.2169, Δ PPL: −0.0283) and uniform (depth: 3.3828, Δ PPL: −0.0997) priors. Poisson local collapses to lower-fixed-depth behavior (sends >90% of its tokens to depth 3, resulting in low cache mismatch), hence beats the dynamic global variant. Please see that response for details and Figure 23 in the updated draft for the complete sweep comparison plot.

difficulty alignment ... in the illustrating trace the commas and semicolons cited as easy nearly all exit at full depth

We believe this was misinterpreted. The depth written adjacent to each token is what the model uses to process the next token's prediction. To assess the model's perceived "difficulty" of commas and semicolons, please the the previous token's exit depth. For example, storing[2] ,[4], ues[2] ;[4], result[1] ,[3], lack[1] of[4] funding[2] for[4], etc. We will make it clearer in the paper. Figure 22 also shows the token-depth mutual information (as broken down in our decomposition of the local KL term). It measures how dependent the exit depth is on the token itself. While local tokens exit kind of independent of the token, global is highly token-dependent, supporting its difficulty awareness.

a difficulty measure such as the correlation between exit depth and full-depth token NLL

Thanks for this valuable suggestion. We measured the correlation between exit-depth and depth-4 NLL, fixing global τ=0.5 and calibrating local τ to match average exit depth.

PriorMean depth (local / global)local ρglobal ρ
Poisson3.082 / 3.0930.0990.386
Geometric3.169 / 3.1780.1730.397
Uniform3.423 / 3.4140.3290.418

global exhibits stronger difficulty alignment across all three priors. The same conclusion holds using depth-1 NLL and the loss reduction from depth 1 to depth 4, supporting alignment with both initial difficulty and the benefit of additional computation in global. We added Figure 24 in the draft to show the complete picture.

conclusions rest on a single 160M model ... no repeats or seed variance are reported

We agree that multiple seeds make findings more reliable. That is precisely why we reported p@1,4 out of up to 16 rollouts for Ouro performance tables instead of a single number. However, pre-training experiments are expensive, and it is standard practice to report single run numbers (Saunshi et al. 2025, Bae et al. 2025, even a reference you cited: Geiping et al. 2025). Each of our 160M training run costs ~50 H100 hours. Additionally, we ran multiple ragged-aware training campaigns (Appendix C) to find optimal training recipes. This along with huge evaluation campaigns in multiple inference modes for Ouro, restricted our ability to perform multiple pre-training runs, or compare with other architectures/algorithms. We also made every effort to make the one seed as fair to each mode as possible (Appendix B). We hope you can appreciate our decision to focus resources on a single important pre-training research question instead of trying to solve everything at once with limited resources.


We recommend reading our response to avNT for new increased throughput results, and our response to Rh5J for more theoretical and empirical analysis of local vs global training. Our currently updated draft contains additional figures in the appendix to support the rebuttal. We are updating it with all promised changes. We hope this resolves your concerns and are happy to take more questions, if any.

Thanks again for your valuable feedback!

A gentle reminder that today is the last day for discussion. We are happy to take any follow-up questions. In case you are satisfied with our rebuttal, we would really appreciate acknowledgement and raising the score.

Reviewer_Rh5JOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper looks at token-level early exiting in looped language models. The authors focus on two issues with early exiting. First, if a token tt exits at some loop l<Ll < L, any subsequent token t′>tt' > t in the generation stream that computes deeper than layer ll will be missing an entry in the KV cache for position tt for all layers l′>ll' > l. They propose "Ragged KV" which simply the KV cache from the exit depth ll to all deeper layers l′>ll' > l. Second, for the particular type of "Ouro" family of looped models that they consider, they propose a change to the early exit distribution regularizer that they claim improves exit depth sensitivity with respect to token difficulty. Empirically the paper evaluates on 1.4B and 2.6B models (both for zero-shot with Ragged KV, and after pretraining with the new loss function and RaggedKV).

Critical strengths

Early exiting with looped models is an exciting and impactful area for continuing to improve LLM serving efficiency. Figuring out exactly the tricks required to get it to work, efficiently, is therefore timely and important. This paper provides a useful empirical study of some of the main questions related to learning when to exit and how to propagate state.

Critical weaknesses
  1. I'm not quite convinced by the theoretical motivation or empirical results for the global vs. local regularizer.

First, the paper claims that per-token KL regularization against a prior πp\pi_p forces similar exit distributions across all tokens, regardless of difficulty. This is not quite clear to me. Suppose that we take the loss for a particular token ii, and minimize just the exit depth distribution πi\pi_i over the simplex:

arg⁡min⁡πi∑dπi(d)ℓi,d+βKL(πi∥πp)\arg\min_{\pi_i} \sum_d \pi_i(d) \ell_{i, d} + \beta \mathrm{KL}(\pi_i \Vert \pi_p)

This gives πi∗∝πp(d)exp⁡(−ℓi,d/β)\pi_i^* \propto \pi_p(d) \exp(-\ell_{i,d} / \beta), where β\beta is acting as a "temperature" so-to-speak on the loss differences between depths. If token ii benefits from greater depth, where ℓi,L≪ℓi,1\ell_{i,L} \ll \ell_{i,1}, then more exit mass will be put on deeper loops. Whether this ends up looking "homogenous" or not in practice mainly depends on how to set β\beta (which, I agree, can be tricky --- and both in form and in spirit it shares a lot of similarity with other such finnicky KL regularizers like in RLHF/DPO). But even so, empirically the results in Section 3 on pretrained Ouro (which is trained with the local objective) still shows that even though only ~20% of decode tokens exit at depth > 2, you still get a ~10pp pass@1 average improvement. So the claim still doesn't seem quite supported? Of course there is the caveat that the pretrained Ouro model does not only use the objective stated in this paper; it also finetunes the exit depth a "stage II". So I'm also not quite sure how that fits into the picture.

Now, when we go to the global objective, the KL divergence is computed using the batch mean πˉ(d)=1N∑iπi(d)\bar{\pi}(d) = \frac{1}{N} \sum_i \pi_i(d). Treat πˉ(d)\bar{\pi}(d) as an auxiliary variable, and minimize πi(d)\pi_i(d) subject to the constraint that πˉ(d)=1N∑iπi(d)\bar{\pi}(d) = \frac{1}{N} \sum_i \pi_i(d). Then for piˉ(d)\bar{pi}(d) fixed at its optimum, each token solves min⁡πi∑dπi(d)(ℓi,d+βcd)\min_{\pi_i} \sum_d \pi_i(d) (\ell_{i,d} + \beta c_d), where cdc_d is a depth-specific constant. Without ties, the solution is a vertex (one-hot exit layer).

So indeed I would expect the exit distribution to be peaky -- I would also expect this peakiness to also potentially hurt training the full backbone by cutting off loss signal to deeper, non-selected layers. It seems empirically that the global objective indeed has worse full-depth (Fixed@4) validation perplexity across all priors. Then, even with ragged-aware training, [email protected] validation perplexity at best roughly ties local (while remaining worse than local's Fixed@4) on the Poisson and Geometric priors where the avg. exit depth is comparable (on the Uniform prior the exit depth is lower, but the perplexity is worse, so it is hard to compare).

  1. There is a substantial amount of related prior work on early-exit KV handling that is understated or not covered in the paper. Copying the exited token's states to deeper layers was used in Depth-Adaptive Transformer (Elbayad et al., 2020) and CALM (Schuster et al., 2022), as well as others thereafter. None of these are cited. Additionally, prior work such as Mixture-of-Recursions (Bae et al., 2026) and Relaxed Recursive Transformers (Bae et al., 2025) that is cited still isn't quite positioned properly, or compared to. In particular, MoR directly studies token-level adaptive depth in recursive transformers, including various strategies for how to handle the KV cache. A comparison against the MoR strategies seems needed.
Questions for authors

I would like to see perplexity at matched average exit depth (e.g., by sweeping \tau for local). Is there a setting where global gives lower PPL than local at the same average exit depth?

Does the collapse at \beta = 0.1 still happen if you keep Ouro's RMSNorm? It seems like there were a couple of tweaks that changed things at the same time.

3 comments

Thanks for your careful reading and detailed review of our paper! We apologize for the delay in our rebuttal.

Before we address your concerns, we want to highlight that our contributions are:

  1. Showing that Ragged KV works out-of-the-box for Ouro, so anyone can use it today for real inference efficiency without any training.
  2. Fixing a mis-specification in Ouro's training objective, that enables truly dynamic exits without overheads as originally intended.

Importantly, we do not claim that our method beats all alternative architectures/algorithms to train looped LMs (which is an active area of research). Our work aims to highlight fundamental limitations in existing work and provide solutions for future work to use/improve. Now, about your concerns:


Theoretical motivations for global vs local

We appreciate your analysis, and agree that keeping the backbone (and hence the per-depth token loss) fixed, minimizing the local loss will simply choose depth according to token losses with β\beta acting like temperature; and minimizing the global loss incentivizes sharp exits. However, this analysis does not completely characterize the joint training of both the exit gate and the backbone together. To simplify, we decompose the local depth regularization term though this identity:

1N∑iKL(πi∥πp)=KL(πˉ∥πp)+1N∑iKL(πi∥πˉ)\frac{1}{N}\sum_i \mathrm{KL} (\pi_i \| \pi^p) =\mathrm{KL}(\bar\pi\|\pi^p) +\frac{1}{N}\sum_i\mathrm{KL}(\pi_i\|\bar\pi)

The local loss is exactly equal to the global loss and an additional token-depth mutual information term. This term penalizes token-dependent exits, and is only zero when all tokens exit with identical distributions. Therefore, we claimed that local objective forces similar exit distributions.

However, as you pointed out, whether this objective causes true homogeneous exits remains an empirical question depending primarily on the β\beta schedule. Hence, we will add this analysis in the paper and re-word our claims to say incentivize instead of force.


Empirical evidence for global vs local

Another reason why we made a strong claim is due to empirical findings. Figure 5 clearly shows more than 90% of tokens exit at depth 3 with a fixed τ=0.5\tau=0.5 (a strong case for identical exit distributions). With the analysis above, we additionally calculated the KL(πˉ∥πp)\mathrm{KL}(\bar\pi\|\pi^p) and mutual information (MI) 1N∑iKL(πi∥πˉ)\frac{1}{N}\sum_i\mathrm{KL}(\pi_i\|\bar\pi) terms on validation tokens.

Priorlocal: KLlocal: MIglobal: KLglobal: MI
Poisson0.77190.14870.45041.1461
Geometric0.91050.15490.54721.1370
Uniform0.72520.17320.37540.9896

global is able to reduce the aggregate KL-to-prior term significantly more than local because local is implicitly reducing the MI term as well. Since global is not, we see a high token-dependent MI term which supports its highly heterogeneous exits. Please see Figure 22 in the updated draft for a complete evolution of these terms across training.

even though only ~20% of decode tokens exit at depth > 2, you still get a ~10pp pass@1 average improvement

Since local penalizes heterogeneous exits, we posit that tokens that still exit deeper at a given threshold would provide very high value compared to others. Hence, a small portion of tokens can have disproportionate impact on performance. global relaxes the homogeneity constraint, providing a way to freely distribute tokens across depths. And like you mentioned, some stage-II training may also affect this, which we were unable to analyze.

perplexity at matched average exit depth ... Is there a setting where global gives lower PPL than local at the same average exit depth?

Thanks for suggesting this evaluation. We matched average exit depth by sweeping τ\tau and found that global beats local in the case of Geometric and Uniform priors (see table below). It loses in the Poisson case, where local behaves like a fixed lower-depth model with very little depth variation (>90% exit at depth 3), resulting in better cache alignment. Please see Figure 23 in the updated draft for the complete sweep comparison plot.

PriorAvg. exit depthlocal PPLglobal PPLΔ PPL (global − local)
Geometric3.216917.353817.3255−0.0283
Uniform3.382817.342417.2427−0.0997
Poisson3.133517.307417.3496+0.0423

For reference, we trained a no-early-exit 3-loop model which achieved 17.07 PPL, compared with the Poisson local's 17.18 at mean depth 3.02, despite the latter training with four loops (extra training FLOPs). So if performance is the only concern, early-exit should probably not be used. But if we want early-exit, global objective enables truly dynamic difficulty-aware exits, which potentially scale better.

expect this peakiness to also potentially hurt training the full backbone by cutting off loss signal to deeper, non-selected layers

Yes, we discuss this in Section 5. In the dense loss steps, higher heterogeneity cuts off loss signal to deeper layers (resulting in lower fixed@4 performance) and also causes more training-inference mismatch with ragged KV. With our proposed alternate ragged aware training steps, this effect is significantly reduced.

Does the collapse at \beta = 0.1 still happen if you keep Ouro's RMSNorm?

Yes. The collapse run in Figure 7 retained Ouro’s per-loop RMSNorm. With RMSNorm retained and β ramped early to 0.1, the local-regularized models’ Fixed@1 versus Fixed@4 perplexities were 19.855/19.828 (Poisson), 19.827/19.819 (geometric), and 19.904/19.808 (uniform), showing almost no benefit from additional loops. A β=0 control retaining the per-loop norm showed substantial depth benefit: 34.630 at Fixed@1 versus 19.713 at Fixed@4. We removed the norm in later experiments that showed faster learning. We will clarify this chronology in the paper. Apart from removing per-loop norm, the architecture remained identical to original Ouro (including per-layer norms).


Positioning among prior work

We regret that some key references you highlighted were missed earlier. We will re-write our related work, introduction and abstract to better position our work in relation to these works.

Regarding Mixture-of-Recursions, there are several archictecutral ablations like cycle vs sequence looping, full vs middle looping, expert-choice routing vs token-choice routing, and corresponding KV caching strategies. Moreover, their expert routing uses top-k routing and hierarchical filtering to enforce early-exit like mechanism in a static manner unlike ours which is enforced dynamically through a prior, making their comparison non-trivial. With the compute budget for our study (a lot of which was used in evaluating Ouro models on diverse datasets with Ragged KV cache), we chose to restrict our training study to a fixed architecture (Ouro) and early-exit mechanism. Training each 160M model alone costs ~50 H100 hours of compute for our study. We also do not claim a universally winning training recipe for looped models. We hope that you will appreciate our choice to perform an in-depth but quite exhaustive study on a focused problem rather than covering breadth of architectures and not answering any research questions meaningfully.


We recommend reading our response to avNT for new increased throughput results, and our response to A5TB for additional empirical analysis of local vs global. Our currently updated draft contains additional figures in the appendix to support the rebuttal. We are updating it with all promised changes. We hope this resolves your concerns and are happy to take more questions, if any.

Thanks again for your valuable feedback!

A gentle reminder that today is the last day for discussion. We are happy to take any follow-up questions. In case you are satisfied with our rebuttal, we would really appreciate acknowledgement and raising the score.

Reviewer_avNTOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

A compelling and well-written paper on a ragged KV-cache aware strategy for dynamic looped LLMs. The paper describes Ouro's dynamic early exit strategy for looped-LLMs, where certain tokens are early-exited based on some gating function. However, Ouro does not exploit this with efficient KV-cache support, and accordingly does not reduce FLOPs or equally important, memory during inference.

This paper plugs in this gap, by proposing a ragged KV-cache approach: early exited tokens' KV-cache should be aliased such that tokens that aren't early exited attend to aliased early exited tokens. This is a simple approach, however, surprisingly works out of the box. However, the authors even went beyond and proposed a ragged KV-cache aware training strategy (section 5), that further plugs the gap between baseline and early-exit looped LLMs.

Critical strengths

The paper is well motivated, and communicates its content well. The convincing set of experiments is also nice.

Critical weaknesses

One pending question from my side: The ragged KV-cache method would introduce some system challenges, e.g. attention now has to index some complicated KV-cache structure. This probably even requires novel kernel support. Can the authors describe some of these challenges and what they have addressed in this paper (and if not, why).

Questions for authors

Systems challenges in indexing a ragged KV-cache structure, what was addressed in this work and what wasn't and why.

1 comment

Thanks for your positive assessment of our paper!

Appendix A contains full-detail of our system contributions that make Ragged KV save real memory and FLOPs. We describe it briefly here for your reference, and add a new result.

KV Memory

We express Ragged KV as a per-depth page table in vLLM's paged cache, Ad[i]=Pi,min⁡(d,ei)A_d[i]=P_{i,\min(d,e_i)} (Eq. 11). Paged attention already reads KV through a block table, so the kernel is unchanged. Only the table passed at depth dd differs. Physical memory is CKVNEˉC_{KV}N\bar E instead of CKVNUC_{KV}NU (Eq. 12). The logical table adds only one integer per token-depth, against roughly 196 KB per bf16 KV entry for Ouro-1.4B, making it practically free in comparison.

FLOPs

We implement depth batching with one queue per depth (like described in Bae et al, 2025). In a traced run it executes 823 depth passes, against 1020 for Fixed@4 (Table 3). It is faster than Fixed@4 at batch sizes 1/4/8: 41.1 vs. 25.0, 135.3 vs. 100.5 and 241.2 vs. 205.5 tokens/s. In our initial vLLM integration (in the paper) depth micro-batches are nested inside one vLLM runner step, so the scheduler is not work-conserving: 823 passes against an ideal 574.2. As a result, throughput at batch size 32 is slightly below Fixed@4 (773.2 vs. 807.9 tokens/s). For maximum throughput gains, the micro-batch scheduler needs to be global, not nested.

Throughput (new)

Since the paper deadline, we have implemented a minimal global depth batching implementation for Ouro to continuously remove tokens as they exit and add new ones to keep batches dense. We simulated this implementation under three scenarios: batch size 1 (as it would be used in many edge devices), 4 (for parallel agentic work) and adaptive batching (up to 64) under a fixed memory cap (32 GB) for maximum possible throughput in resource constrained environments. For workload, we used 64 MATH-500 questions, 8,192-token output cap, EOS stopping and 9,152-token context, 4 rollouts each. BF16 model, activations and KV with supported native BF16 attention, compilation, CUDA graphs and sampling optimizations. Greedy outputs uses temperature 0. Sampled use temperature 1, top-p 0.7, no top-k (as suggested by Ouro), with seeds 42–45. We run all tests on a single RTX Pro 6000 GPU. All inference modes use torch.compile with CUDA graphs enabled. We also compare our implementation with native vLLM for Ouro for fairness. Here are the throughput numbers (each column shows greedy / sampled numbers):

ModeBatch 1Batch 4Adaptive
Native vLLM fixed@469.7 / 67.4171.7 / 186.8376.6 / 392.4
Ours fixed@492.8 / 91.9246.9 / 246.3425.8 / 403.6
Ours Ragged τ=0.3, dense prefill157.3 / 150.3405.4 / 376.2730.3 / 803.1
Ours Ragged τ=0.5, dense prefill126.5 / 121.9334.3 / 309.3576.5 / 572.0

To make sure that the performance does not come at a cost of correctness, we first ensured bit-identical fp32 computations on a small set and then evaluated the rollouts with p@1 / p@4:

ModeBatch 1Batch 4Adaptive
Native vLLM fixed@486.33 / 95.3188.67 / 95.3191.41 / 96.88
Ours fixed@488.67 / 95.3189.45 / 96.8886.33 / 93.75
Ours Ragged τ=0.3, dense prefill84.38 / 93.7583.98 / 95.3184.38 / 95.31
Ours Ragged τ=0.5, dense prefill87.50 / 96.8886.72 / 93.7585.94 / 95.31

Note that the performance does not match because bf16 precision often causes divergences (first in exit decisions and then in token choices; especially in adaptive mode where vLLM uses different samplers for different batch sizes).

Finally, we also implemented a ragged-prefill mode for faster prefill (which it does, as seen below) but did not notice a big improvement in overall throughput since the prompt is quite small in this case.

Prompt modeInput tokens/sSpeedupKV saved
Dense fixed@411,1631.00×0.0%
Ragged τ=0.317,9291.61×45.4%
Ragged τ=0.514,9851.34×31.6%
fixed@2, control22,1081.98×50.0%

We hope that this addresses your queries on systems implementation.

Comments on the paper

1 comment

We thank all reviewers for their truly valuable constructive feedback (in these times of AI slop, it is rare). All reviewers recognized the usefulness of our study in making early-exit a reality for better inference efficiency. There were some misunderstandings and concerns (Rh5J and A5TB) that our rebuttal clarifies. It strengthens our central contributions:

1.⁠ Correcting the training objective. Our added KL decomposition shows that Ouro’s per-token ⁠ local ⁠ objective implicitly adds an unwanted penalty on token-dependent exits, which our ⁠ global ⁠ objective removes. Our new matched-depth evaluations and quantitative difficulty-alignment measurements support the resulting improvement in adaptive computation. We also clarify that the ⁠ local ⁠ objective incentivizes, rather than forces homogeneous exits (we do see homogeneous exits in our experiments).

2.⁠ ⁠Demonstrating immediately usable inference benefits. Ragged KV works on existing Ouro checkpoints without retraining. We showed this by delivering measured memory and throughput gains, which no-one else had shown yet. Our additional comparisons demonstrated better ragged accuracy than matched-depth fixed-iteration mixtures on the thinking models, and we clarifed the limitations of dense-prefill ragged on non-thinking models.

We acknowledge the shared concern about missing references and specified our novelty claim: the contribution is demonstrating substantial, immediately accessible benefits of an established KV-reuse mechanism for Ouro, alongside diagnosing and correcting the training objective. The rebuttal provides additional theoretical, empirical, and systems evidence for these contributions.

This dialogue helped our paper substantially. We hope that the reviewers would consider revisiting their scores and the AC sees the potential impact of this work in training more inference efficient LLMs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.