- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In looped language models such as Ouro, later tokens attend to earlier tokens' KV entries at every depth, so per-token early exit saves neither compute nor memory. The paper proposes Ragged KV, in which deeper iterations read an exited token's deepest available KV entry. Implemented through vLLM page aliasing, it is applied without training to four Ouro checkpoints, mainly with full-depth prefill (Dense-Ragged). On Ouro-2.6B-Thinking it matches four-iteration accuracy at an average decode depth of 2.83. The paper also argues that Ouro's per-token exit objective makes nearly all tokens exit at one depth, and proposes a batch-level ("global") prior and ragged-aware training, evaluated by perplexity on a 160M looped model.
Critical strengths
-
Ragged KV recovers much of the accuracy that truncation loses at low depth, without any training. Between two and three decode iterations it keeps far more accuracy than simply cutting depth on both Ouro sizes, and a traced run shows real KV memory savings. That makes it a cheap, drop-in option for serving looped models, not just a FLOP count.
-
The diagnosis of Ouro's exit objective is useful in its own right. Showing that a per-token KL pulls every token toward one exit distribution explains why a training-free cache change works on Ouro at all, and shows what a better exit objective must fix.
Critical weaknesses
-
The efficiency claim rests on a weak reference point and a mechanism close to established practice. The headline "up to 30% reduction in FLOPs and KV memory" is relative to four fixed iterations, yet three already nearly match four on both thinking models, and the claimed operating point (decode depth 2.83) sits barely below three. Counting prefill, Dense-Ragged falls a few points below a mix of three and four fixed iterations on non-thinking BBH (Fig. 16), and memory and speed are measured at other operating points, so no reported configuration shows full-depth accuracy with measured savings. Reusing an exited token's deepest state is established (Elbayad et al., 2020; Schuster et al., 2022; Geiping et al., 2025), and Ouro's own last-step KV reuse, which its authors report near full-cache accuracy, is dismissed without per-benchmark results. Efficiency is the main selling point, and against sensible baselines the advantage is much narrower than claimed.
-
The case for the global prior and ragged-aware training is too thin for its general claims. Conclusions about training objectives rest on one 160M model, perplexity only, where the deciding differences are hundredths of a point and no repeats or seed variance are reported. The prior is not even reached: global's average exit depth stays about one iteration above the prior's mean, so the effects cannot be credited to it. Ragged-aware training brings global below its own four-iteration perplexity, but that baseline is already worse than local's, and at similar depth global only roughly matches local. Global does produce far more large depth jumps between adjacent tokens, yet difficulty alignment rests on a few qualitative samples, and in the illustrating trace the commas and semicolons cited as easy nearly all exit at full depth. The conclusion's "true dynamic exits with difficulty awareness" is therefore not yet supported.
Questions for authors
- On all four models, how does Dense-Ragged compare in overall depth with a request-level mixture of three and four fixed iterations?
- Could you add Ouro's last-step KV reuse as a baseline in Tables 1 and 2, and explain how Ragged KV differs from the deepest-state reuse in Elbayad et al., Schuster et al. and Geiping et al. (Remark 6.1)?
- For the 160M runs, can you report several seeds, the measured KL to the prior, and a difficulty measure such as the correlation between exit depth and full-depth token NLL, comparing local and global at matched depth?
- Your four-iteration MATH500 score for Ouro-1.4B is well below the 82.40 that Ouro reports, and on this model both Ragged modes trail full depth on MMLU-Pro and BBH. Do these gaps persist under Ouro's evaluation protocol?
4 comments
how does Dense-Ragged compare in overall depth with a request-level mixture of three and four fixed iterations?
Request-level mixture is a great suggestion to compare Ragged with fixed-depth modes at matched avg. depths. Assuming the difficulty of queries is uniformly distributed, we simulate the mixture by randomly routing queries to depth {2, 3, 4} to achieve the desired avg. depth. We average the simulated performance over 200 draws for reliability. We report numbers at in order separated by / (0.45 was not calculated for non-thinking models).
| Model | Matched depth | Dense-Ragged p@1 (%) | Mixture p@1 (mean ± SD, %) | Δ (pp) |
|---|---|---|---|---|
| Ouro-1.4B | 3.625 / 3.784 | 59.54 / 62.88 | 62.94±0.09 / 63.49±0.08 | −3.40 / −0.61 |
| Ouro-2.6B | 3.550 / 3.757 | 69.68 / 72.27 | 71.90±0.12 / 72.13±0.10 | −2.23 / +0.14 |
| Ouro-1.4B-Thinking | 2.364 / 2.586 / 2.949 | 49.63 / 50.28 / 52.27 | 43.17±0.72 / 46.39±0.66 / 50.81±0.30 | +6.46 / +3.90 / +1.46 |
| Ouro-2.6B-Thinking | 2.268 / 2.493 / 2.855 | 58.38 / 59.34 / 62.26 | 52.48±0.55 / 55.55±0.62 / 59.99±0.46 | +5.89 / +3.79 / +2.27 |
Like mentioned earlier, Dense-Ragged doesn't offer good value for non-thinking models and we would recommend using fixed-depth when possible. But for thinking models, Ragged provides real performance gains compared to fixed-depth mixtures. For efficiency critical cases in 1.4B, ETE-ragged still provides better value than fixed-depth mixtures. 2.6B has a known prefix instability issue causing catastrophic performance drops in ETE ragged (see Figure 21 in the draft). We will update the paper with these results to further strengthen our case.
| Model | Matched depth | ETE-Ragged p@1 (%) | Mixture p@1 (mean ± SD, %) | Δ (pp) |
|---|---|---|---|---|
| Ouro-1.4B | 2.626 / 3.221 | 58.01 / 62.45 | 51.41±0.16 / 61.24±0.07 | +6.60 / +1.21 |
| Ouro-2.6B† | 3.100 / 3.677 | 52.72 / 52.93 | 70.67±0.09 / 71.92±0.11 | −17.95 / −18.99 |
| Ouro-1.4B-Thinking | 2.329 / 2.536 / 2.909 | 46.67 / 50.19 / 52.05 | 42.85±0.71 / 45.87±0.68 / 50.48±0.37 | +3.82 / +4.32 / +1.57 |
| Ouro-2.6B-Thinking | 2.259 / 2.564 / 2.968 | 58.46 / 60.57 / 61.98 | 52.63±0.55 / 56.74±0.60 / 61.30±0.30 | +5.82 / +3.83 / +0.68 |
"up to 30% reduction in FLOPs and KV memory" is relative to four fixed iterations, yet three already nearly match four on both thinking models
We agree: fixed@3 performance follows fixed@4 closely in Ouro thinking models. But our headline is calculated relative to Ouro's officially recommended inference setting of fixed@4. And note that we never claimed out-of-the-box ragged performance should beat fixed@n. It just happens to be the case sometimes and we only report it for reference.
Related Work
Thanks for highlighting these related works, some of which we had missed earlier. In our literature survey, we had found AdaPonderLM the closest to Ragged KV, but Elbayad et al. and other subsequent works already proposed similar caching mechanisms. We will re-write portions of the paper to remove any claims of Ragged KV being a novel caching mechanism proposal, and properly cite these works. Note that, our central contributions remain original and orthogonal to this.
- Showing that Ragged KV works out-of-the-box for Ouro, enabling true inference efficiency without any re-training.
- Fixing the training objective to make it ragged KV aware and enabling truly dynamic exits.
Ouro 160M experiments: local vs global
The prior is not even reached ... can you report the measured KL to the prior
The prior is only our expectation of what ideal token-routing should look like. What exit distribution is actually achieved depends on the tension between LM loss and early-exit regularization, controlled by . With a higher , we were easily able to reach the prior, but training collapses to poor performance (Figure 7). Even the pre-trained Ouro models trained with trillions of tokens never reach their intended uniform exit distribution (in fact, they hardly ever exit at depth 1, see Figure 2). But we find that global is much more effective in reducing the average exit-distribution's KL-to-prior than local. The table below shows numbers for the 20B checkpoints. A more detailed analysis can be seen in our response to Rh5J, and the corresponding training evolution Figure 22 added to the draft.
| Prior | local: KL | global: KL |
|---|---|---|
| Poisson | 0.7719 | 0.4504 |
| Geometric | 0.9105 | 0.5472 |
| Uniform | 0.7252 | 0.3754 |
comparing global and local at matched depths
We followed Rh5J's suggestion and swept to get depth-match performance numbers. In summary, we found that at matched avg. depth, global beats local under geometric (depth: 3.2169, Δ PPL: −0.0283) and uniform (depth: 3.3828, Δ PPL: −0.0997) priors. Poisson local collapses to lower-fixed-depth behavior (sends >90% of its tokens to depth 3, resulting in low cache mismatch), hence beats the dynamic global variant. Please see that response for details and Figure 23 in the updated draft for the complete sweep comparison plot.
difficulty alignment ... in the illustrating trace the commas and semicolons cited as easy nearly all exit at full depth
We believe this was misinterpreted. The depth written adjacent to each token is what the model uses to process the next token's prediction. To assess the model's perceived "difficulty" of commas and semicolons, please the the previous token's exit depth. For example, storing[2] ,[4], ues[2] ;[4], result[1] ,[3], lack[1] of[4] funding[2] for[4], etc. We will make it clearer in the paper. Figure 22 also shows the token-depth mutual information (as broken down in our decomposition of the local KL term). It measures how dependent the exit depth is on the token itself. While local tokens exit kind of independent of the token, global is highly token-dependent, supporting its difficulty awareness.
a difficulty measure such as the correlation between exit depth and full-depth token NLL
Thanks for this valuable suggestion. We measured the correlation between exit-depth and depth-4 NLL, fixing global τ=0.5 and calibrating local τ to match average exit depth.
| Prior | Mean depth (local / global) | local ρ | global ρ |
|---|---|---|---|
| Poisson | 3.082 / 3.093 | 0.099 | 0.386 |
| Geometric | 3.169 / 3.178 | 0.173 | 0.397 |
| Uniform | 3.423 / 3.414 | 0.329 | 0.418 |
global exhibits stronger difficulty alignment across all three priors. The same conclusion holds using depth-1 NLL and the loss reduction from depth 1 to depth 4, supporting alignment with both initial difficulty and the benefit of additional computation in global. We added Figure 24 in the draft to show the complete picture.
conclusions rest on a single 160M model ... no repeats or seed variance are reported
We agree that multiple seeds make findings more reliable. That is precisely why we reported p@1,4 out of up to 16 rollouts for Ouro performance tables instead of a single number. However, pre-training experiments are expensive, and it is standard practice to report single run numbers (Saunshi et al. 2025, Bae et al. 2025, even a reference you cited: Geiping et al. 2025). Each of our 160M training run costs ~50 H100 hours. Additionally, we ran multiple ragged-aware training campaigns (Appendix C) to find optimal training recipes. This along with huge evaluation campaigns in multiple inference modes for Ouro, restricted our ability to perform multiple pre-training runs, or compare with other architectures/algorithms. We also made every effort to make the one seed as fair to each mode as possible (Appendix B). We hope you can appreciate our decision to focus resources on a single important pre-training research question instead of trying to solve everything at once with limited resources.
We recommend reading our response to avNT for new increased throughput results, and our response to Rh5J for more theoretical and empirical analysis of local vs global training. Our currently updated draft contains additional figures in the appendix to support the rebuttal. We are updating it with all promised changes. We hope this resolves your concerns and are happy to take more questions, if any.
Thanks again for your valuable feedback!
A gentle reminder that today is the last day for discussion. We are happy to take any follow-up questions. In case you are satisfied with our rebuttal, we would really appreciate acknowledgement and raising the score.
Thanks for your deliberate review and for highlighting the intended strengths of our paper. We apologize for the delay in sending our rebuttal.
Before we address your concerns, we want to underline: Ragged KV performing close to fixed@4 out-of-the-box with Ouro is a welcome by-product of the way it is trained. People can use it today without any training for better inference efficiency. Our contribution is finding this out-of-the-box ability. We do not claim that Ouro models are most performant in this mode. That would require correctly training the model and making it ragged KV aware, which we discussed in section 4 and 5.
Now about your concerns:
Reproducing Ouro's reported numbers
All our reported numbers follow the protocol described in the Ouro paper (Tables 16, 17), as much as possible. This is mentioned in our appendix section D, Tables 4,5,6. First and most importantly, Ouro reports a single number performance, while we measured p@1 and p@4 with up to 16 roll-outs for high reliability. Second, for many benchmarks (especially logprob eval cases), Ouro uses the standard lm-eval-harness which we follow down to the number of CoT shots, temperature, top_p, etc; but in many generation cases, Ouro uses an "in-house harness" which is not publicly accessible. The Ouro paper also does not mention the LLM used as judge (therefore we used the best open-sourced model that we could run: Qwen 3.8-27B, judge details in D.3). We used the standard lm-eval-harness to generate responses and evaluated all answers with this judge. With this setup, we are unable to reproduce some numbers (like MATH500) reported in the Ouro paper. To maintain fairness, all inference-modes (fixed/ragged) use the same evaluation setup in our study.
This was the first baseline we tested. Unfortunately, using a single last-step KV cache drops performance to near zero in our reproduction attempts. In Table 17, we reported BBH performance using a single shared KV cache under ragged decode at various thresholds. We tested both: the last-step KV (like Ouro), as well as the exit-step KV. In both cases, the performance dropped to near zero. Our finding is consistent with Vendrell et. al. (https://arxiv.org/abs/2605.07721) Appendix B, Table 6, who find a similar performance drop to zero with shared KV. It is also noteworthy that using the last-step KV requires computing all 4 loops for each token (so it only saves KV memory, no FLOPs). Since this setting fails so severely, we did not run all benchmarks in this setting.
Clarification regarding Dense-Ragged and avg. exit depth
There is a stark difference between the thinking and non-thinking models when it comes to overall avg. exit depths for dense-ragged modes. It is because non-thinking models are evaluated in few-shot CoT setting. Prompt tokens constitute majority of all processed tokens in non-thinking models, but are a tiny fraction in thinking models (see table below). We will add the avg. overall depth for dense-ragged in the tables (like Tables 14-16) to make this clear.
Since the savings vs performance tradeoff for dense-ragged modes in non-thinking models seems not worth it, we will revise the paper to recommend using fixed-depth when possible in non-thinking models. ETE-ragged mode may still be preferred when inference efficiency is critical, especially if the desired avg. depth is between 2 and 3, since ETE-ragged comfortably sits much higher than fixed@2 performance (even better than request-level mixture as discussed later in this rebuttal).
We posit that each few-shot example adds (potentially surprising) valuable information for the non-thinking model to perform In-Context Learning, and hence benefits from full depth prompt processing. This is not the case with thinking models, where most of the work is done in decode, and overall avg. depth is pretty much the same as avg. decode depth in dense-ragged mode. We recommend all ragged modes for thinking models.