- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks how per-token early exits in looped language models can become wall-clock speedups under batched serving, where continuous batching schedules whole forward passes. It implements continuous depth batching (CDB) in a full serving engine: prefill, prelude, recurrent core and coda get separate queues, and batches are re-formed between loop steps, either shrinking as tokens exit or refilled with new tokens. Depth-aware KV-cache layouts and a lookahead gate that decides exits one step early complete the design. A prefill-corrected FLOP bound and a latency model frame experiments on Ouro and Huginn over three workloads with replayed exits. With refill, Ouro reaches up to 1.53x the throughput of full-depth continuous batching, close to the bound.
Critical strengths
- The first end-to-end measurement of depth batching in a real serving engine. Earlier estimates came from simulation or from ablations that leave out prefill, scheduling and KV-cache updates; here all of these are included, so the gains reflect what a deployed engine actually pays. The paper also explains where results fall short of the bound: adaptive depth cannot shorten prefill, which weighs most on long contexts, and Huginn's costly boundary stages favor no-refill.
- A mechanistic latency account and a reusable fix for exits on the critical path. Showing that the saturation batch size shrinks with context explains why refill helps most where decoding is memory bound, so the results can be anticipated rather than just observed. The lookahead gate nearly removes the GPU idle time of synchronous exits at about the accuracy of Ouro's own gate.
Critical weaknesses
- For Ouro, the headline speedups are timed under a cache layout whose task quality is not validated, and the layout that preserves accuracy is never timed. The main experiments give each recurrent layer a single shared KV slot, yet in the paper's own fixed-depth evaluation this layout scores 0.23% on GSM8K against 78.09% with the full depth-indexed cache (Table 4), and the authors could not reproduce the near-lossless sharing reported for Ouro. Because the throughput runs replay recorded output lengths and exits, they cannot expose this. The last-exited layout, which the paper says generally preserves accuracy, costs more memory and copies state on every exit, but it is never timed. For Ouro the evidence therefore shows an efficient scheduler for a given exit pattern; the conclusion's claim of near-optimality would be more convincing if the timed configuration were shown to preserve task quality.
- Throughput and accuracy are never joined, and there is no truncated fixed-depth baseline at matched quality. The serving claim is that adaptive depth buys speed at little cost in quality, which needs both measured in one configuration. The paper measures accuracy on GSM8K and throughput on other workloads; for Huginn, the threshold used in the batch-size sweep roughly halves mean depth but gives up a substantial part of full-depth accuracy (Fig. 14). The simplest alternative, looping fewer times for every token, needs no new system, and Ouro run one step short of its maximum seems to keep nearly all its accuracy, with a FLOP bound of about 4/3 on decode speedup. Since the Pareto claim is made along mean depth while serving is judged in wall-clock time, the paper cannot yet say what CDB's added complexity buys.
Questions for authors
- Can you report Ouro throughput on ShareGPT and ArXiv with the last-exited or depth-indexed layout, including the copy-on-exit cost and any effect of the larger cache on the feasible batch size?
- For one model and one workload, how do measured throughput and GSM8K accuracy compare between two truncated fixed depths and one or two CDB thresholds, with exits recorded on the accuracy data?
- Since no gate runs live during the experiments, which policy generated the replayed exits: Ouro's native gate or the lookahead gate, and for Huginn the immediate or the delayed convergence rule?
- How many repetitions underlie each throughput point, and what is the run-to-run spread? Without it, small differences near the bound may be noise.
If the accuracy-preserving Ouro layout keeps most of the reported speedup, and a matched-quality comparison shows CDB ahead of truncated fixed-depth serving for at least one model, I would find the serving claims more convincing and raise my rating.
2 comments
As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.
Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.
Thank you all for your time and constructive engagement.
Thank you for your thorough review and constructive suggestions. We respond to your comments below.
We also uploaded a revised manuscript based on reviewer feedback with the main changes colored in blue.
Weaknesses & Questions
W1 & Q1: Throughput of shared and last-exited KV cache and Ouro accuracy.
Thank you for pointing this out. The throughput gains are nearly the same with the accuracy-preserving last-exited cache, so the reported improvements align with useful accuracy for Ouro. Specifically, at the same batch size, switching from shared to last-exited KV changes throughput by less than 2%.
Our main experiments use single-slot sharing only because its lower memory cost lets us test larger batch sizes and compare across a wider range of compute regimes. We clarified this now in the experimental setup (Sec. 6.1).
Yes, we added the following results to the revised manuscript:
W2 & Q2: Accuracy-throughput curve and fewer fixed loops.
Thank you for this suggestion. On GSM8K, CDB with Ouro 1.4B achieves comparable accuracy with 13% higher throughput than 4-loop CB and 7.5% higher throughput than 3-loop CB (see Table R.1 below). With Ouro 1.4B Thinking, the speedups are 21% and 14%, respectively (Table R.2).
The larger gains for Ouro Thinking come from its longer outputs (about 100 tokens for Ouro and 800 for Ouro Thinking). This is consistent with our results in Sec. 6. CDB gives larger end-to-end speedups when prefill accounts for a smaller share of the runtime.
An additional benefit of CDB is that tuning the exit threshold gives finer control over the accuracy-throughput trade-off. For example, an intermediate point between 2-loop and 3-loop CB (Table R.1 and R.2).
We will continue to test more thresholds and include full accuracy-throughput curves for Ouro, Ouro Thinking, and Huginn in the final manuscript.
Table R.1: Ouro 1.4B GSM8K Accuracy vs. throughput speedup.
Table R.2: Ouro 1.4B Thinking GSM8K Accuracy vs. throughput speedup.
Q3: Exit gate in workload recording
For Ouro, we used the original gate. For Huginn, we used the delayed convergence rule. We clarified this in the revised manuscript (Appendix D.2).
Note that we still execute the gate during replay and only overwrite its decision with the recorded one, so all reported numbers include its cost (Sec. 6.1).
Additionally, CDB’s speedup depends mainly on how many loops each token runs on average (Sec. 5). Other gates can reach similar accuracy at comparable average depths, but they may change the reported thresholds (Fig. 16).
Q4: Run-to-run spread.
We average throughput points over three runs. The run-to-run standard deviation is below 0.3%, which is negligible relative to the measured speedups.