acceptodds
Under review as a conference paper at ICLR 2027

Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching

Abstract

A main promise of looped language models is depth-adaptive inference. By looping a block of shared layers a variable number of times, the model can use less compute for "easy" tokens and more for "hard" ones. However, tokens with different numbers of loops cannot share a uniform forward pass and therefore cannot be handled by standard batching systems such as vLLM. The practical value of depth-adaptive inference thus hinges on whether batching can be made efficient. We introduce the first efficient method for depth-adaptive looped LMs via continuous depth batching (CDB), which forms new batches between loop steps. Our method dynamically schedules looped and non-looped parts of the architecture, manages looped KV-caching, and predicts which tokens will exit the loop in advance so it can prepare batches asynchronously. Experiments on Ouro 1.4B and Huginn 3.5B show that fully looped architectures are best suited to depth-adaptive inference, as large non-looped layers outside the recurrent core (e.g., token embedding, LM head, and unshared transformer blocks) slow down and complicate scheduling. Overall, CDB realizes up to 99% of the estimated maximum speedup available, leaving further gains primarily dependent on model architecture and exit behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM inference, looped language models, continuous depth batching, continuous batching, early-exit inference, adaptive computation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_V78eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how per-token early exits in looped language models can become wall-clock speedups under batched serving, where continuous batching schedules whole forward passes. It implements continuous depth batching (CDB) in a full serving engine: prefill, prelude, recurrent core and coda get separate queues, and batches are re-formed between loop steps, either shrinking as tokens exit or refilled with new tokens. Depth-aware KV-cache layouts and a lookahead gate that decides exits one step early complete the design. A prefill-corrected FLOP bound and a latency model frame experiments on Ouro and Huginn over three workloads with replayed exits. With refill, Ouro reaches up to 1.53x the throughput of full-depth continuous batching, close to the bound.

Critical strengths
  1. The first end-to-end measurement of depth batching in a real serving engine. Earlier estimates came from simulation or from ablations that leave out prefill, scheduling and KV-cache updates; here all of these are included, so the gains reflect what a deployed engine actually pays. The paper also explains where results fall short of the bound: adaptive depth cannot shorten prefill, which weighs most on long contexts, and Huginn's costly boundary stages favor no-refill.
  2. A mechanistic latency account and a reusable fix for exits on the critical path. Showing that the saturation batch size shrinks with context explains why refill helps most where decoding is memory bound, so the results can be anticipated rather than just observed. The lookahead gate nearly removes the GPU idle time of synchronous exits at about the accuracy of Ouro's own gate.
Critical weaknesses
  1. For Ouro, the headline speedups are timed under a cache layout whose task quality is not validated, and the layout that preserves accuracy is never timed. The main experiments give each recurrent layer a single shared KV slot, yet in the paper's own fixed-depth evaluation this layout scores 0.23% on GSM8K against 78.09% with the full depth-indexed cache (Table 4), and the authors could not reproduce the near-lossless sharing reported for Ouro. Because the throughput runs replay recorded output lengths and exits, they cannot expose this. The last-exited layout, which the paper says generally preserves accuracy, costs more memory and copies state on every exit, but it is never timed. For Ouro the evidence therefore shows an efficient scheduler for a given exit pattern; the conclusion's claim of near-optimality would be more convincing if the timed configuration were shown to preserve task quality.
  2. Throughput and accuracy are never joined, and there is no truncated fixed-depth baseline at matched quality. The serving claim is that adaptive depth buys speed at little cost in quality, which needs both measured in one configuration. The paper measures accuracy on GSM8K and throughput on other workloads; for Huginn, the threshold used in the batch-size sweep roughly halves mean depth but gives up a substantial part of full-depth accuracy (Fig. 14). The simplest alternative, looping fewer times for every token, needs no new system, and Ouro run one step short of its maximum seems to keep nearly all its accuracy, with a FLOP bound of about 4/3 on decode speedup. Since the Pareto claim is made along mean depth while serving is judged in wall-clock time, the paper cannot yet say what CDB's added complexity buys.
Questions for authors
  1. Can you report Ouro throughput on ShareGPT and ArXiv with the last-exited or depth-indexed layout, including the copy-on-exit cost and any effect of the larger cache on the feasible batch size?
  2. For one model and one workload, how do measured throughput and GSM8K accuracy compare between two truncated fixed depths and one or two CDB thresholds, with exits recorded on the accuracy data?
  3. Since no gate runs live during the experiments, which policy generated the replayed exits: Ouro's native gate or the lookahead gate, and for Huginn the immediate or the delayed convergence rule?
  4. How many repetitions underlie each throughput point, and what is the run-to-run spread? Without it, small differences near the bound may be noise.

If the accuracy-preserving Ouro layout keeps most of the reported speedup, and a matched-quality comparison shows CDB ahead of truncated fixed-depth serving for at least one model, I would find the serving claims more convincing and raise my rating.

2 comments

Thank you for your thorough review and constructive suggestions. We respond to your comments below.

We also uploaded a revised manuscript based on reviewer feedback with the main changes colored in blue.

Weaknesses & Questions

W1 & Q1: Throughput of shared and last-exited KV cache and Ouro accuracy.

For Ouro, the headline speedups are timed under a cache layout whose task quality is not validated, and the layout that preserves accuracy is never timed.

Thank you for pointing this out. The throughput gains are nearly the same with the accuracy-preserving last-exited cache, so the reported improvements align with useful accuracy for Ouro. Specifically, at the same batch size, switching from shared to last-exited KV changes throughput by less than 2%.

Our main experiments use single-slot sharing only because its lower memory cost lets us test larger batch sizes and compare across a wider range of compute regimes. We clarified this now in the experimental setup (Sec. 6.1).

Can you report Ouro throughput on ShareGPT and ArXiv with the last-exited or depth-indexed layout, including the copy-on-exit cost and any effect of the larger cache on the feasible batch size?

Yes, we added the following results to the revised manuscript:

  1. At matched batch sizes, switching from shared to last-exited KV changes CDB throughput by less than 2% (Appendix B, Fig. 11). The copying version was at most 0.32% slower, so the copy-on-exit cost is small. The largest tested batch sizes that fit were 128 with last-exited KV versus 512 with shared KV on ShareGPT, and 8 versus 32 on ArXiv.
  2. At matched memory budgets, last-exited KV fits smaller batches and gives lower absolute throughput than shared KV. But both CB and CDB need accuracy-preserving KV layouts, so both run smaller batches, and CDB keeps the full speedup over CB (Sec. 6.4, Fig. 8).
  3. Our new memory-saving last-exited variant frees unused KV slots after early exits (Sec. 4.2). This lets CDB fit larger batches than the copying variant at the same memory budget and gets larger speedup over CB than shared KV (Appendix B, Fig. 12).

W2 & Q2: Accuracy-throughput curve and fewer fixed loops.

How do measured throughput and GSM8K accuracy compare between CDB and looping fewer times for every token?

Thank you for this suggestion. On GSM8K, CDB with Ouro 1.4B achieves comparable accuracy with 13% higher throughput than 4-loop CB and 7.5% higher throughput than 3-loop CB (see Table R.1 below). With Ouro 1.4B Thinking, the speedups are 21% and 14%, respectively (Table R.2).

The larger gains for Ouro Thinking come from its longer outputs (about 100 tokens for Ouro and 800 for Ouro Thinking). This is consistent with our results in Sec. 6. CDB gives larger end-to-end speedups when prefill accounts for a smaller share of the runtime.

An additional benefit of CDB is that tuning the exit threshold gives finer control over the accuracy-throughput trade-off. For example, an intermediate point between 2-loop and 3-loop CB (Table R.1 and R.2).

We will continue to test more thresholds and include full accuracy-throughput curves for Ouro, Ouro Thinking, and Huginn in the final manuscript.

Table R.1: Ouro 1.4B GSM8K Accuracy vs. throughput speedup.

MethodGSM8K accuracySpeedup vs. 4-loop CBComment
CB, 4 loops78.09%1.00x
CDB, lookahead78.39%1.13x13% more than loop 4
CB, 3 loops75.59%1.33x
CDB, lookahead75.51%1.43x7.5% more than loop 3
CDB, lookahead72.78%1.57xintermediate point
CB, 2 loops63.38%1.96x

Table R.2: Ouro 1.4B Thinking GSM8K Accuracy vs. throughput speedup.

MethodGSM8K accuracySpeedup vs. 4-loop CBComment
CB, 4 loops84.38%1.00x
CDB, lookahead84.76%1.21x21% more than loop 4
CB, 3 loops82.64%1.33x
CDB, lookahead82.41%1.52x14% more than loop 3
CDB, lookahead80.97%1.68xintermediate point
CB, 2 loops77.79%1.81x

Q3: Exit gate in workload recording

Which policy generated the replayed exits?

For Ouro, we used the original gate. For Huginn, we used the delayed convergence rule. We clarified this in the revised manuscript (Appendix D.2).

Note that we still execute the gate during replay and only overwrite its decision with the recorded one, so all reported numbers include its cost (Sec. 6.1).

Additionally, CDB’s speedup depends mainly on how many loops each token runs on average (Sec. 5). Other gates can reach similar accuracy at comparable average depths, but they may change the reported thresholds (Fig. 16).

Q4: Run-to-run spread.

How many repetitions underlie each throughput point, and what is the run-to-run spread?

We average throughput points over three runs. The run-to-run standard deviation is below 0.3%, which is negligible relative to the measured speedups.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Reviewer_gjJxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an end-to-end implementation of Continuous Depth Batching (CDB) for depth-adaptive looped language models. The proposed system supports dynamic scheduling across recurrent depths, depth-aware KV caching, and asynchronous batch preparation with lookahead exit prediction. Experiments on Ouro 1.4B and Huginn 3.5B demonstrate throughput improvements over standard continuous batching, approaching the estimated theoretical speedup bounds.

Critical strengths
  1. The paper addresses an important practical challenge in serving depth-adaptive looped language models, where conventional continuous batching cannot efficiently handle different recurrent depths.
  2. The system design is comprehensive, integrating stage-wise scheduling, KV cache management, and asynchronous execution to reduce GPU idle time.
  3. The hardware-aware latency analysis provides useful insights into when refill is beneficial and how model architecture affects inference efficiency.
Critical weaknesses
  1. Accuracy degradation under KV sharing. Table 4 shows that Ouro's GSM8K accuracy drops dramatically from 78.09% to 0.23% when using single-slot shared KV caching, which is also adopted in the main throughput experiments. Do the authors have a practical solution to mitigate this degradation? I would also encourage evaluation on more diverse and larger looped models to determine whether this issue is specific to Ouro. If similar degradation occurs across multiple models, it would significantly limit the practical applicability of the proposed system.
  2. Lack of fully end-to-end inference evaluation. The main serving experiments replay pre-recorded exit decisions and use reference output lengths rather than performing actual autoregressive generation with online exit gating. Although this setup enables controlled comparisons, it does not fully capture the overhead and behavior of a realistic serving pipeline. I would appreciate additional experiments with real-time exit decisions and autoregressive generation to validate the reported efficiency gains under practical deployment conditions.
  3. Limited sclae and model architecture.* As the authors stated in the limitation seciton, the scale of the model, number of GPUs and components of model architecutres is limited.
Questions for authors

Please refer weaknesses.

3 comments

Thank you for your constructive and thoughtful review. We address your comments below.

We also uploaded a revised manuscript based on reviewer feedback with the main changes colored in blue.

Weaknesses

W1: Accuracy degradation under KV sharing for Ouro.

Do the authors have a practical solution to mitigate this degradation?

Yes, CDB already supports last-exited KV (Sec. 4.2), which retains useful task accuracy on Ouro. CB can use depth-indexed KV (Table 6). The throughput improvements are nearly the same with single-slot shared KV or last-exited KV, so the results from the main throughput experiments carry over. We only use single-slot sharing in the main experiments because its lower memory cost lets us test larger batch sizes and compare across a wider range of compute regimes. We clarified this now in the experimental setup (Sec. 6.1).

We also added the following results to the revised manuscript:

  1. At matched batch sizes, switching from shared to last-exited KV changes CDB throughput by less than 2%. But last-exited KV uses more memory, which limits the largest batch we can test (Appendix B, Fig. 11).
  2. At matched memory budgets, last-exited KV fits smaller batches and gives lower absolute throughput than shared KV. But both CB and CDB need accuracy-preserving KV layouts, so both run smaller batches, and CDB keeps the full speedup over CB (Sec. 6.4, Fig. 8).
  3. Our new memory-saving last-exited variant frees unused KV slots after early exits (Sec. 4.2). This lets CDB fit larger batches than the copying variant at the same memory budget and gets larger speedup over CB than shared KV (Appendix B, Fig. 12).

If shared KV accuracy degradation occurs across multiple models, it would significantly limit the practical applicability of the proposed system.

No, CDB also works without shared KV and can use last-exited KV for those models, which preserves accuracy.

Evaluation on more diverse and larger looped models to determine whether this issue is specific to Ouro.

Thank you for this suggestion. Concurrent work found that Huginn and smaller Parcae models work well with single-slot shared KV, while Ouro and larger Parcae models lose accuracy [R1]. NVIDIA’s recent model Dory1.0-4B also supports it [R2]. We expect more looped models to support it in the future, but as mentioned above, CDB also supports other cache layouts.

W2: Full end-to-end inference evaluation.

The main serving experiments replay pre-recorded exit decisions and use reference output lengths rather than performing actual autoregressive generation with online exit gating. Although this setup enables controlled comparisons, it does not fully capture the overhead and behavior of a realistic serving pipeline.

Our original description was misleading here. Our replay experiments include the full end-to-end cost of autoregressive generation and serving, including scheduling and running the exit gate. We only overwrite each request’s output length and exit decision during live generation, but the scheduler does not know this and thus cannot take advantage of this. We rewrote this part of the experimental setup (Sec. 6.1 and Appendix D).

Additional experiments with real-time exit decisions.

Thank you for this suggestion. In replay experiments, CDB already runs the exit gate in real time and just overwrites its prediction, so its cost is included. We still ran additional experiments on GSM8K without replay and saw similar speedups to our main results, validating our setup (Table R.1).

These experiments also let us measure accuracy and throughput together. They show that CDB with last-exited cache gives useful accuracy for Ouro at multiple thresholds with different accuracy-speed trade-offs. We will continue to test more thresholds and include full accuracy-throughput curves for Ouro, Ouro Thinking, and Huginn in the final manuscript.

Table R.1: Ouro 1.4B GSM8K Accuracy vs. throughput speedup.

MethodGSM8K accuracySpeedup
CB, 4 loops78.09%-
CDB, lookahead78.39%1.13x
CDB, lookahead75.51%1.43x
CDB, lookahead72.78%1.57x

W3: Scale and model architecture.

As the authors stated in the limitation section, the scale of the model, number of GPUs and components of model architectures is limited.

We agree that we tested on models of limited scale, but Ouro and Huginn are representative of the scale and architecture of publicly available looped models at the moment. Currently, looped models are about scaling compute of small models and not parameters, due to their parameter efficiency. We are interested to see how this evolves. Our full implementation supports multi-GPU tensor parallelism, so the community can easily evaluate CDB on future larger models.

References

[R1] BeNice! Closing Looping’s Inference-Memory Gap via KV-Sharing (https://arxiv.org/abs/2610.31633)

[R2] Dory1.0-4B: A Looped Normalized-GPT (https://arxiv.org/abs/2611.03998).

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Thank you for your response and clarification. I will maintain my original score and supported position as discussed.

Reviewer_HoU8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to translate adaptive recurrence in looped language models into faster inference. It implements continuous depth batching (CDB) through stage-wise scheduling, depth-aware KV caching, and asynchronous lookahead scheduling. Experiments on Ouro and Huginn measure offline throughput and online serving latency. The analysis examines how boundary-stage costs, batch size, and hardware utilization affect the benefits of refill.

Critical strengths
  1. The paper is, in general, well-written and easy to follow.
  2. The paper addresses an important real-world obstacle to deploying adaptive-depth models: lower computation does not automatically produce faster inference.
  3. The analysis provides useful insight into when refill helps and why its benefits differ across architectures.
  4. Experimental results show promising efficiency gains across multiple models and tasks. They also provide detailed and effective ablations to justify design choices.
Critical weaknesses
  1. The reported speedups have not been established at acceptable task accuracy under live inference. This leaves the practical end-to-end benefit insufficiently supported. See Q1.
  2. The paper does not establish a better accuracy–throughput trade-off than standard batching with fewer fixed loops. This comparison is needed to justify the added complexity of adaptive scheduling. See Q2.
Questions for authors
  1. Table 1 specifies shared KV caching, while Table 4 reports 0.23% GSM8K accuracy for Ouro with single-slot sharing. Which layout is used in Fig. 5? Did the reported throughput improvement align with the useful accuracy? What throughput gain does a full depth-indexed or first-then-shared KV layout provide that delivers usable performance for Ouro?

  2. Does adaptive CDB outperform standard CB with fewer fixed loops at comparable accuracy? For example, accuracy-throughput curves for both methods, including fixed 2/3/4 loop baselines for Ouro.

  3. What overhead does the implemented lookahead gate introduce when enabled, and how does it affect the reported throughput gains?

3 comments

Thank you for your constructive review and suggestions. We address your comments below.

We also uploaded a revised manuscript based on reviewer feedback with the main changes colored in blue.

Weaknesses & Questions

W1 & Q1: Throughput of shared and last-exited KV cache and Ouro accuracy.

Did the reported throughput improvement align with useful accuracy?

Yes, the throughput gains are nearly the same with the accuracy-preserving last-exited cache, so the reported improvements align with useful accuracy also for Ouro. Specifically, at the same batch size, switching from shared to last-exited KV changes throughput by less than 2%.

Which KV cache layout was used in Fig. 5?

Our main experiments (including Fig. 5) use single-slot sharing only because its lower memory cost lets us test larger batch sizes and compare across a wider range of compute regimes. The last-exited cache shows similar results. We clarified this now in the experimental setup (Sec. 6.1).

What throughput gain does a depth-indexed (last-exited) layout provide for Ouro?

With last-exited KV, CDB reaches 1.56 to 1.66x the throughput of CB on ShareGPT, compared to 1.52 to 1.54x with shared KV. So the accuracy-preserving layout keeps the full speedup. We added the following results to the revised manuscript:

  1. At matched batch sizes, switching from shared to last-exited KV changes CDB throughput by less than 2%. But last-exited KV uses more memory, which limits the largest batch we can test (Appendix B, Fig. 11).
  2. At matched memory budgets, last-exited KV fits smaller batches and gives lower absolute throughput than shared KV. But both CB and CDB need accuracy-preserving KV layouts, so both run smaller batches, and CDB keeps the full speedup over CB (Sec. 6.4, Fig. 8).
  3. The additional gain comes from our new memory-saving last-exited variant, which frees unused KV slots after early exits (Sec. 4.2). This lets CDB fit larger batches than the copying variant at the same memory budget (Appendix B, Fig. 12).

W2 & Q2: Accuracy-throughput curve and fewer fixed loops.

Does adaptive CDB outperform standard CB with fewer fixed loops at comparable accuracy?

Yes. On GSM8K, CDB with Ouro 1.4B achieves comparable accuracy with 13% higher throughput than 4-loop CB and 7.5% higher throughput than 3-loop CB (see Table R.1 below). With Ouro 1.4B Thinking, the speedups are 21% and 14%, respectively (Table R.2).

The larger gains for Ouro Thinking come from its longer outputs (about 100 tokens for Ouro and 800 for Ouro Thinking). This is consistent with our results in Sec. 6. CDB gives larger end-to-end speedups when prefill accounts for a smaller share of the runtime (Sec. 7).

An additional benefit of CDB is that tuning the exit threshold gives finer control over the accuracy-throughput trade-off. For example, an intermediate point between 2-loop and 3-loop CB (Table R.1 and R.2).

Thank you for suggesting this comparison. We will continue to test more thresholds and include full accuracy-throughput curves for Ouro, Ouro Thinking, and Huginn in the final manuscript.

Table R.1: Ouro 1.4B GSM8K Accuracy vs. throughput speedup.

MethodGSM8K accuracySpeedup vs. 4-loop CBComment
CB, 4 loops78.09%1.00x
CDB, lookahead78.39%1.13x13% more than loop 4
CB, 3 loops75.59%1.33x
CDB, lookahead75.51%1.43x7.5% more than loop 3
CDB, lookahead72.78%1.57xintermediate point
CB, 2 loops63.38%1.96x

Table R.2: Ouro 1.4B Thinking GSM8K Accuracy vs. throughput speedup.

MethodGSM8K accuracySpeedup vs. 4-loop CBComment
CB, 4 loops84.38%1.00x
CDB, lookahead84.76%1.21x21% more than loop 4
CB, 3 loops82.64%1.33x
CDB, lookahead82.41%1.52x14% more than loop 3
CDB, lookahead80.97%1.68xintermediate point
CB, 2 loops77.79%1.81x

Q3: Overhead from lookahead gate.

What overhead does the implemented lookahead gate introduce when enabled, and how does it affect the reported throughput gains?

Compared with the original gate, the lookahead gate introduces no additional overhead. For Ouro, both gates use a single linear layer, just with different weights. Compared with fixed depth CB (without gate), the extra compute is just that small linear layer. The reported CDB throughput already includes this cost, as during exit replay, we run the gate as usual and replace only its decision with the recorded one. We clarified this in the revised experimental setup (Sec. 6.1 and Appendix D).

Furthermore, lookahead lets the scheduler process the exit decision in parallel with the next loop step, which reduces GPU idle time from 25.80% to 1.13% compared to the original gate (Fig. 3, right). This roughly translates to 20% higher throughput for Ouro.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Thank you for the thoughtful response, the additional experiments, and the expanded discussion. Your response addresses my main concerns and answers my questions. I remain supportive of the paper and am happy to maintain my original score.

Comments on the paper

2 comments

Dear Reviewers, thank you again for your detailed reviews. As the discussion period ends November 18, please let us know whether there are any outstanding questions. We are happy to address them before the deadline. If our response has resolved your main concerns, we would appreciate if you would consider updating your score accordingly.

AuthorsOpenReview ↗

Summary of revisions

Dear Reviewers,

We have completed the additional experiments mentioned in our responses and added them to the revised manuscript (main changes in blue). The method and main conclusions are unchanged. But we believe the paper is now clearer and better supported thanks to your feedback.

The revision addresses the main concerns as follows:

  1. Accuracy-preserving KV cache for Ouro: The last-exited cache preserves Ouro's accuracy, and we now explicitly show that CDB's speedup remains unchanged with it. At matched batch sizes, throughput changes by less than 2% compared to shared KV. Additionally, a new ablation limiting cache memory shows that CDB's speedup over CB even grows with last-exited (Sec. 6.4, Fig. 8, Appendix B).
  2. Accuracy-throughput comparison with fixed-depth baselines: At comparable GSM8K accuracy, CDB is ~12% faster than 4-loop CB and ~6% faster than 3-loop CB for Ouro, and ~30-40% and ~23% faster for Ouro Thinking*. CDB also adds many useful intermediate points with high accuracy. For Huginn, the gains are small, consistent with our finding that its costly boundary stages limit the gains from CDB (Sec. 6.4, Fig. 9).
  3. Experimental setup: We clarified that our measurements include the full end-to-end cost of autoregressive generation and serving, including scheduling and running the exit gate, and we clarified which exit policies we used. We also report mean exit depth instead of exit thresholds for easier interpretation (Sec. 6.1, Sec. 6.4, Appendix D, Appendix E.2).

Thank you again for taking the time to review our paper.

*final measurements differ from the preliminary rebuttal tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.