acceptodds
Under review as a conference paper at ICLR 2027

Experts of Exit: Risk-Controlled Halting in Looped Language Models

Abstract

Looped language models allow recurrent depth to be adjusted at inference time, but adaptive depth requires choosing a stopping policy for each deployment. Candidate policies are heterogeneous: checkpoints may include trained gates, whereas confidence, convergence, and fixed-depth rules can be added post hoc. Our cross-model, cross-task evaluation of fixed-threshold stopping rules finds no reliable default among the tested signal families: their quality-cost rankings vary with maximum recurrent depth, model scale, and task. We formulate adaptive-depth inference as risk-controlled selection among exit experts. Given a user-specified quality-loss budget, the selector chooses the least expensive candidate that passes a simultaneous validation test. We implement this formulation in Experts of Exit (EoE) with EoE-Safe, which evaluates each candidate on its own autoregressive generations and constructs paired bootstrap max- lower bounds simultaneously calibrated over the candidate pool. Across 160 held-out selection trials on Ouro, point-estimate selection violates the quality budget in 55 trials, with 23 exceeding twice the budget. EoE-Safe records 13 violations, none exceeding twice the budget, while retaining a mean recurrent-step reduction. On all 20 splits of Ouro-2.6B/MATH500, where no candidate meets the validation criterion, it returns full depth. The simultaneous bounds have bootstrap-asymptotic validity. Deployment measurements distinguish analytical depth savings from hardware speedups. In our single-GPU prototype, deterministic fixed-depth skipping achieves – decode throughput over full-step execution, while the native gate and a JSD rule achieve and relative to their corresponding full-compute baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped language models, early exiting, adaptive inference, recurrent-depth transformers, test-time compute, risk-controlled selection, uncertainty calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wxVQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EoE-SAFE, a method for selecting an early-exit policy for looped language models. Based on a task-specific validation dataset and a maximum acceptable accuracy reduction, EoE-SAFE conducts a grid search across multiple exit policies and thresholds, and returns the combination with the lowest estimated compute cost that passes a statistical bootstrap test. Experiments on Ouro 1.4B and 2.6B show that the accuracy of exit policies depends substantially on the task, and that task-dependent selection with EoE reduces average loop depth.

Critical strengths

While the paper has some strengths (e.g., well-written, a coherent selection protocol, and transparent treatment of limitations), I do not consider any of them substantial enough to support acceptance.

Critical weaknesses

1. Limited novelty of EoE-SAFE. The method combines grid search with an established bootstrap method using a validation dataset. I would expect a practitioner to consider this straightforward approach before selecting an exit policy. Additionally, related work already calibrates early-exit policies under quality constraints [R1] and selects efficient model configurations through statistical testing [R2]. Thus, I find the methodological contribution limited.

2. Limited new empirical insights due to incomplete experiments. The paper gives a good summary of different exit gates and their task-dependent performance, but the main experiments focus on basic settings already studied in the original works on Ouro [R3] and Huginn [R4]. Both prior works examine different exit gates, thresholds, and task-dependent performance. I did not get a new empirical understanding from reading this paper.

Additionally, the experiments do not account for important settings that may change the conclusions: The main evaluation computes KV-cache entries for all steps, without evaluating KV cache sharing as recommended in the Ouro paper [R3, Sec. 5.4.2]. Later tokens can therefore attend to states whose computation is excluded from the reported mean loop steps. Your Section 6.3 discusses this limitation and clearly shows why mean selected loop steps alone are insufficient. I would like to see actual executed computation, including any cache back-fill, used as the main compute metric. This would add some new insights beyond what is known from [R3,R4] and give the actual quality-compute trade-offs. The evaluation also allows exits after a single loop, despite Ouro’s poor performance at this depth [R3, Sec. 5.3]. This may explain part of the highlighted accuracy degradation and weaken the comparison with the original gate (clearly stated: the minimum selected exit depth for Ouro should be set to 2).

3. Limited assessment of generalization of the selected gate policy. The cost of the method can be very high if a selected policy does not generalize across tasks or token budgets. A user would then need to redo the selection whenever the task or token budget changes. There is insufficient analysis of how the selected policies generalize, only validation-test comparisons within the same task and some discussion of limitations.

References

[R1] Schuster et al. Confident Adaptive Language Modeling.

[R2] Laufer-Goldshtein et al. Efficiently Controlling Multiple Risks with Pareto Testing.

[R3] Zhu et al. Scaling Latent Reasoning via Looped Language Models.

[R4] Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.

Questions for authors
  1. Related to weakness 1: What does the EoE-SAFE method add beyond applying existing prior works to looped LMs? What guidance does this give other researchers and practitioners beyond the established approach of testing on validation data based on a statistical selection method?
  2. Related to weakness 2: Which new empirical understandings does the paper add beyond what is already known from prior works?
  3. Related to weakness 2: If the cost includes all executed computation, including cache back-fill and stopping-rule overhead, does EoE still select the same policies? Does the ranking of exit policies change and is even fixed-depth the preferred approach then, since it does not require any per-token exit overheads like back-fill?
  4. Related to weakness 2: You report per-token mean depth: do early-exit policies generate more tokens than full-depth decoding? If yes, then the cost savings from looping may be reduced by more token predictions, again changing the order of approaches.
  5. Related to weakness 3: What is the compute cost of selecting a policy, including evaluating all candidates on the validation data? How does this cost compare with the compute saved during deployment?
  6. Related to weakness 3: Do you have evidence how results generalize to other tasks and token budgets? Eg, does the exit policy selected for GSM8k generalize to MATH500? Can I select a policy for a domain like "math" or "coding", or do I need a different one for every narrow task?
4 comments

W1 and Q1. Contribution and evidence

EoE-Safe jointly selects among native gates, confidence, convergence, committee rules and fixed depths on frozen looped models, using each policy's own outputs. It compares 111 prespecified candidates via signed quality differences and shared-resample simultaneous bounds. CALM [R1] uses clipped excess loss and descending-threshold tests for risk consistency; Pareto Testing [R2] orders configurations on one split and tests them on another.

To measure candidate-pool value on Ouro [R3], we retrospectively applied CALM's Hoeffding–Bentkus fixed-sequence test to our frozen outputs and 160 splits (1 percentage-point [pp] budget, overall level 0.05):

Candidate poolMean selected-depth reduction
Native-gate chain4.86%
Max-softmax, best single chain in hindsight8.54%
All 16 predefined chains, 0.05/16 per chain10.09%

Each had 0/160 held-out accuracy deficits above 1 pp. The 1.55 pp gain over max-softmax came from MMLU; all three fell back to full depth on every MATH500 and HumanEval+ split. This is a search-scope ablation on our outputs, not a CALM system replication or latency comparison.

For practitioners: compare zero-training rules and fixed depths alongside the native gate, score their generated answers, and measure cost in the intended executor. With one-loop exits allowed, max-softmax 0.9 agrees on 99.8% of tokens along full-depth 1.4B GSM8K outputs, yet loses 6.0 pp on its own generations (§5).

W2 and Q2–Q4. Empirical findings and actual execution cost

Minimum depth 2. On §5's 1.4B GSM8K examples, this minimum changes max-softmax 0.9's accuracy deficit from 5.99 to 0.91 pp, but gate 0.1's from 4.09 to 3.49 pp. Removing one-loop exits largely resolves the confidence-rule example, while the native-gate deficit persists.

Measured execution. We timed seed-0 selections at batch size 1 on an RTX A6000, including prefill, exit checks, decoding and exact KV back-fill. With prompt stopping, MMLU speedups are 1.233× on 1.4B (88/128 vs. full depth's 90/128) and 1.298× on 2.6B (94/128 each); executed token×loop work (with prefill) falls 18.9% and 20.8%. Six generation settings give 0.992–1.010×; executed work falls 0.06%–0.77% in the five non-fallback selections, where back-fill offsets 96.2%–98.9% of nominal decode savings.

Policy ranking (Q3). Ouro [R3] and Huginn [R4] compared exit rules. Among policies meeting the original 1 pp criterion, ranking by measured validation latency instead of selected depth changed 2/6 seed-0 choices across GSM8K, MATH500, and MMLU. On the test subsets, speedup changed from 0.992× to 1.004× on 1.4B GSM8K and from 1.233× to 1.163× on 1.4B MMLU. Full back-fill-inclusive re-selection across all 160 splits was not run. Fixed-3 has higher speed point estimates in all eight settings, but failed the original seed-0 1 pp validation criterion in each. On timed 2.6B GSM8K, it scores 170/192 at 1.334× vs. our selection's 166/192 at 1.003×; on all 1,319 batched items, fixed-3 trails full depth by 1.74 pp.

Two-GPU overlap. On 96 1.4B GSM8K items, 3+1 and 2+2 two-GPU overlap schedules give 1.111× and 1.165× over matched waiting with identical outputs (0.963× and 0.862× vs. single-GPU full depth, including background completion).

Output length (Q4). In the timing study, the largest mean-length increase among selected policies is ~1.7% (183.4 to 186.5 tokens on 1.4B HumanEval+). On the same items, fixed-2 averaged 415.0 tokens and ran at 0.868× full-depth speed.

KV sharing. Testing per-step prompt KV with shared step-4 decode KV (Ouro §5.4.2 [R3]), our executor still completes step 4 for shallow tokens if decoding continues. On 192 1.4B GSM8K items at full depth, switching from per-step to shared decode KV changed accuracy from 147/192 to 132/192 and mean output length from 107.4 to 324.7 tokens. Full-request speed was 0.329× relative to per-step KV. The difference from Ouro Table 14 remains unexplained; this result concerns our integration, not KV sharing in general.

W3 and Q5–Q6. Calibration cost and reuse

Logs attribute ~84 A6000 GPU-hours to 111 candidates across eight validation halves, excluding loading, retries and CPU work. For 2.6B MMLU, 1.03 GPU-hours breaks even after ~149k requests (126k–181k), conditional on confirming quality on the prompt-stopping path. At 0.992×, 1.4B GSM8K has no finite break-even point.

For each transfer (GSM8K → MATH500 and MATH500 cap 256 → 512), transferred policies exceed the 1 pp budget in 0/20 splits on 1.4B and 4/20 on 2.6B; on 1.4B, reuse retains 37%–43% selected-depth reductions on target settings. Separately, 9/24 directed seed-0 cross-task pairs exceed 1 pp. Reuse should be validated on the target task and cap; a shared “math” or “coding” label alone does not establish transfer.

We will make executed work and request latency the main cost metrics, retain selected depth as a diagnostic, and add these comparisons to the revision.

Thank you for the comments and additional experiments. However, I feel like my questions are left unanswered. You are only giving me many numbers without a clear response to my questions or interpretation.

I would appreciate more direct and understandable answers my questions.

A few additonal clarifying questions based on my interpretation:

  1. Contribution: This means your methodological contribution is the idea of having within 1pp performance? What new insight do we get from your work/results that we didn’t see before?
  2. What do you mean by “prompt stopping”? Do you also do early exit during prefill? Why only batch 1 and what is this two GPU overlap experiment? In real settings I’d assume batches are larger.
  3. GSM8k speedup: So there is no GSM8k speedup at all (1.004)?
  4. Transfer: So transfer to other tasks does not work?

Overall, it feels like all the points I raised and all the additional experiments you made substantially change your paper results. Which specific changes would you make to the manuscript?

Thank you for following up. Here are direct answers:

1. Contribution. No. The 1 pp budget is a user-chosen tolerance (Appendix A varies it from 0.2 to 5 pp). Our contribution is an evaluation and selection protocol for exit policies in looped LMs, built from standard statistical tools rather than a new test. The new findings are empirical. (i) Selecting an exit policy by validation accuracy alone exceeded the budget on 55/160 held-out splits. (ii) Measuring executed computation including cache back-fill, as you suggested, changes the picture: per-token exits cut selected depth yet save almost no generation time, because skipped steps are still computed later for the cache. (iii) You were right about one-step exits: with a minimum exit depth of 2, the §5 loss drops from 6.0 to 0.9 pp.

2. Prompt stopping, batch size, two GPUs. Yes, prompt stopping is early exit during prefill, used only on MMLU. Because the answer is a single letter, we run the prompt one recurrent step at a time and stop once the final position meets the exit rule (1.23× and 1.30× speedups). Generation tasks always run the prompt at full depth. Our adaptive executor supports only batch size 1, so larger batches remain untested, and we expect smaller gains from adaptive exit there. The two-GPU setup tests whether overlap can hide back-fill: one GPU continues decoding while the other completes the skipped steps for the cache. This overlap is 1.11–1.17× faster than waiting.

3. GSM8K. Correct. Cache back-fill leaves no meaningful speedup (0.99–1.00×). Fixed depth runs faster, but it exceeds the 1 pp budget on GSM8K.

4. Transfer. Not reliably. Transfer held on Ouro-1.4B (0/20 splits over budget) but not on Ouro-2.6B (4/20), so any reused policy must be re-validated on the new task and output cap.

Manuscript changes. The submission's runtime point in §1 (iv) and §6.3, that speed depends on executable skipped work rather than mean depth, holds and is now measured directly on the selected policies. In the revision, we will (a) report selected depth and measured latency separately, using the eight-setting runtime table as the main cost result, and (b) report both the submitted rule (33.6% depth reduction) and the zero-variance-corrected rule (9.0%) in §7. We will also (c) add the minimum-depth-2 control to §5, and (d) replace the open transfer question in §8 with these results while stating the batch-1 scope and calibration cost.

Thank you again for the detailed responses and new experiments. However, I will keep my score.

I think the methodological contribution is limited and fixed depth is now mostly the best option and not early exit. I think the paper needs some substantial rework to make throughput the main metric (replacing mean exit depth), which would need another complete review round.

Reviewer_xJYmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies adaptive stopping policies for looped language models and introduces Experts of Exit (EoE), a framework for selecting early-exit policies under a specified quality-loss budget. Its EoE-Safe procedure uses paired evaluations and simultaneous bootstrap confidence bounds to select among different stopping strategies. Experiments on several datasets examine policy reliability, quality-cost trade-offs, and the relationship between recurrent-depth savings and actual inference speedups.

Critical strengths
  • The paper evaluates stopping policies on their actual autoregressive generations. The analysis highlights how small token-level differences can lead to substantial changes in final answer quality.

  • EoE-Safe substantially reduces quality-budget violations compared with point-estimate selection while preserving meaningful recurrent-depth savings. The discussion of KV-cache backfilling and measured throughput also provides valuable practical insights.

Critical weaknesses
  • The framework largely combines existing stopping strategies with established multiple-testing and bootstrap procedures. In Table 1(b), EoE-Safe and Holm have identical violation counts, while their mean depth savings differ by only 0.9 percentage points. This leaves the additional benefit of the proposed selection procedure unclear.

  • The main 33.6% depth reduction is measured analytically, while the evaluation harness still executes all recurrent steps. The runtime experiments do not directly evaluate the policies selected by EoE-Safe across the benchmark tasks, and the cost of evaluating all candidates during task-specific calibration is not included in the efficiency comparison.

Questions for authors
  1. How much actual inference speedup would the policies selected by EoE-Safe achieve once KV-cache backfilling and other runtime overheads are included?

  2. Since all candidates need to be evaluated on validation data, how expensive is the calibration process, and when would this cost be worth the resulting savings?

4 comments

W1: Contribution and the Holm Comparison

EoE evaluates and safely selects among heterogeneous stopping rules (native gates, confidence/convergence metrics, committees, fixed depths) on frozen looped models using their own generations under a specified quality budget. Our contribution includes measuring when selected-depth savings translate into runtime speedups across execution schedules.

In Table 1(b), simultaneous statistical control (both max-t and paired-normal Holm) reduces >1 pp test violations from 55/160 under point-estimate selection (23 exceeding 2 pp) to 13/160 (0 exceeding 2 pp), with mean selected-depth cuts of 33.61% and 32.72%. Max-t and Holm select identical policies in 144/160 splits (agreeing 20/20 on LoopCoder-V2 7.6B). Both provide sound control; the primary practical gain comes from evaluating heterogeneous candidate pools rather than a single exit family.

In an ablation across the same 160 splits (Hoeffding–Bentkus tests at 1 pp budget / 0.05 level, 0/160 violations):

  • Native gate alone: 4.86% mean depth cut
  • Max-softmax alone (best single chain): 8.54% depth cut
  • Full 16-chain pool: 10.09% depth cut Expanding candidate diversity unlocks clear efficiency gains under certified statistical safety.

Finite-sample diagnostic: Among non-fallback selections, 75 for max-t and 89 for Holm matched full depth on all validation items (zero empirical variance) and were admitted without margin, accounting for 12 of the 13 violations in each method (all on HumanEval+, 82 items/half). Regularizing zero-variance bounds (score-studentized max-t: 1/160 violations, 9.01% cut; signed Clopper–Pearson: 0/160, 10.58% cut) provides a sound recipe for small validation sets.

W2 & Q1: Measured Speedup and Pipelined Execution

We timed full requests for seed-0 selected policies across eight model-task settings on single RTX A6000 GPUs at batch size 1, covering prefill, exit checks, decoding, and exact KV back-fill:

Cell (items)SpeedupWork CutPick ΔppFixed-3 Speedup / Δpp
1.4B MMLU (128)1.233×18.87%−1.561.327× / −4.69
2.6B MMLU (128)1.298×20.80%0.001.334× / 0.00
1.4B GSM8K (192)0.992×0.12%0.001.304× / −3.65
2.6B GSM8K (192)1.003×0.12%−2.081.334× / 0.00
1.4B MATH500 (192)1.010×0.77%−0.521.077× / −4.17
2.6B MATH500 (192)1.000× (fallback)0.00%0.001.455× / −5.21
1.4B HumanEval+ (82)0.999×0.06%−2.441.315× / +12.20
2.6B HumanEval+ (82)1.008×0.38%0.001.064× / +7.32

Key findings across execution schedules:

  1. Classification readout: Stopping prompt recurrence at exit yields genuine wall-clock speedups of 1.233× (1.4B) and 1.298× (2.6B), cutting executed token×loop work by 18.9%–20.8%.
  2. Generative execution bottleneck: In serial generation, exact KV back-fill offsets 96.2%–98.9% of nominal decode cuts. This explains why analytical step savings (33.6%) do not equal wall-clock latency at batch size 1, pinpointing KV cache maintenance as the core systems bottleneck in looped models.
  3. Pipelined overlap schedules: Early exits enable overlapping exact KV maintenance with subsequent token generation. On a fixed prompt (Appendix D), max-softmax 0.9 and JSD 0.02 achieve 1.41–1.84× decode throughput over serial execution with identical generations. On 96 1.4B GSM8K items, pipelined schedules (3+1 and 2+2) achieve 1.111× and 1.165× over matched two-GPU waiting (0.963× and 0.862× vs. single-GPU full depth including background completion). Overlapping cache maintenance is key to unlocking early-exit decode speedups.
  4. Fixed-depth trade-off: Fixed depth 3 has higher speed point estimates (1.064–1.455×) but causes severe quality drops (−3.65 to −5.21 pp on math tasks), failing the 1 pp validation test in all eight cells. Statistical selection is required to prevent erratic degradation.

W2 & Q2: Calibration Cost and Break-Even Volume

Evaluating all 111 candidates on eight validation halves costs 84.00 A6000 GPU-hours (from 1.03 GPU-h for 2.6B MMLU to 11.97 for 1.4B GSM8K). When does this investment pay off?

  • High-throughput classification (MMLU): At 1.03 GPU-hours calibration and 1.298× speedup, 2.6B MMLU breaks even at ~149,000 requests (95% CI: 126k–181k). For persistent query endpoints, this is readily amortized.
  • Generative workloads: Under single-GPU serial execution with exact back-fill, small latency gains (1.008× on 2.6B HumanEval+) break even at ~162,000 requests (59k–4.0M), while cells near ~1.0× require pipelined overlap or asynchronous schedules to yield positive savings.

In revision, we will clarify the Holm comparison in §7 and Fig 4(a), disclose zero-variance handling and post-hoc checks, and incorporate candidate-pool ablations, 8-cell latency benchmarks, pipelined overlap throughput, and break-even analysis.

Thank you for the additional experiments and clarifications. I have two remaining questions:

  1. After correcting the zero-variance issue, does EoE-Safe still provide a meaningful advantage over Holm and the best single-family policy under the same risk-control setting? Can this be verified on fresh held-out evaluations?

  2. Can the corrected EoE-Safe demonstrate a clear end-to-end quality–efficiency advantage over strong early-exit baselines on generative tasks, including KV-cache backfilling?

Thank you for following up. Here are answers to both questions:

1. Advantage over Holm, best single family, and fresh held-out data. Against Holm, the answer is no, and we do not claim an advantage. Before the zero-variance correction, max-t and Holm already picked the same policy on 144/160 splits and had the same 13/160 violations. We have not rerun Holm with the zero-variance correction.

Searching across policy families gives only a small gain over the best single family, and only on one task. To compare pools under a shared test without the zero-variance issue, we ran CALM's Hoeffding–Bentkus fixed-sequence test at a 1 pp budget and level 0.05. The full candidate pool cuts selected depth by 10.09%, versus 8.54% for the best single family in hindsight (max-softmax) and 4.86% for the native gate alone. All three incur 0/160 held-out violations, and MMLU accounts for the entire 1.55 pp gain.

We have not yet tested on fresh held-out data. Policy selection never uses the 160 test halves. Still, the 20 splits reuse the same benchmark items, so this check is retrospective. On these splits, the corrected max-t test yields 1/160 violations and a 9.01% mean selected-depth reduction, down from 33.6% under the submitted rule.

2. End-to-end advantage on generative tasks. No. After the zero-variance correction, the selector falls back to full depth in all six generative settings for seed 0, so it runs at full-depth speed. Even under the submitted rule, the selected policies ran at 0.99–1.01× full-depth speed because exact KV back-fill recomputes 96–99% of the skipped steps. Fixed depth 3 ran faster in all eight timed settings. It exceeded the 1 pp budget on GSM8K, but on HumanEval+ it was also more accurate on the test items. We get an end-to-end speedup only on MMLU (1.23× and 1.30×). There, prompt recurrence stops at exit, and the corrected selector keeps the same policy.

In the two-GPU setup, the fair baseline is loop-level pipeline parallelism on the same two GPUs (recurrent steps split 1–3 | 4 or 1–2 | 3–4, same total computation as full depth). Overlapping cache completion with decoding makes this pipeline 1.11× and 1.17× faster on GSM8K with identical outputs. This setup is useful only when a looped model must be pipelined anyway. For a 1.4B model that fits on one GPU, single-GPU full depth remains faster.

Manuscript changes. We will present EoE-Safe as a protocol where max-t and Holm are interchangeable tests, and report both the submitted and corrected selection rules in §7. In §6.3, we will make measured latency the main cost metric and state that per-token early exit does not yet pay off on generative tasks in our implementation.

Thank you for the response. After reviewing the additional results, I still have concerns about the limited methodological contribution and the lack of meaningful end-to-end speedups on generative tasks, which would require more substantial work to address, so I will maintain my current rating.

Reviewer_d1onOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to decide the number of recurrent steps per token in looped language models. It first shows that no stopping rule is a reliable default. The rules tested are a trained gate, confidence, output convergence, hidden-state convergence, committees and fixed depth. Their quality-cost behaviour changes with the model, the task and the maximum depth. It also shows that token-level agreement under teacher forcing does not predict final answer quality.

It then proposes EoE-Safe. Each of the candidate policies is run autoregressively on a validation set and compared with full-depth decoding on the same examples. A shared-bootstrap max-t procedure gives lower bounds that hold simultaneously for all candidates. The method returns the cheapest candidate whose bound stays within the quality budget ε, and falls back to full depth if none qualifies

Critical strengths
  1. Rigorous evaluation that targets the real failure modes. Every candidate is scored on its own autoregressive generations rather than under teacher forcing. Comparisons are paired, the bounds account for searching across candidates, and results come from held-out splits with severe violations reported separately. The paper shows that both naive approaches fail in practice: teacher-forced diagnostics miss a 5–6 point accuracy drop, and point-estimate selection exceeds the budget in 34% of trials. The method needs no retraining, so it is usable on existing checkpoints.

  2. Separation of analytical step savings from real speedups. The paper shows that per-token exits lose much of their nominal saving to KV-cache back-fill (1.31× for the native gate at a mean depth of 2.48 out of 4). Early-exit work often overstates speedups, and this result tells practitioners which exit patterns actually reduce latency.

Critical weaknesses
  1. Limited Model Scale and Evaluations
  • Only small models (≤ ~3.5B) are tested, with short, greedy chain-of-thought (256–512 token caps) and no long-reasoning models.
  • The depth savings are analytical, and runtime is measured on one model and one prompt. End-to-end speedup of the policies EoE-Safe actually selects is not reported.
  1. Limited compute matched comparisons
  • There is no compute-matched comparison with alternative uses of compute, such as a smaller fixed depth or spending the saved compute on more tokens or samples.
  • Longer output caps already remove the savings on MATH500 for Ouro-2.6B, which suggests the gains may shrink in realistic reasoning settings.
Questions for authors
  1. What is the total calibration cost (GPU-hours to generate with all the candidates on the validation set)?
  2. What is the measured wall-clock speedup of the policies EoE-Safe actually selects (for example, max-softmax and the margin–agreement committee), not only the gate, JSD and fixed-depth rules?
  3. Could the authors do a compute-matched comparison: spend the saved recurrent steps on longer chain-of-thought or self-consistency, and compare against the best fixed depth?
  4. Could authors evaluate on larger models and long chain-of-thought reasoning, with output caps that avoid truncation.
  5. Could the authors test of whether a policy selected on one task carries over to related tasks or shifted data?
2 comments

W1 and Q2. Measured speed of the selected policies

We timed seed-0 selections across eight Ouro settings on a single RTX A6000 at batch size 1, including prefill, exit checks, decoding, and exact KV back-fill. Policies are max-softmax for generation and margin–agreement for MMLU, falling back to full depth on 2.6B MATH500. Speedups and accuracy Δ (pp) are relative to full depth.

Task (items per model)1.4B speedupAccuracy Δ (pp)2.6B speedupAccuracy Δ (pp)
MMLU (128)1.233×−1.561.298×0.00
GSM8K (192)0.992×0.001.003×−2.08
MATH500 (192)1.010×−0.521.000× (fallback)0.00
HumanEval+ (82)0.999×−2.441.008×0.00

The MMLU rows stop prompt recurrence at exit; without prompt stopping, speedups were 0.98× and 0.99×. On 2.6B, both the selected policy and full depth scored 94/128. For generation in this single-GPU implementation, exact back-fill offsets 96.2%–98.9% of nominal decode savings across the five non-fallback selections.

Two-GPU execution overlaps higher-loop KV completion with subsequent token generation. On the fixed prompt in Appendix D, the prototype achieved 1.41–1.84× decode throughput relative to serial execution of the same policies. For the selected 1.4B GSM8K policy on 96 items, the 3+1 and 2+2 schedules achieved 1.111× and 1.165× over matched two-GPU waiting with identical outputs. Including background completion, these schedules achieved 0.963× and 0.862× relative to single-GPU full depth.

Fixed-3 had higher speedup estimates than the selected policy in all eight settings but never passed the seed-0 1 pp test. On the 2.6B GSM8K timing subset, it scores 170/192 at 1.334× versus the selected policy's 166/192 at 1.003×. Across all 1,319 items in the original batched evaluation, fixed-3 trails full depth by 1.74 pp.

W2 and Q3. Equal-time comparison and additional samples

On 96 Ouro-1.4B GSM8K items, we compared policies under matched per-request wall-clock budgets on the same GPU, including prefill, decoding, back-fill, and answer extraction. Each first answer is greedy; remaining time is used for independent samples at temperature 0.7 and answer voting. Unfinished first answers score 0; only completed samples vote. We selected the best fixed depth on validation data for each budget. All samples had a 256-token cap.

Seconds per itemFull-depth correctEoE-Safe correctValidation-chosen fixed depth
4.5530/9630/96Depth 2: 60/96
9.1067/9667/96Depth 3: 71/96

At 4.55 s, fixed-2 finishes more first answers before the deadline (6 uncompleted vs. 57 for EoE-Safe and 56 for full depth). At 9.10 s, the EoE-Safe-minus-fixed-3 accuracy difference is −4.17 pp (paired 95% interval [−12.50, +4.17]). In this experiment, EoE-Safe did not improve accuracy over full depth at either time budget. Matched-time longer CoT caps were not tested.

W1 and Q4. Larger model and long reasoning

We evaluated seven policies on LoopCoder-V2 (7.6B, two recurrent steps) using 1,319 GSM8K items, 20 splits, and a 724-token cap. EoE-Safe reduced splits with quality deficits above 1 pp from 11/20 under point-estimate selection to 1/20, with a mean selected-depth reduction of 1.71%. On that split, the selected policy scored 2.28 pp below full depth. This extends the evaluation to another looped architecture; architecture, prompt, and loop count differ from Ouro, and early-exit latency was not measured.

On Ouro-2.6B-Thinking on GSM8K, we used 48 validation and 96 test items. Extending unfinished 2,048-token runs to 8,192 gives 90/96 for full depth, 90/96 for the transferred base policy, and 91/96 after recalibration. All three finish naturally on the same 95 items; the remaining item reaches 8,192 under all three. On the 95 items completed by all three policies, the transferred policy used 0.92% fewer decode steps than full depth, while the recalibrated policy used 1.27% more. For Ouro-2.6B MATH500 at a 512-token cap, recalibration selects full depth in all 20 splits (§7).

Q1. Calibration cost

From generation logs, we estimate that evaluating 111 candidates across the eight validation halves required 84 A6000 GPU-hours (0.54–25.53 h per setting, e.g., 1.03 h for 2.6B MMLU), excluding model loading, retries, and CPU evaluation.

Q5. Transfer across tasks and token budgets

We tested GSM8K → MATH500 and MATH500 cap 256 → 512 on 20 frozen splits. For each transfer setting, the source-selected policy fell >1 pp below target-task full depth on 0/20 splits at 1.4B and 4/20 at 2.6B. On 1.4B, source validation selected early-exit policies in all 20/20 splits, yielding 37%–43% selected-depth reduction on the target settings. Separately, 9/24 directed seed-0 cross-task pairs exceed 1 pp. These results support validating a reused policy on the target task and token budget.

We will add these evaluations and their protocols to the revised experimental section.

I thank the authors for additional experiments and for responding to my questions. My primary concerns about a budget matched study is not fully resolved and the speedup gains of the method are still unclear to me. I keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.