- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes EoE-SAFE, a method for selecting an early-exit policy for looped language models. Based on a task-specific validation dataset and a maximum acceptable accuracy reduction, EoE-SAFE conducts a grid search across multiple exit policies and thresholds, and returns the combination with the lowest estimated compute cost that passes a statistical bootstrap test. Experiments on Ouro 1.4B and 2.6B show that the accuracy of exit policies depends substantially on the task, and that task-dependent selection with EoE reduces average loop depth.
Critical strengths
While the paper has some strengths (e.g., well-written, a coherent selection protocol, and transparent treatment of limitations), I do not consider any of them substantial enough to support acceptance.
Critical weaknesses
1. Limited novelty of EoE-SAFE. The method combines grid search with an established bootstrap method using a validation dataset. I would expect a practitioner to consider this straightforward approach before selecting an exit policy. Additionally, related work already calibrates early-exit policies under quality constraints [R1] and selects efficient model configurations through statistical testing [R2]. Thus, I find the methodological contribution limited.
2. Limited new empirical insights due to incomplete experiments. The paper gives a good summary of different exit gates and their task-dependent performance, but the main experiments focus on basic settings already studied in the original works on Ouro [R3] and Huginn [R4]. Both prior works examine different exit gates, thresholds, and task-dependent performance. I did not get a new empirical understanding from reading this paper.
Additionally, the experiments do not account for important settings that may change the conclusions: The main evaluation computes KV-cache entries for all steps, without evaluating KV cache sharing as recommended in the Ouro paper [R3, Sec. 5.4.2]. Later tokens can therefore attend to states whose computation is excluded from the reported mean loop steps. Your Section 6.3 discusses this limitation and clearly shows why mean selected loop steps alone are insufficient. I would like to see actual executed computation, including any cache back-fill, used as the main compute metric. This would add some new insights beyond what is known from [R3,R4] and give the actual quality-compute trade-offs. The evaluation also allows exits after a single loop, despite Ouro’s poor performance at this depth [R3, Sec. 5.3]. This may explain part of the highlighted accuracy degradation and weaken the comparison with the original gate (clearly stated: the minimum selected exit depth for Ouro should be set to 2).
3. Limited assessment of generalization of the selected gate policy. The cost of the method can be very high if a selected policy does not generalize across tasks or token budgets. A user would then need to redo the selection whenever the task or token budget changes. There is insufficient analysis of how the selected policies generalize, only validation-test comparisons within the same task and some discussion of limitations.
References
[R1] Schuster et al. Confident Adaptive Language Modeling.
[R2] Laufer-Goldshtein et al. Efficiently Controlling Multiple Risks with Pareto Testing.
[R3] Zhu et al. Scaling Latent Reasoning via Looped Language Models.
[R4] Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.
Questions for authors
- Related to weakness 1: What does the EoE-SAFE method add beyond applying existing prior works to looped LMs? What guidance does this give other researchers and practitioners beyond the established approach of testing on validation data based on a statistical selection method?
- Related to weakness 2: Which new empirical understandings does the paper add beyond what is already known from prior works?
- Related to weakness 2: If the cost includes all executed computation, including cache back-fill and stopping-rule overhead, does EoE still select the same policies? Does the ranking of exit policies change and is even fixed-depth the preferred approach then, since it does not require any per-token exit overheads like back-fill?
- Related to weakness 2: You report per-token mean depth: do early-exit policies generate more tokens than full-depth decoding? If yes, then the cost savings from looping may be reduced by more token predictions, again changing the order of approaches.
- Related to weakness 3: What is the compute cost of selecting a policy, including evaluating all candidates on the validation data? How does this cost compare with the compute saved during deployment?
- Related to weakness 3: Do you have evidence how results generalize to other tasks and token budgets? Eg, does the exit policy selected for GSM8k generalize to MATH500? Can I select a policy for a domain like "math" or "coding", or do I need a different one for every narrow task?
4 comments
Thank you for the comments and additional experiments. However, I feel like my questions are left unanswered. You are only giving me many numbers without a clear response to my questions or interpretation.
I would appreciate more direct and understandable answers my questions.
A few additonal clarifying questions based on my interpretation:
- Contribution: This means your methodological contribution is the idea of having within 1pp performance? What new insight do we get from your work/results that we didn’t see before?
- What do you mean by “prompt stopping”? Do you also do early exit during prefill? Why only batch 1 and what is this two GPU overlap experiment? In real settings I’d assume batches are larger.
- GSM8k speedup: So there is no GSM8k speedup at all (1.004)?
- Transfer: So transfer to other tasks does not work?
Overall, it feels like all the points I raised and all the additional experiments you made substantially change your paper results. Which specific changes would you make to the manuscript?
Thank you for following up. Here are direct answers:
1. Contribution. No. The 1 pp budget is a user-chosen tolerance (Appendix A varies it from 0.2 to 5 pp). Our contribution is an evaluation and selection protocol for exit policies in looped LMs, built from standard statistical tools rather than a new test. The new findings are empirical. (i) Selecting an exit policy by validation accuracy alone exceeded the budget on 55/160 held-out splits. (ii) Measuring executed computation including cache back-fill, as you suggested, changes the picture: per-token exits cut selected depth yet save almost no generation time, because skipped steps are still computed later for the cache. (iii) You were right about one-step exits: with a minimum exit depth of 2, the §5 loss drops from 6.0 to 0.9 pp.
2. Prompt stopping, batch size, two GPUs. Yes, prompt stopping is early exit during prefill, used only on MMLU. Because the answer is a single letter, we run the prompt one recurrent step at a time and stop once the final position meets the exit rule (1.23× and 1.30× speedups). Generation tasks always run the prompt at full depth. Our adaptive executor supports only batch size 1, so larger batches remain untested, and we expect smaller gains from adaptive exit there. The two-GPU setup tests whether overlap can hide back-fill: one GPU continues decoding while the other completes the skipped steps for the cache. This overlap is 1.11–1.17× faster than waiting.
3. GSM8K. Correct. Cache back-fill leaves no meaningful speedup (0.99–1.00×). Fixed depth runs faster, but it exceeds the 1 pp budget on GSM8K.
4. Transfer. Not reliably. Transfer held on Ouro-1.4B (0/20 splits over budget) but not on Ouro-2.6B (4/20), so any reused policy must be re-validated on the new task and output cap.
Manuscript changes. The submission's runtime point in §1 (iv) and §6.3, that speed depends on executable skipped work rather than mean depth, holds and is now measured directly on the selected policies. In the revision, we will (a) report selected depth and measured latency separately, using the eight-setting runtime table as the main cost result, and (b) report both the submitted rule (33.6% depth reduction) and the zero-variance-corrected rule (9.0%) in §7. We will also (c) add the minimum-depth-2 control to §5, and (d) replace the open transfer question in §8 with these results while stating the batch-1 scope and calibration cost.
Thank you again for the detailed responses and new experiments. However, I will keep my score.
I think the methodological contribution is limited and fixed depth is now mostly the best option and not early exit. I think the paper needs some substantial rework to make throughput the main metric (replacing mean exit depth), which would need another complete review round.
W1 and Q1. Contribution and evidence
EoE-Safe jointly selects among native gates, confidence, convergence, committee rules and fixed depths on frozen looped models, using each policy's own outputs. It compares 111 prespecified candidates via signed quality differences and shared-resample simultaneous bounds. CALM [R1] uses clipped excess loss and descending-threshold tests for risk consistency; Pareto Testing [R2] orders configurations on one split and tests them on another.
To measure candidate-pool value on Ouro [R3], we retrospectively applied CALM's Hoeffding–Bentkus fixed-sequence test to our frozen outputs and 160 splits (1 percentage-point [pp] budget, overall level 0.05):
Each had 0/160 held-out accuracy deficits above 1 pp. The 1.55 pp gain over max-softmax came from MMLU; all three fell back to full depth on every MATH500 and HumanEval+ split. This is a search-scope ablation on our outputs, not a CALM system replication or latency comparison.
For practitioners: compare zero-training rules and fixed depths alongside the native gate, score their generated answers, and measure cost in the intended executor. With one-loop exits allowed, max-softmax 0.9 agrees on 99.8% of tokens along full-depth 1.4B GSM8K outputs, yet loses 6.0 pp on its own generations (§5).
W2 and Q2–Q4. Empirical findings and actual execution cost
Minimum depth 2. On §5's 1.4B GSM8K examples, this minimum changes max-softmax 0.9's accuracy deficit from 5.99 to 0.91 pp, but gate 0.1's from 4.09 to 3.49 pp. Removing one-loop exits largely resolves the confidence-rule example, while the native-gate deficit persists.
Measured execution. We timed seed-0 selections at batch size 1 on an RTX A6000, including prefill, exit checks, decoding and exact KV back-fill. With prompt stopping, MMLU speedups are 1.233× on 1.4B (88/128 vs. full depth's 90/128) and 1.298× on 2.6B (94/128 each); executed token×loop work (with prefill) falls 18.9% and 20.8%. Six generation settings give 0.992–1.010×; executed work falls 0.06%–0.77% in the five non-fallback selections, where back-fill offsets 96.2%–98.9% of nominal decode savings.
Policy ranking (Q3). Ouro [R3] and Huginn [R4] compared exit rules. Among policies meeting the original 1 pp criterion, ranking by measured validation latency instead of selected depth changed 2/6 seed-0 choices across GSM8K, MATH500, and MMLU. On the test subsets, speedup changed from 0.992× to 1.004× on 1.4B GSM8K and from 1.233× to 1.163× on 1.4B MMLU. Full back-fill-inclusive re-selection across all 160 splits was not run. Fixed-3 has higher speed point estimates in all eight settings, but failed the original seed-0 1 pp validation criterion in each. On timed 2.6B GSM8K, it scores 170/192 at 1.334× vs. our selection's 166/192 at 1.003×; on all 1,319 batched items, fixed-3 trails full depth by 1.74 pp.
Two-GPU overlap. On 96 1.4B GSM8K items, 3+1 and 2+2 two-GPU overlap schedules give 1.111× and 1.165× over matched waiting with identical outputs (0.963× and 0.862× vs. single-GPU full depth, including background completion).
Output length (Q4). In the timing study, the largest mean-length increase among selected policies is ~1.7% (183.4 to 186.5 tokens on 1.4B HumanEval+). On the same items, fixed-2 averaged 415.0 tokens and ran at 0.868× full-depth speed.
KV sharing. Testing per-step prompt KV with shared step-4 decode KV (Ouro §5.4.2 [R3]), our executor still completes step 4 for shallow tokens if decoding continues. On 192 1.4B GSM8K items at full depth, switching from per-step to shared decode KV changed accuracy from 147/192 to 132/192 and mean output length from 107.4 to 324.7 tokens. Full-request speed was 0.329× relative to per-step KV. The difference from Ouro Table 14 remains unexplained; this result concerns our integration, not KV sharing in general.
W3 and Q5–Q6. Calibration cost and reuse
Logs attribute ~84 A6000 GPU-hours to 111 candidates across eight validation halves, excluding loading, retries and CPU work. For 2.6B MMLU, 1.03 GPU-hours breaks even after ~149k requests (126k–181k), conditional on confirming quality on the prompt-stopping path. At 0.992×, 1.4B GSM8K has no finite break-even point.
For each transfer (GSM8K → MATH500 and MATH500 cap 256 → 512), transferred policies exceed the 1 pp budget in 0/20 splits on 1.4B and 4/20 on 2.6B; on 1.4B, reuse retains 37%–43% selected-depth reductions on target settings. Separately, 9/24 directed seed-0 cross-task pairs exceed 1 pp. Reuse should be validated on the target task and cap; a shared “math” or “coding” label alone does not establish transfer.
We will make executed work and request latency the main cost metrics, retain selected depth as a diagnostic, and add these comparisons to the revision.