- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies how to allocate test-time compute between recurrent loops and chain-of-thought tokens in looped language models. The authors propose SETTLE, which uses answer stability and a small labeled calibration set to estimate accuracy and select loop counts and token budgets. Experiments on six models and ten reasoning benchmarks show that joint allocation can improve accuracy under limited compute budgets, with most gains coming from task-level rather than prompt-level allocation.
Critical strengths
- clear motivation and good presentation, both writings and figures, making the paper easy to understand.
Critical weaknesses
- likely marginal contribution of the core method.
My main concern is that the key components of SETTLE provide rather limited improvements over simpler allocation strategies. In Table 2, replacing direct accuracy lookup with the settling-based estimator improves average accuracy by only about 0.1 points across budgets. Table 8 also shows that the advantage of settling estimation largely disappears with 50 or more labeled examples.
Equation (6) also uses a task-level accuracy estimate, so the per-prompt allocation mainly depends on estimated input costs rather than question difficulty. While joint allocation is useful, I suppose the additional contribution of answer settling and per-prompt allocation relatively limited.
Questions for authors
Suggestions: maybe provide difficulty-aware per-question routing via hidden-state probes for eq.6?
2 comments
Thank you for the detailed rebuttal and the additional experiments.
Based on these clarifications, I have raised my Soundness and Confidence scores from 2 to 3.
Thank you for the review and the suggestion to test difficulty-aware routing. We ran the suggested experiment and added the results to the paper.
Weakness 1
Our comment. The improvement from settling depends on how many labeled questions are available. Table 2 uses the full calibration set, where settling and direct lookup perform similarly. We ran a matched comparison with fewer calibration questions to measure the effect of the estimator (new experiment, App. E.2, Table 9).
In this comparison, each SETTLE variant uses both estimators, and the candidate settings, average budget, λ search, cross-validation, and calibration questions are identical. For each calibration size, we randomly sampled questions ten times and reran the full procedure across the 60 model–task cases in Table 2. The table reports accuracy differences (settling minus lookup), averaged over four budgets, with 95% bootstrap confidence intervals across cases. SETTLE-E_cap is column Ē in Table 2, the comparison you cite.
For SETTLE-E_cap, the difference grows from 0.08 points with the full calibration set to 3.05 points with 6 questions. Direct lookup estimates accuracy separately for each loop–token setting, so its estimates are noisy with few labels. Eq. (5) instead uses labels to measure how often settled answers are correct. Since these answers remain unchanged at longer token limits, the same correctness estimates can be used across those limits (Section 4). Across the 64 comparisons in Table 9, none significantly favor lookup, while 22 of the 40 comparisons with 50 or fewer questions significantly favor settling.
Compared with Token-only, a simpler strategy that adjusts only the token limit, SETTLE-H reaches baseline accuracy at a median 68% of baseline cost against 74%, and is 9.8 points [5.5, 15.3] more accurate at half cost.
Weakness 2
Our comment. Equation (6) uses the same accuracy estimate for every prompt in a task. The only prompt-specific term is the cost C_i(k,T), which increases with prompt length. Under an average budget, this shifts compute from longer prompts to shorter ones. Per-prompt allocation adds under one point at half cost (lines 81–82).
The per-task bound picks the best single setting for each task using evaluation labels. At half cost, SETTLE-H achieves 94.1% [85.1, 100.0] of the accuracy gain that this bound has over Token-only. At quarter cost, SETTLE-H is more accurate than the bound, because it varies the setting across prompts (App. D.2).
Suggestion
Our comment. We tested difficulty-aware routing with a hidden-state probe (new experiment, App. E.7, Table 13). We added a per-question difficulty score to SETTLE-E's accuracy estimate in Eq. (6), without changing the allocation procedure.
We trained a ridge probe on the loop-1 hidden state using calibration labels. The score adjusts estimated accuracy on the logistic scale, with one coefficient per loop count. The probe uses the first-loop state, which the model computes at every setting, so it costs nothing extra. Setting the score to zero gives back SETTLE-E, so any change in accuracy comes from the score. We tested Ouro-1.4B base and Thinking on ten tasks each.
As an upper bound, we used each question's true difficulty, which is its mean correctness across all settings, computed from evaluation labels. It improves accuracy by 6.61 points [4.03, 9.51] at quarter cost and 1.55 [0.25, 2.99] at half cost. The learned probe improves accuracy by up to 0.97 points [−0.44, 2.59] at quarter cost. With prompt length added, its Spearman correlation with true difficulty is 0.44–0.55 on GSM8K, SVAMP, MATH500, and BBH, but at most 0.16 on AQuA, CSQA, ARC, StrategyQA, and HellaSwag. The upper bound suggests room for better difficulty estimates.
Revision. We added Algorithm 1 to Section 4, the matched comparison and calibration experiments to App. E.2 (Tables 9–10), and the difficulty experiment to App. E.7 (Table 13). We also clarified the label requirements in Section 4. We hope these results address your concerns and would appreciate your reconsideration of the score.