acceptodds
Under review as a conference paper at ICLR 2027

Let it Settle: Allocating Test-Time Compute in Looped Language Models

10 views

Abstract

Inference compute in looped language models can be allocated to additional loops or longer chains of thought. Under a fixed budget, this allocation involves a trade-off because each additional loop increases both prompt-processing and per-token generation costs. We address this with SETTLE, an allocator based on answer settling, which separates when intermediate answers stop changing from whether they are correct. Using these measurements and a small labelled calibration set, SETTLE selects a loop count and chain-of-thought token limit under an average compute budget. We evaluate six models from the Ouro, Huginn, and Recurrent-Llama families on ten reasoning benchmarks, bound selection loss in terms of accuracy-estimation error, and isolate the contributions of estimation, budget sharing, and validation. We find that the best depth and chain length vary by model, task, and budget, and that most of the improvement comes from task-level allocation, with smaller benefits from varying settings across prompts. At half the baseline cost, SETTLE-H is points more accurate than token-only allocation where both are feasible. Including prompt processing, it reaches baseline accuracy at a median of of baseline cost, allowing a one-point tolerance in four cases. Budget-conditioned self-distillation further improves GSM8K accuracy by points at a -token budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Compute, Looped Transformers, Recurrent Depth, Chain-of-Thought Reasoning, Compute Allocation, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5iaDOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies how to allocate test-time compute between recurrent loops and chain-of-thought tokens in looped language models. The authors propose SETTLE, which uses answer stability and a small labeled calibration set to estimate accuracy and select loop counts and token budgets. Experiments on six models and ten reasoning benchmarks show that joint allocation can improve accuracy under limited compute budgets, with most gains coming from task-level rather than prompt-level allocation.

Critical strengths
  1. clear motivation and good presentation, both writings and figures, making the paper easy to understand.
Critical weaknesses
  1. likely marginal contribution of the core method.

My main concern is that the key components of SETTLE provide rather limited improvements over simpler allocation strategies. In Table 2, replacing direct accuracy lookup with the settling-based estimator improves average accuracy by only about 0.1 points across budgets. Table 8 also shows that the advantage of settling estimation largely disappears with 50 or more labeled examples.

Equation (6) also uses a task-level accuracy estimate, so the per-prompt allocation mainly depends on estimated input costs rather than question difficulty. While joint allocation is useful, I suppose the additional contribution of answer settling and per-prompt allocation relatively limited.

Questions for authors

Suggestions: maybe provide difficulty-aware per-question routing via hidden-state probes for eq.6?

2 comments

Thank you for the review and the suggestion to test difficulty-aware routing. We ran the suggested experiment and added the results to the paper.

Weakness 1

My main concern is that the key .....

Our comment. The improvement from settling depends on how many labeled questions are available. Table 2 uses the full calibration set, where settling and direct lookup perform similarly. We ran a matched comparison with fewer calibration questions to measure the effect of the estimator (new experiment, App. E.2, Table 9).

In this comparison, each SETTLE variant uses both estimators, and the candidate settings, average budget, λ search, cross-validation, and calibration questions are identical. For each calibration size, we randomly sampled questions ten times and reran the full procedure across the 60 model–task cases in Table 2. The table reports accuracy differences (settling minus lookup), averaged over four budgets, with 95% bootstrap confidence intervals across cases. SETTLE-E_cap is column Ē in Table 2, the comparison you cite.

Calibration questionsSETTLE-E_capSETTLE-ESETTLE-H
6+3.05 [2.37, 3.79]+2.69 [2.07, 3.34]+0.29 [0.09, 0.51]
10+1.88 [1.37, 2.41]+1.52 [1.07, 1.99]+0.19 [0.03, 0.34]
20+0.81 [0.51, 1.16]+0.50 [0.20, 0.82]+0.22 [−0.00, 0.46]
30+0.65 [0.38, 0.91]+0.48 [0.22, 0.74]+0.06 [−0.08, 0.23]
50+0.34 [0.16, 0.54]+0.27 [0.05, 0.50]+0.03 [−0.10, 0.15]
100+0.13 [−0.03, 0.29]+0.07 [−0.14, 0.29]−0.02 [−0.22, 0.16]
All+0.08 [−0.08, 0.23]+0.07 [−0.14, 0.29]−0.07 [−0.27, 0.11]

For SETTLE-E_cap, the difference grows from 0.08 points with the full calibration set to 3.05 points with 6 questions. Direct lookup estimates accuracy separately for each loop–token setting, so its estimates are noisy with few labels. Eq. (5) instead uses labels to measure how often settled answers are correct. Since these answers remain unchanged at longer token limits, the same correctness estimates can be used across those limits (Section 4). Across the 64 comparisons in Table 9, none significantly favor lookup, while 22 of the 40 comparisons with 50 or fewer questions significantly favor settling.

Compared with Token-only, a simpler strategy that adjusts only the token limit, SETTLE-H reaches baseline accuracy at a median 68% of baseline cost against 74%, and is 9.8 points [5.5, 15.3] more accurate at half cost.

Weakness 2

Equation (6) also uses a task-level accuracy ...

Our comment. Equation (6) uses the same accuracy estimate for every prompt in a task. The only prompt-specific term is the cost C_i(k,T), which increases with prompt length. Under an average budget, this shifts compute from longer prompts to shorter ones. Per-prompt allocation adds under one point at half cost (lines 81–82).

The per-task bound picks the best single setting for each task using evaluation labels. At half cost, SETTLE-H achieves 94.1% [85.1, 100.0] of the accuracy gain that this bound has over Token-only. At quarter cost, SETTLE-H is more accurate than the bound, because it varies the setting across prompts (App. D.2).

Suggestion

maybe provide difficulty-aware per-question routing via hidden-state probes for eq.6?

Our comment. We tested difficulty-aware routing with a hidden-state probe (new experiment, App. E.7, Table 13). We added a per-question difficulty score to SETTLE-E's accuracy estimate in Eq. (6), without changing the allocation procedure.

We trained a ridge probe on the loop-1 hidden state using calibration labels. The score adjusts estimated accuracy on the logistic scale, with one coefficient per loop count. The probe uses the first-loop state, which the model computes at every setting, so it costs nothing extra. Setting the score to zero gives back SETTLE-E, so any change in accuracy comes from the score. We tested Ouro-1.4B base and Thinking on ten tasks each.

As an upper bound, we used each question's true difficulty, which is its mean correctness across all settings, computed from evaluation labels. It improves accuracy by 6.61 points [4.03, 9.51] at quarter cost and 1.55 [0.25, 2.99] at half cost. The learned probe improves accuracy by up to 0.97 points [−0.44, 2.59] at quarter cost. With prompt length added, its Spearman correlation with true difficulty is 0.44–0.55 on GSM8K, SVAMP, MATH500, and BBH, but at most 0.16 on AQuA, CSQA, ARC, StrategyQA, and HellaSwag. The upper bound suggests room for better difficulty estimates.

Revision. We added Algorithm 1 to Section 4, the matched comparison and calibration experiments to App. E.2 (Tables 9–10), and the difficulty experiment to App. E.7 (Table 13). We also clarified the label requirements in Section 4. We hope these results address your concerns and would appreciate your reconsideration of the score.

Thank you for the detailed rebuttal and the additional experiments.

Based on these clarifications, I have raised my Soundness and Confidence scores from 2 to 3.

Reviewer_kXXFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks: Under a fixed compute budget, should looped language models use more loops or generate longer chains of thought? The authors propose SETTLE, which uses answer stability and a small amount of labeled data to select loop counts and reasoning lengths. It achieves baseline accuracy at a median of approximately 68% of the baseline compute cost. The main finding is that jointly tuning loop depth and reasoning length is effective, but most gains come from task-level tuning; per-question allocation adds relatively little. The method also requires substantial upfront calibration.

Critical strengths
  1. The paper systematically characterizes the trade-off between recurrent depth and chain-of-thought length. Experiments across six models from three families and ten tasks show that longer chains cannot always compensate for insufficient recurrent depth. This provides valuable empirical guidance for inference under fixed compute budgets.

  2. The proposed joint allocation method delivers meaningful compute savings. SETTLE models answer stability separately from correctness and uses calibration data to select loop counts and reasoning lengths. It achieves baseline accuracy at a median of approximately 68% of baseline compute, without probing multiple configurations at deployment, giving the approach practical potential.

Critical weaknesses
  1. The independent contribution of answer-stability estimation is insufficiently established. Most gains come from task-level joint tuning of loop counts and reasoning lengths; at half the compute budget, SETTLE-H improves over the best single configuration by only about 0.8 percentage points. A fully matched comparison against direct accuracy estimation using the same allocation and validation procedures is missing. This makes it difficult to establish whether the core method offers meaningful advantages over simpler alternatives, weakening the methodological contribution.

  2. Substantial upfront calibration costs limit the practical benefits. The paper reports a median calibration cost equivalent to 839 baseline queries, requiring approximately 5,565 queries to break even. Consequently, the savings primarily benefit large volumes of similar queries from stable task distributions. The overall efficiency advantage remains unclear for smaller workloads or shifting distributions, limiting the practical scope of the compute-saving claims.

Questions for authors
  1. Core methodological contribution (additional evidence): When candidate configurations, average-budget allocation, cross-validation procedures, and label budgets are held identical, how does replacing answer-stability estimation with direct accuracy estimation affect final performance? This comparison would directly affect my assessment of the independent value of stability estimation.

  2. Low-label experiments (clarification): Does “using only ten labeled examples” cover all label-dependent steps, including correctness estimation, candidate-family selection, and cross-validation? Were any additional calibration labels used to select the final policy?

  3. Overall compute savings (clarification): Which comparator is used to calculate the break-even point of approximately 5,565 queries? When both SETTLE’s and the best single configuration’s calibration costs are included, how many queries are needed to amortize SETTLE’s additional cost?

3 comments

Thank you for the review and for mentioning the comparison that is important to your assessment.

Weakness 1

Independent contribution of answer-stability .......

Our comment. We added a matched comparison against direct accuracy estimation (App. E.2, Table 9; details under Question 1). We also clarify what the 0.8-point improvement measures.

The 0.8 points do not isolate the contribution of settling. Best single setting already selects both the loop count and token limit for each task using SETTLE's calibration grid, and both methods use the full calibration set, where settling and direct lookup perform similarly. The additional 0.8 points [0.1, 1.5] from SETTLE-H at half cost come mainly from per-prompt allocation (lines 81–82). This difference increases to 3.9 points at quarter cost, and 1.95 at half cost with 10 calibration questions. Compared with Token-only, which adjusts only the token limit, SETTLE-H is 9.8 points [5.5, 15.3] more accurate at half cost. It also reaches baseline accuracy at a median 68% of baseline compute, compared with 74% for Token-only.

Settling contributes when only a few labeled questions are available. Direct lookup estimates accuracy separately for each loop–token setting, so its estimates are noisy with few labels. Eq. (5) instead uses labels to measure how often settled answers are correct. Since these answers remain unchanged at longer token limits, the same correctness estimates can be used across those limits (Section 4). In the matched comparison, direct lookup is not significantly better in any of the 64 comparisons. With 6 calibration questions, settling improves SETTLE-E by 2.69 points [2.07, 3.34] on average across four budgets. For SETTLE-H, the improvement is 0.29 points [0.09, 0.51] with 6 questions and 0.19 [0.03, 0.34] with 10. Settling also provides a label-free measure of answer stability. Tasks whose answers settle later tend to show larger improvements over Token-only (Spearman −0.54 [−0.74, −0.30], App. E.6).

Weakness 2

Substantial upfront calibration ..........

Our comment. The 839 queries and the 5,565-query break-even are for the full calibration set. Calibration cost grows with the number of calibration questions, and settling needs few of them (Question 1). We tested smaller calibration sets and whether SETTLE can be used on a different task without recalibration.

With 30 calibration questions, calibration costs a median of 99 baseline queries and breaks even after 1,159 queries. With the full set, the same two figures are 897 and 10,152 (Table 11, lower panel, half cost). With 30 questions, SETTLE-H still reaches 61.4% accuracy at half cost, compared with 60.8% for Best single setting and 43.2% for Baseline (Tables 10 and 2).

We also tested transfer across tasks (App. E.8, Table 14). For each model, we calibrated SETTLE on one task and applied it to another without using any labels from the target task. The accuracy estimates come from the source task, and the budget multiplier λ is fitted on the costs of the target prompts, which needs no labels. Across 59 target cases and 522 source–target pairs, the transferred policy is 17.9 points [13.2, 22.8] more accurate than Baseline at quarter cost and 13.5 [8.3, 19.0] at half cost. At three-quarter cost, the two are similar (+1.1 [−1.4, 3.5]). At full cost, Baseline is 4.1 points [2.6, 5.8] more accurate. SETTLE therefore still improves accuracy under tight budgets on a new task without labels from that task, and recalibration is useful at higher budgets.

Q1

When candidate configurations, average-budget allocation, cross-validation procedures and label budgets are held identical, how does replacing answer-stability estimation with direct accuracy estimation affect final performance? "This comparison would directly affect my assessment."

Our comment. We ran the matched comparison for each SETTLE variant and changed only the accuracy estimator (new experiment, App. E.2, Table 9). Direct lookup was not significantly better in any comparison, while settling was more accurate when labels were scarce.

We kept the candidate settings, average budget, λ search, cross-validation, and calibration questions identical. For each calibration size, we randomly sampled questions ten times and reran the full procedure across the 60 model–task cases in Table 2. The table reports settling minus lookup in accuracy points, averaged over the four budgets, with 95% bootstrap confidence intervals across cases. SETTLE-E_cap uses per-prompt cost limits (column Ē in Table 2). Table 9 reports each budget separately.

Calibration questionsSETTLE-E_capSETTLE-ESETTLE-H
6+3.05 [2.37, 3.79]+2.69 [2.07, 3.34]+0.29 [0.09, 0.51]
10+1.88 [1.37, 2.41]+1.52 [1.07, 1.99]+0.19 [0.03, 0.34]
20+0.81 [0.51, 1.16]+0.50 [0.20, 0.82]+0.22 [−0.00, 0.46]
30+0.65 [0.38, 0.91]+0.48 [0.22, 0.74]+0.06 [−0.08, 0.23]
50+0.34 [0.16, 0.54]+0.27 [0.05, 0.50]+0.03 [−0.10, 0.15]
100+0.13 [−0.03, 0.29]+0.07 [−0.14, 0.29]−0.02 [−0.22, 0.16]
All+0.08 [−0.08, 0.23]+0.07 [−0.14, 0.29]−0.07 [−0.27, 0.11]

With 6 calibration questions, settling improves accuracy over direct lookup by 2.69 points for SETTLE-E and 3.05 for SETTLE-E_cap, averaged across four budgets. With 10 questions, the improvements are 1.52 and 1.88 points, respectively. Direct lookup estimates accuracy separately for each loop–token setting, so its estimates are noisy with few labels. Eq. (5) instead uses labels to measure how often settled answers are correct. Since these answers remain unchanged at longer token limits, the same correctness estimates can be used across those limits (Section 4). Settling therefore works better with few labels, and the difference shrinks as more labeled questions become available (line 83).

For SETTLE-E_cap, settling is ahead at every tested size and budget. Of the 64 per-budget comparisons in Table 9, 23 significantly favor settling and none significantly favor lookup. The difference is smaller for SETTLE-H because cross-validation often selects the same reference policy with either estimator.

Q2

Does "using only ten labeled examples" cover all label-dependent steps, including correctness estimation, candidate-family selection and cross-validation? Were additional calibration labels used to select the final policy?

Our comment. Yes. In Tables 8, 9 and 10, every step uses only the same n questions: the settling distributions, the correctness rates, the λ search, the choice of family and reference, and cross-validation. We select the final policy with these n questions alone, and none of them is an evaluation question. The cross-validation threshold (c = 0.5), the four families and the rule that pools settle times with fewer than eight labels are fixed at every n. App. B.4 also mentions subsampling only the labels. That describes a different experiment, the estimator comparison in App. E.2 (4.49 against 7.20 points).

Q3

Which comparator is used for the ~5,565 break-even? When both SETTLE's and the best single configuration's calibration costs are included, how many queries are needed to amortise SETTLE's additional cost?

Our comment. The comparator is the uncapped Baseline, which needs no calibration. At the full budget and under prefix-cached cost, SETTLE needs a median of 5,565 queries to recover its calibration cost, and 48 of 49 cases break even (Table 11). Under generation-only cost at equal accuracy, the median is 1,638 queries, and 36 of 49 cases break even.

SETTLE has no additional calibration cost over Best single setting, so zero queries are needed to amortise it. Best single setting needs the calibration accuracy of every setting. SETTLE uses those same runs and computes the settle times from the answers already collected.

We have provided the matched comparison and additional evidence on the value of settling with limited labels. Direct accuracy estimation is not significantly better in any comparison, while settling is significantly better in 22 of the 40 comparisons with 50 or fewer calibration questions. We hope these results address your concerns, and we would be grateful if you would reconsider your score.

Thank you for the detailed rebuttal. Most of my concerns have been resolved, rasing the score to 3.

Reviewer_brRwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3
Read review
Paper summary

Looped language models can spent resources on internal loops, which aim to solve parts of the task, or on longer chains of thought. Both can be useful, but are costly. This paper suggests the SETTLE guidance: the method tracks when intermediate internal model loop, repeating the same task, stop changing, instead of looking to the correctness of the final answer. Given a compute budget, SETTLE determines the resources for looping. Experiments show efficacy.

Critical strengths

The idea is simple, internal settling, but the solution is clever. It requires a small labelled calibration set, to determine the optimal loop count and chain-of-thought token limit for a given compute budget. It is tested on the Ouro, Huginn, and Recurrent-Llama families on different reasoning benchmarks, with interesting results.

Critical weaknesses

I am not sure about the role of the calibration questions. " it fits a policy on calibration questions that uses the settling estimate of Section 3.4 to assign each prompt a loop count and chain-of-thought token limit before generation, with cross-validation against a simple reference." I would like to see a more precise description of this, and how robust are results to the calibration set. This is not clear.

Questions for authors

Can you make a sensitivity analysis of the calibration? You find evidence that settling early does not allow to improve performance with more allocation, independently of correctness. Can then settling be perturbed, to improve correctness? How does the complexity of the task impact SETTLE?

2 comments

Thank you for the review.

Weakness

I am not sure about the role of the calibration questions. ........

Our comment. Algorithm 1 in Section 4 now describes the calibration steps. The calibration set (D) is a small labeled sample of each task, with 100 ≤ |D| ≤ 300, separate from evaluation (App. B.4). Its labels mainly tell us how often a settled answer is correct. We use it in two steps:

(1) Estimate the accuracy of each setting (loop count k, token limit T). For every calibration question, we record at which token limit its answer stops changing. This needs no labels. We then use the labels to measure how often settled answers are correct and how often unsettled answers are correct. Eq. (5) combines the settled share with these rates into one accuracy estimate per setting.

(2) Decide whether to trust these estimates. In a candidate policy, Eq. (6) chooses each prompt's loop count and token limit. The reference policy is the most accurate on calibration among three simple policies: the baseline setting when it fits the budget, the largest loop count that fits the budget on average with its longest affordable token limit, and the baseline setting with a per-prompt token limit. We fit each candidate on 70% of the set, test it on the other 30%, and repeat with the two parts swapped (two folds). A candidate passes if it is more accurate than the reference by more than 0.5 bootstrap SD in both folds. If several candidates pass, we use the one with the largest margin. If none passes, we use the reference.

Q1

Can you make a sensitivity analysis of the calibration?

Our comment. We tested how sensitive SETTLE-H is to the choice of calibration questions ( App. E.2, Table 10). For each calibration size, we randomly sampled the questions ten times and reran the full calibration procedure on all 60 model-task cases in Table 2, at half the baseline cost. The table reports mean accuracy (%) ± standard deviation over the ten runs.

Calibration questions10203050100200 (42 cases)All
SETTLE-H58.6 ± 1.5460.8 ± 0.8261.4 ± 0.7062.1 ± 0.5162.8 ± 0.2364.2 ± 0.2763.5
Best single setting56.6 ± 0.6859.8 ± 0.6260.8 ± 0.5361.6 ± 0.5562.5 ± 0.2064.2 ± 0.1362.8

The choice of calibration questions has little effect on the results. We randomly selected 100 calibration questions ten times, and SETTLE-H's accuracy varied by only 0.23 percentage points (SD) across these samples. It also performs well with fewer questions: 58.6% accuracy with 10 and 62.1% with 50, compared with 63.5% with the full set. At every calibration size, SETTLE-H matches or outperforms Best single setting. We also tested five redrawn splits. At half cost, SETTLE-H remained 4.6 to 6.2 points more accurate than Baseline, compared with 6.3 points with the original split (Table 12). The split therefore also has little effect.

Q2

You find evidence that settling early does not allow to improve performance with more allocation, independently of correctness. Can then settling be perturbed, to improve correctness?

Our comment. Yes. We trained the model to settle earlier, and its accuracy at small budgets increases. We also tested steering at inference time. We fine-tuned Ouro-1.4B on its own correct answers from higher test-time compute, with the token budget written in the prompt (Section 4). After training, half of the GSM8K questions settle within 32 tokens. Before training, this took 128 tokens. Accuracy with a 32-token limit rises from 19.7% to 57.3% (App. G). At a quarter of the base model's cost, SETTLE-H with the trained model is 19.1 points more accurate on GSM8K and 15.2 on MATH500 than with the base model. At half cost, it is 6.5 points more accurate on GSM8K and 4.3 on CSQA (Table 3, row "Budget-conditioned, uniform loops").

We also tried to change the model at inference, with steering. On AQuA, we shifted the hidden state in the direction that a linear probe associates with correct answers. This corrects up to 9.0% of wrong answers, against 4.5% for a random direction (App. G).

Q3

How does the complexity of the task impact SETTLE?

Our comment. SETTLE contributes most on tasks that need long, multi-step reasoning, and least on tasks that the model answers almost directly. At half cost, SETTLE-H is more accurate than Token-only by 18.4 points on math word problems, 4.0 on reasoning tasks and 1.3 on multiple-choice knowledge tasks (Table 4, App. D.1, cases where Token-only fits the budget). Tasks whose answers settle later show larger improvements over token-only (Spearman −0.54 at half cost, Section 5.5 and App. E.6). Settle times are measured without labels.

We hope these results address your questions and would appreciate it if you reconsidered your score.

Thank you for your good answers to my questions. I have increased the scores on presentation and informativeness, as you will improve both in the paper according to the sketched lines. I stay with my score on acceptance of the paper, without moving it up to a 4 because while your paper opens interesting research lines and makes clear contributions, it does not qualify in my opinion for a 4, which is aimed to the most intersting contributions overall.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.