acceptodds
Under review as a conference paper at ICLR 2027

What Does Another Loop Buy? Attention Reallocation Tracks the Marginal Value of Test-Time Compute in Looped Transformers

Abstract

Looped transformers repeatedly apply shared weights, making iteration count a natural control for test-time compute. However, the benefit of additional iterations is non-monotonic, while existing adaptive stopping methods typically require training or rely on output-level heuristics. We show that the marginal value of a loop is reflected in attention reallocation between successive iterations. Across controlled synthetic tasks and pretrained looped language models, reallocation peaks near the largest marginal gain and decays as performance saturates; alternative signals based on entropy or hidden-state change do not consistently exhibit both behaviors. Intervention experiments further support a functional role for this reallocation: freezing attention updates near the reallocation peak causes substantial accuracy loss, whereas removing attention produces a distinct, position-dependent damage profile. This observation yields a training-free stopping rule that stops after the last iteration whose reallocation exceeds 10% of its peak, followed by one margin iteration. Both constants are fixed on synthetic tasks with known required depths and transferred unchanged to pretrained models. The rule matches maximum-depth accuracy on all eight synthetic runs while using roughly half the computation, and transfers across two model families and three scales, saving 25% of FLOPs on Ouro-2.6B and 48% on Huginn-3.5B. On GSM8K and ARC-Challenge, the selected stops satisfy pre-registered equivalence criteria relative to maximum-depth inference. Reallocation profiles are also highly consistent across inputs on the evaluated short-answer tasks, enabling a shared model-level stopping decision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped transformer, efficient inference, language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kaqTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether the marginal value of an extra loop in a looped transformer can be read off an ordinary forward pass, with no retraining, no halting head and no labels. It measures attention reallocation between successive iterations at the answer position, finds that it peaks at the iteration of largest marginal gain and collapses as accuracy saturates, and derives a training-free rule: stop one iteration after the last one whose signal is at least 10% of its peak. The two constants are fixed on pointer-chase tasks with known depth and transferred unchanged, saving 25% of FLOPs on Ouro-2.6B and 48% on Huginn-3.5B.

Critical strengths
  1. A genuinely post-hoc signal: no retraining, no halting head, no calibration, a few lines of code, and it is benchmarked against Huginn's shipped exits and an output-stability baseline rather than against weak surrogates.

  2. Two model families with a real architectural contrast, three scales, four benchmarks, plus controlled synthetic tasks where required depth is fixed by construction and is used to fix the constants before any pretrained model is touched.

Critical weaknesses
  1. The causal evidence is weaker than the framing: freeze damage does not track the signal on Ouro-1.4B, and freezing half the heads hurts more than freezing all of them.

  2. The zero-loss claim is narrower than the abstract suggests. Equivalence holds in two of six settings, and on MATH the rule is significantly worse than full depth even though the profile looks like GSM8K's. The paper offers no label-free way to tell which case you are in.

  3. Ouro is a weak test of the rule: with a four-iteration budget and the signal defined only from t=2, both peaks land on the first measurable step, one ratio decides the outcome, and the 25% saving is the largest that clears the paper's own margin, one a labeled depth sweep also finds.

  4. Some supporting claims are not carried by data: no hidden-state profile is reported for a pretrained model, and Figure 1a reads the synthetic result off a "90% plateau" that is never defined.

Minor: the fonts of the titles seem different than the standard font in ICLR templete.

Questions for authors
  1. Freeze damage on Ouro-1.4B peaks at iteration 3 while the signal peaks at iteration 2. Does that ordering survive at the answer level, rather than only in teacher-forced log-probability?

  2. On MATH the rule loses about four points while the profile looks the same as on GSM8K. Is there anything label-free that tells you in advance whether value extends past the reallocation window?

  3. What does the "90% plateau" in Figure 1a mean, and how does it relate to the definition of the stopping point in Section 2.3?

  4. Table 6's Ouro-1.4B row appears to use the n=100 pilot as its reference rather than the n=300 result in Table 1. Which reference does it use, and where does the "4.3-14 pt" range come from?

3 comments

We thank the reviewer for the constructive assessment. We conducted targeted experiments.

W1/Q1. Freeze effects on Ouro-1.4B

The apparent mismatch does not persist at the answer level. On 300 GSM8K problems, clean accuracy is 77.67%; Freeze@2/3/4 yields 65.00%/69.67%/77.67%. The largest loss is at iteration 2 (−12.67 pp), matching the reallocation peak rather than the iteration-3 maximum in teacher-forced log-probability. Remove@2/3 yields 10.67%/33.67%.

We also tested freeze confounds. Self-consistent reuse of the preceding attention output A₁V₁ (instead of A₁V₂) reduces Ouro-2.6B Freeze@2 damage from 17.33 to 4.33 pp on n=300. Importantly, our preregistered extension to n=1,000 confirms a residual 3.90-pp loss (82.20% clean vs. 78.30% C1; paired 95% CI [−6.10, −1.70], McNemar p=9.13e−4). Sensitivity remains without weight/value mismatch. However, an L2-calibrated random-attention control causes much greater damage (70.67 pp; 93.5% realized perturbation amplitude), so neither freezing nor non-monotonic partial-head effects isolate a reallocation-specific causal mechanism. The answer-level peak correspondence remains valid, without proving specific causality.

W2/Q2. What explains MATH's final-loop gain?

We investigated MATH's extra 4.2-pp gain at d4. At 64 actual generated positions for each of 200 MATH and 200 GSM8K questions, all 12,800 token-level attention profiles per dataset peak at iteration 2. MATH's t3/t4-to-peak ratios (5.57%/1.65%) are below GSM8K's (6.44%/2.13%). Thus, the gain is not accompanied by renewed attention reallocation during generation.

Crucially, our preregistered prefill analysis on the same 200 questions per dataset reveals stronger final-loop changes outside attention: MATH has greater hidden-state cosine change from t3 to t4 (0.03207 vs. 0.01193; Holm p=5.16e−17, g=0.926), and larger output-distribution change KL(P₃||P₄) (0.06205 vs. 0.01842 nats; Holm p=1.21e−8, g=0.592). MLP-output change is also greater (0.5121 vs. 0.3373). This reveals continued hidden-state/output changes missed by attention-only measurement, without establishing causation.

Spatial-attention features offer complementary evidence. On 50 examples from each of MATH/GSM8K/SVAMP/ARC, a 54-feature model achieved 0.9776 out-of-fold AUC (repeated five-fold CV; permutation p=0.001) for MATH-vs-rest. Its improvement over a training-fold-selected single feature was inconclusive (p=0.092), and dataset labels are only a proxy for residual loop value. We therefore report a promising diagnostic direction, not an already validated stopping gate.

W3. Does Ouro's four-loop budget limit the method?

Ouro's limited budget restricts its attainable saving, not the method's recurrence range. Across eight converged synthetic runs with required depths of 2–6 loops, the rule saves 41.7–62.5% compute while remaining within 0.1 pp of full-depth accuracy. On longer-depth Huginn, R=16 versus R=32 saves ~48% FLOPs with an observed 2.0-pp gap (not statistically equivalent). These extend beyond Ouro's d3/d4 choice.

We also quantified the advantage over labeled depth sweeps. With 10 unlabeled GSM8K prompts, the rule recovered Ouro-2.6B's full-profile d3 choice in 1,000/1,000 resamples. A measured four-depth labeled generation sweep costs 275.6 s for 10 prompts, versus 1.52 s for ten complete attention probes (~181× less profiling time, using conservative full-probe accounting). Full-data labeled accuracy argmax selects d4, while compute-efficient calibration and our rule select d3; these objectives differ.

W4/Q3/Q4. Hidden-state evidence and clarifications

We measured paired attention and hidden-state profiles on Ouro-2.6B/1.4B across GSM8K, SVAMP, and ARC (n=100 per setting). At t3, attention reallocation retains 4.75–9.08% of its peak, versus 44.73–69.63% for hidden-state change norm and 20.10–59.82% for hidden-state cosine change. All three signals peak at t2, but hidden-state changes decay much more slowly. Applying the same 10%-plus-margin rule diagnostically selects d3 from attention and d4 from either hidden-state signal; this comparison is not threshold-optimized. Thus, reallocation reveals dynamics distinct from generic state movement.

In Fig. 1a, “90% plateau” means the earliest t satisfying Acc(t) ≥ 0.9 Acc(Tmax), a descriptive synthetic reference distinct from the rule and Sec. 2.3's statistical criterion. Table 6's Ouro-1.4B row uses n=100 (75.0% versus its 74.0% full-depth reference); Table 1 uses n=300 (77.3%). We will label these references and correct the n=300 d2-to-full loss range to 4.3–12.3 pp.

Overall, the new evidence strengthens the central contribution: a label-free, inexpensive signal for model-level stopping that captures an informative aspect of recurrent attention dynamics, without requiring a universal accuracy or reallocation-specific causal claim.

Thanks for the rebuttal. The revision and the new results address most of my concerns. I think the contribution is real but narrow, so I keep my positive rating.

We thank the reviewer for the positive assessment and constructive feedback. We agree that the main contribution lies in the empirical analysis of how attention reallocation relates to the marginal value and saturation of recurrent computation, with the label-free stopping rule serving as a practical application. We have revised the manuscript to clarify the scope of our findings: restricting the zero-loss/equivalence claims to settings that pass the pre-registered tests, explicitly acknowledging the accuracy gap on MATH, and avoiding causal interpretations of the intervention results. We believe these revisions provide a clearer account of both the findings and their limitations. We appreciate the reviewer's time and thoughtful comments.

Reviewer_3zvmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies when to stop recurrent iterations in looped transformers at test time. The authors measure attention reallocation between loop t−1t-1 and loop tt at the final prompt token, averaged across all layers and heads. They define two metrics: the ℓ1\ell_1 distance (patℓ1pat_{\ell_1}) and the cosine distance (patcos⁡pat_{\cos}). They also propose a stopping rule that needs no training. The rule finds the peak reallocation loop t∗t^*, finds the last loop above 10%10\% of that peak, and adds one margin loop.

The authors test this rule on synthetic pointer chasing tasks and three pretrained models: Ouro1.4B, Ouro2.6B, and Huginn3.5B. On GSM8K, SVAMP, and ARC Challenge, the rule selects depth 3 out of 4 on Ouro, which saves 25% of FLOPs. On Huginn, the rule selects depth 16 out of 32, which saves 48% of FLOPs. The authors also run causal interventions that freeze or zero the attention weights at one loop. Finally, they show that reallocation varies very little across inputs, so the rule picks one fixed stopping depth per model.

Critical strengths
  1. The paper asks a clear question about looped transformers and reports negative results honestly. Section 5.3 and Appendix H show that per sample stopping thresholds do not beat a random permutation null on GSM8K or SVAMP. Many adaptive compute papers claim per sample gains without testing against a random shuffle at matched FLOPs. Showing that the stopping point is a model property rather than an input property is a useful finding.

  2. The statistical reporting is also thorough. The authors use paired McNemar tests, report confidence intervals, and test equivalence with TOST at a pre registered margin.

  3. In Appendix D, they disclose an off-by-one bug from an earlier sweep. In Appendix G, they also report the 12 synthetic runs that failed to converge.

Critical weaknesses
  1. the stopping rule is not an input adaptive testtime method. It selects depth 3 for all 300 GSM8K problems, all 300 SVAMP problems, and 496 of 500 MATH problems on Ouro2.6B. In addition, the method measures reallocation only on the prompt prefill, even though GSM8K and MATH generate up to 512 COT tokens. Because the rule outputs a constant depth for the model, we do not need runtime hooks or eager attention during deployment. We can measure 10 unlabeled prompts once offline and fix the depth hyperparameter. Worse, because the signal does not sense problem difficulty, it still stops at depth 3 on MATH and loses 4.2 points of accuracy (Table 7).

  2. On Ouro, the signal mostly measures attention sink movement on the 4 shot prefix rather than task reasoning. Appendix K and Table 11 show that at loop 2, attention mass drains from the 4 shot exemplars to token 0 (the sink). Token 0 jumps from 4.4% of attention at loop 1 to 59.8% at loop 2, while the question tokens barely change. When the authors exclude the sink tokens in Appendix K, the loop 3 reallocation rises from 4.8% to 12.7% of the peak. Because 12.4% exceeds the 10% threshold, removing the sink makes the rule select depth 4 and saves zero compute on Ouro2.6B. This explains why the signal has a coefficient of variation of 0.01 across different math problems: every prompt shares the same 4 shot prefix.

  3. The "two fixed constants" claim hides an extra rule that switches between patcos⁡pat_{\cos} and patℓ1pat_{\ell_1}. Appendix F admits that patcos⁡pat_{\cos} and patℓ1pat_{\ell_1} cannot be swapped. If we use patℓ1pat_{\ell_1} on Ouro, loop 3 stays at 17-31% of the peak, so the rule stops at depth 4 and saves nothing. If we use patcos⁡pat_{\cos} on Huginn, the signal crosses 10% at loop 10, where accuracy is much lower. To fix this, the authors add a third threshold (ρ=10\rho = 10) on the decay ratio to pick patcos⁡pat_{\cos} for Ouro and patℓ1pat_{\ell_1} for Huginn. Moreover, Ouro only has 4 loops total. Loop 2 is always the peak, and the rule adds a +1+1 margin loop, so any threshold that cuts off at loop 2 will output depth 3.

  4. the causal intervention (Freeze@t) in Section 4.4 has an activation mismatch confound. At loop tt, Freeze@t multiplies the old attention matrix At−1A_{t-1} with the new value vectors VtV_t. Because the model updates residual states between loops, At−1A_{t-1} and VtV_t come from different iterations. At loop 2, where A1A_1 and A2A_2 are nearly orthogonal (1−cos⁡=0.871 - \cos = 0.87), multiplying A1A_1 with V2V_2 creates scrambled features. This does not prove that loop 2 reallocation encodes reasoning value. It only shows that mixing attention weights from loop 1 with value vectors from loop 2 breaks the forward pass.

  5. the equivalence and speedup claims in the abstract are stronger than the data in the appendices. In Table 7, only two settings (GSM8K 2.6B full set at δ=3\delta=3 points and ARC2.6B) pass the preregistered TOST equivalence test. SVAMP, Ouro1.4B, Huginn R=16R=16, and Huginn R=14R=14 all fail TOST because their confidence intervals are too wide (up to −6.0-6.0 points on Huginn). Also, Huginn3.5B gets only 21.3% accuracy on GSM8K at full depth, which makes it a noisy baseline. Finally, because eager attention hooks disable FlashAttention and leftpadding bugs prevent batching (Appendix C), the actual wallclock speedup on Ouro2.6B is only about 1.5%.

Questions for authors
  1. In Appendix K, excluding the sink tokens raises Ouro2.6B's loop3 reallocation to 12.412.7% of the peak. That value exceeds the 10% threshold, so the rule runs to full depth (d=4d=4) and saves 0% FLOPs. What happens to the reallocation profile and stopping depth on Ouro2.6B in a zero shot prompt without the 4 shot exemplar prefix?
  2. On GSM8K and MATH, you measure reallocation only at the last prompt token during prefill. Did you measure reallocation across loops at the generated reasoning tokens, and does that separate MATH from GSM8K?
  3. At loop 2, Freeze@t multiplies A1A_1 with V2V_2. What happens if you copy both A1A_1 and V1V_1 from loop 1, or skip the attention block while keeping the MLP?
  4. The rule selects depth 3 on Ouro and R=14R=14 or 1616 on Huginn for almost every prompt. Why should an engineer run attention hooks at inference time instead of calibrating one fixed loop count on a small validation set?
3 comments

We thank the reviewer for the constructive feedback. We conducted experiments addressing prefix effects, generation-stage reallocation, freeze controls, and deployment.

W1/Q1. Is stopping an artifact of the shared few-shot prefix?

We tested zero-shot prefill on 300 GSM8K questions with Ouro-2.6B, removing exemplars while retaining the Question/Answer format. Iteration-2/3/4 profiles were 0.810479/0.038732/0.009603 (4-shot) and 0.760854/0.039998/0.020907 (zero-shot). Both peaked at iteration 2 and selected d3. All 300 zero-shot examples peaked at iteration 2; 298/300 individually selected d3.

The peak-signal coefficient of variation rose from 0.009908 to 0.025908 (2.61×), showing that the shared prefix reduces variability but is not necessary for the peak or stopping decision. Other sink-related effects remain possible. We will add this control to the appendix.

W2/Q2. Does generation-stage reallocation distinguish MATH from GSM8K?

We measured 64 autoregressive decoding positions for 200 questions per dataset (12,800 profiles each). All profiles peaked at iteration 2. Iteration-3/4-to-peak ratios were 5.57%/1.65% (MATH) and 6.44%/2.13% (GSM8K); secondary peaks occurred at only 0.203%/0.328% of positions. Longer generations showed the same pattern. Thus, MATH's additional 4.2-point d4 accuracy gain occurs without renewed generation-stage reallocation.

A prefill analysis on the same questions found larger iteration-3-to-4 changes on MATH than GSM8K:

  • Hidden-state cosine change: 0.03207 vs. 0.01193 (Holm-adjusted p=5.16e−17, Hedges' g=0.926).
  • Output KL(P3||P4): 0.06205 vs. 0.01842 nats (p=1.21e−8, g=0.592).
  • MLP-output change: 0.5121 vs. 0.3373 (g=1.246).

These results identify internal changes persisting after attention reallocation saturates, providing a correlate of MATH's additional computation without establishing causality. Attention reallocation captures one aspect of recurrence, not a universal measure of marginal accuracy gain. We will clarify this in the revision.

W3/Q3. Activation-mismatch controls for Freeze@2

We tested both requested controls on 300 Ouro-2.6B GSM8K questions with matched settings. C1 reuses the preceding complete attention output A1V1, avoiding stale-weight/current-value mismatch. C2 (Remove@2) zeros attention output while retaining residual and MLP computation.

Accuracy was 81.67% (clean), 64.33% (Freeze@2), 77.33% (C1), and 31.00% (C2). C1 reduced freeze damage from 17.33 to 4.33 points (C1 vs. Freeze p=7.46e−7; C1 vs. clean p=0.0533). C2 lost 50.67 points (p=1.36e−38). An entropy- and sparsity-matched random attention-row permutation scored 7.67%, highlighting the damage from generic structural disruption.

A preregistered 1,000-question C1 extension confirmed a residual effect: clean 82.20% vs. C1 78.30% (−3.90 points; paired 95% CI [−6.10, −1.70]; McNemar p=9.13e−4).

Thus, activation mismatch explains much of the freeze damage, but not all. Alongside Ouro-1.4B's largest answer-level freeze loss at iteration 2, these findings support a relationship between recurrent attention dynamics and performance, not a reallocation-specific causal claim.

W4/Q4. Deployment and labeled calibration

Our method profiles a model once offline, fixes its depth, and runs inference without attention hooks or per-request routing. Across 1,000 unlabeled 10-prompt GSM8K subsets, d3 agreement with full-profile selection was 1,000/1,000 on Ouro-2.6B and 999/1,000 on Ouro-1.4B.

On Ouro-2.6B, four-depth labeled calibration costs 27.56 s/prompt (275.6 s for 10), versus 1.52 s for ten complete attention probes, a 181× reduction using total probe cost rather than incremental overhead.

On the same 300 GSM8K questions, measured latency was 11.051 s/request for d4 (SDPA, no hooks), 8.271 s for d3 (prefill probe, hook-free generation; 1.336× speedup), and 9.977 s for d3 with generation-time hooks (1.108×). Hook-free d3 reduces request latency by 25.16% and token-normalized latency by 23.56%, recovering 94.23% of the theoretical 25% FLOPs saving. The request-level reduction also reflects 3.94% fewer generated tokens, while net prefill-probe overhead is only 34.0 ms.

Only two of six settings pass preregistered TOST equivalence: Ouro-2.6B on GSM8K (3-point margin) and ARC-Challenge (2-point margin). We make no equivalence claim for the others. MATH loses 4.2 points at d3, while Huginn demonstrates cross-architecture recurrence-profile behavior and compute savings rather than strong accuracy preservation given its low GSM8K baseline.

Overall, these results rule out the shared prefix as a necessary explanation, clarify freeze confounds and residual computation beyond attention reallocation, and establish an efficient label-free profiling workflow. Our method selects a shared model-level depth without labeled calibration or runtime attention hooks, while its task-dependent accuracy limitations remain explicit.

Reviewer_3zvmOpenReview ↗

Official Response from Reviewer 3zvm

I thank the reviewer for their response and detailed rebuttal, If i view this paper primarily as a mechanistic study of how attention relocatetion behaves relavtive to MLP refiement in looped transformers, your response is quite strong there and rebuttal supports it.

However, the authors have no addresses the answered patcospat_{cos} vs patl1pat_{l1} metric switching rule and clarify whether excluding 0 attention sinks breaks the 10% threshold as it did in 4 shot. Hence, I am keeping my original score.

We thank the reviewer for raising these two questions. We would like to clarify two aspects of our method.

1. Metric selection is independent of the stopping rule.

The choice between (patcos⁡)and(patℓ1)(\mathrm{pat}_{\cos}) and (\mathrm{pat}_{\ell_1}) is a model-level measurement choice, not an additional stopping criterion. It is determined from the decay characteristics of the attention reallocation profile, using the ratio:

rho = pat_cos(t*) / pat_cos(t*+1)

This procedure does not use accuracy labels or task-specific performance. The observed ratios are 20.9, 20.5, 13.7, and 11.0 across the Ouro settings, compared with 1.30 for Huginn. Furthermore, any ρ\rho threshold between 2 and 10 produces the same metric partition in our experiments. Thus, the metric choice is robust to the precise selection threshold. Once the metric is selected, the stopping rule itself remains unchanged, using only the 10% threshold and one margin iteration.

2. Sink exclusion does not invalidate the original stopping rule.

The key distinction is that our stopping rule is defined on full attention rows, where sink positions are part of the attention distribution. The reviewer asks whether the same threshold remains applicable after removing these positions. However, sink exclusion changes the attention distribution being measured, so the resulting reallocation profile is not directly comparable to the original one under the same numerical threshold.

Our additional experiment confirms this distinction. With full attention rows, the iteration-3 residual is 5.26% of the peak, resulting in d3. Excluding position 0 increases it to 16.65%, while excluding positions 0–3 gives 15.81%. These changes affect the stopping decision under the original cutoff, but do not demonstrate a failure of the proposed rule under its intended definition.

Importantly, the temporal behavior remains consistent: the peak still occurs at iteration 2, followed by a substantial decrease in reallocation at iteration 3 and a further decrease at iteration 4. This supports our observation that attention reallocation captures the transition in recurrent computation. The stopping rule exploits this behavior using the full attention distribution, rather than requiring the same numerical threshold to hold after arbitrary modifications to the measured attention rows.

We therefore maintain the original stopping rule and its reported results. We will add a brief clarification of the metric-selection procedure and the distinction between full and sink-excluded attention measurements.

Reviewer_n32uOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether attention reallocation across successive iterations of looped transformers reflects the marginal benefit of additional computation. Experiments on synthetic tasks and pretrained looped transformer models show that attention reallocation often coincides with accuracy improvements and subsequent saturation. Based on this observation, the authors propose a training-free stopping rule for selecting the number of recurrent iterations.

Critical strengths
  1. Interesting empirical finding. The paper shows that attention reallocation often aligns with the marginal accuracy gains from additional recurrent iterations. The experiments on pretrained looped transformer models demonstrate that the signal can identify stopping depths similar to those obtained through labeled calibration (Table 8).

  2. Useful mechanistic analysis. The attention-freezing and removal experiments provide evidence that reallocation plays a functional role in recurrent computation. The additional analysis of per-prompt adaptation provides an interesting finding that selecting different depths for individual prompts does not consistently improve over using a fixed model-level depth.

Critical weaknesses
  1. The proposed signal does not consistently reflect the marginal value of computation. A central claim of the paper is that attention reallocation indicates when additional recurrent iterations cease to be useful. However, Appendix F shows that stopping at depth 3 on MATH reduces accuracy from 55.4% to 51.2%, despite a reallocation profile similar to GSM8K. This demonstrates that additional computation can remain beneficial even after attention reallocation has saturated. The paper acknowledges this limitation but does not sufficiently explain when or why the relationship breaks down, limiting the reliability of the proposed stopping criterion.

  2. Unclear practical contribution of the stopping rule. Another central contribution is a training-free method for allocating test-time computation. However, Section 5.3 finds little benefit from adapting the stopping depth per prompt, suggesting that the method primarily selects a fixed model-level depth without labels. Table 8 shows that conventional labeled calibration identifies similar stopping depths. While avoiding labeled data is useful, the practical advantage remains unclear, particularly given that the signal does not consistently identify when additional computation is beneficial.

Questions for authors
  1. Reliability of the signal. Could the authors further investigate why attention reallocation saturates on MATH while additional iterations still improve accuracy? Are there other internal signals that could explain these gains? More importantly, under what conditions can attention reallocation be expected to reliably estimate the marginal benefit of additional computation?

  2. Practical contribution of the stopping rule. Given the limited benefits of per-prompt adaptation, what advantage does the proposed method offer over selecting a fixed depth using a small labeled calibration set? Could the authors compare the calibration costs and resulting accuracy–compute trade-offs of both approaches to clarify when the proposed label-free method would be preferable?

3 comments

We thank the reviewer for recognizing our empirical findings and mechanistic analyses. We conducted experiments addressing signal reliability and practical utility.

W1/Q1. Understanding the MATH counterexample and signal reliability

We investigated why MATH benefits from the final iteration despite a reallocation profile similar to GSM8K. Our results identify substantial internal differences beyond attention reallocation.

Generation-stage attention analysis. We measured reallocation at the first 64 generated-token positions for 200 MATH and 200 GSM8K questions (12,800 profiles each). All profiles peaked at iteration 2. The iteration-3/4-to-peak ratios were 5.57%/1.65% on MATH and 6.44%/2.13% on GSM8K. Thus, MATH's additional 4.2-point gain at d4 occurs without renewed generation-stage reallocation.

Hidden-state and output-distribution analysis. We then conducted a preregistered prefill experiment on the same 200 questions per dataset. Crucially, MATH exhibited significantly greater changes from iteration 3 to 4 in both hidden states and output distributions:

  • Hidden-state cosine change: 0.03207 vs. 0.01193 (MATH vs. GSM8K; Holm-adjusted p=5.16e-17, Hedges' g=0.926).
  • Output-distribution KL(P3||P4): 0.06205 vs. 0.01842 nats (p=1.21e-8, g=0.592).
  • MLP-output change: 0.5121 vs. 0.3373 (g=1.246).

These results show that hidden-state, output-distribution, and MLP changes persist on MATH after reallocation largely saturates. They provide an internal correlate of computation missed by the attention-only signal, without establishing causality for the accuracy gain.

We also explored spatial-attention features. Using 50 examples each from MATH, GSM8K, SVAMP, and ARC-Challenge, a 54-feature classifier achieved an out-of-fold AUC of 0.9776 under repeated five-fold cross-validation (permutation p=0.001). Although improvement over a training-fold-selected single-feature baseline was not significant (p=0.092), this suggests information beyond scalar reallocation magnitude. Since labels encode dataset identity, this is not a validated predictor of per-example computational benefit.

Together, these findings distinguish attention reallocation from later hidden-state and output-distribution changes. Our signal informs stopping in evaluated saturation regimes but is not a universal measure of marginal accuracy gain.

W2/Q2. Practical advantage over labeled depth calibration

The primary contribution of our stopping rule is low-cost, label-free model-level depth selection, rather than per-prompt adaptivity. A shared stopping depth also preserves regular batching and eliminates the need for per-request routing.

We conducted a new matched-prompt calibration comparison on GSM8K. For each subset size k=10, 20, and 50, we sampled 1,000 subsets and applied unlabeled reallocation profiling and labeled depth calibration to exactly the same prompts.

On Ouro-2.6B, the unlabeled rule recovered the full-profile stopping depth d3 in 100% of trials at every subset size. On Ouro-1.4B, the corresponding agreement rates were 99.9%, 100%, and 100%.

For comparison, labeled accuracy-maximizing calibration on Ouro-2.6B recovered its own full-data reference d4 in only 28.7%, 38.4%, and 56.7% of trials, respectively. Maximizing accuracy and selecting a depth that balances accuracy and computational efficiency are different objectives. On the full dataset, accuracy-maximizing labeled calibration selects d4, whereas compute-efficient labeled calibration selects d3, consistent with our stopping rule. Therefore, these methods should be compared with their respective optimization objectives in mind.

The calibration cost difference is substantial. Measured four-depth generation requires 27.56 seconds per prompt on Ouro-2.6B. For 10 prompts, labeled calibration therefore costs 275.6 seconds, compared with 1.52 seconds for ten complete attention probes, a 181× reduction in profiling time. This comparison uses the conservative measured total probe cost rather than only its incremental overhead.

The selected d3 depth uses 25% fewer FLOPs than d4. On the full GSM8K test set (n=1,319), its accuracy difference from full depth is −1.4 points and satisfies the preregistered 3-point TOST equivalence criterion (p=0.030).

These experiments further demonstrate the calibration-cost advantage of label-free depth selection. When labeled data are unavailable, our method can determine a stable model-level stopping depth using only a small number of unlabeled prompts, without generating and evaluating answers at multiple candidate depths. Labeled calibration remains a reasonable alternative when sufficient annotated data are available, while the MATH results also show that our current stopping rule does not preserve full-depth accuracy across all tasks.

We will incorporate these results to clarify both the complementary internal signals underlying residual loop value and the efficiency advantage of label-free model-level depth selection.

Thank you for the additional experiments and clarifications.

I am still not convinced that the stopping rule has a real practical advantage over labeled calibration. Calibration is usually performed only once, labeled data is often available, and cost-aware labeled calibration selects the same depth on GSM8K. Furthermore, attention reallocation does not always reflect marginal accuracy gains, as demonstrated by the experiments on MATH.

Despite that, I find the empirical relationship between attention reallocation and accuracy saturation interesting and would encourage the authors to emphasize this analysis rather than the practical stopping rule.

Given the value of these empirical findings, I have raised my score to 3.

We thank the reviewer for the reassessment and the helpful suggestion to place greater emphasis on the empirical findings. We address the remaining concerns below.

1. Practical advantage over labeled calibration.

We agree that labeled calibration is a reasonable choice when graded labels are available and the deployment distribution is stable. On GSM8K, a cost-aware labeled sweep also selects d3, consistent with our method. We have clarified this in Sec. 5.4 and do not claim that our approach outperforms labeled calibration in this setting.

Our method offers two practical benefits in other settings. First, it does not require labeled data. On Ouro-2.6B, just 10 unlabeled prompts are sufficient to recover d3 in all 1,000 subset-resampling trials, without answer grading or generation at multiple depths. This is useful when labels are expensive or unavailable, particularly if the deployment distribution changes. Second, the signal can be obtained from a single prefill forward pass, before generation begins. This could potentially support depth-batched scheduling at serving time [Schwethelm et al., 2026], whereas conventional one-time labeled calibration provides only a fixed depth. We agree that the one-time cost of calibration is not a major concern, and have revised the discussion to focus on these two distinctions rather than calibration cost.

2. MATH and the scope of the signal.

We agree that attention reallocation does not capture every improvement from additional recurrent computation. On MATH, stopping at the selected depth leaves a 4.2-point accuracy gap, which we explicitly acknowledge. We restrict our zero-loss/equivalence claims to GSM8K and ARC-Challenge, the two settings that pass the pre-registered TOST, and report confidence intervals for the remaining results.

Our additional analysis (Exp-3b) helps explain this limitation. Even after attention reallocation has largely subsided, hidden states and predictive distributions on MATH continue to change. The final-iteration hidden-state distance, measured by 1−cos(h₄,h₃), is 0.032 on MATH compared with 0.012 on GSM8K (p = 5.2×10⁻¹⁷). GSM8K shows much clearer signs of saturation. This suggests that attention reallocation captures one aspect of recurrent computation, while further improvements on MATH may arise from continued hidden-state and logit refinement. We will make this distinction clearer in the paper.

3. Reframing the contribution.

We appreciate the suggestion to emphasize the empirical analysis. We will revise the Abstract, Introduction, and Conclusion to focus on the observed relationship between attention reallocation and recurrent computation, including its consistency across model families, its correspondence with answer-level interventions, and its distinction from hidden-state refinement. The stopping rule will be presented as a practical application of these findings, rather than the central contribution, with its limitations clearly stated.

We believe this framing better reflects the evidence and addresses the reviewer's remaining concerns. Thank you again for the constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.