- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether the marginal value of an extra loop in a looped transformer can be read off an ordinary forward pass, with no retraining, no halting head and no labels. It measures attention reallocation between successive iterations at the answer position, finds that it peaks at the iteration of largest marginal gain and collapses as accuracy saturates, and derives a training-free rule: stop one iteration after the last one whose signal is at least 10% of its peak. The two constants are fixed on pointer-chase tasks with known depth and transferred unchanged, saving 25% of FLOPs on Ouro-2.6B and 48% on Huginn-3.5B.
Critical strengths
-
A genuinely post-hoc signal: no retraining, no halting head, no calibration, a few lines of code, and it is benchmarked against Huginn's shipped exits and an output-stability baseline rather than against weak surrogates.
-
Two model families with a real architectural contrast, three scales, four benchmarks, plus controlled synthetic tasks where required depth is fixed by construction and is used to fix the constants before any pretrained model is touched.
Critical weaknesses
-
The causal evidence is weaker than the framing: freeze damage does not track the signal on Ouro-1.4B, and freezing half the heads hurts more than freezing all of them.
-
The zero-loss claim is narrower than the abstract suggests. Equivalence holds in two of six settings, and on MATH the rule is significantly worse than full depth even though the profile looks like GSM8K's. The paper offers no label-free way to tell which case you are in.
-
Ouro is a weak test of the rule: with a four-iteration budget and the signal defined only from t=2, both peaks land on the first measurable step, one ratio decides the outcome, and the 25% saving is the largest that clears the paper's own margin, one a labeled depth sweep also finds.
-
Some supporting claims are not carried by data: no hidden-state profile is reported for a pretrained model, and Figure 1a reads the synthetic result off a "90% plateau" that is never defined.
Minor: the fonts of the titles seem different than the standard font in ICLR templete.
Questions for authors
-
Freeze damage on Ouro-1.4B peaks at iteration 3 while the signal peaks at iteration 2. Does that ordering survive at the answer level, rather than only in teacher-forced log-probability?
-
On MATH the rule loses about four points while the profile looks the same as on GSM8K. Is there anything label-free that tells you in advance whether value extends past the reallocation window?
-
What does the "90% plateau" in Figure 1a mean, and how does it relate to the definition of the stopping point in Section 2.3?
-
Table 6's Ouro-1.4B row appears to use the n=100 pilot as its reference rather than the n=300 result in Table 1. Which reference does it use, and where does the "4.3-14 pt" range come from?
3 comments
Thanks for the rebuttal. The revision and the new results address most of my concerns. I think the contribution is real but narrow, so I keep my positive rating.
We thank the reviewer for the positive assessment and constructive feedback. We agree that the main contribution lies in the empirical analysis of how attention reallocation relates to the marginal value and saturation of recurrent computation, with the label-free stopping rule serving as a practical application. We have revised the manuscript to clarify the scope of our findings: restricting the zero-loss/equivalence claims to settings that pass the pre-registered tests, explicitly acknowledging the accuracy gap on MATH, and avoiding causal interpretations of the intervention results. We believe these revisions provide a clearer account of both the findings and their limitations. We appreciate the reviewer's time and thoughtful comments.
We thank the reviewer for the constructive assessment. We conducted targeted experiments.
W1/Q1. Freeze effects on Ouro-1.4B
The apparent mismatch does not persist at the answer level. On 300 GSM8K problems, clean accuracy is 77.67%; Freeze@2/3/4 yields 65.00%/69.67%/77.67%. The largest loss is at iteration 2 (−12.67 pp), matching the reallocation peak rather than the iteration-3 maximum in teacher-forced log-probability. Remove@2/3 yields 10.67%/33.67%.
We also tested freeze confounds. Self-consistent reuse of the preceding attention output A₁V₁ (instead of A₁V₂) reduces Ouro-2.6B Freeze@2 damage from 17.33 to 4.33 pp on n=300. Importantly, our preregistered extension to n=1,000 confirms a residual 3.90-pp loss (82.20% clean vs. 78.30% C1; paired 95% CI [−6.10, −1.70], McNemar p=9.13e−4). Sensitivity remains without weight/value mismatch. However, an L2-calibrated random-attention control causes much greater damage (70.67 pp; 93.5% realized perturbation amplitude), so neither freezing nor non-monotonic partial-head effects isolate a reallocation-specific causal mechanism. The answer-level peak correspondence remains valid, without proving specific causality.
W2/Q2. What explains MATH's final-loop gain?
We investigated MATH's extra 4.2-pp gain at d4. At 64 actual generated positions for each of 200 MATH and 200 GSM8K questions, all 12,800 token-level attention profiles per dataset peak at iteration 2. MATH's t3/t4-to-peak ratios (5.57%/1.65%) are below GSM8K's (6.44%/2.13%). Thus, the gain is not accompanied by renewed attention reallocation during generation.
Crucially, our preregistered prefill analysis on the same 200 questions per dataset reveals stronger final-loop changes outside attention: MATH has greater hidden-state cosine change from t3 to t4 (0.03207 vs. 0.01193; Holm p=5.16e−17, g=0.926), and larger output-distribution change KL(P₃||P₄) (0.06205 vs. 0.01842 nats; Holm p=1.21e−8, g=0.592). MLP-output change is also greater (0.5121 vs. 0.3373). This reveals continued hidden-state/output changes missed by attention-only measurement, without establishing causation.
Spatial-attention features offer complementary evidence. On 50 examples from each of MATH/GSM8K/SVAMP/ARC, a 54-feature model achieved 0.9776 out-of-fold AUC (repeated five-fold CV; permutation p=0.001) for MATH-vs-rest. Its improvement over a training-fold-selected single feature was inconclusive (p=0.092), and dataset labels are only a proxy for residual loop value. We therefore report a promising diagnostic direction, not an already validated stopping gate.
W3. Does Ouro's four-loop budget limit the method?
Ouro's limited budget restricts its attainable saving, not the method's recurrence range. Across eight converged synthetic runs with required depths of 2–6 loops, the rule saves 41.7–62.5% compute while remaining within 0.1 pp of full-depth accuracy. On longer-depth Huginn, R=16 versus R=32 saves ~48% FLOPs with an observed 2.0-pp gap (not statistically equivalent). These extend beyond Ouro's d3/d4 choice.
We also quantified the advantage over labeled depth sweeps. With 10 unlabeled GSM8K prompts, the rule recovered Ouro-2.6B's full-profile d3 choice in 1,000/1,000 resamples. A measured four-depth labeled generation sweep costs 275.6 s for 10 prompts, versus 1.52 s for ten complete attention probes (~181× less profiling time, using conservative full-probe accounting). Full-data labeled accuracy argmax selects d4, while compute-efficient calibration and our rule select d3; these objectives differ.
W4/Q3/Q4. Hidden-state evidence and clarifications
We measured paired attention and hidden-state profiles on Ouro-2.6B/1.4B across GSM8K, SVAMP, and ARC (n=100 per setting). At t3, attention reallocation retains 4.75–9.08% of its peak, versus 44.73–69.63% for hidden-state change norm and 20.10–59.82% for hidden-state cosine change. All three signals peak at t2, but hidden-state changes decay much more slowly. Applying the same 10%-plus-margin rule diagnostically selects d3 from attention and d4 from either hidden-state signal; this comparison is not threshold-optimized. Thus, reallocation reveals dynamics distinct from generic state movement.
In Fig. 1a, “90% plateau” means the earliest t satisfying Acc(t) ≥ 0.9 Acc(Tmax), a descriptive synthetic reference distinct from the rule and Sec. 2.3's statistical criterion. Table 6's Ouro-1.4B row uses n=100 (75.0% versus its 74.0% full-depth reference); Table 1 uses n=300 (77.3%). We will label these references and correct the n=300 d2-to-full loss range to 4.3–12.3 pp.
Overall, the new evidence strengthens the central contribution: a label-free, inexpensive signal for model-level stopping that captures an informative aspect of recurrent attention dynamics, without requiring a universal accuracy or reallocation-specific causal claim.