acceptodds
Under review as a conference paper at ICLR 2027

Improving Test-Time Scaling with Adaptive Looped Transformers

25 views

Abstract

Looped transformers have demonstrated promising parameter efficiency by reusing layers for latent computation. Prior studies compare looped and non-looped models at matched parameters or per-token FLOPs. However, to the best of our knowledge, whether looping improves test-time scaling as outputs grow longer remains underexplored. Through post-training looped transformers, we study the accuracy–compute slope, measured as the accuracy gain per doubling of test-time decoding FLOPs. We find that existing looped transformers often yield steeper slopes than their non-looped baseline, yet underperform it at matched compute. Our token-level analysis shows that fixed-depth looping spends extra iterations on every token, although many tokens do not benefit from them. We therefore propose TaH2, which enables the model to focus extra iterations on the tokens that benefit from looping. It jointly post-trains the backbone and an iteration decider through lookahead depth supervision, with online labels indicating whether further iteration improves the prediction. A learned updater reinjects token embeddings between iterations, while the decider's stopping probabilities weight predictions across the executed depths. TaH2 improves both the efficiency and attainable accuracy of test-time scaling. On challenging AIME benchmarks, TaH2 improves the accuracy–compute slope by 53% (2.74 vs. 1.79) over the non-looped baseline, exceeding the baseline's peak accuracy by about 3.4 points at matched compute. As the maximum iteration depth increases, existing looped models largely plateau, while TaH2's gain over the non-looped baseline grows from +2.8 points at depth 2 to +3.9 points at depth 8.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped transformers, Adaptive computation, Test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bRznOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is interested in post-training standard LLM into looped models to improve their test-time scaling capabilities. It proposes TaH2, which adds a "decider" module that predicts, for each token, whether to continue looping over the whole model or to stop. The decider is supervised using the per-token change in validation loss to determine when to stop, and is trained jointly with the post-training of the backbone. Validation compares TaH2 against post-trained versions of Ouro and Huginn, on various sizes of Qwen3, evaluating both absolute performance and scaling trends.

Critical strengths

The main strength of the paper is the analysis of post-training models into looped models. The experiments show that Ouro and Huginn, when adapted to post-train a standard model, do not improve over base performance. The proposed TaH2 show both improvement of performance and better scaling.

Critical weaknesses

Major:

  • The comparison is limited to Ouro and Huginn, which were both designed to learn looped LMs from scratch, not from post-training. The relevant comparisons would be to other test-time scaling approaches, and methods that convert pretrained LLMs into Looped LMs. The paper itself discusses at length about such work in the appendix. Without these, the claim of improved scaling is not adequately supported.
  • The technical novelty seems limited with respect to TaH. The only apparent difference is jointly post-training the backbone and the decider, and a direct comparison to TaH is missing in the paper.

Minor:

  • The description in section 4 is not enough to clearly grasp the method. I had to refer to Appendix D to better understand.
Questions for authors

I would like to see a proper comparison with other test-time training method, not only reimplementation in a different setup of Ouro and Huginn:

  • In terms of scaling, does TaH2 outperform state-of-the-art test-time scaling methods?
  • Could you add comparisons to methods that were designed to convert pretrained LLMs into looped models (e.g., those mentioned in the appendix)?
  • How does TaH perform in this setting?
3 comments

We sincerely thank Reviewer bRzn for the constructive suggestions. We appreciate your recognition of our analysis of post-training looped models and TaH2's improvements in performance and test-time scaling. We address your concerns below.

W1, Q1: Comparison with SOTA test-time scaling methods

In terms of scaling, does TaH2 outperform state-of-the-art test-time scaling methods?

We added three comparisons using Qwen3-1.7B on AIME24–26 (Appendix C.4):

  • Length scaling: vary the output cutoff from 4K to 32K.
  • Sampling scaling: majority voting over up to 32 responses at a 32K cap.
  • Pareto frontier: combine length and sampling scaling to obtain Standard's best accuracy at each compute budget, which is compared with TaH2's length scaling.
Model / comparisonDecode TFLOPs / problemAccuracy (%)
(a) Length scaling
Standard19211.98
TaH2 (M=2)19215.41
(b) Sampling scaling
Standard6,13521.92
TaH2 (M=2)6,13527.17
(c) TaH2 vs Pareto frontier scaling
Standard23013.88
TaH2 (M=2)23015.49

TaH2, with matched decoding FLOPs, improves accuracy by 3.43, 5.26, and 1.61 points across these scaling setups.

W1, Q2: Compare with other post-trained loop models

The comparison is limited to Ouro and Huginn, which were both designed to learn looped LMs from scratch, not from post-training. The relevant comparisons would be to other test-time scaling approaches, and methods that convert pretrained LLMs into Looped LMs. The paper itself discusses at length about such work in the appendix. Without these, the claim of improved scaling is not adequately supported.

Could you add comparisons to methods that were designed to convert pretrained LLMs into looped models (e.g., those mentioned in the appendix)?

We add post-trained loop models from Retrofitted Recurrence [1], which increase Huginn's core loops from 1 to 3 with two curriculum setups. Other training and evaluation settings follow Table 1 in the paper.

MethodTraining loopsInference depth (avg)AIME24 (%)AIME25 (%)AIME26 (%)Mean (%)
Standard11.0011.0413.0211.8811.98
Huginn3 + curriculum (1-sqrt, 75%)1 → 32.0012.0814.3711.5612.67
Huginn3 + curriculum (linear, 25%)1 → 32.0013.5414.2711.8713.23
TaH2 (M=2)1–21.2315.7316.4614.2715.49

Training depth counts loops (core loops for Huginn); inference depth counts equivalent full-backbone passes (3 Huginn core loops = 2 passes).

TaH2 exceeds the best post-trained baseline by 2.26 points, with lower mean inference depth (1.23 vs. 2.00).

[1] Sean McLeish et al. Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence. arXiv preprint arXiv:2511.07384, 2025.

W2, Q3: Comparison with TaH

The technical novelty seems limited with respect to TaH. The only apparent difference is jointly post-training the backbone and the decider, and a direct comparison to TaH is missing in the paper.

How does TaH perform in this setting?

We add a direct comparison with TaH. All models below use Qwen3-1.7B-Base and the same SFT data. We compare TaH with different routing-label sources.

MethodRouting-label sourceAIME24 (%)AIME25 (%)AIME26 (%)Mean (%)
Standard—11.0413.0211.8811.98
TaH (M=2)Offline: Standard SFT model13.6511.6712.0812.47
TaH (M=2)Offline: official Qwen3-8B15.3114.6912.6014.20
TaH2 (M=2)Online: current model's loss gain15.7316.4614.2715.49

TaH2 outperforms the stronger TaH adaptation by 1.29 points. Note that TaH2 does not require an additional LLM for routing-label generation, as TaH does. This makes TaH2 more practical if such an LLM is not available, which is typical in SOTA model training. TaH uses offline routing labels from a reference model and trains its decider separately. TaH2 derives labels online from the current model's one-step loss gain and jointly trains the backbone, updater and decider, so routing supervision adapts as the model changes.

Minor weakness: clarify Section 4

The description in section 4 is not enough to clearly grasp the method. I had to refer to Appendix D to better understand.

We have updated Section 4 to be clearer and more intuitive, with Figure 2 as a clear illustration.

Thank you for the additional experiments and for revising the paper. The new comparisons address my first concern, and I appreciate the effort. However, I still find the novelty with respect to TaH limited, and the rebuttal reinforces this view. I understand that deriving routing labels online from the current model, rather than offline from a reference model as in TaH, is more practical and can improve performance. Beyond this change, however, the central question, the analysis, and the motivation are essentially the same as in TaH.

We sincerely thank you for the thoughtful follow-up. We summarize the core differences between TaH2 and TaH below:

AspectTaHTaH2 (Ours)
1. Central QuestionHow to improve looped LLMs' accuracy at matched parameters.How to improve looped LLMs' accuracy and its scaling behavior at matched FLOPs.
2. FindingLatent overthinking: 2nd pass flips some correct 1st-pass tokens into errors.Test-time scaling behavior: looped models have steeper scaling slopes than Standard, but lower matched-FLOPs accuracy.
3. Depth ScalingMainly M=2M=2.Scales from M=2M=2 to M=8M=8 with continuous accuracy gains.
4. Training2-stage decoupled SFT under a static oracle (train–inference mismatch).End-to-end joint training along actual routes; extends to On-Policy Distillation.
5. Routing PrincipleStatic 1st-pass mismatch: iterates if pass 1 is wrong, ignoring whether iteration actually helps.Dynamic lookahead gain: continues only where another iteration actually reduces the current model's loss.
6. InfraNaive Transformers implementation.Mini-SGLang integration with continuous depth batching.

In short, TaH2 extends looping from a binary (1-or-2 loop) choice to scalable loop depth, and shifts the focus from fixed-budget accuracy to scaling gains and steeper slopes as test-time compute grows.

Reviewer_gTCCOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to posttrain pretrained language models into adaptive looped transformers for test-time scaling. The authors first post-train Qwen31.7B using existing looped designs (Ouro and Huginn) and a non-looped baseline (Standard). By truncating generated responses at token cutoffs from 4K to 16K, they fit accuracy against log⁡2\log_2 of decoding FLOPs. They observe that Ouro and Huginn achieve steeper slopes than Standard, but lower accuracy at matched compute.

To fix this gap, the authors propose TaH2. TaH2 adds a learned SwiGLU input updater, a token-level iteration decider, a stopping-weighted output mixture, and extended duo-causal attention. During SFT, TaH2 supervises the decider online using a no-gradient lookahead iteration. That lookahead step measures whether one more loop lowers next-token cross-entropy loss. Across Qwen3 1.7B, 4B, and 8B backbones, TaH2 improves accuracy over Standard on math, code, QA, and tool-use benchmarks.

Critical strengths

The paper addresses a practical problem that is real and not solved yet, retrofitting adaptive latent depth into pretrained checkpoints during SFT without pretraining from scratch. The online lookahead supervision in Section 4.2 is a sensible improvement over two-stage training or offline mismatch labels. Supervising the halting gate with actual one-step loss reductions δt(m)=ℓt(m)−ℓt(m+1)\delta_t^{(m)} = \ell_t^{(m)} - \ell_t^{(m+1)} aligns the decider with next-token improvement (Figure 6).

The empirical evaluation at 1.7B is broad and well-documented. The authors evaluate ten benchmarks across four domains and provide detailed FLOPs accounting in Appendix A.4. They also implement continuous depth batching in Mini SGLang to measure real wall-clock latency (Table 2). The component ablations in Table 4 clearly show the value of cost-sensitive decider weights and the learned updater.

Critical weaknesses

I do figured some weakness in the paper as shared below:

  1. TaH2 and TaH2-fixed use different attention mechanisms, which confounds the central claim that adaptive depth drives the gains. In Appendix A.2.1 (line 1069) and Equation 18, TaH2-fixed uses standard causal attention. There, a query at depth mm sees only the depth-1 KV cache of earlier tokens (P+t−1+mP + t - 1 + m keys). In contrast, TaH2 uses extended duo causal attention (Equation 15). There, a query at depth mm attends to all executed KV streams up to depth mm across all earlier tokens. Thus, comparing TaH2 with TaH2-fixed in Table 1 and Figure 4 changes both the halting policy and cross token KV access. Furthermore, Table 4 never ablates extended duo causal attention against singlestream KV sharing.

  2. the paper never reports the KV cache memory overhead of extended duocausal attention during longcontext decoding. Because every executed iteration stores its own KV cache (Appendix A.2.1), TaH2 multiplies the KV cache footprint by the mean iteration depth. At M=8M = 8, TaH2 increases KV cache memory by 2.2×2.2\times to 2.4×2.4\times on average (and up to 8×8\times worstcase) over 32K tokens. In addition, Equation 15 shows that attention FLOPs grow with the sum of depths across all prior tokens. Yet Table 2 only benchmarks serving latency at M=2M = 2 (where 78% of tokens exit at depth 1) and omits memory numbers.

  3. the test time scaling slope in Section 3.2 and Figure 1a is an artifact of truncating unfinished 32K chains. As Appendix A.3 (line 1130) explains, the authors generate 32K token responses once and truncate them at 4K to 16K tokens. When a chain of thought ends before its final answer, it scores zero. Thus, sweeping the truncation cutoff measures the cumulative length distribution of finished solutions rather than budget aware testtime scaling. That artifact explains why TaH2fixed (M=2) and Ouro (M=2) get steeper slopes than Standard (2.42 and 2.12 vs. 1.79). In Figure 1a, both models have worse accuracy than Standard across the entire 4K to 16K range, and simply start lower at 4K tokens.

  4. the comparisons against Standard do not control for posttraining compute or added parameters, and the 4B/8B evaluation is incomplete. In Table 10, TaH2 uses 1.87×1.87\times (M=2M=2), 2.73×2.73\times (M=4M=4), and 5.17×5.17\times (M=8M=8) as many training FLOPs as Standard. Equation 23 reruns all NmN_m active tokens at depth m+1m+1 to reconstruct the lookahead KV context. At M=8M=8, label computation alone costs more FLOPs (52.0 EFLOPs) than the entire Standard training run (41.8 EFLOPs). TaH2 also adds 46.2M to 184.6M parameters via its updater and decider (Table 7). At 4B and 8B (Table 3), the authors test only M=2M=2 and omit all looped baselines. They also compare against Standard at a fixed 32K token limit rather than matched decoding FLOPs.

Questions for authors

based on the above points, I do have few questions based on which i will consider raising the score:

  1. How accurate is TaH2-fixed when it uses the same extended duo-causal attention (Equation 15) as TaH2, instead of base-stream causal attention (Equation 18)?
  2. What is the peak GPU KV-cache memory usage of TaH2 at M=4M = 4 and M=8M = 8 during 32K-token decoding? What is the corresponding wall-clock latency in Table 2?
  3. How does Standard perform when post-trained for matched training FLOPs (6 to 15 epochs in Table 10) or with +46.2M adapter parameters at 1.7B?
  4. In Table 3 (4B and 8B), what are the FLOPs/token ratios for TaH2 (M=2)? How does TaH2 compare against Standard at matched decoding FLOPs rather than at a fixed 32K token limit?
4 comments

We sincerely thank Reviewer gTCC for the positive assessment. We appreciate your recognition of our practical significance, training-method design, and comprehensive evaluation. We address your concerns below.

W1, Q1: Compare TaH2 and TaH2-fixed with the same attention mechanism

TaH2 and TaH2-fixed use different attention mechanisms, ... Furthermore, Table 4 never ablates extended duo causal attention against singlestream KV sharing. How accurate is TaH2-fixed when it uses the same extended duo-causal attention (Equation 15) as TaH2 , instead of base-stream causal attention (Equation 18)?

We add a direct comparison between TaH2 and TaH2-fixed across causal and duo-causal attention mechanisms, using the same backbone, training recipe and evaluation recipes (AIME avg@32, 32K output budget).

VariantAttentionAIME24 (%)AIME25 (%)AIME26 (%)Mean (%)Mean depth
TaH2-fixedCausal15.7315.0013.2314.652.00
TaH2-fixedDuo14.4814.4814.1714.382.00
TaH2Causal15.6215.4212.4014.481.23
TaH2Duo15.7316.4614.2715.491.23

With the same duo attention, TaH2 exceeds TaH2-fixed by 1.11 points while reducing mean depth from 2.00 to 1.23. The original paper reports each variant with its best-performing attention mechanism: causal for TaH2-fixed and duo for TaH2.

W2, Q2: Report KV cache memory and latency

the paper never reports the KV cache memory overhead of extended duocausal attention during longcontext decoding. ... Yet Table 2 only benchmarks serving latency at 𝑀 = 2 (where 78% of tokens exit at depth 1) and omits memory numbers. What is the peak GPU KV-cache memory usage of TaH2 at 𝑀 = 4 and 𝑀 = 8 during 32K-token decoding? What is the corresponding wall-clock latency in Table 2?

We report the KV-cache memory and latency of the 1.7B Standard, TaH2-fixed (M=2), and TaH2 (M=2/4/8) models on AIME26, using a single A800 80GB GPU with BF16, TP=1, and a 32K total-context limit.

ModelAttentionDepth (M)AIME score (%)Batch size = 1Batch size = 4
Mean depthPeak KV cache (GiB)Mean E2E latency (s/request)Mean depthPeak KV cache (GiB)Mean E2E latency (s/request)
StandardCausal111.981.0003.50139.81.00012.66220.3
TaH2-fixedDuo214.382.0007.00332.92.00025.71576.2
TaH2Duo215.491.2374.69187.01.22414.57285.4
TaH2Duo415.761.3896.52229.51.48218.45350.9
TaH2Duo816.222.27810.01343.62.24529.37610.6

Compared with the fixed-depth baseline (TaH2-fixed), TaH2 at M=2/4 uses less KV-cache memory, runs faster, and achieves higher accuracy. TaH2 at M=8 shows comparable latency, achieving additional accuracy improvements, with moderately higher KV-cache memory.

W3: Test-time scaling slope may be an artifact of length truncation

the test time scaling slope in Section 3.2 and Figure 1a is an artifact of truncating unfinished 32K chains. ... In Figure 1a, both models have worse accuracy than Standard across the entire 4K to 16K range, and simply start lower at 4K tokens.

We added budget forcing to show that our test-time scaling slope is not an artifact of length truncation. As detailed in Appendix B.3 (Fig. 8), unfinished responses are prompted to answer immediately, with up to 512 extra tokens. Below are AIME24–26 avg@32 accuracies at a 16K budget and slopes fitted over 4K–16K (accuracy points per doubling of decoding FLOPs). Here, delta reports accuracy gains in percentage points and relative slope increases over Standard.

SetupMethodAcc. (%)Slope
TruncatedStandard10.971.79
TruncatedTaH213.782.74
TruncatedDelta (TaH2 − Standard)+2.81+53.1%
Force responseStandard12.601.61
Force responseTaH214.862.47
Force responseDelta (TaH2 − Standard)+2.26+53.4%

As shown in the table, the gains remain similar across different test-time control methods, validating that the test-time slope is not an artifact of length truncation.

W4.1, Q3: Performance comparison under matched training compute

the comparisons against Standard do not control for posttraining compute or added parameters. ... TaH2 also adds 46.2M to 184.6M parameters via its updater and decider (Table 7).

How does Standard perform when post-trained for matched training FLOPs (6 to 15 epochs in Table 10) or with +46.2M adapter parameters at 1.7B?

We increase the training compute of Standard to match that of TaH2 (M=2) by continuing training from its 3-epoch checkpoint to 5.6 epochs in total. Results use Qwen3-1.7B and AIME avg@32 with a 32K output budget.

ModelTraining setupAIME24 (%)AIME25 (%)AIME26 (%)Mean (%)
Standard3 epochs11.0413.0211.8811.98
Standard (continued training)5.6 epochs14.1714.5813.3314.03
TaH2 (M=2)3 epochs15.7316.4614.2715.49

TaH2 still exceeds continued-training Standard by 1.46 points at matched training FLOPs.

W4.2: Add an Ouro-4B baseline

At 4B and 8B (Table 3), the authors test only 𝑀 = 2 and omit all looped baselines.

We added Ouro-4B [1] under the settings of Table 3, evaluated on AIME24–26 with avg@32 and a 32K output budget.

Model (4B)Output budgetAIME24 (%)AIME25 (%)AIME26 (%)Mean (%)
Standard32K51.9840.3145.5245.94
Ouro32K53.7539.9048.1347.26
TaH2 (M=2)32K57.7143.1352.4051.08

Ouro improves over Standard, while TaH2 retains the highest mean accuracy among the three models.

[1] Rui-Jie Zhu et al. Scaling Latent Reasoning via Looped Language Models. arXiv preprint arXiv:2510.25741, 2025.

W4.3, Q4: Compare 4B and 8B models under matched decoding FLOPs

They also compare against Standard at a fixed 32K token limit rather than matched decoding FLOPs.

In Table 3 (4B and 8B), what are the FLOPs/token ratios for TaH2 (M=2)? How does TaH2 compare against Standard at matched decoding FLOPs rather than at a fixed 32K token limit?

Across the nine benchmarks used in Table 3, TaH2 (M=2) has 1.37× (4B) and 1.35× (8B) decoding FLOPs/token relative to Standard, using total decoding FLOPs / total generated tokens.

We extend the output budget to 64K to probe further test-time scaling. The table reports AIME24–26 mean accuracy (avg@32), with TaH2 matched to the decoding FLOPs at Standard's peak accuracy:

ScaleModel / pointAcc. (%)Decode TFLOPs / responseAvg. tokens
4BStandard (peak)46.1128521,287
4BTaH2 (matched FLOPs)50.6128517,525
4BTaH2 (peak)51.4238120,293
8BStandard (peak)59.7945919,282
8BTaH2 (matched FLOPs)62.8045915,716
8BTaH2 (peak)63.7556018,040

TaH2 improves accuracy at matched decoding FLOPs by 4.50 points (4B) and 3.01 points (8B).

Reviewer_gTCCOpenReview ↗

Official Response from Reviewer gTCC

I thank the authors for the thorough rebuttal, the extensive new experiments, and for already incorporating these updates into the revised manuscript (Appendices B.3–B.7 and Section 5.4 5.5). Your new results directly resolve the majority of my technical concerns. Two minor follow-up points remain for clarity in the final revision:

  1. The Force response table confirms that TaH2 retains its +53.4% slope advantage over Standard (2.47 vs. 1.61). Did you also evaluate Ouro (M=2) and TaH2-fixed (M=2) under Force response? In Section 3.2 / Figure 1a, part of the motivating puzzle was that fixed/existing looped models had steeper slopes than Standard despite lower accuracy across 4K–16K; knowing whether that holds under budget forcing would clarify Section 3.2.

Thanks for raising your score and for your valuable feedback! We have added results for three looped baselines (Ouro, Huginn and TaH2-fixed) under the same protocol. Increases are relative to Standard:

ModelTruncated: slopeIncreaseForce response: slopeIncrease
Standard1.79—1.61—
Ouro (M=2)2.12+18.6%1.95+21.4%
Huginn (M=3)2.26+26.2%1.95+21.1%
TaH2-fixed (M=2)2.42+35.5%2.20+36.8%
TaH2 (M=2)2.74+53.1%2.47+53.4%

All looped models retain steeper slopes than Standard under Force response, though Ouro, Huginn and TaH2-fixed still largely fall below Standard at matched compute. We also include test-time scaling curves for Truncated and Force response in Appendix B.3, Figure 8, for a direct visual comparison.

Reviewer_6f8GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose the TaH2, which makes the model to focus extra iterations on the tokens that benefit from looping. More specifically, it jointly post-trains the backbone and an iteration decider through lookahead depth supervision, which uses online labels indicating whether further iteration improves the prediction. The experiments show that the proposed TaH2 improves both the efficiency and attainable accuracy of test-time scaling.

Critical strengths
  1. This paper is well-motivated. The focused research promblem is relative clear.
  2. The proposed method alleviates the computational waste caused by the fixed depth.
  3. The paper conducted relatively extensive experiments, and the paper also report the actual service latency.
Critical weaknesses
  1. TaH2 introduces two modules: a learned updater and an adaptive depth prediction mechanism. However, it does not appear to provide a specific analysis of which component contributes the majority of the performance gains.

  2. The general logic behind mining TaH2 tags involves checking whether the predicted probability of a tag improves after an additional round of computation. However, an improvement in the overall tag probability does not necessarily guarantee that the final answer is correct. Could we empirically demonstrate that using token loss to decide whether to continue or stop computation actually leads to better overall results, rather than merely improving local predictions?

  3. Is TaH2 limited to the current SFT (Supervised Fine-Tuning) paradigm, or does it have the potential to be extended to RL and on policy distillation?

Questions for authors

Please see the Critical Weeknesses.

3 comments

We sincerely thank Reviewer 6f8G for the positive assessment. We appreciate your recognition of our clear motivation, improved efficiency, and extensive evaluation. We address your concerns below.

W1: Contributions of each component: updater and adaptive depth

TaH2 introduces two modules: a learned updater and an adaptive depth prediction mechanism. However, it does not appear to provide a specific analysis of which component contributes the majority of the performance gains.

We added two ablations: replacing adaptive depth with fixed depth, and replacing the learned updater with direct hidden-state feedback.

MethodUpdaterDepthMean depthAIME24 (%)AIME25 (%)AIME26 (%)Mean (%)
Standard—Fixed1.0011.0413.0211.8811.98
TaH2LearnedAdaptive1.2315.7316.4614.2715.49
TaH2LearnedFixed2.0014.4814.4814.1714.38
TaH2Last hidden state*Adaptive1.2315.4215.0014.7915.07

Both ablations reduce accuracy: −1.11 percentage points without adaptive depth, versus −0.42 without the learned updater. The adaptive configuration contributes more to accuracy in this setting, while also improving efficiency by reducing mean depth from 2.00 to 1.23.

* Last hidden state means directly passing the hidden state to the next iteration, as in Ouro [1]. All TaH2 methods use duo attention; all other settings match Table 1 in the paper.

[1] Rui-Jie Zhu et al. Scaling Latent Reasoning via Looped Language Models. arXiv preprint arXiv:2510.25741, 2025.

W2: Will reduced next-token prediction loss translate into higher final task accuracy?

The general logic behind mining TaH2 tags involves checking whether the predicted probability of a tag improves after an additional round of computation. However, an improvement in the overall tag probability does not necessarily guarantee that the final answer is correct. Could we empirically demonstrate that using token loss to decide whether to continue or stop computation actually leads to better overall results, rather than merely improving local predictions?

TaH2 reduces training loss and held-out validation loss, and improves downstream task accuracy:

Model (1.7B)Training loss ↓Validation loss ↓Mean accuracy, 10 benchmarks (%) ↑
Standard0.29150.339337.7
TaH2 (M=2)0.28600.332940.6
TaH2 (M=4)0.28700.331440.9
TaH2 (M=8)0.27970.328742.5

In this table, training loss refers to the final-step next-token prediction loss; validation and benchmark results follow Fig. 4(a) and Table 1. We agree that better next-token prediction does not guarantee a correct final answer. The downstream gains support the effectiveness of token-loss-based supervision.

W3: Extension to RL and on-policy distillation

Is TaH2 limited to the current SFT (Supervised Fine-Tuning) paradigm, or does it have the potential to be extended to RL and on policy distillation?

We extend TaH2 to OPD and validate its effectiveness, with details in Section 5.5 and Appendix E. TaH2 has two types of sampling nodes: output tokens and stop/continue decisions. Starting from expected student–teacher reverse KL, we derive the following immediate-reward surrogate:

LOPD=−∑t[∑vsg⁡[qˉt(v)]log⁡qˉt(v),rt(v)+∑m=1min⁡(mt,M−1)log⁡πϕ(bt(m)∣Ht(m)),At(m)].\mathcal L_{\mathrm{OPD}}=-\sum_t\left[\sum_v\operatorname{sg}[\bar q_t(v)]\log\bar q_t(v)\\,r_t(v)+\sum_{m=1}^{\min(m_t,M-1)}\log\pi_\phi(b_t^{(m)}\mid H_t^{(m)})\\,A_t^{(m)}\right].

The first term gives the full-vocabulary token update, and the second gives the decider update. Here qˉt\bar q_t is the student output distribution, rt(v)=sg⁡[log⁡pT(v∣st)−log⁡qˉt(v)]r_t(v)=\operatorname{sg}[\log p_T(v\mid s_t)-\log\bar q_t(v)], and At(m)A_t^{(m)} is the detached routing advantage defined in Appendix E. The operator sg⁡\operatorname{sg} stops gradients. Domain scores are averaged across the corresponding benchmarks; Avg. @ 9 tasks is the reported average across all nine tasks (Table 5).

ModelMethodMath (%)QA (%)Code (%)Avg. (%, 9 tasks)
StandardSFT35.550.563.745.1
TaH2SFT38.251.265.047.1
StandardSFT + OPD38.451.765.247.3
TaH2SFT + OPD41.352.069.249.8

With Qwen3-1.7B students and a Qwen3-4B teacher, OPD improves TaH2 from 47.1% to 49.8%, exceeding Standard + OPD by 2.5 points. General RL for TaH2 can be formulated the same way, with larger-scale validation left for future work.

Dear Reviewer,

Thank you again for your feedback. We have added component ablations, provided downstream evidence supporting token-loss-based supervision, and extended TaH2 to support on-policy distillation. As the discussion period draws to a close, we would like to ask whether you have any remaining questions or concerns. We would be happy to clarify further.

Thank you for the author's response. The author's response partially addresses my concerns, and I have decided to maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.