- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the adaptive update of looped transformers. To address the limitation of fixed update each step,this paper proposes TAPS to determine the update with persistent-progress and centered-fluctuation. Extensive experiments shows TAPS improves the accuracy and efficiency on Sudoku and Maze.
Critical strengths
- This paper treats update magnitute as an independent variable for recurrent inference, which fills the gap of the research of architecture and loop count.
- TAPS can be appiled to pretrained models without training.
- Theoretical analysis provides the necessary condition for TAPS.
- The experimental results include Sudoku and Maze on multiple variants of recurrent language models. The comparisons of different controller variants and inference methods are also conducted
Critical weaknesses
- Update consistency does really reflect task progress. TAPS may amplify the error if the model keeps on updating on wrong direction. There is a gap that could be filled with an experimental study of proxy alignment.
- Why looped transformers? It seems like this method can be applied to traditional transformers. Is there any specialty on looped transformers with TAPS?
- There is a lack of significance analysis, such as Maze-Hard in table 1 where the improvement on accuracy is tiny.
Questions for authors
See Weaknesses.
5 comments
Rebuttal (continued1)
2. Why Looped Transformers?
Thanks for this insightful question. We focus on looped transformers because their recurrent structure makes adaptive update-scale control particularly natural. Specifically, loop transformers repeatedly apply a shared operator to an evolving hidden state: This shared operator produces a sequence of recurrent updates, allowing TAPS to track their persistence and fluctuations and adapt their magnitudes accordingly: In contrast, standard transformers typically apply different operators across layers and do not exhibit the same shared-operator recurrent dynamics. Moreover, looped transformers have emerged as an important direction for scaling test-time reasoning by reusing model parameters across recurrent steps. Improving their inference efficiency is therefore a timely problem for the community.
We also appreciate your observation that TAPS can extend beyond models with native recurrence. Indeed, as shown in Table 4, we have already applied TAPS to a frozen Qwen3-4B-Instruct model with intermediate-layer recurrencem and results demonstrate the applicability of TAPS beyond models originally designed with native recurrence. Since this observation suggests that TAPS can have a broader impact, we will highlight this broader applicability in the revised manuscript.
Rebuttal (continued2)
3. Significance of the Imrovement Brought by TAPS
Thank you for raising this concern. To demonstrate that the improvement brought by TAPS is significant, we first provide significance tests on the evaluation metrics including Acc. and Speedup. Then we show in which situations TAPS can bring greater advantages.
(1) Significance tests on Acc. and Speedup
To show the improvement is statistically significant, we add significance tests on Maze-Hard using 1,000 paired puzzles at , where Accuracy p-values are computed by one-sided paired McNemar tests and Timing p-values are based on seven CUDA timing repeats.
| Method | Acc. (%) | Acc. p-value | GPU Speedup | Timing p-value |
|---|---|---|---|---|
| Standard | 78.8 | — | 1.000× | — |
| TAPS (GD) | 79.8 | 0.055 | 1.518× | <0.001 |
| TAPS (P/S-Sign) | 79.9 | 0.068 | 1.566× | <0.001 |
| TAPS (Momentum) | 79.8 | 0.055 | 1.517× | <0.001 |
| TAPS (RMSProp) | 79.7 | 0.121 | 1.518× | <0.001 |
| TAPS (Adam) | 79.8 | 0.097 | 1.498× | <0.001 |
| TAPS (BB) | 79.6 | 0.128 | 1.476× | <0.001 |
Generally speaking, the accuracy improvements of most variants of TAPS are significant under the nominal level 10%. Most importantly, All TAPS variants achieve substantially larger gains in inference efficiency (1.46–1.56× GPU speedups) and these improvements are all significant (p-value <0.001). These results are sufficient to support the advantage of TAPS, since our primary objective is to improve the accuracy–compute trade-off of recurrent inference.
(2) When TAPS would have more significant advantages
To reveal the greater potential of TAPS in several important situations, we add experiments of TAPS (RMSProp) on 1,000 Sudoku problems. Briefly speaking, we find that TAPS achieves more gains when we have more loop budgets or tasks become more challenging.
- TAPS delivers larger efficiency gains with more recurrent loops. We extend the evaluation in Table 1 to larger loop budgets under the same setup. TAPS consistently improves accuracy under the same budgets, and its GPU-time speedup at matched accuracy increases from 1.15× to 1.34× as the loop budget grows from 32 to 96. Such observed trend indicates that the performance improvement may grow given larger loop budgets.
| Budget | Accuracy (%) (Base → TAPS) | Matched-Acc. Loops (Base → TAPS) | GPU Speedup |
|---|---|---|---|
| 32 | 90.4 → 91.1 | 32 → 27 | 1.15× |
| 48 | 91.7 → 92.4 | 48 → 37 | 1.26× |
| 64 | 92.6 → 93.2 | 64 → 51 | 1.22× |
| 96 | 93.6 → 94.5 | 96 → 70 | 1.34× |
- TAPS achieves more speedups when tasks are more challenging. We further evaluate inference efficiency at fixed accuracy targets. At 95% accuracy, TAPS achieves 1.96× and 1.80× GPU-time speedups on easy and medium Sudoku problems. On hard problems, TAPS reaches 90% accuracy in 400 loops, compared to 565 loops for the baseline, yielding a 1.38× speedup. These results show that the benefits of TAPS become more pronounced at higher accuracy targets and on more challenging problems.
| Target Acc. | Difficulty | Loops (Base → TAPS) | GPU Speedup |
|---|---|---|---|
| 90% | Easy | 3 → 3 | 0.98× |
| 90% | Medium | 33 → 30 | 1.08× |
| 90% | Hard | 565 → 400 | 1.38× |
| 95% | Easy | 12 → 6 | 1.96× |
| 95% | Medium | 350 → 190 | 1.80× |
| 95% | Hard | Not reached | — |
Note: Difficulty is defined by the number of empty cells: Easy (46–54), Medium (55–58), and Hard (59–64).
The above results together demonstrate that TAPS can indeed achieve better accuracy-efficiency tradeoff, where the improvements are statistically significant. These improvements are even more pronounced in several important scenarios.
Rebuttal
We sincerely thank the reviewer for the helpful comments and for recognizing
Clear and easy to follow,fills the gap of the research of architecture and loop count,be appiled to pretrained models without training,Theoretical analysis provides the necessary condition for TAPS,The inclusion of ... further broadens the empirical evaluation. Our point-by-point responses are as follows.1. Validation of the Progress–Fluctuation Principle
Thank you for this insightful suggestion. We address this concern from three complementary perspectives: (1) empirically examining the alignment between the progress–fluctuation score and realized task progress; (2) demonstrating the practical utility of trajectory stability for adaptive inference; and (3) clarifying why correcting persistently misdirected updates requires additional task-dependent information beyond step-size control.
(1) Empirical alignment with realized task progress To examine whether the progress–fluctuation score reflects actual task progress, we evaluate TAPS on 1,000 Sudoku puzzles. For each puzzle i and recurrent step h, we measure the realized one-step loss reduction as Gi,h=Li,h−1−Li,h, where positive values indicate progress. We then compute the Pearson correlation between Bi,h and Gi,h over the corresponding puzzle–step observations. The results show strong positive correlations:
Therefore, the progress–fluctuation score can be strongly associated with realized task progress across recurrent steps.
(2) Practical utility of trajectory stability We have also demonstrated the practical value of trajectory-based inference control in the original manuscript. Our TAPS-based adaptive-exit rule stops only when predictions remain unchanged and step-size-adjusted residual activity stays below a threshold for consecutive loops (Appendix C.3). As shown in Fig. 2, it allocates more computation to harder Sudoku instances while maintaining accuracy comparable to the unit-step baseline across difficulty levels. Notably, TAPS achieves 91.2% accuracy with only 316.5 effective updates, compared with 91.1% using 1,497 updates for FPRM, yielding a 4.7× reduction in computation. This demonstrates the usefulness of observable trajectory signals for adaptive recurrent inference.
(3) Why correcting update direction is not feasible from the information perspective? From an information perspective, determining whether an update is task-improving requires task-dependent information, such as the ground-truth target, the corresponding loss gradient, or external verifier feedback. Such information is unavailable during standard inference. As discussed in Appendix B.3.1, this is precisely why our theory distinguishes task-dependent oracle quantities from observable proxies constructed from recurrent updates.
TAPS instead builds on the premise that the balance between persistent motion and fluctuations provides a useful signal for regulating the magnitude of learned updates, without requiring every persistent update to be beneficial. The effectiveness of this principle is supported by our extensive experiments and the additional proxy-alignment analysis above. Moreover, Theorems 3–5 characterize sufficient conditions under which TAPS identifies loss-improving adjustment directions, reduces expected terminal loss, and reaches target performance with fewer recurrent steps. In the revised manuscript, we will clarify this distinction, together with the motivation and scope of TAPS.