- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how recursive computation interacts with token commitment in masked diffusion models. It repeatedly applies a shared Transformer block within each denoising step and evaluates recursive depth as a parameter-efficient scaling axis. An existing dependence–fidelity decomposition is used to distinguish improvements in model marginals from the dependence cost of parallel commitment. Controlled Sudoku experiments measure fidelity improvements and their saturation. The paper then combines entropy-based loop stopping with entropy-budgeted unmasking and evaluates decoding policies on Sudoku, Countdown, RNA infilling, and executable mathematical program generation.
Critical strengths
-
The controlled experiments provide a useful quantitative account of recursive refinement. On enumerable multi-solution Sudoku instances, the paper measures a 6–43-fold reduction in fidelity error from one to five loops, followed by saturation. Comparison with a sampler using exact marginals helps distinguish inaccurate marginal predictions from errors caused by independent parallel sampling. These measured improvements do not follow automatically from the decomposition and constitute useful empirical evidence.
-
The empirical evaluation is substantial within structured generation. The paper studies parameter efficiency, effective depth, compute allocation, supervision, and training-depth schedules. The decoding experiments include the four combinations of fixed/adaptive looping and commitment, as well as alternative stopping and commitment rules. In particular, the Sudoku and Countdown results provide positive evidence that adaptive loop stopping can avoid unnecessary computation at large loop budgets.
Critical weaknesses
The analysis provides limited additional understanding of what is specific to recursive computation. Section 3.2 provides positive evidence that recursion can outperform untied depth scaling at matched effective depth/FLOPs: on Sudoku at , the model achieves 93.6% VPR versus 77.8% for the 30-layer model. However, the interpretation that iterative refinement before commitment offers benefits beyond depth does not explain the source of this advantage. A deeper nonrecursive model also transforms hidden representations before committing tokens. The training ablations in Appendix D are useful, but they study choices within recursive models and do not establish how much of the advantage comes from weight sharing itself versus accompanying choices such as supervision at every loop.
The dependence–fidelity analysis clarifies a different distinction: improving marginal predictions at a fixed state and commit set versus changing the commitment schedule. This decomposition applies to marginal predictors generally; a deeper nonrecursive predictor could also reduce the fidelity term. Consequently, it does not explain why shared-weight recurrence should be preferable to untied depth. Appendix A.8 appropriately states that the analysis neither guarantees fidelity improvement with additional loops nor identifies an optimal allocation between loops and denoising. My concern is therefore about the extent of the understanding gained about recursion, rather than the correctness of the decomposition.
The most distinctive analytical contribution is the controlled quantification of fidelity improvement, saturation, and residual dependence errors on enumerable Sudoku instances. This is useful evidence, but the paper establishes less about what makes recursion particularly effective or how those findings predict its benefits in other settings. The additional task results demonstrate practical gains without measuring the two error components there. Given that LoopMDM already studies within-step recurrence and adaptive loop depth, and that entropy-budgeted commitment builds on prior sampling work, the proposed combination of entropy-gain stopping and adaptive commit counts is a modest methodological extension. Taken together, the experiments are informative, but I remain unconvinced that the additional insight into recursive computation is sufficiently substantial.
Questions for authors
Can the authors provide further insight into why recursive models outperform deeper nonrecursive models at matched effective depth and comparable compute in Section 3.2? Section 4 distinguishes marginal refinement from token commitment, but does not appear to explain why repeatedly applying shared weights is advantageous over stacking untied layers. Is there analysis or evidence that sheds light on the source of this advantage?
3 comments
Answer to reviewer (part 2/2)
4. Novelty. LoopMDM adapts loops at a fixed number of steps, and entropy-budgeted samplers adapt commits without loops. Our contribution is adapting both jointly, which matters in practice:
- Commit-only adaptation is not enough at large L. On Countdown-4, the commit-adaptive decoder's cost rises from 37 to 98 NFE between L=3 and L=7, while our joint decoder stays near 40.
- Joint adaptation reaches the same quality far more cheaply. On Sudoku it reaches near-perfect VPR at single-digit NFE, versus ~L×T for fixed schedules.
- Adaptive commits drive savings on long sequences. On TinyGSM, the entropy commit rule matches the best fixed decoder at 69 instead of 768 NFE.
- Our stopping rule is cheaper. Our entropy-gain rule matches LoopMDM's hidden-state rule at lower NFE (~60 vs. ~110 on Countdown-5, App. G.6).
In summary, the paper's insight concerns how recursion interacts with decoding: recursion reduces fidelity error but cannot reduce dependence error. Its benefit is therefore largest where fidelity dominates (unique constraint chains, few denoising steps) and reduces where dependence dominates. An efficient decoder must adapt both axes, since commit adaptation alone fails at large and loop adaptation alone misses the savings on long sequences.
We thank the reviewer again and are happy to answer further questions.
References
- Yang et al. (2024). Looped Transformers are Better at Learning Learning Algorithms. ICLR 2024. arXiv:2311.12424
- Saunshi et al. (2025). Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025. arXiv:2502.17416
- Geiping et al. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv preprint (2025). arXiv:2502.05171
- Zhu et al. (2025). Scaling Latent Reasoning via Looped Language Models. arXiv preprint (2025). arXiv:2510.25741
Thank you for clarifying the scope and providing the additional ablations and Table 5. I understand that the paper focuses on the interaction between recursive refinement and token commitment, rather than a general explanation of why weight sharing outperforms untied depth.
The additional results provide useful evidence that the advantage on uniquely constrained Sudoku instances persists under final-only supervision. I also acknowledge the reported efficiency benefits of adaptive looping, particularly at large loop budgets.
My main concern nevertheless remains the magnitude of the contribution beyond existing work. The exact characterization of fidelity and dependence errors is informative, but its direct empirical validation is confined to Sudoku. The entropy analyses on Countdown and the performance results on RNA and TinyGSM provide supporting evidence of practical utility, without directly establishing that the same error-based explanation accounts for the gains in those settings.
Given the existing work on recursive MDMs, decoding-error decomposition, and entropy-based commitment, I view the contribution primarily as a more detailed empirical characterization and an incremental extension through joint adaptive decoding. The rebuttal strengthens the empirical support and addresses some concerns about the interpretation of the results, but I remain unconvinced that the additional insight and methodological advance are sufficiently substantial to change my assessment.
Answer to reviewer (part 1/2)
We thank the reviewer. First, we would like to clarify the scope of our paper. Our work studies recursion as a new scaling axis for MDMs, particularly at decoding time: loops reduce fidelity error, denoising steps control dependence error (Section 4), and a decoder should allocate compute accordingly (Section 5). The iso-FLOP comparison (Section 3) is aimed at establishing recursion as a competitive way to spend compute in this setting. Explaining why weight sharing outperforms untied depth is not our main goal, and it remains an open research question and an active area of research for looped transformers in general (Yang et al., 2024; Saunshi et al., 2025; Geiping et al., 2025; Zhu et al., 2025). Below, we address the reviewer’s question as far as our experiments allow.
1. Is the advantage due to weight sharing or due to our training choices?
Besides weight sharing, our recursive and untied models differ in two training choices. Our default recursive model is supervised at every loop, whereas the untied models are supervised only at their final layer. The recursive model also adds a step embedding indicating the current loop. All other settings are identical: width, data, and number of gradient updates. However, neither of the two design choices explains the advantage since:
Therefore, the observed advantage of the recursive model over the untied baseline is not explained by these training choices.
2. What does recursion enable that untied depth cannot?
(a) Depth chosen at inference: because every loop applies the same operator, depth becomes a decoding-time variable: for each input, the same weights can run more or fewer loops at each denoising step than the number of loops used at train time. An untied network's depth is fixed at training; it can stop early only with added exit heads, and cannot refine the hidden representations beyond its trained depth. Our results rely on this property: at T=1, running more loops than at training improves all recursive models (App. G.3), and adaptive loop stopping saves compute where fixed depth wastes it (Countdown-4 at L=7: ~40 vs. 98 NFE, with higher RTR, 82% vs. 72%; Sec. 5).
(b) Why does recursion beat untied depth in our experiments? We match compute by effective depth and number of gradient updates, so training and inference FLOPs are equal. Recursion wins in this regime, but we do not claim it would in every regime: untied models may benefit from more training or data. For example, our 30-layer model underperforms the 24-layer one at equal updates. We have revised Section 3.2 to state this scope.
Within this regime, we can identify where the advantage arises. We added an additional experiment where we measure Dfid (T=8, 50 boards per group) with the setup of Section 4.2 (detailed in App. E, where these new results have been added too):
On unique boards, where cell values are determined by chains of constraints, recursion lowers Dfid 3–4 times. With final-only supervision, the same supervision as the untied baselines, it still lowers it 2.5–3×, so this gain does not come from loop supervision only. With multiple completions, recursion gives no advantage. Similarly, in Section 3.2 we observe that the VPR gap is large at T=5 (93.6% vs. 77.8%) and vanishes at T=40 (98.0% vs. 97.2%). This localizes the advantage to resolving constraint chains within a single step, but the question of why shared weights do this better remains open, as noted above.
3. Generality: exact D_dep/D_fid requires an enumerable p∗, which only Sudoku provides. Elsewhere, we report two proxies: the token entropy which falls with loops and predicts errors on Sudoku and Countdown-3/4/5 (AUROC 0.86 / 0.84 / 0.91 / 0.93, App. G.9); and the parallel-clash rate (Section 4.2, Remark A.9).