acceptodds
Under review as a conference paper at ICLR 2027

Recursive Scaling in Masked Diffusion Models

Abstract

Masked diffusion models (MDMs) generate sequences by iteratively refining a partially masked state and committing tokens in parallel. We introduce recursion in MDMs and propose new Recursive Masked Diffusion Models (R-MDMs), which apply a shared denoising transformer times within each denoising step, adding recursive depth as an additional compute axis without increasing parameter count. Across structured generation tasks, recursive depth improves quality at fixed parameter budget, matches substantially larger non-recursive models at matched FLOPs, and can reduce the number of denoising steps needed to reach a target quality. We interpret these gains with a dependence–fidelity decomposition of parallel decoding error: recursion refines model marginals at a fixed masked state, whereas denoising steps change that state by committing tokens. Building on this analysis, we propose to treat decoding as a two-axis decision (how many loops to run and which tokens to commit) and show that entropy-guided adaptive rules improve the quality–compute frontier over fixed schedules, transferring across various tasks on Sudoku, Countdown, RNA, and executable math generation. Together, these results establish recursive depth as a practical, complementary test-time scaling mechanism for MDMs.

Primary area:
generative models
Keywords:
discrete diffusion, recursion, looped models, decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RhzzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how recursive computation interacts with token commitment in masked diffusion models. It repeatedly applies a shared Transformer block within each denoising step and evaluates recursive depth as a parameter-efficient scaling axis. An existing dependence–fidelity decomposition is used to distinguish improvements in model marginals from the dependence cost of parallel commitment. Controlled Sudoku experiments measure fidelity improvements and their saturation. The paper then combines entropy-based loop stopping with entropy-budgeted unmasking and evaluates decoding policies on Sudoku, Countdown, RNA infilling, and executable mathematical program generation.

Critical strengths
  1. The controlled experiments provide a useful quantitative account of recursive refinement. On enumerable multi-solution Sudoku instances, the paper measures a 6–43-fold reduction in fidelity error from one to five loops, followed by saturation. Comparison with a sampler using exact marginals helps distinguish inaccurate marginal predictions from errors caused by independent parallel sampling. These measured improvements do not follow automatically from the decomposition and constitute useful empirical evidence.

  2. The empirical evaluation is substantial within structured generation. The paper studies parameter efficiency, effective depth, compute allocation, supervision, and training-depth schedules. The decoding experiments include the four combinations of fixed/adaptive looping and commitment, as well as alternative stopping and commitment rules. In particular, the Sudoku and Countdown results provide positive evidence that adaptive loop stopping can avoid unnecessary computation at large loop budgets.

Critical weaknesses

The analysis provides limited additional understanding of what is specific to recursive computation. Section 3.2 provides positive evidence that recursion can outperform untied depth scaling at matched effective depth/FLOPs: on Sudoku at T=5T=5, the 6⊗56\otimes5 model achieves 93.6% VPR versus 77.8% for the 30-layer model. However, the interpretation that iterative refinement before commitment offers benefits beyond depth does not explain the source of this advantage. A deeper nonrecursive model also transforms hidden representations before committing tokens. The training ablations in Appendix D are useful, but they study choices within recursive models and do not establish how much of the advantage comes from weight sharing itself versus accompanying choices such as supervision at every loop.

The dependence–fidelity analysis clarifies a different distinction: improving marginal predictions at a fixed state and commit set versus changing the commitment schedule. This decomposition applies to marginal predictors generally; a deeper nonrecursive predictor could also reduce the fidelity term. Consequently, it does not explain why shared-weight recurrence should be preferable to untied depth. Appendix A.8 appropriately states that the analysis neither guarantees fidelity improvement with additional loops nor identifies an optimal allocation between loops and denoising. My concern is therefore about the extent of the understanding gained about recursion, rather than the correctness of the decomposition.

The most distinctive analytical contribution is the controlled quantification of fidelity improvement, saturation, and residual dependence errors on enumerable Sudoku instances. This is useful evidence, but the paper establishes less about what makes recursion particularly effective or how those findings predict its benefits in other settings. The additional task results demonstrate practical gains without measuring the two error components there. Given that LoopMDM already studies within-step recurrence and adaptive loop depth, and that entropy-budgeted commitment builds on prior sampling work, the proposed combination of entropy-gain stopping and adaptive commit counts is a modest methodological extension. Taken together, the experiments are informative, but I remain unconvinced that the additional insight into recursive computation is sufficiently substantial.

Questions for authors

Can the authors provide further insight into why recursive models outperform deeper nonrecursive models at matched effective depth and comparable compute in Section 3.2? Section 4 distinguishes marginal refinement from token commitment, but does not appear to explain why repeatedly applying shared weights is advantageous over stacking untied layers. Is there analysis or evidence that sheds light on the source of this advantage?

3 comments
AuthorsOpenReview ↗

Answer to reviewer (part 1/2)

We thank the reviewer. First, we would like to clarify the scope of our paper. Our work studies recursion as a new scaling axis for MDMs, particularly at decoding time: loops reduce fidelity error, denoising steps control dependence error (Section 4), and a decoder should allocate compute accordingly (Section 5). The iso-FLOP comparison (Section 3) is aimed at establishing recursion as a competitive way to spend compute in this setting. Explaining why weight sharing outperforms untied depth is not our main goal, and it remains an open research question and an active area of research for looped transformers in general (Yang et al., 2024; Saunshi et al., 2025; Geiping et al., 2025; Zhu et al., 2025). Below, we address the reviewer’s question as far as our experiments allow.

1. Is the advantage due to weight sharing or due to our training choices?

Besides weight sharing, our recursive and untied models differ in two training choices. Our default recursive model is supervised at every loop, whereas the untied models are supervised only at their final layer. The recursive model also adds a step embedding indicating the current loop. All other settings are identical: width, data, and number of gradient updates. However, neither of the two design choices explains the advantage since:

  • Supervision: a 6⊗56\otimes5 model supervised only at the final loop, like the untied baseline, still reaches ~91% VPR at T=5T=5 (Fig. 11), against 82.8% for the strongest untied model (24⊗124\otimes1) and 77.8% for the 30⊗130\otimes1 model (Table 9).
  • Step embedding: a 6⊗56\otimes5 model without a step embedding reaches ~95% VPR at T=5T=5 (Fig. 10, App. D.3), on par with the learned embedding and well above the untied 24⊗124\otimes1 (82.8%) and 30⊗130\otimes1 (77.8%) models (Table 9).

Therefore, the observed advantage of the recursive model over the untied baseline is not explained by these training choices.

2. What does recursion enable that untied depth cannot?

(a) Depth chosen at inference: because every loop applies the same operator, depth becomes a decoding-time variable: for each input, the same weights can run more or fewer loops at each denoising step than the number of loops used at train time. An untied network's depth is fixed at training; it can stop early only with added exit heads, and cannot refine the hidden representations beyond its trained depth. Our results rely on this property: at T=1T=1, running more loops than at training improves all recursive models (App. G.3), and adaptive loop stopping saves compute where fixed depth wastes it (Countdown-4 at L=7L=7: ~40 vs. 98 NFE, with higher RTR, 82% vs. 72%; Sec. 5).

(b) Why does recursion beat untied depth in our experiments? We match compute by effective depth and number of gradient updates, so training and inference FLOPs are equal. Recursion wins in this regime, but we do not claim it would in every regime: untied models may benefit from more training or data. For example, our 30-layer model underperforms the 24-layer one at equal updates. We have revised Section 3.2 to state this scope.

Within this regime, we can identify where the advantage arises. We added an additional experiment where we measure DfidD_{\mathrm{fid}} (T=8T=8, 50 boards per group) with the setup of Section 4.2 (detailed in App. E, where these new results have been added too):

Solutions per board6⊗56\otimes5 (ALL-supervision)6⊗56\otimes5 (FINAL-supervision)24-layer30-layer
1 (unique)0.0540.0640.2030.163
2–50.1880.2080.2030.177
6–500.1930.2420.2160.201
51–5000.2370.2830.2190.216

On unique boards, where cell values are determined by chains of constraints, recursion lowers DfidD_{\mathrm{fid}} 3–4 times. With final-only supervision, the same supervision as the untied baselines, it still lowers it 2.5–3×, so this gain does not come from loop supervision only. With multiple completions, recursion gives no advantage. Similarly, in Section 3.2 we observe that the VPR gap is large at T=5T=5 (93.6% vs. 77.8%) and vanishes at T=40T=40 (98.0% vs. 97.2%). This localizes the advantage to resolving constraint chains within a single step, but the question of why shared weights do this better remains open, as noted above.

3. Generality: exact D_dep/D_fid requires an enumerable p∗p ^*, which only Sudoku provides. Elsewhere, we report two proxies: the token entropy which falls with loops and predicts errors on Sudoku and Countdown-3/4/5 (AUROC 0.86 / 0.84 / 0.91 / 0.93, App. G.9); and the parallel-clash rate (Section 4.2, Remark A.9).

AuthorsOpenReview ↗

Answer to reviewer (part 2/2)

4. Novelty. LoopMDM adapts loops at a fixed number of steps, and entropy-budgeted samplers adapt commits without loops. Our contribution is adapting both jointly, which matters in practice:

  • Commit-only adaptation is not enough at large L. On Countdown-4, the commit-adaptive decoder's cost rises from 37 to 98 NFE between L=3 and L=7, while our joint decoder stays near 40.
  • Joint adaptation reaches the same quality far more cheaply. On Sudoku it reaches near-perfect VPR at single-digit NFE, versus ~L×T for fixed schedules.
  • Adaptive commits drive savings on long sequences. On TinyGSM, the entropy commit rule matches the best fixed decoder at 69 instead of 768 NFE.
  • Our stopping rule is cheaper. Our entropy-gain rule matches LoopMDM's hidden-state rule at lower NFE (~60 vs. ~110 on Countdown-5, App. G.6).

In summary, the paper's insight concerns how recursion interacts with decoding: recursion reduces fidelity error but cannot reduce dependence error. Its benefit is therefore largest where fidelity dominates (unique constraint chains, few denoising steps) and reduces where dependence dominates. An efficient decoder must adapt both axes, since commit adaptation alone fails at large LL and loop adaptation alone misses the savings on long sequences.

We thank the reviewer again and are happy to answer further questions.

References

  • Yang et al. (2024). Looped Transformers are Better at Learning Learning Algorithms. ICLR 2024. arXiv:2311.12424
  • Saunshi et al. (2025). Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025. arXiv:2502.17416
  • Geiping et al. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv preprint (2025). arXiv:2502.05171
  • Zhu et al. (2025). Scaling Latent Reasoning via Looped Language Models. arXiv preprint (2025). arXiv:2510.25741

Thank you for clarifying the scope and providing the additional ablations and Table 5. I understand that the paper focuses on the interaction between recursive refinement and token commitment, rather than a general explanation of why weight sharing outperforms untied depth.

The additional results provide useful evidence that the advantage on uniquely constrained Sudoku instances persists under final-only supervision. I also acknowledge the reported efficiency benefits of adaptive looping, particularly at large loop budgets.

My main concern nevertheless remains the magnitude of the contribution beyond existing work. The exact characterization of fidelity and dependence errors is informative, but its direct empirical validation is confined to Sudoku. The entropy analyses on Countdown and the performance results on RNA and TinyGSM provide supporting evidence of practical utility, without directly establishing that the same error-based explanation accounts for the gains in those settings.

Given the existing work on recursive MDMs, decoding-error decomposition, and entropy-based commitment, I view the contribution primarily as a more detailed empirical characterization and an incremental extension through joint adaptive decoding. The rebuttal strengthens the empirical support and addresses some concerns about the interpretation of the results, but I remain unconvinced that the additional insight and methodological advance are sufficiently substantial to change my assessment.

Reviewer_iEWXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies recursive scaling of MDM. It introduces looped computations into MDM and studies how MDM improves with increasing depth. Finally, the author proposes a decoder that jointly adapts loop computation and token filling with entropy analysis.

Critical strengths
  1. The paper is easy to follow, and the idea is clearly presented.
  2. The paper analyses how recursive scaling improves marginal predictions and reduces dependency errors.
  3. The paper studies parameter-matched and effective-depth-matched results.
Critical weaknesses
  1. Limited novelty in Looped transformers. The architecture and training recipe design are common strategies.
  2. Limited experiments. The model parameters and tasks in the paper are small-scale. More experiments on standard LM benchmarks are crucial for the paper. The effectiveness of the method and the scaling is not well-supported.
  3. The entropy-based decoder also combines common entropy-based criteria into deciding loop computation or token commitment.
Questions for authors
  1. Can the author provide more results on the standard LM benchmark?
  2. Can the reviewer distinguish the method in the paper from standard Looped transformers?
4 comments
AuthorsOpenReview ↗

Answer to reviewer

We thank the reviewer for their feedback. Below, we address their concerns and questions, with relevant changes marked in blue in the revised version of the manuscript.

W1/Q2: Novelty and looped transformers: we agree the looped architecture is not new, we do not claim it (Section 2) and we discuss prior work on looped and recurrent-depth models extensively in App. B.3. The architecture is a means to our study: we examine architectural choices (App. D) only insofar as they yield a strong and well-understood model for the analysis that follows. Our contribution is studying recursion inside MDMs. In AR models, loops deepen the computation behind each individual next-token prediction. In MDMs, each loop refines the marginals of all masked positions at a fixed state, while denoising steps commit tokens. This creates a second test-time axis that interacts with the first, and that interaction is our object of study:

  • Analysis (Sec. 4): we observe that loops reduce only the fidelity error and the commit schedule controls the dependence error. We measure both exactly on multi-solution Sudoku. Loops cut fidelity error by 6–43$\times$ (when L=1 →\rightarrow 5), and failure rates approach those of a sampler with perfect marginals (64% vs. 59% at T=8). Survival more than doubles, from 14% to 36% at T=8. This explains why recursion helps most at small T.
  • Decoding (Sec. 5): since loops and commits act on different errors, they must be chosen jointly. LoopMDM (concurrent) adapts loops at a fixed denoising budget but, to our knowledge, no prior method adapts both.

Looping the full network, with deep supervision and a step embedding, is what gives per-loop marginals for both studies (ablated in App. D).

W2/Q1: Effectiveness, scale, and standard LM benchmarks: we address the reviewers concerns below:

  • Breadth of experiments: we evaluate on four domains: Sudoku (9×9, 25×25 and multi-solution boards); Countdown-3/4/5; RNA infilling; and TinyGSM. We include extensive ablations on architecture, supervision, training depth and decoding (App. D, F, G).
  • Small scale: our largest model has 161M parameters, a 49k vocabulary and 512-token sequences, and was trained on 8 H200 GPUs. This is comparable in parameter count to the small configurations standard in the masked diffusion literature (e.g., MDLM, ~110M parameters; SEDD-small, ~110M). Billion-scale models and pretrained MDMs are beyond our compute budget, and we have updated Section 6 to state this more explicitly.
  • Standard LM benchmarks: we report GSM8K, evaluated by execution accuracy, and our evaluation follows standard practice for MDMs. Sudoku and Countdown are commonly used to evaluate MDMs (Ye et al., 2025; Kim et al., 2025; He et al., 2026) and training on TinyGSM and evaluating on GSM8K is likewise an established setup for DLMs at our scale (Liu et al., 2023; Deschenaux & Gulcehre, 2026a; Deschenaux & Gulcehre, 2026b). Furthermore, our central question, how loops and commits trade off at inference time, can only be measured through generation quality (meanwhile, perplexity-based benchmarks have been shown to be misleading across diffusion families (Sahoo et al., 2026)). We therefore chose tasks with verifiable correctness and strong inter-token dependencies, where the two error sources can be separated.
  • Effectiveness and scaling: recursion scales at fixed parameters (Countdown-5: 15.6% → 56.4%; GSM8K: 13.7% → 24.0%) and outperforms untied depth at matched FLOPs (Sudoku: 93.6% vs. 77.8% with 5× fewer parameters). Our decoder lies on or near the quality–NFE Pareto front in all four domains (Fig. 8), reaching the same GSM8K accuracy with 11× fewer NFE.

W3: Entropy-based criteria: we agree that entropy-based commit rules come from prior work and do not claim them as new (Sections 2 and 5.1). Our contribution is using entropy also to decide how many loops to run, and showing why this is principled:

  • Loop stopping on entropy gain across loops: entropy-based loop stopping reaches similar quality to LoopMDM's hidden-state rule at a much lower NFE (App. G.6).
  • The coupling of the two rules: Prop. A.3 shows the entropy commit guarantee is bounded by DfidD_{fid}, which is exactly what loops reduce. Empirically, entropy decreases across loops and predicts token errors (AUROC 0.86 on Sudoku; 0.84/0.91/0.93 on Countdown-3/4/5; Fig. 7 and App. G.9).

Using one shared signal for both decisions follows from that analysis, and the resulting decoder lies on or near the quality–NFE Pareto front in all domains (Fig. 8). The two components contribute in different regimes: the commit rule drives most savings on long sequences (TinyGSM: 69 vs. 768 NFE at matched accuracy), while loop stopping matters at large L (Countdown-4: about 40 vs. 98 NFE).

We hope this addresses the reviewer's concerns and would appreciate a reconsideration of the score. We are happy to engage further if the reviewer has follow-up questions.

AuthorsOpenReview ↗

References

  • Ye et al. (2025). Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning. ICLR 2025. arXiv:2410.14157
  • Kim et al. (2025). Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions. ICML 2025. arXiv:2502.06768
  • He et al. (2026). Reasoning with Latent Tokens in Diffusion Language Models. arXiv preprint (2026). arXiv:2602.03769
  • Liu et al. (2023). TinyGSM: Achieving >80% on GSM8k with Small Language Models. arXiv:2312.09241
  • Deschenaux & Gulcehre (2026a). BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers. arXiv preprint (2026). arXiv:2606.02241
  • Deschenaux & Gulcehre (202b6). Language Modeling with Hyperspherical Flows TMLR (2026). arXiv:2605.11125
  • Sahoo et al. (2026). Scaling Beyond Masked Diffusion Language Models. arXiv preprint (2026). arXiv:2602.15014

I thank the authors for their detailed rebuttal. However, my main concern remains. I agree with the authors that 161M is okay. However, current evaluations rely mainly on small synthetic tasks, which makes it hard to tell whether this works on real LM setup. I think more evaluations on language modeling benchmarks such as OWT and LM1B are still necessary. For now, I keep my score.

We thank the reviewer for engaging with our rebuttal. Our evaluation is not limited to synthetic tasks, but considers structured datasets inspired by real-world problems alongside real-world datasets such as RNA infilling on RNAcentral sequences and executable program generation evaluated on the GSM8K test set.

We agree that open-ended language modeling (LM1B or OWT) would be a valuable extension, and we note it as future work in Section 6. However, language is not the only domain of interest for MDMs, which are also applied to biological sequences, reasoning and planning, and focusing on these domains first does not invalidate our contributions. Our loop–step error analysis (Section 4) and joint decoder (Section 5) concern how recursion interacts with decoding, which perplexity does not evaluate, and therefore motivates our choice of tasks. We hope the reviewer will weigh these contributions.

Reviewer_3vaAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discussed and analyzes two complementary directions for inference-time improvement of recursive masked diffusion models. (1) Recursive Length (L) and (2) Denoising Steps (T).

Contributions:

  1. Recursive architecture and parameter-efficient scaling.
  2. Dependence–fidelity analysis.
  3. Adaptive two-axis decoding.
Critical strengths
  1. Clear formulation of two distinct dimensions of inference-time computation.
  2. Strong empirical evidence on many tasks.
  3. Experiments support the theory.
  4. Extensive ablations.
Critical weaknesses
  1. Limited architectural novelty.
  2. Lack of direct comparison with closely related works. Maybe LoopMDM and Fixed-Point Masked Generative Modeling could have been used.
  3. The jointly adaptive decoder is not consistently better than simpler alternatives.
  4. Limited evidence of generalization to realistic language modeling and larger-scale settings.
  5. Predictive entropy is an imperfect proxy for marginal fidelity and decoding improvement: The proposed adaptive method assumes that reductions in predictive entropy indicate beneficial recursive refinement. However, lower entropy does not necessarily imply lower prediction error. A model may become increasingly confident in an incorrect prediction, and two significantly different predictive distributions can have nearly identical entropy. The authors acknowledge some of these limitations, but the empirical justification primarily relies on correlations between entropy and correctness in Sudoku.
Questions for authors

Check the weakness. Answering each point is the question for the author.

2 comments
AuthorsOpenReview ↗

Answer to reviewer (part 1/2)

We thank the reviewer for their feedback. Below, we address their concerns and questions, with relevant changes marked in blue in the revised version of the manuscript.

W1: Architectural novelty: we keep the architecture intentionally simple, and architectural novelty is not our main claim. The architecture is a means to our study and we study architectural choices (App. D) only to build a strong and well-understood model for the analysis that follows. These choices also differ from concurrent looped MDMs in:

  • We loop the entire network rather than middle layers, so every loop produces a prediction. This allows deep supervision across loops (App. D.4) and gives the decoder per-loop marginals.
  • We add a recursive step embedding (App. D.3) to inform the model of the loop progress.
  • This minimal architecture enables our controlled study: it lets us isolate recursion in the dependence–fidelity analysis (Section 4) and explore the role of looping as a decoding axis (Section 5). These two points are specific to MDMs, and, to our knowledge, have not been studied in any other work.

W2: Comparison with LoopMDM / FP-MGM: we consider both as concurrent works, and they introduce recursion differently in training: LoopMDM loops a subset of layers, and FP-MGM embeds a fixed-point solver. An architecture-level comparison is therefore not directly transferable to our setting and would not test our claims, since our goal is not to find the most efficient recipe for training recursive MDMs, but rather to study how recursion interacts with token commitment at test time (Sections 4 and 5). We study architectural choices only as a means to develop a strong model for our subsequent analysis.

Moreover, our error decomposition and entropy-based decoding rules require per-loop marginals, which a middle-layer loop (as proposed by other works) does not produce natively. While they could be recovered by running the remaining layers and output head after every loop, this adds their cost to every stopping decision. It also changes the compute accounting and departs from the original method, so it would no longer be a faithful comparison to either work. Where the methods overlap at test time, we compare directly: our hidden stopping rule is LoopMDM’s hidden-state-drift criterion, which reaches similar quality to our entropy rule but at higher NFE (Section 5.1, App. G.6). We have also extended our discussion in Appendix B.3 with this explanation.

W3: Joint decoder not consistently better: our claim is Pareto efficiency, not dominance in absolute quality. The entropy_entropy decoder lies on or near the quality–NFE Pareto front in all four domains (Fig. 8), and it matches the best quality on Sudoku and Countdown-3/4 at a fraction of the compute. Comparing it with one-axis decoders also shows when each axis matters (Section 5.2). On long sequences, the commit rule drives the savings: on TinyGSM at L=3, fixed_entropy matches the best fixed decoder (24.0% vs. 24.2%) at 69 instead of 768 NFE, and adding loop stopping does not improve on it. As L grows, loop stopping becomes necessary: on Countdown-4, fixed_entropy grows from 37 to 98 NFE between L=3 and L=7 while entropy_entropy stays near 40, and on TinyGSM at L=5, entropy_entropy needs roughly half the NFE of fixed_entropy (63 vs. 136) for similar accuracy. The joint decoder’s cost therefore stays flat as L grows, whereas one-axis decoders are competitive only at a well-chosen L. We now state this more explicitly in Section 5.

W4: Large-scale / language settings: This limitation reflects our compute budget, and we have added a limitations paragraph to Section 6 to better reflect that. Our focus is the interaction between recursion and decoding, which requires tasks with well-defined accuracy metrics (open-ended diffusion language generation relies on proxy metrics, such as generative perplexity, that can be misleading (Sahoo et al., 2026)). Our evaluation follows standard practice for MDMs: Sudoku, Countdown and GSM8K are commonly used (Ye et al., 2025; Kim et al., 2025; He et al., 2026), and our tasks also go beyond toy settings:

  • RNA infilling uses real RNAcentral sequences with long-range base-pairing dependencies.
  • TinyGSM uses a 161M-parameter model with a 49k vocabulary and 512-token sequences, trained on 8 H200 GPUs and evaluated by execution accuracy on GSM8K, an established setup for diffusion language models at our scale (Liu et al., 2023; Deschenaux & Gulcehre, 2026a; Deschenaux & Gulcehre, 2026b).

Under the same decoder, recursion improves both (Δ\DeltaZ 0.453 →\rightarrow 0.140; GSM8K 13.7% →\rightarrow 24.0%), and the adaptive commit rule reaches 24.0% on TinyGSM with 11× fewer NFE than the fixed decoder (69 vs. 768). While absolute GSM8K accuracy remains modest, our claim concerns the relative gain from recursion at fixed parameter count and not state-of-the-art performance.

AuthorsOpenReview ↗

Answer to reviewer (part 2/2)

W5: Entropy as a proxy: We agree that low entropy does not by itself imply correctness, hence we do not use it as a guarantee but rather as a proxy. Our decoder uses entropy in two distinct ways, which is supported by different evidence:

  • Per-token entropy for commits:
    • Theory: Prop. A.3 shows that the gap between model and true entropies is controlled by DfidD_{fid}, under a bounded-KL condition. Entropy therefore becomes a more faithful signal as recursion reduces fidelity error. Since DfidD_{fid} is unobservable at test time, this result motivates the rule. We have updated the manuscript to state this more clearly in Section 5.1.
    • Prior work: The rule builds on established entropy-based commit rules (EB-Sampler; Cai & Li, 2026).
    • Empirical study: Entropy predicts token errors beyond Sudoku. To address the reviewer's concern, we extended the entropy–error analysis of Fig. 7 to Countdown-3/4/5 (new App. G.9), where entropy predicts token errors with AUROC 0.84/0.91/0.93, against 0.86 on Sudoku.
  • Entropy change for loop stopping: mean entropy decreases with recursive depth, with most of the reduction in the first loops (Fig. 7a; App. G.9). We present this as a practical stopping signal (Section 5.1). The alternative, hidden-state drift, does not improve quality and has higher NFE cost (App. G.6).

We hope this addresses the reviewer's concerns and would appreciate a reconsideration of the score. We are happy to engage further if the reviewer has follow-up questions.

References

  • Sahoo et al. (2026). Scaling Beyond Masked Diffusion Language Models. arXiv preprint (2026). arXiv:2602.15014
  • Ye et al. (2025). Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning. ICLR 2025. arXiv:2410.14157
  • Kim et al. (2025). Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions. ICML 2025. arXiv:2502.06768
  • He et al. (2026). Reasoning with Latent Tokens in Diffusion Language Models. arXiv preprint (2026). arXiv:2602.03769
  • Liu et al. (2023). TinyGSM: Achieving >80% on GSM8k with Small Language Models. arXiv:2312.09241
  • Deschenaux & Gulcehre (2026a). BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers. arXiv preprint (2026). arXiv:2606.02241
  • Deschenaux & Gulcehre (2026b). Language Modeling with Hyperspherical Flows TMLR (2026). arXiv:2605.11125

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.