acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Reasoning Through Masks

Abstract

Reasoning is built from intermediate results: each step of a proof, program or iterative computation rests on what earlier steps established. Masked diffusion models (MDMs) generate by repeatedly filling in blanks, in any order and with the option to revise, and this flexibility has made them a promising alternative to left-to-right language models for reasoning. Yet their gains have fallen short, and it is unclear whether better decoding orders, larger models or stronger revision can close the gap. Here we provably show that none of them removes the fundamental bottleneck: between rounds, an MDM remembers only its tokens, and a token must be either an answer or a blank. Without revision, an MDM giving a short answer is effectively one forward pass deep, however many rounds it runs and however large it grows. Revision lets rounds build on one another, but only by paying for extra rounds or scratch tokens that grow with input length. And when the intermediate results a task needs are not yet answers, no amount of output space can hold them. Passing hidden states from one round to the next removes these limits. This Latent-State MDM, trained on at most 40 steps of the Rule 110 cellular automaton, executes 96 steps without error, where GPT-6 Astra solves only 6 of 10 cases, and lifts Sudoku-Extreme accuracy from 8.95% to 73.95% on the same 6.6M-parameter backbone, surpassing MDMs 46 times larger. Reasoning requires unfinished computation to survive before it becomes an answer.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Masked Diffusion Models, Reasoning, Iterative Computation, Computational Complexity, Persistent Hidden States, Circuit Complexity

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_9u75OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper suggest a strategy of learning for reasoning tasks, which has a recursive structure in the sense that its own output is feed several time to the same model sequentially. This technique is compared to MDMs, where the goal is to approximate a continuous time Markov process in discrete state space, where on starts in a masked token and over time each of this masks is unmasked. Their main contribution is a theoretical analysis on what the hidden state representation, that can be changed for each round, can bring in terms of expressivity.

Critical strengths

The theoretical analysis of the role of the hidden representations in their recursive model is interesting and relevant.

Critical weaknesses

The presentation is really not easy to follow, in the main text there is no concise description of the model. E.g. in the main text something like h0=E(x,[M]m),hr+1=Fθ(hr)...h_0=E(x,[M]^m),h_{r+1}=F_\theta(h_r)... is missing as well as a loss function. Also there is a statements referring to the appendix "Parity, the XOR of a binary sequence, makes this separation exact; the formal result appears in Section ..." where the content is not understandable without checking the appendix.

Further the statement of Theorem 2: "Under the fixed-depth AC0 round model, an answer-faithful MDM that follows the ready-position protocol can compute a task in T rounds only if the task admits an answer derivation chain of length at most T . If no such chain exists, neither MDMmonoMDM_{mono} nor MDMrevMDM_{rev} succeeds in any finite number of rounds." is really hard to understand, because it uses too many vague adjectives. It would be much easier if there was a short paragraph where everything is introduced with the corresponding formulas, especially since the theoretical part is the main contribution of the paper. The positioning of the paper is a bit unclear to me, since it is called Latent-State MDM but it is more related to recursive model than masked diffusion, since it doesn't try to learn any stochastic process or denoising problem.

Questions for authors
  • Without a clear description of the method including formulas and losses, the paper is hard do understand.
  • Could you please also give a short introduction on what the AC0AC^0 circuit is, how it is related to the method as well as to the tasks in the experiments?
  • For explaining better the bottleneck of the MDM, it might be helpful to give an elementary example with S5S_5, like composing (g1,g2,g3,g4)(g_1,g_2,g_3,g_4) which one can do with the recursive model in two steps i.e. (g2∘g1,g4∘g3,e,e)(g_2\circ g_1,g_4\circ g_3,e,e) and (g4∘g3∘g2∘g1,e,e,e)(g_4\circ g_3\circ g_2 \circ g_1,e,e,e), while for a MDM, where one only has one changable position, the computation has to be done directly, needing more expressivity of the network. This is because the sequence length for MDM is 1, only one function evaluation makes sense if there is no remasking. (At least thats how i understood the statement heuristically, please correct it, if its wrong)
  • Is the literature cited sufficiently? E.g. for Theorem 1, how is the relation to "Saunshi et al,REASONING WITH LATENT THOUGHTS: ON THE POWER OF LOOPED TRANSFORMERS", Theorem 5.1.
Reviewer_x2cuOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that token-only feedback limits iterative reasoning in masked diffusion models and proposes preserving hidden states across rounds. The idea is compelling, and the results are promising.

Critical strengths

The paper identifies a clear and useful distinction between producing answer tokens and preserving intermediate computation. The theoretical analysis and controlled tasks provide complementary ways to examine this idea, and the proposed implementation is refreshingly simple.

Critical weaknesses
  1. The reliability and interpretation of the experimental gains need stronger support. The main table gives an incomplete picture of variability across training runs. At Rule 110 depth 128, the reported 66.7% exact-match accuracy averages three seed-level results of 100%, 0%, and 100%. These outcomes indicate a substantial stability problem: one trained model fails on every evaluated trajectory, while the other two solve all of them. Three seeds provide limited evidence about how reliably training produces a model that generalizes to longer computation horizons. The paper should explain this failure and show where performance begins to break down as depth increases.

The baseline results also raise questions about what drives the gap. At depth 64, Revise-MDM achieves 56.54% token accuracy, exactly matching the reported majority baseline. This suggests that the comparison may involve a baseline that has largely failed to learn the task. A revision-capable token model can, in principle, store the current binary row and apply the local Rule 110 update repeatedly using the same vocabulary and no padding. Its empirical failure therefore needs further investigation before attributing the improvement to an expressivity limitation. The prompted-language-model comparison is also based on only ten examples per depth, which limits the strength of that comparison.

  1. Theorem 10’s simulation proof does not consistently account for the latent-state size. B.1 explicitly defines the recurrent state as ((n+m+P)w) bits. With zero padding, this becomes ((n+m)w), but the proof in B.9 repeatedly treats the complete state as containing only (nw) bits. The additional (mw) bits associated with output positions are not accounted for in the proposed encoding.

This matters for the stated padding guarantee. For example, with (|A|=4), (w=4), and (m=n), the theorem’s bound supplies (2n) padding tokens. Even including the (n) writable output tokens, the token canvas provides only (6n) representation bits, whereas the formally defined latent state contains (8n) bits. Thus, the described direct encoding cannot store the full state. This establishes a gap in the given proof; another simulation might still establish containment. The authors should reconcile the definitions, correct the workspace bound, and clarify which resource comparisons are affected.

Questions for authors
  1. Could you report additional training seeds and depth-by-depth results between 96 and 128 steps? What causes one seed to collapse completely, and how consistently does training produce successful long-horizon generalization?

  2. Could you include a trained discrete-state rollout baseline that stores and updates the current binary row, with matched training and inference budgets? This would help distinguish the effects of state representation from those of the denoising objective and decoding procedure.

  3. Where are the output positions’ additional (mw) latent bits stored in Theorem 10’s construction? Please provide a complete encoding argument and a corrected sufficient padding bound.

  4. Does this correction change the capacity-matched accounting in B.5 or any subsequent conclusions? Releasing checkpoints, test inputs, and per-instance predictions would also make the extreme accuracy results easier to independently verify.

1 comment

It is a pity that I didn't have a chance to have a deeper communication with the authors. Therefore, given the weaknesses I mentioned above, I decided to lower my rating.

Reviewer_EbxFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper deals with the problem of masked diffusion's aspect of reusing the tokens for the later inference stage. The authors propose LS - MDM, with some theoretical guarantees, where we put the hidden state of an MDM (not the tokens) into the next iteration. This shows an improvement over MDM (with an adaptive decoding strategy) empirically on Sudoku - extreme.

Critical strengths

The idea of 'MDM needs to use the previous information of the masked prediction' sounds useful and theoretically justified.

Critical weaknesses
  1. The proposed method appears to be essentially a looped Transformer, an architecture with an extensive existing literature. Although the authors mention this connection in Appendix A, it should be made explicit in the main text, ideally by presenting the method as starting from a looped Transformer. The current framing risks obscuring the distinction between the paper’s contributions.

  2. I find it surprising that the proposed method outperforms MDMs equipped with confidence-based adaptive decoding and remasking. Adaptive unmasking and remasking are central capabilities of MDMs, but the looped Transformer implementation evaluated here outperforms these models without using either mechanism. This finding deserves more investigation into when and why adaptive token-level decoding falls short of latent-state iteration.

  3. The experimental scope is also a concern. The main practical evaluation is on Sudoku-Extreme, which remains a relatively small-scale and specialized benchmark. A broader set of experiments is needed to establish the claim.

I would have viewed the paper more favorably if it had explicitly built on looped Transformers, explored how latent-state iteration could be combined with MDMs’ native unmasking and remasking capabilities, and demonstrated the benefits across a more diverse set of tasks. Given the concerns above, I assign a score of 1.

Questions for authors

See the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.