acceptodds
Under review as a conference paper at ICLR 2027

Order-Asymmetry in Diffusion Language Models: Discrete Curl and Conditional Compatibility

Abstract

Diffusion language models decode tokens in flexible orders, but their denoisers provide only local conditionals for partially observed contexts, which need not be compatible with a single joint distribution over an unresolved block. As a result, different commitment orders can induce different order-specific pseudo-joints. We call this model-side failure conditional incompatibility, distinct from data-side dependence. For any two unresolved positions, we define curl as the log-ratio between the pseudo-joints induced by reversing the commitment order; it vanishes exactly when the two orders agree. We then prove that the order-dependent part of sequential decoding error is a data-weighted curl path integral, separate from conditional total correlation and marginal model error. Empirically, pairwise Fisher–Rao curl energy distinguishes list-operation regimes, recovers Sudoku constraint structure, and identifies a left-to-right preferred direction on GSM8K for both LLaDA-8B-Instruct and Dream-7B. Order sensitivity is therefore a measurable compatibility property of learned local conditionals.

Primary area:
interpretability and explainable AI
Keywords:
diffusion language models, decoding order

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Anw4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how different generation orders induce different distributions in diffusion language models. It introduces discrete curl to measure local order asymmetry and connects it to differences in sequential decoding KL error. The paper distinguishes model-side conditional incompatibility from data-side dependence and provides diagnostic experiments on list operations, Sudoku, GSM8K, and LAMBADA.

Critical strengths
  1. The distinction between data dependence and model-side conditional incompatibility is useful. It clarifies that strongly dependent tokens can still be sampled in different sequential orders without changing the distribution when the conditionals are compatible.

2.The experiments provide interesting observations about order sensitivity. For example, the Sudoku serialization experiment shows that the measured asymmetry depends on how the input is represented. The LAMBADA experiment in Appendix C also provides evidence that the proposed metric can predict output sensitivity beyond confidence and entropy baselines.

Critical weaknesses
  1. My main concern is what new understanding the theory provides beyond formalizing order asymmetry. Different generation orders assigning different probabilities is an expected consequence of imperfectly learned conditionals. Theorem 1 follows directly from the definitions, while the error decomposition and swap-path identity largely follow from probability identities and telescoping. These results provide a clear mathematical description, but it is less clear which additional, nontrivial conclusions follow from this description.

  2. The implications for decoding or training remain unclear. Appendix C shows that the metric can help predict whether outputs change across orders, which is useful. However, predicting sensitivity does not tell us which order produces better answers or whether reducing incompatibility improves generation quality. Likewise, the GSM8K experiment measures order preferences on model-predicted tokens rather than correctness. I do not think a new decoder is necessary, but the paper would benefit from a clearer explanation of what decisions or insights this framework enables.

Questions for authors
  1. What is the main nontrivial insight enabled by the curl framework beyond directly comparing the probabilities induced by different orders? Could you clarify which conclusion would be difficult to obtain from the standard chain-rule and KL identities alone?

  2. Beyond predicting output sensitivity, do the current results establish any connection between the proposed metric and choosing a better decoding order or improving training?

Reviewer_fow4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the order asymmetry in diffusion language models. Different commit orders will produce different joint distributions from the learned conditional distributions. The paper defines curl as the log-ratio of two order-specific factorizations and their connections to KL error. It also proposes Fisher-Rao as a symmetric incompatibility diagnostic. Experiments are conducted on list-operation, sudoku and math tasks.

Critical strengths
  1. This paper points out the failure from consistency among learned conditionals. The adjacent-swap formulation gives an interpretable local description which is useful for determining order sensitivity in learned models.
  2. The Sudoku experiment is convincing.The LAMBADA results also indicate that the measure carries information beyond marginal confidence and entropy, especially for predicting the NLL gap.
Critical weaknesses
  1. The incremental contribution over existing consistency analyses is insufficiently established. Theorem 1 follows directly from the definition of curl, the order-comparison identities follow by telescoping, and the KL decomposition follows by introducing a factorized reference distribution. These are useful identities, but less convincing as theoretical contributions. In [1], it also examine inconsistent joint probabilities under reversed masked-token completion orders. [2] also analyzed unmasking order through pathwise log-likelihood and KL divergence. The paper does not adequately explain or demonstrate what its diagnostic reveals beyond them.
  2. The experiments do not validate the stronger interpretations of the theory. The GSM8K experiments measures different orders on fixed assignments. However, it didn’t establish connections to lower data-distribution KL or higher accuracy
  3. Applications are limited. The paper does not establish if curl can help improve decoding performance or order selection.

[1] Wagner, et al., A Framework for Consistency Testing of Span Probabilities in Language Models. EMNLP, 2024. aclanthology.org/2024.emnlp-main.866

[2] Xu, et al., Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models. ICML, 2026. arxiv.org/abs/2606.23567

Questions for authors
  1. Greedy decoding: Greedy outputs can depend on order even under a perfectly compatible joint distribution. Thus, results under greedy do not show that curl explains output differences. How should the Appendix C results be interpreted in light of this?
  2. Applications: Can curl be used to guide decoding, and does doing so improve accuracy?
Reviewer_LQTyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies order dependence in diffusion language models through conditional compatibility and discrete curl. It separates statistical dependence in the data, prediction accuracy, and inconsistency between conditionals evaluated in different generation orders. It derives local swap identities and an order-dependent KL decomposition, then develops top-k diagnostics based on these ideas. Experiments on controlled tasks, Sudoku, GSM8K, and LAMBADA examine how the diagnostics vary with task structure, serialization, and generation order. The main contribution is an analytical perspective on diffusion language models rather than a new decoder.

Critical strengths
  1. The conceptual distinction is valuable. A model can make confident predictions while its conditionals remain inconsistent across generation orders; conversely, compatibility does not imply accurate modeling of the data. Making these distinctions explicit provides a useful way to reason about diffusion language models beyond token confidence alone.
  2. The analysis is supported by informative controlled experiments. In particular, the serialization comparisons and the study of local order sensitivity connect the formal quantities to observable model behavior. These results provide an interesting diagnostic perspective even without a new decoding algorithm.
Critical weaknesses
  1. The evidence remains local and does not yet establish how much the diagnostic helps with complete generation. The empirical procedure uses a truncated candidate set, and the GSM8K experiment studies permutations of small blocks at fixed candidate assignments. This demonstrates local differences in model probabilities, but does not establish that the preferred order improves correctness or identifies a good order over the full generation trajectory. The LAMBADA analysis offers useful evidence of order sensitivity, yet the step from diagnosing that sensitivity to making better decoding decisions remains unresolved. I find the analysis interesting, but the demonstrated scope is not yet strong enough to meet the acceptance bar.
  2. The cost and actionable value of the diagnostic need to be considered together. Estimating pairwise conditional distributions requires multiple intervened forward contexts for the active positions and candidate tokens; batching and reuse can help, but this is not a one-forward-pass signal. The paper has not demonstrated a low-cost curl-based controller that improves end-to-end accuracy or speed. A new decoder is not necessary for every analysis paper, but in this case a clearer account of what the diagnostic enables at a practical cost would materially strengthen the significance of the contribution. The interpretation linking Dream’s directional preference to autoregressive initialization should also remain a hypothesis, since the causal explanation has not been isolated.
Questions for authors
  1. Clarification: Which existing results establish that the preferred local order predicts better ground-truth likelihood or task correctness, rather than only a higher probability for a fixed model-selected assignment? How far do the authors intend the conclusions to extend from small blocks to complete generation?
  2. Clarification: What is the measured cost of the diagnostic relative to confidence or entropy, including batching and reuse, and what is the strongest demonstrated benefit at that cost? Evidence that it supports useful decoding choices within a matched budget would materially strengthen my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.