acceptodds
Under review as a conference paper at ICLR 2027

Raster or Random? Let the Data Decide

11 views

Abstract

Autoregressive (AR) and masked diffusion models differ fundamentally in generation order, yet what determines the best order remains unclear. We show that the answer depends on the inner structure of the data. Prior comparisons contrast complete systems that differ in attention pattern, training objective, and decoding schedule at once, so the effect of order alone cannot be isolated. We introduce a unified framework that makes order the only variable: a shared causal-attention architecture takes the token order as an explicit input, holds everything else fixed, and decouples the training order from the inference order. Linear dynamical systems (LDS) and their shuffled counterpart further provide controlled proxies whose dependencies are aligned or misaligned with the token order. Across modalities, the AR order wins on language and ordinary LDS, whereas the random order wins on images and shuffled LDS. To reveal the underlying mechanisms, we decouple training and inference stages and reveal two distinct mechanisms: during training, order diversity acts as data augmentation on image-like data but not on language; during inference, scattered orders prevent the accumulation of shift along the generation. An information-theoretic analysis on LDS explains how the augmentation benefit depends on the data dependency structure. These results clarify when and why each order succeeds and guide the choice of order for a given modality.

Primary area:
generative models
Keywords:
Generation order, masked diffusion, generalization and memorization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_d8vjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when raster or random token order works better. It changes the training and sampling orders separately while keeping the architecture fixed within each modality. In the main experiments, left-to-right order works better for language and standard LDS, while random order works better for images and shuffled LDS. The analysis examines the benefits of varied training orders and how distribution shift builds up during generation. It also includes an information-theoretic result for a restricted LDS setting.

Critical strengths
  • I like that the authors compare orders without changing the architecture, model size, or training budget within each modality. Testing all training–sampling combinations, rather than just matching each model with its training order, makes it much easier to see where the differences come from.
  • The fixed-random comparison is the part I find most informative. It separates the benefit of seeing different orders during training from that of visiting scattered positions during sampling.
  • The findings are tested on both real and synthetic data. The LDS experiments help examine the role of dependency structure while keeping the underlying dynamics fixed.
Critical weaknesses
  • My main concern is how Theorem 2 relates to the actual models. A smaller lower bound does not necessarily imply a lower actual error, and the analysis assumes single-token conditioning with restricted lag observations. Appendix B.2 is a useful check, but I would like the authors to explain how tight the bounds are and what they tell us about full-prefix Transformers.
  • I am not sure how much of the image–text difference comes from dependency structure and how much comes from data reuse. The image models are trained for 800 epochs, while the text models see the corpus only slightly more than once. Could random-order training help more on images because the same data are reused heavily? The LDS controls partly address this, but I would still like to see experiments with fewer training epochs on images or multiple passes over a smaller text dataset. This would help separate the effects of dependency structure and data reuse.
Questions for authors

Please see "Critical Weaknesses" part for reference.

4 comments
AuthorsOpenReview ↗

Response to Reviewer d8vj (1/2)

W1: A smaller lower bound does not necessarily imply a lower actual error, and the analysis assumes single-token conditioning with restricted lag observations. How tight are the bounds, and what do they tell us about full-prefix Transformers?

A-W1. Thank you for the constructive comments. Overall, the bound is attainable and reasonably tight, and both simplifications are mild. Theorem 2 is meant to give mechanistic insight in a simplified setting without heavy notation and derivation. Below we address your concerns one by one in detail.

Single-token conditioning. This assumption does not change the comparison in Theorem 2, because the information it discards is the same under AR and random training. Let z1,…,zTz_1, \dots, z_T be the tokens in the inference order and p^\hat p a single-token model. Apart from the error on the first token, Appendix B.1.8 (Eq. (40)) splits the joint KL of p^\hat p exactly into two parts:

KL(q,∥,p^)=∑tElog⁡q(zt∣z<t)q(zt∣zt−1)⏟A: information lost+∑tE,KL(q(zt∣zt−1),∥,p^(zt∣zt−1))⏟conditional errors (Theorem 2).\mathrm{KL}(q \\,\Vert\\, \hat p) = \underbrace{\sum_t \mathbb E \log \frac{q(z_t \mid z_{< t})}{q(z_t \mid z_{t-1})}}_{\mathcal A:\ \text{information lost}} + \underbrace{\sum_t \mathbb E\\, \mathrm{KL}\big(q(z_t \mid z_{t-1}) \\,\Vert\\, \hat p(z_t \mid z_{t-1})\big)}_{\text{conditional errors (Theorem 2)}}.

The first part A\mathcal A measures how much the earlier tokens tell about ztz_t beyond the previous token zt−1z_{t-1}. It depends only on the data and the inference order, not on the model or on how it was trained. For ordinary LDS under AR order, A=0\mathcal A = 0: by the Markov property, q(zt∣z<t)=q(zt∣zt−1)q(z_t \mid z_{<t}) = q(z_t \mid z_{t-1}), so single-token conditioning is exact. Under shuffled LDS or random orders, A>0\mathcal A > 0, but its average Aˉ\bar{\mathcal A} enters the lower bounds of AR and random training as the same constant (Eqs. (41)–(42)). It therefore does not change our conclusion.

Restricted lag observations. This assumption is a reasonable approximation for real Transformers: transfer across lags is modest and confined to the adjacent lag. We test it on full-prefix Transformers. We train on a fixed set of lag-1 sequences combined with 20k–60k extra lag-kk sequences (x1,x1+k,x1+2k,… )(x_1, x_{1+k}, x_{1+2k}, \dots), k∈2,4,8k\in\\{2,4,8\\}, and measure whether the lag-1 test KL decreases with additional lag-kk data. Each setting is run with 3 random seeds; parentheses give the relative change from the baseline.

Extra sequences0 (baseline)+20k+40k+60k
lag 20.0449 ± 0.00090.0397 ± 0.0017 (−11.6%)0.0409 ± 0.0015 (−9.1%)0.0384 ± 0.0014 (−14.6%)
lag 40.0449 ± 0.00090.0445 ± 0.0012 (−0.9%)0.0436 ± 0.0010 (−2.9%)0.0437 ± 0.0019 (−2.8%)
lag 80.0449 ± 0.00090.0442 ± 0.0023 (−1.7%)0.0452 ± 0.0019 (+0.5%)0.0456 ± 0.0016 (+1.4%)

Lag-2 data lower the lag-1 KL by 9–15%, while lag-4 and lag-8 data have no effect beyond seed noise. We have added this experiment and the discussion to Appendix B.2 of the revised paper.

Tightness of the bound. The bound is attainable and of the same order of magnitude as the errors we measure: the MLE reaches it to within 1%, and single-token Transformers stay within a factor of 3. We use models trained with random orders and the corresponding bound Bkrand(E)\mathcal{B}^{\mathrm{rand}}_{k}(E) with E→∞E \to \infty (every lag-kk pair is observed). The table reports the ratio of the measured lag-kk conditional KL to this bound.

lag kk124
MLE1.001.001.01
Single-token Transformer, ordinary LDS2.951.781.30
Single-token Transformer, shuffled LDS2.651.841.31

MLE. The MLE achieves KL which is very close to the predicted bound at every lag tested, so the bound is attainable and a smaller bound corresponds to a smaller achievable error.

Transformers. The trained Transformers do not reach the bound, but the gap stays within the same order of magnitude and is well controlled: the measured KL is 1.3–3.0 times the bound, and the ratio shrinks as the lag grows (from 2.95 and 2.65 at k=1k=1 to about 1.3 at k=4k=4).

We have added this table to Appendix B.2 of the revised paper. We have also rewritten the Remarks after Theorem 2 in Section 5 as three corresponding points, which summarize the evidence and point to Appendix B.2.

AuthorsOpenReview ↗

Response to Reviewer d8vj (2/2)

W2: Random-order training may help more on images because the image data are reused heavily (800 epochs, against about one pass for text); multiple passes over a smaller text dataset would separate data reuse from dependency structure.

A-W2. We appreciate the reviewer's valuable comment. Under heavy data reuse, our observations on language remain unchanged: the scan orders still beat the random order, and random-order training still brings no augmentation benefit. The image–text difference therefore comes from the dependency structure, not from data reuse. As suggested, we trained the text models for 40 passes over a 225M-token subset of ClimbMix (compared with about 1 pass in the paper), so that text is reused heavily, as images are.

We first show the end-to-end quality at the best checkpoints (rows: training order; columns: inference order). MAUVE (↑\uparrow):

Train ↓ / Infer →rand.ARreversefixed-rand.
rand.0.6920.3430.2540.634
AR0.0080.7390.0140.011
reverse0.0120.0260.8370.014
fixed-rand.0.0530.0290.0470.039

Generation perplexity (↓\downarrow):

Train ↓ / Infer →rand.ARreversefixed-rand.
rand.278.0313.8367.4260.5
AR1720.076.41166.41782.6
reverse1896.0707.680.82010.5
fixed-rand.*22.510.611.725.7

∗^* The fixed-random model degenerates into repetition under every order, which explains its low perplexity.

As in Table 1 of the paper, random-order training still lags behind AR-order training, even when trained for many passes.

Generalization gap (validation minus training bpb) of each model under its own training order, at the best checkpoints:

Training orderRandomARReverseFixed-random
Val − train bpb (↓\downarrow)0.1380.0860.0820.088

As the table shows, the random model does not have a smaller gap than the single-order models; its gap is in fact the largest (0.138, against 0.082–0.088). On CIFAR-10, by contrast, random-order training shrinks the gap, acting as strong data augmentation (Table 2 of the paper). These results verify our conclusion that the augmentation effect of random-order training largely depends on the data structure.

Convergence. As further evidence, Figure 10 in Appendix A.14 of the revised paper shows the validation bpb and the training loss of each model during training. Every model reaches its best validation bpb well within the 40 passes and degrades afterwards while its training loss keeps falling, which confirms that 40 passes is enough for each order to fully exploit the data. All models converge as the figure shows. The AR and reverse models overfit after 2 passes, while the random model keeps improving until pass 21. Even so, the best random model (1.058) is worse than the AR model after a single pass (1.029).

We have added these results to Appendix A.14 of the revised paper and mention them in Section 4.2.1.

Thanks for the authors' response. After carefully reviewing the response, I think the additional analysis of Theorem 2 and the experiments on data reuse address my main concerns. Thus, I will keep my score of 3.

AuthorsOpenReview ↗

Thank you for your response

Dear Reviewer d8vj,

Thank you for getting back to us! We are glad to hear that the new experiments and additional analysis have resolved your main concerns. If you have any further questions or concerns, we would be more than happy to clarify.

We sincerely appreciate your insightful feedback, which has helped us substantially strengthen the paper.

Best regards, The Authors

Reviewer_2QXyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the relationship between token order and the underlying dependency structure of different data modalities. The authors introduce a unified causal Transformer framework to isolate the effect of generation order from other modeling choices and independently investigate the roles of training and inference orders. Experiments show that random ordering performs better on images and shuffled linear dynamical systems (LDS), whereas left-to-right autoregressive ordering is preferable for language and ordinary LDS. Further analysis suggests that random-order training acts as a form of data augmentation, improving generalization and reducing overfitting on images, while scattered inference orders mitigate error accumulation during image generation. Finally, the paper provides an information-theoretic analysis of ordinary and shuffled LDS to explain how the benefits of random-order training depend on the underlying data dependency structure.

Critical strengths

1. Important and well-motivated research question. The paper investigates an important question in generative modeling: why autoregressive generation remains dominant in language, while both autoregressive and masked diffusion approaches are widely explored for images. Understanding whether the choice of token order is related to the underlying dependency structure of different data modalities is an interesting and meaningful research direction.

2. Well-controlled experiments and thorough analysis. The experimental design is carefully controlled, isolating the effects of token generation order from other factors, such as attention mechanisms, within a unified framework. Beyond conventional image and language datasets, the authors introduce ordinary and shuffled LDS as synthetic settings to further investigate the role of data dependency structure. The paper also independently analyzes training and inference orders, with additional experiments examining the regularization effect of random-order training and error accumulation during sampling. Overall, the experimental analysis provides useful insights into the effects of token order.

Critical weaknesses

1. Limited experimental scope and generality. The image experiments are limited to CIFAR-10 and FFHQ at 32×32 resolution, without validation on larger-scale datasets such as ImageNet or at higher image resolutions. The observed regularization benefits of random-order training may be influenced by overfitting in these relatively small datasets, and it remains unclear whether similar benefits hold at larger data scales and higher resolutions. Furthermore, the experiments focus on causal Transformers with per-token diffusion heads, without exploring VQ-based autoregressive models or bidirectional masked generative models. These limitations may affect the generality of the conclusions across different data scales and model architectures.

Questions for authors
  1. Larger-scale image generation. Does the regularization benefit of random-order training persist on larger datasets and at higher image resolutions? Could the authors provide further experimental evidence or analysis to support this?

  2. Generality across model architectures. Do the conclusions extend to other generative modeling architectures, such as bidirectional-attention masked models or VQ-based discrete image generation models? Could the authors provide additional experiments or analysis to validate their generality?

  3. Inference order sensitivity. How sensitive is fixed-random inference to the choice of permutation? Would different fixed-random permutations or structured scattered orders lead to similar generation quality and error propagation?

I am open to updating my assessment if the authors can satisfactorily address these concerns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2QXy (1/2)

Q1: Does the regularization benefit of random-order training persist on larger datasets and at higher resolutions?

A-Q1. Thank you for the constructive comment. We have added experiments on ImageNet-100 (126k images at 256×256), which show that the regularization benefit of random-order training persists on this larger, higher-resolution dataset.

We train a 32-layer causal transformer (MAR-L size) on KL-16 VAE latents with the random, AR and fixed-random orders; reverse is skipped as it behaves like AR in Tables 1 and 2. The three models share all other settings and, as in the paper, are evaluated at their best checkpoints.

We first compare the end-to-end generation quality. FID (↓\downarrow), in the format of Table 1:

Train ↓ / Infer →rand.ARfixed-rand.
rand.20.5139.8519.99
AR249.550.90228.9
fixed-rand.234.1179.746.67

As on CIFAR-10, random-order training gives by far the best FID: 20.51 under its own order, against 50.90 for AR. Even on the same fixed-random order, the random-order model reaches FID 19.99, far better than the 46.67 of the model trained on that single order.

We then verify that this advantage comes with less overfitting, using the teacher-forcing MSE in the format of Table 2:

Training orderTrain lossTest, rand.Test, ARTest, fixed-rand.
rand.0.2550.2780.2770.278
AR0.2160.6120.2940.608
fixed-rand.0.2310.5460.5720.309

The single-order models fit the training data better (0.216 and 0.231) but generalize worse (test loss 0.294 and 0.309), a gap of 0.078. The random-order model has the lowest test loss under every inference order and a gap of only 0.023, about 3 times smaller. Together, the two tables reproduce the CIFAR-10 findings of Tables 1 and 2 at a larger scale and a higher resolution: random-order training reduces overfitting and gives better generation quality.

We have added these results to Appendix A.4 of the revised paper and summarized them in Section 4.2.1.

Q2: Do the conclusions extend to other architectures, such as bidirectional masked models or VQ-based discrete image generation models?

A-Q2. Our conclusions carry over to bidirectional masked models: scattered inference orders still achieve better FID than scan orders, and the gap again comes from error propagation, not from worse per-step predictions.

We use the official MAR-L checkpoint (Li et al., 2024), a bidirectional masked model trained with random orders on ImageNet 256×256. All orders use the same sampling settings (no classifier-free guidance, 50k samples); the teacher-forcing (TF) MSE and Rˉ\bar R are measured as in Tables 2 and 3.

Inference orderTF MSERˉ\bar RFID
Random0.25800.542.68
Fixed-random0.25850.602.72
Halton [1]0.25800.562.64
AR (raster)0.25843.064.57
Reverse0.26423.297.35

As on CIFAR-10, the scattered orders (the first three) give nearly the same FID (2.64–2.72), while the scan orders (the last two) are clearly worse (4.57–7.35) and have 5–6 times the Rˉ\bar R. AR and random order have almost the same TF MSE, so the FID gap comes from error accumulation during generation, not from worse conditional predictions. Our conclusions therefore carry over to bidirectional masked models.

Actually, using causal attention degrades the FID little compared with the effect of the order. The paper uses a single causal architecture for all orders, so that the order is the only variable (Section 3.1). To measure the cost of this choice, we trained the original bidirectional MAR with exactly the recipe of our causal model: the only difference is the attention type. The FID (50k samples) is shown below:

ModelCIFAR-10, bidirectionalCIFAR-10, causalFFHQ-32, bidirectionalFFHQ-32, causal
MAR-Tiny (35.6M)15.9220.426.326.57
MAR-Base (206M)10.7014.264.715.55

Although bidirectional attention achieves slightly better FID, the improvement (0.25–4.50 FID) is small. The order matters far more: for MAR-Base on CIFAR-10, the attention type changes FID by 3.6 (10.70 vs. 14.26), whereas switching the same causal model from random to AR inference changes it by 26.1 (14.26 vs. 40.33, Table 1). Comparing a bidirectional random-order model with a causal AR model would mix the two effects; using one causal architecture keeps the order as the only difference.

We have added both results to Appendix A.9 of the revised paper and mention them in Sections 3.1 and 4.2.2.

AuthorsOpenReview ↗

Response to Reviewer 2QXy (2/2)

Q3: How sensitive is fixed-random inference to the choice of permutation, and would other permutations or structured scattered orders give similar generation quality and error propagation?

A-Q3. Thank you for this question. The answer is yes: different fixed-random permutations, as well as a structured scattered order, give nearly the same generation quality and error propagation. We evaluated the random-order CIFAR-10 model of the paper under five additional fixed-random permutations and under the Halton order [1].

Inference orderRˉ\bar RFID
Fixed-random, permutation 01.4814.25
Fixed-random, permutation 11.6214.69
Fixed-random, permutation 21.5214.44
Fixed-random, permutation 31.6714.82
Fixed-random, permutation 41.7314.70
Fixed-random, permutation 51.5314.27
Fixed-random, mean ± std over the 6 permutations1.59 ± 0.1014.53 ± 0.24
Halton [1]1.2414.84
Random1.5214.26
Reverse5.4028.45
AR6.5040.33

All six fixed-random permutations fall in a narrow band, with Rˉ\bar R between 1.48 and 1.73 and FID between 14.25 and 14.82. The scan orders, by contrast, raise Rˉ\bar R to 5.4–6.5 and FID to 28–40. The Halton order is a structured scattered order, where the revealed tokens cover the image evenly at every step.

Those results show that generation quality and error propagation are not sensitive to the choice of permutation; what matters during inference is the scattered geometry of the order, not the particular permutation.

We have added these results to Appendix A.8 of the revised paper and mention them in Section 4.2.2.

[1] Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, Matthieu Cord. Halton Scheduler for Masked Generative Image Transformer. ICLR 2025.

Thank you for the detailed rebuttal and additional experiments. The additional experiments have addressed most of my concerns regarding scalability, architectural generalization, and permutation sensitivity. I appreciate the authors' efforts and will maintain my positive assessment of the paper.

Dear Reviewer 2QXy,

Thank you for your reply! We are delighted that the additional experiments have addressed most of your concerns. If you have any other questions or remaining concerns, we would be very happy to discuss them further. If our responses have addressed all of your concerns, we would greatly appreciate it if you would consider raising your score.

Thank you again for your constructive comments, which have greatly improved our revision.

Best regards, The Authors

Reviewer_DS7xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the question whether generation order should follow the dependency structure of the data. The motivation stems from the fact that comparisons between autoregressive and masked diffusion systems change several design choices at once, making the effect of order difficult to identify. The authors propose a unified causal attention framework that allows training and inference order to vary independently. The experiments on text, CIFAR-10 and synthetic dynamical systems provide empirically supported practical guidance on when to adopt sequential or random ordering.

Critical strengths
  • The study addresses a meaningful experimental gap. Separating training order from inference order, while retaining a common architecture, makes the comparisons easier to interpret than comparisons between complete generation systems. The fixed-random control further distinguishes exposure to multiple orders from the spatial arrangement of a particular order (Tables 1–2).
  • The empirical pattern is sufficiently clear to motivate further study: random ordering improves CIFAR-10 FID from 23.17 to 14.26, while sequential ordering performs better on text and ordinary LDS (Table 1). The contrasting outcomes support the claim that order interacts with the data and training regime. There is also a strong case for treating order as a design choice rather than assuming one ordering is uniformly preferable.
Critical weaknesses
  • Theorem 2 considers a simplified dynamical system in which each prediction conditions on only one preceding token. The evaluated transformers instead condition on every previously generated token, so it can combine information in ways this simplified model cannot represent. The theorem therefore provides an intuition for why order diversity might help, but does not directly explain the transformer results.
Questions for authors
  • How do the paper’s findings extend to diffusion language models trained with random masking but decoded using confidence-based token selection? Can the proposed account of order diversity and error propagation explain this adaptive ordering, and under what conditions would it predict an advantage over random decoding?
4 comments
AuthorsOpenReview ↗

Response to Reviewer DS7x (1/2)

W1: Theorem 2 conditions each prediction on only one preceding token, whereas Transformers condition on the full prefix, so the theorem does not directly explain the Transformer results.

A-W1. We appreciate your constructive comments. Theorem 2 aims to provide a mechanism insight on a simplified setting without introducing too many complex notation and derivation. But we can further bridge the gap and address the your two points in turn.

Single-token conditioning. This assumption does not change the comparison in Theorem 2, because the information it discards is the same under AR and random training. Let z1,…,zTz_1, \dots, z_T be the tokens in the inference order and p^\hat p a single-token model. Apart from the error on the first token, Appendix B.1.8 (Eq. (40)) splits the joint KL of p^\hat p exactly into two parts:

KL(q,∥,p^)=∑tElog⁡q(zt∣z<t)q(zt∣zt−1)⏟A: information lost+∑tE,KL(q(zt∣zt−1),∥,p^(zt∣zt−1))⏟conditional errors (Theorem 2).\mathrm{KL}(q \\,\Vert\\, \hat p) = \underbrace{\sum_t \mathbb E \log \frac{q(z_t \mid z_{< t})}{q(z_t \mid z_{t-1})}}_{\mathcal A:\ \text{information lost}} + \underbrace{\sum_t \mathbb E\\, \mathrm{KL}\big(q(z_t \mid z_{t-1}) \\,\Vert\\, \hat p(z_t \mid z_{t-1})\big)}_{\text{conditional errors (Theorem 2)}}.

The first part A\mathcal A measures how much the earlier tokens tell about ztz_t beyond the previous token zt−1z_{t-1}. It depends only on the data and the inference order, not on the model or on how it was trained. For ordinary LDS under AR order, A=0\mathcal A = 0: by the Markov property, q(zt∣z<t)=q(zt∣zt−1)q(z_t \mid z_{<t}) = q(z_t \mid z_{t-1}), so single-token conditioning is exact. Under shuffled LDS or random orders, A>0\mathcal A > 0, but its average Aˉ\bar{\mathcal A} enters the lower bounds of AR and random training as the same constant (Eqs. (41)–(42)). It therefore does not change our conclusion.

Combining information. This assumption is a reasonable approximation for real Transformers: transfer across lags is modest and confined to the adjacent lag. We test it on full-prefix Transformers. We train on a fixed set of lag-1 sequences combined with 20k–60k extra lag-kk sequences (x1,x1+k,x1+2k,… )(x_1, x_{1+k}, x_{1+2k}, \dots), k∈2,4,8k\in\\{2,4,8\\}, and measure whether the lag-1 test KL decreases with additional lag-kk data. Each setting is run with 3 random seeds; parentheses give the relative change from the baseline.

Extra sequences0 (baseline)+20k+40k+60k
lag 20.0449 ± 0.00090.0397 ± 0.0017 (−11.6%)0.0409 ± 0.0015 (−9.1%)0.0384 ± 0.0014 (−14.6%)
lag 40.0449 ± 0.00090.0445 ± 0.0012 (−0.9%)0.0436 ± 0.0010 (−2.9%)0.0437 ± 0.0019 (−2.8%)
lag 80.0449 ± 0.00090.0442 ± 0.0023 (−1.7%)0.0452 ± 0.0019 (+0.5%)0.0456 ± 0.0016 (+1.4%)

Lag-2 data lower the lag-1 KL by 9–15%, while lag-4 and lag-8 data have no effect beyond seed noise. We have added this experiment and the discussion to Appendix B.2 of the revised paper, and have revised the Remarks after Theorem 2 in Section 5 to summarize these checks.

AuthorsOpenReview ↗

Response to Reviewer DS7x (2/2)

Q1: How do the findings extend to diffusion language models decoded by confidence-based token selection, and when would the paper's account predict an advantage over random decoding?

A-Q1. Thank you for the constructive question. Confidence-based decoding outperforms random-order decoding on generation perplexity but loses on MAUVE, and under either order the random model stays far behind the AR-trained model, as in the paper. We decode the random-order text model of the paper using confidence-based order and compare it with random decoding of the same model. Below, we analyze this result in terms of per-step error and error propagation, and then give the conditions under which confidence decoding should have an advantage.

We first report the generation quality:

Training orderInference orderMAUVE (↑\uparrow)Gen. ppl (↓\downarrow)Entropy
RandomRandom0.653152.34.85
RandomConfidence0.539111.64.54
ARAR0.86835.14.81
Real text–0.89316.04.77

The two metrics thus rank the two orders oppositely: confidence decoding is better on generation perplexity (111.6 vs. 152.3), random decoding on MAUVE (0.653 vs. 0.539).

The disagreement comes from what each metric measures. Generation perplexity mainly measures the coherence between tokens. By committing the most confident position first, confidence decoding avoids local errors as far as possible, but it is prone to collapse: its diversity (entropy 4.54 vs. 4.77 for real text) and its global distribution deviate from those of real text [1, 2]. MAUVE instead measures quality at the level of the distribution. Random decoding avoids error accumulation and thereby keeps the global statistics of its samples consistent with real text.

Error propagation (AA) makes this concrete:

Inference orderTF bpb (↓\downarrow)AA
Random0.89423.7
Confidence0.889118.1

Confidence-based inference propagates errors far more strongly: one wrong token changes 118.1 of the 256 tokens, five times as many as under random inference. This is because confidence-based inference nearly degenerates into AR generation, as also observed in [2]: 57% of its steps fill the position next to the previous one. It therefore retains the error propagation of AR generation.

The AR model remains better under either order. It is the best on both MAUVE and generation perplexity, and its TF bpb is 0.706 against 0.907–0.939 for the random model (Table 2 of the paper), a gap that the 0.005 bpb gain of confidence decoding cannot close. This is consistent with the paper: on language, the random model does not reach the AR model under any inference order.

We expect confidence decoding to beat random decoding under two conditions. Confidence-based inference reduces the per-step error but may cause extra error propagation, so it gains only when the first effect is large and the second is small:

  1. The per-step difficulty depends strongly on the order. On Sudoku, for example, adaptive decoding raises the accuracy from below 7% to about 90% [3]; on text, however, the gain is only 0.005 bpb.
  2. The confident positions are spread over the sequence, so that the order stays scattered and errors stay local. On images, for example, replacing the confidence scheduler of MaskGIT with the spatially uniform Halton order lowers FID [4].

We have added these results and the discussion to Appendix A.13 of the revised paper.

[1] Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, Qinsheng Zhang. Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling. ICLR 2025.

[2] Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeguo Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, Gao Huang. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models. arXiv:2601.15165, 2026.

[3] Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, Sitan Chen. Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions. ICML 2025.

[4] Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, Matthieu Cord. Halton Scheduler for Masked Generative Image Transformer. ICLR 2025.

Thank you for the detailed answer to this question. I think the case of ordering in diffusion LLMs is a particularly useful case for this work. The findings and interpretations are also interesting, and perhaps a more extensive study could be a fruitful follow-up work. I will maintain my weak acceptance rating.

Dear Reviewer DS7x,

Thank you for your follow-up and continued support! We agree that ordering in diffusion LLMs is a particularly informative case, and a more extensive study in this direction would be a fruitful follow-up. We will highlight it as future work in the revision.

If you feel our responses have strengthened the paper, we would be grateful if you would consider raising your score. Thank you again for your thoughtful review and valuable suggestions.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revision

Dear Area Chair and Reviewers,

We sincerely appreciate your valuable comments and constructive discussions during the rebuttal.

Review summary. We are pleased that all three reviewers recommend acceptance, and each has responded to our rebuttal: Reviewer d8vj states that the additional analysis and experiments "address my main concerns"; Reviewer 2QXy states that the additional experiments "have addressed most of my concerns" and maintains a positive assessment; Reviewer DS7x finds the new results interesting and maintains the rating.

Below we summarize the main points raised by the reviewers and how we addressed them.

  • Larger-scale and higher-resolution data (2QXy): We add experiments on ImageNet-100 at 256×256, which show that the regularization benefit of random-order training persists on large-scale dataset and that it still outperforms AR training.
  • Other architectures (2QXy): We show that our conclusions carry over to bidirectional masked models: scattered inference orders still achieve better FID than scan orders, and the attention type matters far less than the order.
  • Sensitivity to the permutation (2QXy): Different fixed-random permutations, as well as a structured scattered order, give nearly the same generation quality and error propagation; what matters is the scattered geometry of the order, not the particular permutation.
  • Theorem 2 and full-prefix Transformers (DS7x, d8vj): We show that the bound of Theorem 2 is attainable and reasonably tight, and that both of its simplifications are mild for full-prefix Transformers.
  • Data reuse vs. dependency structure (d8vj): Under heavy data reuse, our observations on language remain unchanged, so the image–text difference comes from the dependency structure, not from data reuse.
  • Confidence-based decoding (DS7x): Confidence-based decoding improves generation perplexity but loses on MAUVE, as it nearly degenerates into AR generation and inherits its error propagation; we also discuss when it should help.

For the paper revision, we have made the following changes following the reviewer's suggestions (marked in blue):

  • Added ImageNet-100 results at 256×256 (Appendix A.4; summarized in Section 4.2.1).
  • Added the bidirectional MAR-L evaluation and the bidirectional vs. causal comparison (Appendix A.9; Sections 3.1 and 4.2.2).
  • Added the sensitivity study with five further fixed-random permutations and the Halton order (Appendix A.8; Section 4.2.2).
  • Added the heavy data reuse experiment on text, with its training curves (Appendix A.14; Section 4.2.1).
  • Added confidence-based decoding of the text model, analyzed through per-step error and error propagation (Appendix A.13).
  • Added empirical checks of the assumptions of Theorem 2 and the tightness of its bound (Appendix B.2), and rewrote the Remarks after Theorem 2 in Section 5 to summarize them.

Overall, we remain confident in the contributions of this work and thank the AC and reviewers for helping us strengthen it.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.