- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies when raster or random token order works better. It changes the training and sampling orders separately while keeping the architecture fixed within each modality. In the main experiments, left-to-right order works better for language and standard LDS, while random order works better for images and shuffled LDS. The analysis examines the benefits of varied training orders and how distribution shift builds up during generation. It also includes an information-theoretic result for a restricted LDS setting.
Critical strengths
- I like that the authors compare orders without changing the architecture, model size, or training budget within each modality. Testing all training–sampling combinations, rather than just matching each model with its training order, makes it much easier to see where the differences come from.
- The fixed-random comparison is the part I find most informative. It separates the benefit of seeing different orders during training from that of visiting scattered positions during sampling.
- The findings are tested on both real and synthetic data. The LDS experiments help examine the role of dependency structure while keeping the underlying dynamics fixed.
Critical weaknesses
- My main concern is how Theorem 2 relates to the actual models. A smaller lower bound does not necessarily imply a lower actual error, and the analysis assumes single-token conditioning with restricted lag observations. Appendix B.2 is a useful check, but I would like the authors to explain how tight the bounds are and what they tell us about full-prefix Transformers.
- I am not sure how much of the image–text difference comes from dependency structure and how much comes from data reuse. The image models are trained for 800 epochs, while the text models see the corpus only slightly more than once. Could random-order training help more on images because the same data are reused heavily? The LDS controls partly address this, but I would still like to see experiments with fewer training epochs on images or multiple passes over a smaller text dataset. This would help separate the effects of dependency structure and data reuse.
Questions for authors
Please see "Critical Weaknesses" part for reference.
4 comments
Response to Reviewer d8vj (2/2)
W2: Random-order training may help more on images because the image data are reused heavily (800 epochs, against about one pass for text); multiple passes over a smaller text dataset would separate data reuse from dependency structure.
A-W2. We appreciate the reviewer's valuable comment. Under heavy data reuse, our observations on language remain unchanged: the scan orders still beat the random order, and random-order training still brings no augmentation benefit. The image–text difference therefore comes from the dependency structure, not from data reuse. As suggested, we trained the text models for 40 passes over a 225M-token subset of ClimbMix (compared with about 1 pass in the paper), so that text is reused heavily, as images are.
We first show the end-to-end quality at the best checkpoints (rows: training order; columns: inference order). MAUVE ():
| Train ↓ / Infer → | rand. | AR | reverse | fixed-rand. |
|---|---|---|---|---|
| rand. | 0.692 | 0.343 | 0.254 | 0.634 |
| AR | 0.008 | 0.739 | 0.014 | 0.011 |
| reverse | 0.012 | 0.026 | 0.837 | 0.014 |
| fixed-rand. | 0.053 | 0.029 | 0.047 | 0.039 |
Generation perplexity ():
| Train ↓ / Infer → | rand. | AR | reverse | fixed-rand. |
|---|---|---|---|---|
| rand. | 278.0 | 313.8 | 367.4 | 260.5 |
| AR | 1720.0 | 76.4 | 1166.4 | 1782.6 |
| reverse | 1896.0 | 707.6 | 80.8 | 2010.5 |
| fixed-rand.* | 22.5 | 10.6 | 11.7 | 25.7 |
The fixed-random model degenerates into repetition under every order, which explains its low perplexity.
As in Table 1 of the paper, random-order training still lags behind AR-order training, even when trained for many passes.
Generalization gap (validation minus training bpb) of each model under its own training order, at the best checkpoints:
| Training order | Random | AR | Reverse | Fixed-random |
|---|---|---|---|---|
| Val − train bpb () | 0.138 | 0.086 | 0.082 | 0.088 |
As the table shows, the random model does not have a smaller gap than the single-order models; its gap is in fact the largest (0.138, against 0.082–0.088). On CIFAR-10, by contrast, random-order training shrinks the gap, acting as strong data augmentation (Table 2 of the paper). These results verify our conclusion that the augmentation effect of random-order training largely depends on the data structure.
Convergence. As further evidence, Figure 10 in Appendix A.14 of the revised paper shows the validation bpb and the training loss of each model during training. Every model reaches its best validation bpb well within the 40 passes and degrades afterwards while its training loss keeps falling, which confirms that 40 passes is enough for each order to fully exploit the data. All models converge as the figure shows. The AR and reverse models overfit after 2 passes, while the random model keeps improving until pass 21. Even so, the best random model (1.058) is worse than the AR model after a single pass (1.029).
We have added these results to Appendix A.14 of the revised paper and mention them in Section 4.2.1.
Thanks for the authors' response. After carefully reviewing the response, I think the additional analysis of Theorem 2 and the experiments on data reuse address my main concerns. Thus, I will keep my score of 3.
Thank you for your response
Dear Reviewer d8vj,
Thank you for getting back to us! We are glad to hear that the new experiments and additional analysis have resolved your main concerns. If you have any further questions or concerns, we would be more than happy to clarify.
We sincerely appreciate your insightful feedback, which has helped us substantially strengthen the paper.
Best regards, The Authors
Response to Reviewer d8vj (1/2)
A-W1. Thank you for the constructive comments. Overall, the bound is attainable and reasonably tight, and both simplifications are mild. Theorem 2 is meant to give mechanistic insight in a simplified setting without heavy notation and derivation. Below we address your concerns one by one in detail.
Single-token conditioning. This assumption does not change the comparison in Theorem 2, because the information it discards is the same under AR and random training. Let z1,…,zT be the tokens in the inference order and p^ a single-token model. Apart from the error on the first token, Appendix B.1.8 (Eq. (40)) splits the joint KL of p^ exactly into two parts:
KL(q,∥,p^)=A: information lostt∑Elogq(zt∣zt−1)q(zt∣z<t)+conditional errors (Theorem 2)t∑E,KL(q(zt∣zt−1),∥,p^(zt∣zt−1)).
The first part A measures how much the earlier tokens tell about zt beyond the previous token zt−1. It depends only on the data and the inference order, not on the model or on how it was trained. For ordinary LDS under AR order, A=0: by the Markov property, q(zt∣z<t)=q(zt∣zt−1), so single-token conditioning is exact. Under shuffled LDS or random orders, A>0, but its average Aˉ enters the lower bounds of AR and random training as the same constant (Eqs. (41)–(42)). It therefore does not change our conclusion.
Restricted lag observations. This assumption is a reasonable approximation for real Transformers: transfer across lags is modest and confined to the adjacent lag. We test it on full-prefix Transformers. We train on a fixed set of lag-1 sequences combined with 20k–60k extra lag-k sequences (x1,x1+k,x1+2k,…), k∈2,4,8, and measure whether the lag-1 test KL decreases with additional lag-k data. Each setting is run with 3 random seeds; parentheses give the relative change from the baseline.
Lag-2 data lower the lag-1 KL by 9–15%, while lag-4 and lag-8 data have no effect beyond seed noise. We have added this experiment and the discussion to Appendix B.2 of the revised paper.
Tightness of the bound. The bound is attainable and of the same order of magnitude as the errors we measure: the MLE reaches it to within 1%, and single-token Transformers stay within a factor of 3. We use models trained with random orders and the corresponding bound Bkrand(E) with E→∞ (every lag-k pair is observed). The table reports the ratio of the measured lag-k conditional KL to this bound.
MLE. The MLE achieves KL which is very close to the predicted bound at every lag tested, so the bound is attainable and a smaller bound corresponds to a smaller achievable error.
Transformers. The trained Transformers do not reach the bound, but the gap stays within the same order of magnitude and is well controlled: the measured KL is 1.3–3.0 times the bound, and the ratio shrinks as the lag grows (from 2.95 and 2.65 at k=1 to about 1.3 at k=4).
We have added this table to Appendix B.2 of the revised paper. We have also rewritten the Remarks after Theorem 2 in Section 5 as three corresponding points, which summarize the evidence and point to Appendix B.2.