- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how different generation orders induce different distributions in diffusion language models. It introduces discrete curl to measure local order asymmetry and connects it to differences in sequential decoding KL error. The paper distinguishes model-side conditional incompatibility from data-side dependence and provides diagnostic experiments on list operations, Sudoku, GSM8K, and LAMBADA.
Critical strengths
- The distinction between data dependence and model-side conditional incompatibility is useful. It clarifies that strongly dependent tokens can still be sampled in different sequential orders without changing the distribution when the conditionals are compatible.
2.The experiments provide interesting observations about order sensitivity. For example, the Sudoku serialization experiment shows that the measured asymmetry depends on how the input is represented. The LAMBADA experiment in Appendix C also provides evidence that the proposed metric can predict output sensitivity beyond confidence and entropy baselines.
Critical weaknesses
-
My main concern is what new understanding the theory provides beyond formalizing order asymmetry. Different generation orders assigning different probabilities is an expected consequence of imperfectly learned conditionals. Theorem 1 follows directly from the definitions, while the error decomposition and swap-path identity largely follow from probability identities and telescoping. These results provide a clear mathematical description, but it is less clear which additional, nontrivial conclusions follow from this description.
-
The implications for decoding or training remain unclear. Appendix C shows that the metric can help predict whether outputs change across orders, which is useful. However, predicting sensitivity does not tell us which order produces better answers or whether reducing incompatibility improves generation quality. Likewise, the GSM8K experiment measures order preferences on model-predicted tokens rather than correctness. I do not think a new decoder is necessary, but the paper would benefit from a clearer explanation of what decisions or insights this framework enables.
Questions for authors
-
What is the main nontrivial insight enabled by the curl framework beyond directly comparing the probabilities induced by different orders? Could you clarify which conclusion would be difficult to obtain from the standard chain-rule and KL identities alone?
-
Beyond predicting output sensitivity, do the current results establish any connection between the proposed metric and choosing a better decoding order or improving training?