- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper explores the problem of parallel decoding in Masked Diffusion Language Models (MDLMs) through the lens of mutual information. The authors argue that MDLMs are not ideal and can provide different marginal distributions for the same token depending on the order of unmasking due to neural approximations. Based on this idea, the authors name this discrepancy the "self-marginalization defect" and explore methods to quantify it. In practice, the authors estimate this "self-marginalization defect" in several contemporary MDLMs and propose a sampler based on mutual information corrected for the "self-marginalization defect".
Critical strengths
- The observation that MDLM models are not ideal and can induce different joint distributions on tokens depending on the unmasking order is quite interesting and could lead to some useful practical insight.
Critical weaknesses
The paper is very hard to read from start to finish, and that is the main problem.
- The sections lack motivation and conclusions.
- For example, Section 3 is more of a list of technical details rather than a motivated methodology connected to the other sections of the paper.
- Section 2 is very technically complex and very hard to read. It should be simplified and supplemented with useful commentary.
- In addition, some of the notation is not introduced correctly. In the contributions section, , , and are mentioned, but no definitions or explanations are given. Also, for example, is defined in Proposition 4, but there is no explanation of what it means, why we need to estimate it, or why the whole proposition is introduced for this purpose.
- Experimental Sections 4 and 5 are borderline unreadable. The experimental results are mostly presented in text rather than in plots or tables. The references to the appendix are overwhelming. Moreover, the main problem is that there is no clear motivation, meaningful data analysis, or conclusions. These sections are more like lists of facts rather than coherent sections of a paper.
- The conclusion section also does not help much in drawing useful conclusions from the paper; it is a list of facts as well.
The main problem with the paper is its very poor presentation, which makes it almost unreadable and hard to judge. I believe this paper needs to undergo a major revision.
Questions for authors
See weaknesses.
3 comments
As the discussion period closes soon, we would be grateful if you could look at the revised PDF, since your assessment rested mainly on presentation. The general response gives a ten-minute reading route (Fig. 1 → §2.1 → Fig. 4 → §5.2), and our reply above summarises each claim and its evidence. We are happy to clarify anything in the remaining time.
Answer to the Rebuttal
I thank the authors for taking the time to answer my questions and for their thorough rework of the paper.
In particular, I thank the authors for effort taken into the paper revision and the revised version is now much more human readable. I can now see the narrative of the paper, the purpose of sections, the defect definition and the measurement of it. However, as also reviewer JUnb noted:
"The blue text shows that basically the whole paper is re-written, and to me it looks like the framing of results is significantly different and would require a fresh review. I hope this hard work pays off in the next submission."
The entire main body of the paper has been rewritten, which constitutes a major revision and would require a fresh review. In addition, the Appendix remains largely unchanged in the revised version, and the presentation issues I noted about the main text in my initial review still seem to persist there. Revising the Appendix for the next submission would therefore also be beneficial.
I'd like to note that paper has its positive moments:
• I think that studying marginal inconsistency with respect to the generation trajectory and its connection to parallel decoding is an interesting and valuable direction.
• Additionally, I find valuable the observations that imperfect conditionals induce different distributions depending on the unmasking order and, in particular, that this self-marginalization defect actually dominates the parallel-decoding score in important regimes.
But, because the paper had very poor readability prior to the rebuttal, has undergone a major revision, and now needs to be reviewed again in full, I recommend that the authors resubmit this work to a future conference. Therefore I keep my initial score.
Response to Reviewer XUMS
Thank you for the frank assessment. We agree that the presentation obscured the contribution, and we have rewritten the paper.
What the paper claims, and the evidence.
Training-free parallel samplers rank positions by a dependence score that is read as a mutual information. For a deployed model it is not: it equals the dependence of the model's own one-at-a-time sampler plus a non-negative self-marginalization defect.
Scope. The paper does not propose a new sampler. The score-based decoders in Appendix N are ablations: they test whether the score can serve as a decoder on its own, and §5.2 reports that it cannot. The deployed sampler we study is EB-Sampler, unchanged.
The readability changes you asked for.
Soundness.
We would be grateful if you could look at the revision. The general response gives a short reading route.