- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper studies the few-step generation capability of Masked Diffusion Language Models (MDLM), and claims that the performance gaps reported in the literature may come from sampler configurations. The authors show that with the pretrained weights of MDLM fixed, the perplexity can be improved by simply adjusting temperature and precision (quality and diversity judged by LLMs). To this end, the authors introduce GroupEval to separately measure output quality and group-level semantic diversity. A finite-step analysis is also conducted to explain the reason why better denoising predictions may not produce better generated distributions. The validation mainly includes experiments on the OpenWebText dataset with various settings, and comparisons with other benchmark methods.
Critical strengths
The evidence that the few-step generation performance of MDLMs can be clearly improved by simply adjusting the sampler configurations, with the weights of pre-trained models remain unchanged. This suggests that existing models may actually be stronger than reported in the literature, and should be properly tuned before concluding that a new method is better. Besides, the theoretical contribution may also be important for principled explanation (but did not check the correctness carefully).
Critical weaknesses
Aside from the limitations acknowledged by the authors in the “Scope and limitations” section, the main weakness of this manuscript is the clarity of its presentation. From my view, it is somewhat verbose and difficult to follow. This manuscript has considerable room for further improvement, e.g., making the narrative more concise and focused. In addition, the sampler tuning comparison experiments for DUO and SEDD should be included in the main submission, as this directly affects the evaluation of the paper's contribution.
Questions for authors
In the manuscript, the authors note that DUO and SEDD were not given comparable sampler tuning. I wonder if MDLM remain competitive if these models were similarly tuned? Additional experimental results or theoretical discussion would help clarify this point.