- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies remasking in discrete diffusion models. The authors propose random remasking at inference time and trajectory correction training, which they claim and show it improves performance.
Critical strengths
The main strength of the paper is that it studies a simple idea that has been shown (in works like ReMDM) to be useful for improving discrete diffusion models with additional compute.
Critical weaknesses
My main concern is Novelty.
The paper makes minor modifications to ideas already explored in the literature, such as ReMDM. The work feels more like an ablation study, but the ablations are not extensive enough to explain why these changes work better. Instead, the paper places considerable emphasis on theory, whose connection to the practical results is not sufficiently evaluated.
Other concerns:
- Given the limited novelty, the evaluation on a small set of tasks and benchmarks makes it difficult to assess the benefits over existing remasking methods, including random remasking, which is already widely used.
- Large models already work well with random remasking or confidence-based remasking. Although the paper includes ablations, it is still unclear how much of the improvement comes from each proposed change.
- The paper brings ReMDM into its own evaluation setting/setup rather than also evaluating on benchmarks used in the ReMDM paper. The model setups differ as well. This leaves it unclear whether the improvements extend to the settings where ReMDM originally showed benefits.
- Regarding TCT, several existing works, such as ProSeCo, PRISM, and RemeDi, already modify training to improve self-correction. It is unclear what TCT adds beyond these methods if evaluated under same setup. I am also concerned about the practical cost of repeated correction, given that computational efficiency is one motivation for discrete diffusion.
Questions for authors
What do the authors consider the main new contribution beyond the relatively minor changes to existing RemeDi, ReMDM and other methods?