- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates ReMDM, an unmasking and remasking schedule for diffusion language models. With several engineering tricks (optimizing the EoT penalty, and refinement rate annealing), they outperform a different setup of ReMDM and as well as RCR on two dLLMs across several benchmarks spanning knowledge, coding, and mathematical reasoning.
Critical strengths
The authors show generally strong knowledge of prior work and the models as well as the benchmarks (tasks) are extensive with experiments conducted up to statistical significance.
Critical weaknesses
There is little novelty. There is no theory and barely any methodological advancement. While the authors try to reframe their ReMDM setup with a different name, calling it cross-validation refinement, it is conceptually isomorphic to ReMDM under a different refinement scheduler.
The additional engineering tricks (such as EoT penalty and refinement rate annealing) are established in prior art, yet not applied to the baselines (such as RCR and ReMDM). Also, the authors should compare against threshold-based methods such as WINO etc.
Questions for authors
-
The authors claim that "our approach departs from prior work by explicitly decoupling fill-up and refinement.". ReMDM already does this, see Section 4.2 "Design Strategies for ‘Turning On’ ReMDM". In fact, FIRE is isomorphic to ReMDM.
-
The authors write: "Concretely, we penalize the logit of the EoT token by a factor before softmax, using the sign-aware form of the standard repetition penalty (Keskar et al., 2019). While this adjustment does not directly improve draft quality, it effectively prevents early termination and produces drafts that are better suited for subsequent refinement.". Can the authors provide strong evidence that optimizing an EoT penalty does not directly improve draft quality? This would directly contradict the findings on the Dream models where an EoT token penalty was proven crucial for strong parallel generation under confidence/entropy based decoding. The claim on the EoT penalty is repeated in line 424-426, which likewise may have to be revised.
-
The sign-aware choice of multiplicative penalty for the EoS token is odd, since logits are shift invariant. So the behavior of the penalty depends on a degree of freedom (dof) in the convergence of weights that does not affect model behavior, and said dof cannot be factored out by adjusting the penalty. Can the authors ablate on this and compare against a well-tuned additive shift on the logits at EOT positions?
-
The authors do not compare against WINO, arguing that it was designed for block-attention and is threshold based. The argument is not convincing, WINO simply uses different threshold for drafting and verification (without 2-stage separation) and should be included. The threshold can be tuned to achieve comparable average NFEs. Moreover, WINO could also be applied under 2-stage draft-then-refine separation to compare threshold-based with fixed budget generation.
-
In the experiments, the authors compare against "ReMDM" and RCR. While the former is just the same algorithm but for a different draft and refine window, RCR is actually a different algorithm. Moreover, the comparison is not entirely fair, since both methods seem to be missing an EOT penalty (with schedule) and the same annealing for the refinement rate. There are also no ablations identifying which degrees of freedom (changing and , other annealing options, etc.) actually resulted in the accuracy improvements.
-
On line 216 the authors state that they "remask a random subset", yet they do not indicate under what distribution. In this review I assumed they select the subset uniformly at random just like ReMDM. Can the authors provide clarification on this?