- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MLD3, a finite mixture of factorized transition kernels for few-step discrete diffusion distillation. A shared discrete latent captures dependencies between tokens updated in the same step, while finite marginalization enables exact evaluation of the student’s transition likelihood. The paper analyzes the limitations of factorized students and a latent-separable surrogate objective, and evaluates the method on text, molecule, regulatory DNA, and image generation.
Critical strengths
-
This paper explains why token-wise matching misses dependencies in few-step generation. It also shows how the analyzed surrogate objective can cause latent collapse. This motivates training with the joint mixture likelihood.
-
The finite-mixture model is simple and permits exact likelihood evaluation. Results improve over the single-component control on text and molecule generation. Tests across several domains help show where the method is useful.
Critical weaknesses
-
The text experiments currently use sequences of only 128 tokens, and I would like to see whether the improvements persist at length 1024. Could the authors evaluate this setting using an existing pretrained teacher, such as MDLM trained on OpenWebText or the SDTT checkpoints used by Di4C, and compare MLD3 with Di4C under the same teacher and evaluation protocol? This would help assess whether the proposed modeling of cross-token dependencies remains effective for longer sequences.
-
As I understand it, Equation 9 scores candidates using $ \sum_i \log \sum_k \widetilde{w}_k P_{i,k}(x_i),$ whereas the joint mixture log-likelihood has the form The former uses token-wise mixture marginals and therefore does not directly evaluate the dependencies represented by the shared latent. Since capturing these dependencies is central to the paper’s motivation, I would appreciate an explanation of this choice. Is it an intentional decoding heuristic, and how does it compare with scoring candidates by their joint mixture likelihood?
Questions for authors
See weaknesses.
2 comments
Friendly Reminder as Discussion Period closes tomorrow
Dear Reviewer 44LA,
As the discussion period will close tomorrow, we would like to send a friendly reminder to check out our response. We believe that our response establishes MLD3 as a strong competitor to discrete diffusion and flow models through extensive evaluations with OWT (with longer sequences) and the baselines, and that the revised version of the paper addresses the points raised in your review.
We would greatly appreciate your feedback before the discussion period closes and kindly ask you to reconsider your rating in light of our response.
Thank you,
The Authors
Dear Reviewer 44LA,
Thank you for your careful review and the two precise suggestions, and for recognizing that our finite mixture permits exact likelihood evaluation and analysis of missing token-wise dependencies in current methods. We hope that our responses can help you support the paper more affirmatively.
We performed extensive evaluations on OpenWebText with sequences of 1,024 tokens (please refer to Table 1 and the global comments). The results below share the public MDLM-OWT checkpoint as the teacher and our evaluation setup, and SDTT + Di4C is the authors' released checkpoint (gen-PPL / entropy):
Our results show that MLD3 outperforms Di4C at all NFEs and at a similar entropy under the same teacher, both from MDLM and SDTT. The mixture also improves on its own M=1 control at all NFEs, by 11 to 55% from MDLM, 14 to 20% from SDTT, and 7 to 32% from IDLM (Table 1), which confirms that modeling the dependence between tokens remains more effective for long sequences.
It's true that our sampler scores proposals with the token-wise mixture marginals in Eq. 9 rather than with the joint likelihood, and we would like to clarify that this is an intentional decoding heuristic. The dependence between tokens enters through the proposals, since each proposal draws all its tokens from one component, and the score only chooses between two such proposals. To verify that this choice has little effect on sampling, we replaced Eq. 9 with the joint score log∑kwk∏ipk(yi), keeping the same components (gen-PPL):
Our results show that the two scores pick the same proposal in 98 to 99% of the steps, and that the gen-PPL differs by at most 2% in either direction. The share of identical samples falls at 32 NFEs, since a sample stays identical over K steps only if all K choices agree, but the two scores rank such proposals the same way in almost all cases. We will add this comparison to the appendix.
If we have addressed your remaining concerns, we hope you consider raising your score in stronger support of acceptance, and if any concerns remain, we are happy to continue the discussion!
Thank you,
The Authors