acceptodds
Under review as a conference paper at ICLR 2027

Mixture-of-Latents for Few-Step Discrete Diffusion Distillation

Abstract

Discrete diffusion and flow models can generate high-quality sequences, but they usually require many denoising steps. Distilling them into few-step models is challenging because a single student step predicts many positions independently, which can lose important dependencies between tokens. We show that matching each position separately is not enough to reproduce the teacher's joint transition. Prior work addresses this issue with a shared continuous latent variable, but its training requires an approximation that can fail to make effective use of the latent. We propose Mixture-of-Latents for Discrete Diffusion Distillation (MLD3), which instead uses a small finite set of components selected by a learned router. This makes the training objective exactly computable and allows the student to model correlated updates directly. We also propose a novel sampling approach that exploits the mixture components at a cost close to that of a factorized step. Notably, MLD3 naturally reduces to a standard masked diffusion student with one component and adds negligible parameter overhead as new components are added. Across text, molecules, regulatory DNA, and image generation, MLD3 consistently improves few-step generation by roughly 2x over the baselines, with the largest gains on tasks with stronger dependencies across positions.

Primary area:
generative models
Keywords:
Generative models, diffusion models, masked diffusion language models, discrete flow matching, few step generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_44LAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MLD3, a finite mixture of factorized transition kernels for few-step discrete diffusion distillation. A shared discrete latent captures dependencies between tokens updated in the same step, while finite marginalization enables exact evaluation of the student’s transition likelihood. The paper analyzes the limitations of factorized students and a latent-separable surrogate objective, and evaluates the method on text, molecule, regulatory DNA, and image generation.

Critical strengths
  • This paper explains why token-wise matching misses dependencies in few-step generation. It also shows how the analyzed surrogate objective can cause latent collapse. This motivates training with the joint mixture likelihood.

  • The finite-mixture model is simple and permits exact likelihood evaluation. Results improve over the single-component control on text and molecule generation. Tests across several domains help show where the method is useful.

Critical weaknesses
  • The text experiments currently use sequences of only 128 tokens, and I would like to see whether the improvements persist at length 1024. Could the authors evaluate this setting using an existing pretrained teacher, such as MDLM trained on OpenWebText or the SDTT checkpoints used by Di4C, and compare MLD3 with Di4C under the same teacher and evaluation protocol? This would help assess whether the proposed modeling of cross-token dependencies remains effective for longer sequences.

  • As I understand it, Equation 9 scores candidates using $ \sum_i \log \sum_k \widetilde{w}_k P_{i,k}(x_i),$ whereas the joint mixture log-likelihood has the form log⁡∑kw~k∏iPi,k(xi).\log \sum_k \widetilde{w}_k \prod_i P_{i,k}(x_i). The former uses token-wise mixture marginals and therefore does not directly evaluate the dependencies represented by the shared latent. Since capturing these dependencies is central to the paper’s motivation, I would appreciate an explanation of this choice. Is it an intentional decoding heuristic, and how does it compare with scoring candidates by their joint mixture likelihood?

Questions for authors

See weaknesses.

2 comments

Dear Reviewer 44LA,

Thank you for your careful review and the two precise suggestions, and for recognizing that our finite mixture permits exact likelihood evaluation and analysis of missing token-wise dependencies in current methods. We hope that our responses can help you support the paper more affirmatively.

[W1] The text experiments currently use sequences of only 128 tokens, and I would like to see whether the improvements persist at length 1024.

We performed extensive evaluations on OpenWebText with sequences of 1,024 tokens (please refer to Table 1 and the global comments). The results below share the public MDLM-OWT checkpoint as the teacher and our evaluation setup, and SDTT + Di4C is the authors' released checkpoint (gen-PPL / entropy):

OWT, NFE481632
Di4C on MDLM1704 / 5.94613 / 5.85238 / 5.75131 / 5.67
MLD3 M=1M=1 control on MDLM366 / 5.84186 / 5.78135 / 5.73116 / 5.70
MLD3 M=8M=8 on MDLM239 / 5.8094.6 / 5.6964.2 / 5.6351.9 / 5.57
SDTT + Di4C252 / 5.3783.3 / 5.3844.1 / 5.3430.8 / 5.27
MLD3 M=8M=8 on SDTT74.4 / 5.4240.2 / 5.3830.8 / 5.3426.9 / 5.29

Our results show that MLD3 outperforms Di4C at all NFEs and at a similar entropy under the same teacher, both from MDLM and SDTT. The mixture also improves on its own M=1M=1 control at all NFEs, by 11 to 55% from MDLM, 14 to 20% from SDTT, and 7 to 32% from IDLM (Table 1), which confirms that modeling the dependence between tokens remains more effective for long sequences.

[W2] The token-wise mixture marginals and therefore does not directly evaluate the dependencies represented by the shared latent.

It's true that our sampler scores proposals with the token-wise mixture marginals in Eq. 9 rather than with the joint likelihood, and we would like to clarify that this is an intentional decoding heuristic. The dependence between tokens enters through the proposals, since each proposal draws all its tokens from one component, and the score only chooses between two such proposals. To verify that this choice has little effect on sampling, we replaced Eq. 9 with the joint score log⁡∑kwk∏ipk(yi)\log \sum_k w_k \prod_i p_k(y_i), keeping the same components (gen-PPL):

CorpusNFEEq. 9JointIdentical samples
OWT895.495.597.7%
OWT3252.352.390.1%
LM1B840840886.1%
LM1B3235035758.4%
WikiText-103816216288.1%
WikiText-10332979668.9%

Our results show that the two scores pick the same proposal in 98 to 99% of the steps, and that the gen-PPL differs by at most 2% in either direction. The share of identical samples falls at 32 NFEs, since a sample stays identical over KK steps only if all KK choices agree, but the two scores rank such proposals the same way in almost all cases. We will add this comparison to the appendix.


If we have addressed your remaining concerns, we hope you consider raising your score in stronger support of acceptance, and if any concerns remain, we are happy to continue the discussion!

Thank you,

The Authors

AuthorsOpenReview ↗

Friendly Reminder as Discussion Period closes tomorrow

Dear Reviewer 44LA,

As the discussion period will close tomorrow, we would like to send a friendly reminder to check out our response. We believe that our response establishes MLD3 as a strong competitor to discrete diffusion and flow models through extensive evaluations with OWT (with longer sequences) and the baselines, and that the revised version of the paper addresses the points raised in your review.

We would greatly appreciate your feedback before the discussion period closes and kindly ask you to reconsider your rating in light of our response.

Thank you,

The Authors

Reviewer_2qdJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MLD3, which introduces a finite mixture of factorized transition kernels to capture dependencies between tokens during few-step generation. The authors propose an exact mixture-likelihood training objective and a new sampling strategy. Experiments on text, molecules, DNA, and images demonstrate improvements over factorized students and Di4C.

Critical strengths

The paper addresses an important problem of accelerating discrete diffusion models while preserving dependencies between generated tokens. The theoretical analysis is interesting, particularly the characterization of factorization error and the limitations of surrogate objectives. The authors conduct experiments across multiple domains and provide several useful ablations and analyses.

Critical weaknesses

1. Insufficient comparison with related approaches. There are several recent methods for few-step generation that are not included in the comparisons. For example, continuous approaches such as Flow Map Language Models show strong results for few-step text generation and would be interesting to compare with. Another closely related work is Latent-Kernel Discrete Flow Maps, which also uses finite mixtures of factorized kernels with exact marginalization. Although LKF trains models from scratch rather than through distillation, I think the connection between these works deserves more discussion and comparison.

2. Generality. The method is evaluated only with masked diffusion teachers, relatively small models, and unconditional generation. On ImageNet, the student is unconditional even though the MaskGIT teacher is class-conditional. It would be interesting to see whether the approach works for uniform-state discrete diffusion, larger teachers, or conditional tasks, since the paper is framed around discrete diffusion and flow models in general.

3. Evaluation and diversity concerns. The experiments show improvements in several metrics, but in some cases these improvements come at the cost of diversity. For example, on QM9, the eight-component model achieves much higher validity but lower uniqueness than the single-component baseline at two NFEs. It is unclear whether the method actually improves the overall generative distribution or simply produces a smaller set of valid samples. Also, the text evaluation mainly relies on generative perplexity measured by GPT-2 Large and relatively short unconditional sequences. It would be interesting to include more evaluation metrics, longer sequences to better understand the improvements.


The paper presents an interesting theoretical analysis and promising empirical results. The experimental validation is broad. Some concerns remain, but overall, I think the paper makes an interesting contribution to the community.

Questions for authors

see weaknesses

4 comments

Dear Reviewer 2qdJ,

Thank you for recognizing the strengths of our paper, and thank you for finding our analysis of the factorization error and surrogate collapse interesting with broad experimental validation. We address the three weaknesses below with further experiments, and sincerely hope that our responses resolve your remaining concerns.

[W1] Insufficient comparison with related approaches

Regarding the comparison with FMLM, we now include it in Tables 1 and 2 (please refer to the global comments). On OWT, FMLM is ahead of all masked students at 1 and 2 NFEs (gen-PPL 168 and 133), but its PPL decreases more slowly, while MLD3 distilled from SDTT or IDLM drops significantly lower across 4-32 NFEs (74.4 against 111 at 4 NFEs from SDTT and 14.2 against 45.1 at 32 from IDLM). Similarly, on LM1B, MLD3 distilled from our SDTT student is lower than FMLM across 8-32 NFEs (67.3 against 69.2 at 32 NFEs). We note that FMLM works in a continuous relaxation with its own flow teacher, whereas MLD3 keeps the discrete kernel and can start from an existing masked diffusion student, and hence we see the two as complementary methods.

While both MLD3 and LKF use a finite mixture of factorized kernels, LKF trains a few-step model from scratch on data, while MLD3 distills a pretrained teacher, which makes them difficult to compare fairly. In addition, our method targets the optimality of the distillation objective itself, where we show that the convexity bound used by Di4C returns a factorized student for any number of components, while the exact mixture likelihood avoids this collapse (Theorem 2).

[W2] General distillation framework for masked & uniform diffusion and flow models

While our paper focuses on masked diffusion teachers, we note that this covers a significant class of models, and we have now broadened our experiments in the revision to demonstrate the generality of our approach in three ways. First, our results on OpenWebText at 1,024 length sequences with released teachers show that the gain holds for longer sequences (Table 1). Second, MLD3 now improves on three different starting points on text, including the MDLM teacher and the SDTT and IDLM distilled models, which were trained with different objectives, and therefore our framework is not tied to a single teacher. Third, we extended MLD3 to the uniform-state DUO and DUO-DCD on OWT with a uniform-state version of our objective. After five rounds, MLD3 from DUO lowers the gen-PPL of DUO by 37 to 55% at a similar entropy and achieves the lowest gen-PPL of all uniform-state models at 16 and 32 NFEs at the entropy of held-out text (Tables 1 and 4). Extending MLD3 to conditional generation and larger teachers remains open as a direction for future work, which we will state in the conclusion. We also note that the ImageNet student is unconditional because we hold one class fixed along each teacher trajectory, which turns the class-conditional teacher into an unconditional model (App. D.2).

[W3] Evaluation and diversity concerns

We agree that the diversity metrics show tradeoffs in the molecule generation results, and now report uniqueness and novelty next to validity for better readability (Tables 5 to 7 and the global comments). Our results show that the M=8M=8 student has a lower uniqueness than its M=1M=1 control at all NFEs (0.327 against 0.612 on QM9 at 8 NFEs). With top-1 routing, which costs the same as the baselines, the same student is more diverse and reaches the highest uniqueness on QM9 across 4-32 NFEs (0.778 against 0.694 for the teacher at 8 NFEs) and the highest novelty at 4 and 8 NFEs. The mixture thus offers a trade-off between more valid and more diverse molecules, and one can choose between the two through the sampler. The results for novelty and uniqueness on QM9 and ZINC-250k (Table 4 and 5) are also added in the continuation of this response (including Tables 6 & 7 in the revised PDF).

We also report the sample entropy next to the gen-PPL at all NFEs in Tables 1 and 2 on OpenWebText. To verify that MLD3 does not win by narrowing its samples, we raised the temperature of SDTT until its entropy matched ours. MLD3 has a 2.4x lower gen-PPL than SDTT at 8 NFEs at matched entropy and a 1.3x lower one at 32, and it is ahead of IDLM at 8 NFEs, and slightly behind at 32. We also sweep the sampling temperature of each student and plot gen-PPL against entropy (Figure 4 of the revised PDF). MLD3 also lowers the gen-PPL of IDLM and SDTT at all matched entropies, by 45% and 61% at 8 NFEs and by 20% and 25% at 32 NFEs, and the curve of MLD3 distilled from IDLM forms the Pareto frontier at 8, 16, and 32 NFEs, below all baselines.


If we have addressed your remaining concerns, we hope you consider raising your score in support of acceptance, and if any concerns remain, we are happy to continue the discussion!

Thank you,

The Authors

Table 4/5: Molecule generation, uniqueness

We additionally report uniqueness as the fraction of distinct molecules among the valid samples.

ModelTeacherQ1Q2Q4Q8Q16Q32Z1Z2Z4Z8Z16Z32
MDLM, our sampler-0.2640.3870.5350.6940.8210.8700.4150.3020.5290.7480.8610.927
MDLM, MDLM sampler-0.2450.3810.5340.6840.8100.8690.5340.2920.4870.7200.8500.932
Di4CMDLM0.2490.3760.549‾\underline{0.549}0.6880.8010.8690.4480.2960.5250.7660.9150.961
SDTT (7 rounds)MDLM0.378‾\underline{0.378}0.1360.2170.4140.6230.7000.2080.2600.3610.5470.7460.866
IDLMMDLM0.2040.1280.2930.5790.7450.8030.4440.338‾\underline{0.338}0.527‾\underline{0.527}0.7340.8690.933
MLD3 (M=1M=1, control)MDLM0.2760.2160.4650.6120.8000.9040.533‾\underline{0.533}0.3840.6720.9320.9890.997
MLD3 (M=4M=4)MDLM0.2620.2690.5220.711‾\underline{0.711}0.862‾\underline{0.862}0.943‾\underline{0.943}0.5350.2490.4350.6620.8940.961
MLD3 (M=8M=8)MDLM0.2700.0960.1770.3270.6720.8620.4700.2130.2940.5450.7890.926
MLD3 (M=8M=8), top-1MDLM0.2390.306‾\underline{0.306}0.5960.7780.9080.9520.4780.2760.5190.846‾\underline{0.846}0.972‾\underline{0.972}0.983‾\underline{0.983}
MLD3 (M=1M=1, control)SDTT0.0550.0390.0710.1030.5140.7990.0900.1280.1240.2880.5970.839
MLD3 (M=8M=8)SDTT0.2500.0130.0180.0520.4660.7670.2500.0600.0830.1920.5360.849
MLD3 (M=8M=8), top-1SDTT0.6670.0220.0180.0750.5360.8040.2610.0900.0960.2390.6310.897
MLD3 (M=1M=1, control)IDLM0.2510.0780.1870.3300.5870.8210.4540.2590.4710.7880.9330.975
MLD3 (M=8M=8)IDLM0.1140.0470.1510.1970.5070.7950.3620.2530.4460.7570.9320.979
MLD3 (M=8M=8), top-1IDLM0.1040.0560.1440.2450.5800.8350.3760.2470.5080.7850.9380.982

Our results on molecular uniqueness show two key trends:

  1. The M=8M=8 student distilled from MDLM has a lower uniqueness than its M=1M=1 control at all NFEs (0.327 against 0.612 on QM9 at 8 NFEs), and hence part of its gain in validity comes from repeated molecules.
  2. With top-1 routing, which costs the same as the baselines, the same student is more diverse from 2 NFEs on both datasets, and reaches the highest uniqueness on QM9 from 4 NFEs (0.778 against 0.694 for the teacher at 8 NFEs).

We note that SDTT and IDLM also lower the uniqueness of the teacher on QM9 from 2 NFEs, and that all students distilled from SDTT, the M=1M=1 control included, repeat many molecules at 2 to 8 NFEs, which suggests that this loss comes from SDTT rather than from the mixture.

Table 5/5: Molecule generation, novelty

Finally, we report novelty as the fraction of distinct molecules absent from the dataset among the valid samples.

ModelTeacherQ1Q2Q4Q8Q16Q32Z1Z2Z4Z8Z16Z32
MDLM, our sampler-0.1990.2650.2600.2620.2860.2470.4150.3020.5290.7480.8600.926
MDLM, MDLM sampler-0.1660.2510.2400.2760.2730.2410.5340.2920.4870.7190.8480.930
Di4CMDLM0.1790.2360.270‾\underline{0.270}0.281‾\underline{0.281}0.2970.2550.4480.2960.5250.7650.9140.959
SDTT (7 rounds)MDLM0.0000.0290.0540.0700.0650.0630.2080.2600.3610.5450.7440.861
IDLMMDLM0.1630.0750.0830.0790.0990.0910.4440.338‾\underline{0.338}0.527‾\underline{0.527}0.7340.8640.929
MLD3 (M=1M=1, control)MDLM0.198‾\underline{0.198}0.1260.2160.2030.2180.2280.533‾\underline{0.533}0.3840.6720.9320.9890.995
MLD3 (M=4M=4)MDLM0.1760.1730.2220.2300.1890.1740.5350.2490.4350.6600.8920.958
MLD3 (M=8M=8)MDLM0.2080.0550.0570.0560.0690.0950.4700.2130.2940.5450.7870.925
MLD3 (M=8M=8), top-1MDLM0.1780.194‾\underline{0.194}0.3190.3230.258‾\underline{0.258}0.253‾\underline{0.253}0.4780.2760.5190.845‾\underline{0.845}0.972‾\underline{0.972}0.980‾\underline{0.980}
MLD3 (M=1M=1, control)SDTT0.0340.0080.0050.0130.0340.0590.0900.1280.1240.2870.5960.834
MLD3 (M=8M=8)SDTT0.0000.0030.0000.0000.0190.0340.2500.0600.0830.1910.5340.844
MLD3 (M=8M=8), top-1SDTT0.0000.0100.0020.0100.0390.0590.2610.0900.0960.2380.6290.894
MLD3 (M=1M=1, control)IDLM0.198‾\underline{0.198}0.0400.0740.0810.0790.0830.4540.2590.4710.7870.9310.969
MLD3 (M=8M=8)IDLM0.0860.0150.0440.0330.0370.0640.3620.2530.4460.7570.9290.972
MLD3 (M=8M=8), top-1IDLM0.0770.0200.0390.0530.0600.0770.3760.2470.5080.7830.9370.977

Our results show that novelty follows uniqueness on ZINC-250k, where nearly all distinct valid molecules are novel. On QM9, novelty stays below 0.33 for all models, since the dataset enumerates its chemical space nearly exhaustively, and top-1 routing reaches the highest novelty at 4 and 8 NFEs.

AuthorsOpenReview ↗

Friendly Reminder as Discussion Period closes tomorrow

Dear Reviewer 2qdJ,

As the discussion period will close tomorrow, we would like to send a friendly reminder to check out our response. We believe that our response establishes MLD3 as a general distillation framework for discrete diffusion and flow models through extensive evaluations with OWT (with longer sequences) and the baselines, and that the requested clarifications in the revised version of the paper address the points raised in your review.

We would greatly appreciate your feedback before the discussion period closes and kindly ask you to reconsider your rating in light of our response.

Thank you,

The Authors

Reviewer_mPvmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces an improved version of Di4C (Hayakawa et al., 2025, Distillation of Discrete Diffusion through Dimensional Correlations) which does not use a surrogate loss. Given a teacher model ψ\psi, they aim at matching the teacher transition ps∣tψ(xs∣xt)p_{s|t}^\psi(x_s | x_t) to a student transition $ p_{s|t}^\theta(x_s | x_t) $ defined as mixture of factorized distributions. The teacher transition is sampled using a rollout trajectory, i.e several iterations of the factorized teacher transition. In Di4C, people bound the $ KL(p_{s|t}^\psi(x_s | x_t) \mid p_{s|t}^\theta(x_s | x_t)) $ by the average over λ\lambda of KL(ps∣tψ(xs∣xt)∣ps∣tθ(xs∣xt,λ))KL(p_{s|t}^\psi(x_s | x_t) \mid p_{s|t}^\theta(x_s | x_t, \lambda)), the second argument being a factorized and tractable distribution. In the work reviewed, they show that this leads to a surrogate collapse, i.e the optimal transition is still factorized. Therefore, they avoid using the surrogate loss by using a finite mixture with learned weights wθw^\theta, with MM components. Everything becomes tractable and differentiable, so that the exact loss can be optimized. Overall, they report improvements compared to Di4C.

Critical strengths

None

Critical weaknesses

I have serious concerns about the method's performance. My claim is that the method might not work even better than Di4C at a similar sampling compute.

  • The experiments are incomplete. The authors only compare their method (MLD3) against Di4C. The latter is not SOTA at all. Even in its paper, the competitive baseline on text generation is SDTT-Di4C, which starts at the SDTT (Deschenaux and Gulcehre, 2025, Beyond Autoregression: Fast LLMs via Self-Distillation Through Time) checkpoint. Indeed, in MLD3, we can see that Di4C does not even beat the teacher at similar NFE on LM1B. This cannot be taken as a baseline, that does not seem fair. However, for image generation (Appendix D), I know that Di4C is indeed competitive. And we can see in MLD3's results that M=1 does not beat Di4C at all. Only M=4 and M=8 are a bit ahead. But the sampling compute budget is bigger because they need to compute two different distributions. The comparison is not exactly fair. I am honestly not sure that the mixture mechanism helps at equivalent sampling budget.

  • The only baseline used is Di4C, which is, again, not SOTA on Discrete Diffusion Distillation. Why didn't the authors compare against IDLM (Li et al., 2026, IDLM: Inverse-distilled Diffusion Language Models) and D-MMD (Hoogeboom et al., 2026, Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD), or even SDTT? If it was the case, I seriously doubt that MLD3 would be ahead of all baselines. For Language Modeling, they could even compare MLD3 with Flow-Maps (FMLM, Lee et al., 2026, Flow Map Language Models), which is SOTA in 1/2 NFE. This is clearly not enough.

  • Experiments don't show results on OpenWebText, which is the main reference for unconditional text generation. It makes it very difficult to compare the results with other baselines, as these don't always show results on LM1B and WikiText.

  • Even the theoretical results do not seem very novel. Theorem 1 is classical. And Theorem 2 shows a lot of results, among which only the third one is not obvious. In my view, the most novel part is the way they managed to generate the weights and the M components with only one architecture. Theoretical results being quite obvious, it would be smarter to put the practical ablations in the main body.

  • The authors did not mention IDLM and D-MMD in their related work nor in the experiments. These are the best performing distillation methods for Masked Diffusion at the moment. This is a serious mistake for a Discrete Diffusion Distillation paper...

Overall, in my view, the quality of the paper does not deserve an acceptance to a high-level conference. It really lacks convincing experiments/baselines. The proposed algorithm does not introduce enough novelty to compensate for such a flaw in evaluations.

Questions for authors
  • In your related work, you did not mention the recent Distribution Distillation IDLM (Li et al., 2026) and D-MMD (Hoogeboom et al., 2026). They are the best performing distillation methods for discrete diffusion, and especially for MDLM, which suits your work perfectly and would be appropriate baselines. Overall, you only used Di4C as a baseline. Honestly, for text generation, it does not even beat the teacher, or barely. Do you think it is an appropriate baseline to evaluate your MLD3 method? Why didn't you include other distillation methods such as SDTT, ReDi (Yoo et al., 2025, ReDi: Rectified Discrete Flow) (even if Duo teacher), and the two IDLM/D-MMD? With these in mind your method does not seem to be SOTA.
2 comments

Dear Reviewer mPvm,

Thank you for your constructive review. We agree that our evaluations were not fully comprehensive, and address each point below with complete comparisons with all requested baselines and tasks (please refer to the global comments and Tables 1 and 2 of the revised PDF). We sincerely hope that our responses establish MLD3 as a strong competitor against both discrete diffusion distillation and flow map baselines.

[W1-a] The sampling compute budget is bigger because they need to compute two different distributions.

We would like to clarify the computational complexity of our sampler with a new ablation. The first 8 of the 12 transformer blocks are shared, whereas only the last 4 blocks run once per component (shared network) for the MM components. Our chosen sampler evaluates 2 components, and hence costs 16 block passes per step against 12 for Di4C, SDTT, and IDLM, which is 1.33x rather than 2x the inference cost. Top-1 routing evaluates only the most likely component at each step at matched inference cost (12 passes) as the baselines (OWT, gen-PPL results as follows):

OWT, NFE481632
Di4C on MDLM (12 passes)1704613238131
MLD3 M=1M=1 control (12 passes)366186135116
MLD3 M=8M=8, top-1 (12 passes)3771469879
MLD3 M=8M=8, default (16 passes)239956452
SDTT + Di4C (12 passes)25283.344.130.8
MLD3 M=8M=8 on SDTT, top-1 (12 passes)77.941.232.328.7

It can be seen that MLD3 still outperforms Di4C and SDTT + Di4C at all NFEs. We also notice gains with top-1 routing on molecule generation (Table 3 in global comments). Our sampler costs two thirds of Di4C per step on ImageNet, since the Di4C student is a 24-block MaskGIT transformer and ours have 12 blocks. MLD3 improves the FID over Di4C at all NFEs (118.7 against 125.5 at 8 NFEs and 100.4 against 117.5 at 32) at this lower cost.

[W1-b] Di4C does not even beat the teacher at similar NFE on LM1B and can't be taken as a baseline.

We agree that the MLD3 text student models in our original submission (Di4C included) were trained from random initialization which made them weaker, and hence we now train them with a warm start from the teacher. Di4C now outperforms its teacher on OWT (131 against 196 at 32 NFEs). MLD3 outperforms SDTT + Di4C at all NFEs and at a similar entropy when distilled from the same SDTT student (40.2 against 83.3 at 8 NFEs), with one round of 10k steps against two rounds of Di4C. Similarly, MLD3 distilled from SDTT outperforms SDTT + Di4C at all NFEs and reaches the lowest gen-PPL of all models from 8 NFEs (even ahead of FMLM) on LM1B dataset (Table 2).

[W2,W3] The only baseline used is Di4C and no evaluation on OWT dataset.

We further include comparisons with the SOTA baselines, such as IDLM, D-MMD, SDTT, SDTT + Di4C, FMLM, DUO-DCD, IDLM-DCD, and DiDi-Instruct on OWT in Table 1. The results show MLD3 as complementary to these methods. We summarize the comparison as follows.

  • MLD3 distilled from MDLM in one round of 30k steps outperforms SDTT at 4 and 8 NFEs, but trails SDTT from 16 NFEs and IDLM from 8 NFEs. However, they sample at a lower entropy than MLD3 (5.34 and 5.14 against 5.57 at 32 NFEs), and at matched entropy, MLD3 outperforms SDTT at 8 to 32 NFEs and is close to IDLM. MLD3 forms the Pareto frontier of gen-PPL against entropy (Figure 4). MLD3 also reaches the lowest gen-PPL of all models at 8, 16, and 32 NFEs (30.4, 19.2, and 14.2, against 71.7, 31.4, and 18.5 for IDLM) when distilled from the IDLM student.
  • D-MMD is the second best model at 4 NFEs (66.1), behind MLD3 distilled from SDTT over six rounds (62.2), while MLD3 distilled from IDLM is better at 16 and 32 NFEs (gen-PPL 19.2 against 67.7 and 14.2 against 72.1).

[W4] Theoretical results do not seem very novel. Theorem 2 shows a lot of results, among which only the third one is not obvious.

We would like to clarify that our main theoretical result is Theorem 2(iii), which shows that the exact likelihood rewards a mixture as soon as the teacher's step carries dependence, while the convexity bound behind Di4C returns a factorized student for any number of components. We also thank you for acknowledging that producing the router and all components from one network is a novel contribution, and the suggestion to move the ablations into the main text.

[W5] No discussion of IDLM and D-MMD.

Thank you for bringing IDLM and D-MMD to our attention as related work. We now include discussion of IDLM, D-MMD, and DiDi-Instruct in the related works (App. A) and compare against them on OWT (Table 1), and against IDLM also on molecules (Table 3).


If we have addressed your concerns, we hope you consider raising your score in support of acceptance, and if any concerns remain, we are happy to continue the discussion!

Thank you,

The Authors

AuthorsOpenReview ↗

Friendly Reminder as Discussion Period closes tomorrow

Dear Reviewer mPvm,

As the discussion period will close tomorrow, we would like to send a friendly reminder to check out our response. We believe that our response establishes MLD3 as a strong competitor to the distillation of discrete diffusion and flow models through extensive evaluations with OWT and the baselines, and that the revised version of the paper addresses the points raised in your review.

We would greatly appreciate your feedback before the discussion period closes and kindly ask you to reconsider your rating in light of our response.

Thank you,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.