- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes NIPSD for few-step CDLM distillation. It combines a real-valued fake score estimator (FiTScore), teacher-trajectory warmup, and perturbations interpolating between generation noise and independent noise. The experiments are conducted with DI and TDM on FLM and LangFlow and show noticeable gains across most setups.
Critical strengths
The paper addresses an important problem: applying score distillation to CDLMs.
It explores why this can be challenging for CDLMs with categorical outputs (FLM and LangFlow). The resulting analysis motivates the proposed method components, and the ablations help show the their benefits in practice. In particular, NIP is well motivated and noticeably improves performance.
The results are promising for both models across 1-4 NFE budgets.
Critical weaknesses
My main concern is that the paper does not sufficiently justify why its proposed components are needed beyond established score distillation methods, e.g., DMD/DMD2/SiD.
-
If I understand correctly, the discussed restrictions on fake-score estimation are not the case for standard endpoint-based score distillation with independent perturbations. Such methods support both one-step and few-step generation through endpoint prediction (e.g., DMD2). Thus, FiTScore requires additional motivation and evidence of its necessity.
-
I am not quite convinced by the analysis distinguishing image and text domains, especially why a sharper loss of content-aligned response in CDLMs implies less variative outputs and worse initialization for score distillation. For now, the initialization limitation seems general for both domains. Moreover, DMD already uses paired teacher-output regression, CausVid uses teacher-trajectory regression for initialization, and DMD2 uses adversarial supervision. Although these techniques were not originally motivated by the specific problem discussed here, they already mitigate it.
-
While NIP is well motivated and interesting, it still remains unclear if it is needed when score distillation is combined with other objectives or advanced techniques. By the way, the idea seems similarly applicable to the image variants.
Questions for authors
-
Could the authors evaluate several more recent score distillation variants (e.g., DMD/DMD2), ablate their components, and demonstrate the additional gains from the proposed techniques? I believe the proposed method should account for already established techniques rather than starting from the first proposed approach like DI.
-
Since I do not currently see the proposed method as inherently specific to CDLMs, it would help to clarify which of the identified difficulties are specific to them. In particular, please elaborate on the analysis comparing text and image models and explain why it suggests that student warmup is more important for CDLMs than for image DMs, given that DMD-like approaches already use similar techniques.
-
(Not critical but still a noticeable weakness) I find the notation in Section 3 difficult to follow and often unnecessarily complicated. I suggest simplifying it to make the main ideas clearer.
-
(Minor) Could the authors add more baselines for LangFlow, e.g., some flow-map method?
-
(Minor) Evaluating embedding-based variants, e.g., FMLM* and ELF, would also help assess whether the findings extend to a broader range of CDLM families.
9 comments
2. Q2 / W2 Initialization and the image–text comparison
Initialization is also challenging in image diffusion distillation, but images and text differ in how their outputs are represented. One of our main contributions is to explain why CDLM students can struggle to recover sample-specific content lost at initialization. We link this difficulty to the different ways image and language teachers represent their denoising predictions. An object’s outline is often enough to recognize it in an image, whereas the meaning of a statement depends on how its words fit together. An image diffusion teacher produces a denoising estimate in a continuous visual space. The CDLMs we study construct this estimate from token probabilities at each position. At high noise levels, a multistep teacher learns to estimate what the clean sample might be, while a one-step student must map each noise input directly to a complete sample. When the input contains little discernible textual information, the student tends to predict token distributions close to the marginal token frequencies in the corpus.
To examine how this difference manifests at initialization, we compared samples from one-step students initialized with teacher weights against real samples, for both text-to-image diffusion models and CDLMs. In separate two-dimensional PCA projections for the two modalities, the one-step samples show some separation from real samples in both cases. The separation is more pronounced for text. View the PCA plots comparing one-step outputs with real data in both image and text models
The samples themselves illustrate how the outputs degrade in each modality. We used one-step sampling from each teacher to reproduce the behavior of the corresponding student at initialization. Although the generated images are blurry, one can still make out a horse or a cat. On OWT-1024, the one-step CDLM outputs consist mostly of repeated words and punctuation and rarely form a complete statement. View one-step image and text samples side by side across four random seeds
The one-step samples illustrate how much sample-specific content can already be lost at initialization. Figure 1 asks how much guidance remains once this variation has been reduced. We contract real samples toward their mean and measure the frozen teacher’s response along the direction that distinguishes each sample from that mean. At , each sample retains 10% of its original deviation from the mean. LangFlow and FLM retain only 1.81% and 0.145% of their respective full-content responses, compared with 9.41% for EDM and 6.75% for DiT. Figure 1b also shows that the estimated state update is less aligned with the direction that would restore the original content in the two CDLMs. This weaker guidance matters when different noise inputs already produce similar outputs. In the limiting case where two inputs yield the same continuous endpoint, independent perturbations give them identical expected output-space feedback from the distribution-matching term at a fixed noise level. The student then receives little guidance from that term on how those outputs should differ. This motivates giving each input a distinct teacher-trajectory target during warmup.
In our CDLM experiments, we therefore use CD to warm up the student before matching distributions by minimizing the reverse KL divergence. During subsequent distillation, NIP preserves the link between each output and the noise that generated it. This can give the student an update direction that allows its outputs to differ across inputs while bringing the distribution of its noisy states closer to the teacher’s distribution at the same noise level.
Table 5 examines the roles of CD warmup and NIP in practice. On OWT-1024, students trained directly from the teacher weights have repeated 4-gram rates of 98.36% with independent perturbations and 98.34% with NIP. With CD warmup, these rates fall to 0.29% and 0.43%, respectively. CD warmup thus alleviates the severe repetition observed with direct teacher initialization. Given the same warmup, NIP provides a complementary gain in generative perplexity, lowering GenPPL from 266.29 with independent perturbations to 231.36.
3. W1 When FiTScore is needed
In our one-step CDLM experiments, DI and SiD produced highly repetitive text, and DMD’s outputs remained fragmented and repetitive even with paired teacher regression (Q1). These observations motivated us to examine how the distillation signal can distinguish different noise inputs when the student produces similar outputs.
Consider a CDLM endpoint that lies in the token convex hull at each position. Its conditional mean also lies in the hull for any score query . Under independent Gaussian corruption, this conditional mean is exactly the effective mean required by Eq. (5) to recover the student score (Appendix D.1). A categorical head therefore has sufficient output range for independently perturbed endpoints. This holds for both one-step generation and few-step endpoint prediction of the kind illustrated by DMD2.
To see why retaining the generation noise can matter, consider two noise inputs that produce the same continuous endpoint. At a fixed noise level, independent perturbations give them identical expected output-space feedback from the distribution-matching term. One-step NIP mixes the noise that generated each endpoint with fresh Gaussian noise, so the expected feedback can depend on which input produced it. Table 5 tests the value of this pairing. After the same teacher-trajectory warmup, one-step LangFlow–DI on OWT-1024 achieved a GenPPL of 266.29 with independent perturbations and 231.36 with NIP. Shuffling the source noise across samples preserved its marginal distribution but broke the pairing, and GenPPL rose to 261.59. All three settings had repeated 4-gram rates below 0.5%.
For NIP queries, the effective mean in Eq. (5) can differ from and fall outside the token convex hull. At such queries, a categorical head incurs an irreducible error in estimating the score. The same issue can arise in few-step TDM, where we perturb a generated intermediate state that retains information from its trajectory (Section 4.3). Appendix I detected violations of the hull condition in both FLM and LangFlow after Bonferroni correction, with none detected in the independent-corruption controls. FiTScore’s direct ε head uses a real-valued output in embedding space to predict the fresh Gaussian component of the query. The conditional mean of this component determines the query score through Eq. (12).
We added the direct ε head to the separate fixed-budget LangFlow ablation in Appendix I, using the same NIP setup and training budget for all three heads. The table reports differences in GenPPL on OWT-1024 across settings with comparable sample entropy.
| Setting | NFE | Categorical DSM | FiTScore Signed- residual | FiTScore Direct |
|---|---|---|---|---|
| NIPSD | 1 | 201.01 | 231.36 | 174.20 |
| NIPSD | 4 | 150.43 | 157.12 | 136.74 |
When the effective mean in Eq. (5) falls outside the token convex hull, a categorical head incurs an irreducible error in estimating the score at that query. In our matched-budget LangFlow experiments with NIP, FiTScore’s direct ε head achieved the lowest GenPPL among the three heads tested, improving on categorical DSM by 13.34% for one-step generation and 9.10% for four-step generation.
Thank you for your detailed feedback. We address your questions and concerns below.
1. Q1 Stronger baselines
DI, DMD, and SiD all use score information from pretrained diffusion models to match distributions and distill a teacher into a one-step generator. DI was the first of these methods to formulate distillation as minimizing a weighted integral of reverse KL divergences over noise levels. DMD likewise uses reverse KL for distribution matching and adds a regression loss on noise–image pairs obtained from the teacher’s ODE sampler. DMD2 removed this paired regression loss, and subsequent methods such as TDM likewise train without noise–image pairs from the teacher’s ODE sampler. SiD takes the Fisher divergence under the generator’s distribution as its theoretical starting point. DMD2 and TDM extend reverse-KL-based score distillation to few-step generation. TDM is the more recent of the two and reports faster convergence and a lower teacher–student FID than DMD2. We therefore chose DI and TDM as representative methods for one-step and few-step reverse-KL-based distillation, respectively. Although DMD and SiD perform well in image generation, directly applying these methods to CDLMs can lead to collapse, with the generator repeatedly emitting the same few tokens. To examine this transfer empirically, we adapted DMD, SiD, and DMD2 to continuous diffusion language modeling and evaluated them on OpenWebText with sequences of length 1,024 (OWT-1024). We include the one-step DI and NIPSD results for comparison.
The following excerpts illustrate how these differences appear in the generated text.
At one NFE, DI and SiD obtain very low GenPPL while repeatedly producing the same few tokens. DMD uses a wider range of words, but even with paired teacher regression its output remains repetitive and fragmented. These examples illustrate the difficulty of transferring one-step image distillation methods directly to this CDLM setting. Our DI-based NIPSD student produces a less repetitive passage and reaches a mean within-sample unigram entropy of 5.24 nats, close to the LangFlow teacher’s 5.25 at 1,024 function evaluations. At four NFEs, both DMD2 and NIPSD retain substantial token diversity. With the same LangFlow teacher and evaluation protocol, NIPSD achieves 54.8% lower GenPPL than DMD2 (48.06 vs. 106.42), with similar mean within-sample unigram entropy (5.25 vs. 5.17 nats). Tables 3 and 5 isolate the contribution of NIP within DI, and the comparison in W1 tests the choice of score head with NIP held fixed. These ablations complement the comparisons of the full methods above.