acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Enabling Score Distillation in Continuous Diffusion Language Models

Abstract

Score Distillation (e.g., Diff-Instruct and Trajectory Distribution Matching) has substantially accelerated image and video diffusion models, yet transferring it to continuous diffusion language models requires more than reusing the training objective. We identify a central distinction in the relationship between denoising predictions and generated samples: in categorically parameterized language models, continuous distillation updates act on expected token embeddings, whereas generation ultimately requires discrete token decisions. Consequently, increasing prediction confidence can either improve generation or collapse different samples onto a small set of tokens. We propose a unified perspective: effective language distillation should resolve token-level uncertainty while preserving noise-dependent variation across generated samples. A denoiser's conditional mean should therefore not be assumed to provide a suitable generator initialization. This perspective unifies one-step endpoint matching and few-step trajectory matching, guiding generator initialization and score modeling of the student's generated distribution. Building on these principles, we adapt DI and TDM to continuous diffusion language models, enabling effective one-step and few-step generation on LM1B and OpenWebText. Our work clarifies the key considerations that distinguish distribution-matching distillation for language and provides reusable principles for transferring image-diffusion distillation methods to language generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continuous diffusion language model, few-step generation, score distillation, diff-instruct, trajectory distribution matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FWoMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes NIPSD for few-step CDLM distillation. It combines a real-valued fake score estimator (FiTScore), teacher-trajectory warmup, and perturbations interpolating between generation noise and independent noise. The experiments are conducted with DI and TDM on FLM and LangFlow and show noticeable gains across most setups.

Critical strengths

The paper addresses an important problem: applying score distillation to CDLMs.

It explores why this can be challenging for CDLMs with categorical outputs (FLM and LangFlow). The resulting analysis motivates the proposed method components, and the ablations help show the their benefits in practice. In particular, NIP is well motivated and noticeably improves performance.

The results are promising for both models across 1-4 NFE budgets.

Critical weaknesses

My main concern is that the paper does not sufficiently justify why its proposed components are needed beyond established score distillation methods, e.g., DMD/DMD2/SiD.

  1. If I understand correctly, the discussed restrictions on fake-score estimation are not the case for standard endpoint-based score distillation with independent perturbations. Such methods support both one-step and few-step generation through endpoint prediction (e.g., DMD2). Thus, FiTScore requires additional motivation and evidence of its necessity.

  2. I am not quite convinced by the analysis distinguishing image and text domains, especially why a sharper loss of content-aligned response in CDLMs implies less variative outputs and worse initialization for score distillation. For now, the initialization limitation seems general for both domains. Moreover, DMD already uses paired teacher-output regression, CausVid uses teacher-trajectory regression for initialization, and DMD2 uses adversarial supervision. Although these techniques were not originally motivated by the specific problem discussed here, they already mitigate it.

  3. While NIP is well motivated and interesting, it still remains unclear if it is needed when score distillation is combined with other objectives or advanced techniques. By the way, the idea seems similarly applicable to the image variants.

Questions for authors
  1. Could the authors evaluate several more recent score distillation variants (e.g., DMD/DMD2), ablate their components, and demonstrate the additional gains from the proposed techniques? I believe the proposed method should account for already established techniques rather than starting from the first proposed approach like DI.

  2. Since I do not currently see the proposed method as inherently specific to CDLMs, it would help to clarify which of the identified difficulties are specific to them. In particular, please elaborate on the analysis comparing text and image models and explain why it suggests that student warmup is more important for CDLMs than for image DMs, given that DMD-like approaches already use similar techniques.

  3. (Not critical but still a noticeable weakness) I find the notation in Section 3 difficult to follow and often unnecessarily complicated. I suggest simplifying it to make the main ideas clearer.

  4. (Minor) Could the authors add more baselines for LangFlow, e.g., some flow-map method?

  5. (Minor) Evaluating embedding-based variants, e.g., FMLM* and ELF, would also help assess whether the findings extend to a broader range of CDLM families.

9 comments

Thank you for your detailed feedback. We address your questions and concerns below.

1. Q1 Stronger baselines

DI, DMD, and SiD all use score information from pretrained diffusion models to match distributions and distill a teacher into a one-step generator. DI was the first of these methods to formulate distillation as minimizing a weighted integral of reverse KL divergences over noise levels. DMD likewise uses reverse KL for distribution matching and adds a regression loss on noise–image pairs obtained from the teacher’s ODE sampler. DMD2 removed this paired regression loss, and subsequent methods such as TDM likewise train without noise–image pairs from the teacher’s ODE sampler. SiD takes the Fisher divergence under the generator’s distribution as its theoretical starting point. DMD2 and TDM extend reverse-KL-based score distillation to few-step generation. TDM is the more recent of the two and reports faster convergence and a lower teacher–student FID than DMD2. We therefore chose DI and TDM as representative methods for one-step and few-step reverse-KL-based distillation, respectively. Although DMD and SiD perform well in image generation, directly applying these methods to CDLMs can lead to collapse, with the generator repeatedly emitting the same few tokens. To examine this transfer empirically, we adapted DMD, SiD, and DMD2 to continuous diffusion language modeling and evaluated them on OpenWebText with sequences of length 1,024 (OWT-1024). We include the one-step DI and NIPSD results for comparison.

MethodNFEGenPPL ↓Sample entropy
DI11.280.04
DMD151.782.82
SiD13.670.71
NIPSD170.085.24
DMD24106.425.17
NIPSD448.065.25

The following excerpts illustrate how these differences appear in the generated text.

  • DI NFE=1 <|endoftext|> the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the ......
  • DMD NFE=1 <|endoftext|> to to the Cruz. in o. a few percent,, the man of to be in the of the- to the the..s. that ctuary the �-old that have to the the the the of that to to the in of the of, and-,,�. and said. He is. a of " the," in of the� of the people � " the� a�t was the- and young.,,. the problem for that the is the that, for,. the can be, and to to, and the Hunt,�. to will ......
  • SiD NFE=1 <|endoftext|> the the the the the the, the, the the the the the the the the the the the the the the the the the, the the the the the the the the the the, the the the the the the the, the the the, the the the the the the the the the the the the the the the the the the the the the the the, the the the the ......
  • NIPSD NFE=1 <|endoftext|> They’re working here in China because they are have any money for the same time, because They have a lot of companies who tell me there are no “very many companies” in a “new company” don’t come back to them when they work here because they have a lot more than us. They don’t have more than 10% and they want to at any more than one (1) or “big companies”)......
  • DMD2 NFE=4 <|endoftext|> policies of the American Family Movement, carried up by members of the Tea Party part of the ads, include Mitsobino's ProjectV4 Vehicle Target Home. National Crime Law. Counter Watch claimed that "sicious intoxicated and drug-like males were recorded in 2014 on officers involving public safety". ......
  • NIPSD NFE=4 <|endoftext|> surgery because he was not going to get away from a season with the Cubs back in two years. The surgery didn’t lead to his recovery, as Bisby said. But it’s still not like he was able to stay in St. City, and it still wasn’t as that Bisby was leaving. “We all know that he’s coming out for a year now, and so our momentum has changed,” he said. ......

At one NFE, DI and SiD obtain very low GenPPL while repeatedly producing the same few tokens. DMD uses a wider range of words, but even with paired teacher regression its output remains repetitive and fragmented. These examples illustrate the difficulty of transferring one-step image distillation methods directly to this CDLM setting. Our DI-based NIPSD student produces a less repetitive passage and reaches a mean within-sample unigram entropy of 5.24 nats, close to the LangFlow teacher’s 5.25 at 1,024 function evaluations. At four NFEs, both DMD2 and NIPSD retain substantial token diversity. With the same LangFlow teacher and evaluation protocol, NIPSD achieves 54.8% lower GenPPL than DMD2 (48.06 vs. 106.42), with similar mean within-sample unigram entropy (5.25 vs. 5.17 nats). Tables 3 and 5 isolate the contribution of NIP within DI, and the comparison in W1 tests the choice of score head with NIP held fixed. These ablations complement the comparisons of the full methods above.

2. Q2 / W2 Initialization and the image–text comparison

Initialization is also challenging in image diffusion distillation, but images and text differ in how their outputs are represented. One of our main contributions is to explain why CDLM students can struggle to recover sample-specific content lost at initialization. We link this difficulty to the different ways image and language teachers represent their denoising predictions. An object’s outline is often enough to recognize it in an image, whereas the meaning of a statement depends on how its words fit together. An image diffusion teacher produces a denoising estimate in a continuous visual space. The CDLMs we study construct this estimate from token probabilities at each position. At high noise levels, a multistep teacher learns to estimate what the clean sample might be, while a one-step student must map each noise input directly to a complete sample. When the input contains little discernible textual information, the student tends to predict token distributions close to the marginal token frequencies in the corpus.

To examine how this difference manifests at initialization, we compared samples from one-step students initialized with teacher weights against real samples, for both text-to-image diffusion models and CDLMs. In separate two-dimensional PCA projections for the two modalities, the one-step samples show some separation from real samples in both cases. The separation is more pronounced for text. View the PCA plots comparing one-step outputs with real data in both image and text models

The samples themselves illustrate how the outputs degrade in each modality. We used one-step sampling from each teacher to reproduce the behavior of the corresponding student at initialization. Although the generated images are blurry, one can still make out a horse or a cat. On OWT-1024, the one-step CDLM outputs consist mostly of repeated words and punctuation and rarely form a complete statement. View one-step image and text samples side by side across four random seeds

The one-step samples illustrate how much sample-specific content can already be lost at initialization. Figure 1 asks how much guidance remains once this variation has been reduced. We contract real samples toward their mean and measure the frozen teacher’s response along the direction that distinguishes each sample from that mean. At λ=0.1\lambda=0.1, each sample retains 10% of its original deviation from the mean. LangFlow and FLM retain only 1.81% and 0.145% of their respective full-content responses, compared with 9.41% for EDM and 6.75% for DiT. Figure 1b also shows that the estimated state update is less aligned with the direction that would restore the original content in the two CDLMs. This weaker guidance matters when different noise inputs already produce similar outputs. In the limiting case where two inputs yield the same continuous endpoint, independent perturbations give them identical expected output-space feedback from the distribution-matching term at a fixed noise level. The student then receives little guidance from that term on how those outputs should differ. This motivates giving each input a distinct teacher-trajectory target during warmup.

In our CDLM experiments, we therefore use CD to warm up the student before matching distributions by minimizing the reverse KL divergence. During subsequent distillation, NIP preserves the link between each output and the noise that generated it. This can give the student an update direction that allows its outputs to differ across inputs while bringing the distribution of its noisy states closer to the teacher’s distribution at the same noise level.

Table 5 examines the roles of CD warmup and NIP in practice. On OWT-1024, students trained directly from the teacher weights have repeated 4-gram rates of 98.36% with independent perturbations and 98.34% with NIP. With CD warmup, these rates fall to 0.29% and 0.43%, respectively. CD warmup thus alleviates the severe repetition observed with direct teacher initialization. Given the same warmup, NIP provides a complementary gain in generative perplexity, lowering GenPPL from 266.29 with independent perturbations to 231.36.

3. W1 When FiTScore is needed

In our one-step CDLM experiments, DI and SiD produced highly repetitive text, and DMD’s outputs remained fragmented and repetitive even with paired teacher regression (Q1). These observations motivated us to examine how the distillation signal can distinguish different noise inputs when the student produces similar outputs.

Consider a CDLM endpoint MM that lies in the token convex hull at each position. Its conditional mean E[M∣Y]\mathbb{E}[M\mid Y] also lies in the hull for any score query YY. Under independent Gaussian corruption, this conditional mean is exactly the effective mean required by Eq. (5) to recover the student score (Appendix D.1). A categorical head therefore has sufficient output range for independently perturbed endpoints. This holds for both one-step generation and few-step endpoint prediction of the kind illustrated by DMD2.

To see why retaining the generation noise can matter, consider two noise inputs that produce the same continuous endpoint. At a fixed noise level, independent perturbations give them identical expected output-space feedback from the distribution-matching term. One-step NIP mixes the noise that generated each endpoint with fresh Gaussian noise, so the expected feedback can depend on which input produced it. Table 5 tests the value of this pairing. After the same teacher-trajectory warmup, one-step LangFlow–DI on OWT-1024 achieved a GenPPL of 266.29 with independent perturbations and 231.36 with NIP. Shuffling the source noise across samples preserved its marginal distribution but broke the pairing, and GenPPL rose to 261.59. All three settings had repeated 4-gram rates below 0.5%.

For NIP queries, the effective mean in Eq. (5) can differ from E[M∣Y]\mathbb{E}[M\mid Y] and fall outside the token convex hull. At such queries, a categorical head incurs an irreducible error in estimating the score. The same issue can arise in few-step TDM, where we perturb a generated intermediate state that retains information from its trajectory (Section 4.3). Appendix I detected violations of the hull condition in both FLM and LangFlow after Bonferroni correction, with none detected in the independent-corruption controls. FiTScore’s direct ε head uses a real-valued output in embedding space to predict the fresh Gaussian component of the query. The conditional mean of this component determines the query score through Eq. (12).

We added the direct ε head to the separate fixed-budget LangFlow ablation in Appendix I, using the same NIP setup and training budget for all three heads. The table reports differences in GenPPL on OWT-1024 across settings with comparable sample entropy.

SettingNFECategorical DSMFiTScore Signed-ϵ\epsilon residualFiTScore Direct ϵ\epsilon
NIPSD1201.01231.36174.20
NIPSD4150.43157.12136.74

When the effective mean in Eq. (5) falls outside the token convex hull, a categorical head incurs an irreducible error in estimating the score at that query. In our matched-budget LangFlow experiments with NIP, FiTScore’s direct ε head achieved the lowest GenPPL among the three heads tested, improving on categorical DSM by 13.34% for one-step generation and 9.10% for four-step generation.

Reviewer_DpGzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks why DMD-style score distillation (DI, TDM) fails to transfer from images to continuous diffusion language models such as LangFlow and FLM. It identifies two causes:

  • a categorical fake-score head cannot represent the scores of generated states (Prop. 2);
  • the teacher's signal is weak when the student's outputs collapse to the same text (Fig. 1).

The proposed NIPSD has three parts:

  • FiTScore: an ε-prediction fake score trained on the actual queries;
  • a consistency-distillation warm-up on teacher rollouts;
  • NIP: generated states are re-noised while keeping part of their generation noise.

On OpenWebText, NIPSD improves FLM's Gen PPL over FMLM by 21–41% at 1–4 steps. LangFlow+NIPSD reaches 70.1 at one step and 48.1 at four, with entropy around 5.2.

Critical strengths
  1. A careful diagnosis of why image recipes fail for these models: the convex-hull limit of categorical heads (Prop. 2), a gradient sign flip (App. H), and controlled diagnostics. Together they give reusable guidance.
  2. Strong one-step results. They are also ahead of two cited methods missing from Table 1: ELF+PD (136.1) and self-conditioned FMLM (113.9).
Critical weaknesses
  • Metrics. Only Gen PPL and within-sample unigram entropy are reported, with no MAUVE, cross-sample diversity or human judgment. This matters because the paper's guiding principle is to "preserve variation across generated samples", which neither metric measures (App. E). The one-step samples in App. K are also largely incoherent.
    • Baselines. There is no same-teacher LangFlow baseline, and results are single runs.

** Prior work. **

  • DLM-One already distils continuous diffusion LMs to one step with score distillation and hits the same repetition failure.
  • DiDi-Instruct and Di[M]O apply distribution matching to diffusion LMs.
  • CDCD introduced the categorical head analysed here.
  • The NIP kernel is exactly Phased DMD's Eq. 2, and FiTScore resembles its subinterval score matching. What's new is the noise reuse (κ) and the analysis.
Questions for authors

I lean towards acceptance (weak accept); the following clarifications would support that score:

  1. How do NIPSD, FiTScore and NIP differ from DLM-One and Phased DMD? Please position the work against the prior work in Weakness 2, and qualify the claim that score-based distillation "remains underexplored in language".
  2. App. C.4 shows that reusing the generation noise moves the one-step optimum away from the data distribution unless κ→0. How large is this bias as a function of κ? Which κ do you use, and does NIP still help when κ is small?
2 comments

Thank you for your helpful feedback.

1. Q1/W2 Relation to existing methods

To our knowledge, limited studies directly address score distillation for CDLMs, and DLM-One is the only such study among the works you cite. However, its unconditional generation results report only GenPPL, which can be misleading because repetitive outputs may achieve low GenPPL. We believe this area remains underexplored because existing work has yet to explain how differences between images and language affect the distillation signal. As shown in the table, we found that directly transferring score distillation methods from the image domain does not work well. For example, applying either DMD or SiD directly on OWT-1024 leads the model to generate the same few tokens repeatedly. Image denoisers typically predict continuous features directly, whereas the CDLMs we study derive their continuous predictions from token probabilities. These differences led us to examine which scores the student can represent during CDLM distillation and how the teacher’s supervision varies with input content. This analysis guided the design of NIPSD.

MethodNFEGenPPL ↓Sample entropy
DI11.280.04
DMD151.782.82
SiD13.670.71
NIPSD170.085.24
DMD24106.425.17
NIPSD448.065.25

DLM-One adapts SiD to CDLMs and uses adversarial regularization to mitigate degeneration. This requires real text samples to train the discriminator and makes optimization more demanding, since the generator must balance the SiD objective against feedback from a continually updated discriminator. In our OWT-1024 experiments, SiD alone led to collapse, with the student repeatedly generating the same few tokens. Directly transferring DMD2 yielded poor text quality in our CDLM experiments. We also note that DLM-One does not report sample entropy alongside Gen PPL in its unconditional generation comparison. Repetitive outputs can have low GenPPL. Our design of NIPSD starts from the differences between image and language distillation. When student generators are initialized directly from their teachers’ weights, language models are more prone than image models to producing repetitive outputs that are unrepresentative of real data. This motivated our use of a CD warmup to initialize the student. We then match the student distribution to the teacher distribution by minimizing the reverse KL divergence, using NIP to keep each distillation query coupled to the source noise used to generate its sample.

Phased DMD follows TDM in extending distribution matching to intermediate states along a multistep generation trajectory. Phased DMD applies this idea at successive phase boundaries. For a single endpoint with independent perturbation noise, its distribution-matching update takes the Diff-Instruct form. NIPSD provides a unified formulation of one-step endpoint distillation and multistep trajectory distillation by making the coupling to the source noise explicit. The supervision provided by distribution matching depends both on the perturbation noise level and on how the perturbation noise is coupled to the source noise used for generation. Even when the generation trajectory consists of a single endpoint prediction, NIPSD can vary this coupling while keeping the Gaussian marginal distribution of the perturbation noise fixed. FiTScore addresses the resulting score-estimation problem in CDLMs. When source coupling puts the required denoising mean outside the token convex hull, FiTScore uses a real-valued auxiliary head trained on the fresh noise added to each query to estimate the student’s score.

DiDi-Instruct and Di[M]O study the distillation of discrete masked diffusion models over text tokens and image tokens, respectively. Masked diffusion is a form of discrete diffusion in which the forward process replaces tokens with a mask symbol and the reverse process samples token identities at masked positions. In the CDLMs we study, Gaussian noise perturbs real-valued token representations. The model denoises these vectors in continuous space before decoding them into discrete tokens. This difference in state space determines what each method matches during distillation. DiDi-Instruct and Di[M]O match distributions over discrete tokens, whereas our method matches distributions over noisy, continuous representations along the generation trajectory.

CDCD addresses data denoising by converting the posterior over clean tokens into the continuous score of noisy token embeddings. Our NIPSD starts from a fundamentally different problem in reverse KL distillation, where updating the generator requires the score of the student’s own query distribution. When these queries retain source noise, the required denoising mean can lie outside the token convex hull. In the example in Appendix H, even an optimal categorical head then yields an update that increases the true KL divergence.

2. Q2 Noise-reuse bias and the choice of κ\kappa

How large is the bias? In the binary example of Appendix C.4, the endpoint G(U)=sign⁡(U)G(U)=\operatorname{sign}(U) with U∼N(0,1)U\sim\mathcal N(0,1) has the target binary distribution. Reusing UU increases the query variance by exactly 2aσκ2/π2a\sigma\kappa\sqrt{2/\pi} relative to independent perturbation. For any scalar generator whose population query law at a fixed noise level exactly matches that of the independently perturbed data, Gaussian deconvolution and the triangle inequality give W1(PG,Pdata)≤2σκa2π,W_1(P_G,P_{\mathrm{data}})\leq \frac{2\sigma\kappa}{a}\sqrt{\frac{2}{\pi}}, where PGP_G is the generator’s endpoint distribution. At a fixed noise-to-signal ratio σ/a\sigma/a, this upper bound is proportional to κ\kappa.

How do we choose κ\kappa? For one-step LangFlow on OWT-1024, we divide the minimum noise-to-signal ratio in the teacher’s proposal by the query’s noise-to-signal ratio and cap κ\kappa at 0.875≈0.935\sqrt{0.875}\approx 0.935.

Does small κ\kappa help? In the binary example, fix any positive κ<1\kappa<1, however small. Starting from the collapsed point θ=0\theta=0, a sufficiently small increase in θ\theta lowers the query KL divergence (Eq. 33). The initial slope is proportional to −κ-\kappa and vanishes under independent noise (κ=0\kappa=0). Our LangFlow experiments in Table 5 evaluate the default NIP schedule. With the same trajectory initialization, default NIP achieves a mean GenPPL of 231.36 across three training seeds, compared with 266.29 for independent noise and 261.59 for shuffled source noise.

3. W1 Metrics

As noted in Appendix E, the unigram entropy reported alongside GenPPL in Table 1 measures variation in token use within each passage. To measure variation across decoded passages, we computed the generalized Jensen–Shannon divergence among their token unigram distributions (JSD_rows, in nats) for each method and inference budget. A higher value means that token frequencies differ more from one passage to another.

We also evaluated overall text quality through a blinded pairwise comparison using Qwen3.8-27B. At each inference budget, the judge evaluated 1,024 pairs, each containing one NIPSD passage and one FMLM passage. We withheld the method names and presented each pair in both orders to reduce sensitivity to presentation order. We averaged the two judgments within each pair.

NFEMethodJSD_rows ↑Preference Rate ↑
1FMLM0.74622.85%
1NIPSD0.86577.15%
4FMLM1.1218.95%
4NIPSD1.47781.05%

NIPSD’s decoded passages show greater cross-sample lexical diversity than FMLM’s at both budgets. The JSD gap is 0.119 nats at one NFE and 0.357 nats at four NFE.

To examine passage coherence directly, we also report automated ratings of fluency and coherence on a five-point scale. The teacher scores provide a reference across inference budgets. The final column is the mean of the two ratings.

ModelNFEFluency ↑Coherence ↑Average ↑
Teacher11.0001.0001.000
Teacher81.2891.1831.236
Teacher161.5981.2191.409
Teacher641.9381.8091.874
Teacher1281.9451.8481.897
Teacher2561.9771.8841.931
Teacher10241.9841.8981.941
FMLM11.0311.0011.016
FMLM41.5861.1821.384
NIPSD11.6331.1721.403
NIPSD41.9841.8371.911

On OWT-1024, Qwen3.8-27B preferred passages from NIPSD over those from FMLM in 77.15% of the one-NFE comparisons and 81.05% of the four-NFE comparisons. At one NFE, the clearest gain was in fluency. At four NFEs, NIPSD also showed a larger gain in coherence.

The new one-step DMD and SiD baselines use the same LangFlow teacher and evaluation protocol as NIPSD. At one NFE, NIPSD avoids the severe token repetition observed with SiD and maintains a mean within-sample unigram entropy of 5.24 nats, compared with 2.82 for DMD and about 0.7 for SiD. At four NFEs, NIPSD achieves 54.8% lower GenPPL than DMD2 (48.06 vs. 106.42) at comparable entropy (5.25 vs. 5.17 nats). For flow-map distillation, Table 1 provides a separate comparison between NIPSD and FMLM using the same FLM teacher. At 1, 2, and 4 NFEs, NIPSD reduces GenPPL relative to FMLM by 21.0%, 40.9%, and 40.4%, respectively.

We plan to extend this comparison to LangFlow in follow-up work. We will evaluate NIPSD against a flow-map student distilled from the same LangFlow teacher, using matched sampling budgets and multiple training seeds for both methods.

Reviewer_6VzTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the challenges of applying score distillation methods, such as Diff-Instruct and TDM, to continuous diffusion language models. The authors identify two key issues: limitations of categorical predictions for score estimation and reduced generation diversity caused by token averaging. To address these issues, they propose NIPSD, which combines teacher-trajectory initialization, noise interpolation, and FiTScore. Experiments on LM1B and OpenWebText demonstrate improved one-step and few-step generation performance compared to existing distillation methods.

Critical strengths
  1. The paper provides a thoughtful analysis of why score distillation methods developed for image diffusion cannot be directly transferred to language models. In particular, the discussion of categorical representations and score estimation offers useful insights beyond a straightforward adaptation of existing methods.

  2. The experimental results are promising. The method is validated on two different CDLMs, with ablation studies supporting the proposed design choices. In particular, the one-step LangFlow model initialized from a distilled two-step generator achieves perplexity close to that of the 512-step teacher, demonstrating the practical potential of the approach.

Critical weaknesses
  1. The evaluation of generation quality is somewhat limited. The experiments mainly rely on generative perplexity and unigram entropy, which do not fully capture semantic coherence or diversity across samples. Some generated examples in Appendix K still appear unnatural or repetitive.

  2. The experiments are limited to 179M-parameter models and unconditional text generation, so it remains unclear how well the method generalizes to larger models or conditional generation tasks. In addition, Appendix I shows that categorical DSM sometimes outperforms the signed-epsilon variant, which raises questions about the consistency of FiTScore's advantages.

Questions for authors
  1. Could the authors provide additional evaluations of text coherence, diversity across samples, or human judgments to better demonstrate whether the perplexity improvements translate into better generation quality?

  2. In Appendix I, categorical DSM sometimes performs better than the signed-epsilon variant. How do the authors explain this observation, and under what conditions does FiTScore consistently improve performance?

2 comments

Thank you for the constructive feedback on both the analysis and the generation results.

1. Q1 / W1 Generation quality and diversity

Our main evaluation follows the practice used in prior diffusion language modeling work, including FMLM (Lee et al., 2026): we report GPT-2 Large GenPPL together with mean per-sample unigram entropy. This preserves comparability with existing baselines and helps identify repetitive outputs that can yield misleadingly low perplexity. In the within-FLM comparison in Table 1, NIPSD reduces GenPPL by 21–41% relative to FMLM at 1–4 NFE with comparable entropy. To measure the diversity across samples, we introduced the JSD_rows metric, which is the generalized Jensen–Shannon divergence among per-sample token unigram distributions, in nats. To simulate human judgment, we selected Qwen3.8-27B as the expert model and conducted blind evaluations on the generation quality of our method and the baseline method. In a blinded pairwise evaluation, Qwen3.8-27B compared pairs of passages, each containing one NIPSD passage and one FMLM passage generated at the same NFE. We hid the method names, presented each pair in both orders, and averaged the two judgments to reduce sensitivity to presentation order.

NFEMethodJSD_rows ↑Preference Rate ↑
1FMLM0.74622.85%
1NIPSD0.86577.15%
4FMLM1.1218.95%
4NIPSD1.47781.05%

Qwen3.8-27B also rated individual passages generated by the teacher, FMLM, and NIPSD, assigning separate scores from 1 to 5 for fluency and coherence. For each model and inference budget, we report the mean score for each dimension and the arithmetic mean of those two scores in the Average column.

ModelNFEFluency ↑Coherence ↑Average ↑
Teacher11.0001.0001.000
Teacher81.2891.1831.236
Teacher161.5981.2191.409
Teacher641.9381.8091.874
Teacher1281.9451.8481.897
Teacher2561.9771.8841.931
Teacher10241.9841.8981.941
FMLM11.0311.0011.016
FMLM41.5861.1821.384
NIPSD11.6331.1721.403
NIPSD41.9841.8371.911

The experimental results show that our method outperforms the FMLM distillation method in terms of cross-sample diversity. Moreover, in the Qwen3.8-27B simulation of human judgment, our method also outperforms FMLM. On the fluency and coherence evaluation, one-step and four-step NIPSD achieved ratings comparable to those of the teacher at higher sampling budgets.

2. Q2 / W2 (FiTScore): Categorical DSM and score parameterization

In our analysis experiments with LangFlow as the teacher, FiTScore’s direct ϵ\epsilon head achieved lower GenPPL than both categorical DSM and the signed-epsilon residual variant in one-step generation and four-step generation. These new results extend the comparison in Appendix I.

The two FiTScore heads use the same fresh-noise regression target but fit it differently. Both FiTScore heads can help when a student query requires an effective mean outside the convex hull of the token embeddings. A categorical head cannot represent that mean exactly. LangFlow adds Gaussian noise to continuous token embeddings, so the auxiliary head described in Appendix F.2 predicts the added noise ϵ\epsilon directly in those coordinates. The signed-epsilon variant first converts predicted token probabilities into a noise estimate and then learns a residual correction. When that correction makes an otherwise useful score estimate less accurate on states generated by the student, it can hurt the generator update. This offers a possible explanation for the residual variant’s higher GenPPL in Appendix I.

The table compares the three parameterizations at the same training budget. Each configuration used 1,000 steps of CD warmup, 512 fake-only steps for the auxiliary model, and 1,024 steps of joint training. We used GPT-2 Large to compare GenPPL across settings with similar sample entropy (lower GenPPL is better).

SettingNFECategorical DSMFiTScore Signed-ϵ\epsilon residualFiTScore Direct ϵ\epsilon
NIPSD1201.01231.36174.20
NIPSD4150.43157.12136.74

Relative to categorical DSM, the direct ϵ\epsilon head achieved 13.34% lower GenPPL for one-step generation and 9.10% lower GenPPL for four-step generation. We will include both results in the revised manuscript and specify the auxiliary head and training setup for each configuration.

Regarding model scale and conditional generation, we evaluated unconditional text generation using the 179M-parameter versions of FLM and LangFlow. In future work, we plan to evaluate NIPSD on larger continuous diffusion language models and extend the evaluation to conditional text generation tasks to test whether the distillation gains observed here carry over to these settings.

Comments on the paper

1 comment

We thank the Area Chair and all three reviewers for their careful reading of our work on score distillation for continuous diffusion language models (CDLMs).

Our work starts from the fundamental question of what the core difference is between image diffusion and language models from the perspective of extreme sampling acceleration. In image diffusion, distillation acts directly on continuous visual features. In the CDLMs we study, it acts on probability-weighted averages of token embeddings. The student must turn these predictions into discrete token choices while preserving variation across noise inputs. This gap helps explain why directly transferring image distillation methods to CDLMs can lead to repetitive text. To address this gap, we propose NIPSD. We first warm up the student on teacher trajectories so that it learns to produce distinct outputs from different noise inputs. During distillation, noise-interpolated perturbations preserve the link between each output and the noise that generated it. FiTScore, a real-valued auxiliary head, estimates the score at the student’s perturbed states by predicting the freshly added noise. Using DI for one-step distillation and TDM for few-step distillation, we show that the success of image diffusion distillation carries over to language. In just one or a few network evaluations, a distilled CDLM can approach the generative perplexity that its teacher achieves with hundreds of sampling steps. On OpenWebText, our one-step LangFlow student generates all 1,024 tokens in parallel in a single network evaluation. Initialized from a distilled two-step model, it approaches the 512-NFE teacher in GenPPL (43.10 vs. 41.52). At four NFEs, our LangFlow student achieves lower GenPPL than the 256-NFE teacher (48.06 vs. 49.24). As CDLMs are quickly developing, we believe our work offers valuable new insights to both understanding and algorithmic design to make CDLMs sample faster.

  • Reviewer 6VzT wrote that the discussion of categorical representations and score estimation “offers useful insights beyond a straightforward adaptation of existing methods.”
  • Reviewer DpGz highlighted the one-step results and found that the convex-hull analysis, the gradient sign flip, and the controlled diagnostics give “reusable guidance.”
  • Reviewer FWoM noted that “NIP is well motivated and noticeably improves performance.”

Additional Experiments and Improvements

Blinded preference and cross-sample diversity (6VzT, DpGz). Qwen3.8-27B judged 1,024 OWT-1024 passage pairs per budget, comparing NIPSD with FMLM. We hid method names and averaged each pair’s judgments across both orders. JSD_rows is the generalized Jensen–Shannon divergence among per-passage token unigram distributions (nats).

NFEMethodJSD_rows ↑Preference Rate ↑
1FMLM0.74622.85%
1NIPSD0.86577.15%
4FMLM1.1218.95%
4NIPSD1.47781.05%

At both sampling budgets, NIPSD had a substantially higher preference rate than FMLM and showed greater lexical diversity across samples.

Fluency and coherence (6VzT, DpGz). Qwen3.8-27B also scored each dimension on a five-point scale. The last column averages the two ratings.

ModelNFEFluency ↑Coherence ↑Average ↑
Teacher11.0001.0001.000
Teacher81.2891.1831.236
Teacher161.5981.2191.409
Teacher641.9381.8091.874
Teacher1281.9451.8481.897
Teacher2561.9771.8841.931
Teacher10241.9841.8981.941
FMLM11.0311.0011.016
FMLM41.5861.1821.384
NIPSD11.6331.1721.403
NIPSD41.9841.8371.911

At one NFE, fluency improves from 1.031 to 1.633. At four NFE, NIPSD’s average rating (1.911) is close to the teacher’s at 128 NFE (1.897).

DMD, SiD, and DMD2 on OWT-1024 (DpGz, FWoM). We adapted these methods to CDLMs and evaluated them alongside DI and NIPSD. We report GenPPL under GPT-2 Large alongside sample entropy.

MethodNFEGenPPL ↓Sample entropy
DI11.280.04
DMD151.782.82
SiD13.670.71
NIPSD170.085.24
DMD24106.425.17
NIPSD448.065.25

NIPSD achieved the highest sample entropy at one NFE and led on both metrics at four NFEs, with the highest sample entropy and the lowest GenPPL.

What the student generates at initialization (FWoM). With teacher weights, the one-step images in our examples retain recognizable shapes, while the one-step CDLM passages often repeat a few words or punctuation marks. The PCA projections were fitted separately for images and text. PCA plots · Samples by seed

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.