acceptodds
Under review as a conference paper at ICLR 2027

A First-Step Bias Restores Lost Diversity in Diffusion Model Distillation

Abstract

Distribution-matching distillation enables high-quality image generation in only a few denoising steps. However, distilled student models often map different initial noise samples to similar outputs under the same prompt. To locate where diversity is lost, we compare the teacher and student along the sampling trajectory. The largest measured reduction in the response to different initial noises occurs during the first, high-noise step. When we replace the first step with the teacher's corresponding trajectory and leave the final three student steps unchanged, most of the output diversity returns. This suggests that the first step is a key point for restoring lost diversity. Motivated by this finding, we introduce *ReSeed*, a post-distillation method that adds a seed-response bias adapter only to the first student step and aligns its velocity with the teacher's first-interval secant velocity. All student weights and later denoising steps remain unchanged. We evaluate *ReSeed* on three text-to-image models (SD3-Medium, SD3.5-Medium, and FLUX.1-dev) and one text-to-video model (Wan2.1-T2V-1.3B). For a distilled 4-step FLUX.1-dev model, *ReSeed* requires only additional trainable parameters relative to the base model and is trained on 8,000 prompt–noise pairs, while keeping the distilled student fully frozen and requiring no target images, perceptual losses, or adversarial losses. It improves DINO diversity by while maintaining competitive visual quality. These results show that substantial diversity can be recovered through lightweight, data-efficient tuning of only the first denoising step, without fine-tuning the full student model.

Primary area:
generative models
Keywords:
diffusion models, distribution-matching distillation, flow matching, diversity recovery, domain adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_xaCcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the loss of same-prompt diversity in few-step diffusion models trained with distribution-matching distillation. Through a teacher-prefix intervention on a four-step SD3-Medium student, the authors observe that replacing the first student interval with the corresponding teacher trajectory recovers most of the measured diversity gap, even when the remaining student steps are left unchanged. Motivated by this observation, the paper proposes ReSeed, a lightweight post-distillation correction. ReSeed inserts trainable per-channel biases into transformer blocks, enables them only during the first denoising step, and optimizes them to match the teacher’s secant velocity over the corresponding interval. Experiments cover SD3-Medium, SD3.5-Medium, FLUX.1-dev/SenseFlow, and Wan2.1 text-to-video generation. ReSeed substantially increases feature-space diversity and often improves FID

Critical strengths
  1. The teacher-prefix replacement experiment shows that replacing only the first student step recovers most of the lost diversity, while the frozen suffix can preserve the restored seed differences. This observation leads naturally to ReSeed. Repairing diversity in an existing distilled checkpoint, instead of modifying or repeating the original distillation, is both practically useful and relatively novel.
  2. ReSeed keeps the student backbone frozen and trains only a small set of first-step bias parameters. It introduces only 0.002–0.004% additional parameters, requires no target images or perceptual/adversarial losses, and adds no extra denoising steps. The reported inference overhead is also negligible.
  3. The method is evaluated on three image-generation model families and one video generator. The main image experiments use multiple training seeds, while matched prompts and initial noises are used throughout the comparisons. The paper also reports multiple diversity and quality metrics rather than presenting diversity gains alone.
Critical weaknesses
  1. The current controls do not fully isolate whether the proposed bias parameterization is especially effective. Comparisons with a parameter-matched first-step LoRA or adapter, as well as the same bias applied to later steps, would help clarify which design choice matters.
  2. The regression target is constructed by running the teacher for a fixed number of microsteps over the first student interval, but the effect of this choice is unclear. It would be useful to vary the number of teacher steps used to compute the secant target. If fewer steps achieve similar results, the training cost could be reduced substantially
Questions for authors

In the qualitative comparisons in Appendix F, ReSeed appears to produce more complex and cluttered backgrounds than the teacher and other student baselines. This is particularly noticeable in the first case of Figure 11, “a couple of buckets in a white room,” where the ReSeed samples often introduce colorful objects and detailed environments, seemingly weakening the “white room” constraint. Have the authors observed a systematic trade-off between recovered diversity and prompt adherence, especially for prompts specifying simple backgrounds or compositions?

Reviewer_jfhiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the loss of same-prompt diversity in few-step students distilled with distribution matching. A diagnostic on SD3-Medium shows that most of the loss can be recovered by replacing only the student’s first step with the teacher’s trajectory. Building on this, the authors propose ReSeed: small bias adapters, active only in the first step of the frozen student, trained to match the teacher’s average velocity over that interval. With about 0.002–0.004% extra parameters and roughly 8K prompt–noise pairs, ReSeed raises DINOv3 diversity by 27–48% across SD3-M, SD3.5-M, FLUX and Wan2.1, and improves FID, at some cost in preference and alignment metrics.

Critical strengths
  • Reduced diversity is a well-known weakness of few-step students. A repair that works on an already-distilled, frozen model is attractive in practice. It is very efficient, plug-and-play and its α\alpha lets users trade quality for diversity at inference.
  • The diagnosis motivates the method well. The teacher-prefix intervention (Fig. 2) and the observed link between first-endpoint seed-response error and final-image diversity (Fig. 4, Table 9) make a clear case for why the first step is the right place to intervene.
  • The paper covers three image models and one video model, uses three training seeds with standard deviations for the main image results, reports per-prompt and bootstrap statistics, and includes useful controls (α = −1, random bias)
Critical weaknesses
  • Naively matching the student’s first step to the teacher with L2 loss appears to cost quality. I think this the most critical weakness of this approach
  • Gandikota et. al [1] already report the finding that diversity is lost in the first step, and can be recovered by using the base model. So this work novelty is therefore mainly the post-hoc, frozen, tiny-adapter repair rather than the finding, and the contribution list should be framed accordingly.

Reference: [1] Rohit Gandikota, David Bau. "Distilling Diversity and Control in Diffusion Models." arXiv preprint arXiv:2503.10637 (2025).

Questions for authors
  • Does a higher-capacity first-step adapter recover more diversity and/or reduce the quality drop?
  • Could the drop in VBench dynamic degree (0.531 → 0.329) indicate reduced motion rather than a quality improvement?
1 comment

Thank authors for detail reply. I've had a good faith in this paper already hence its score of 3. Unfortunately, I don't think the paper is super good enough to raise to 4. So I've decided to maintain my score of 3.

Reviewer_gjeSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReSeed, which restores same-prompt diversity in distilled models using small trainable biases activated only during the first denoising step. Training matches the teacher’s interval-average velocity while freezing the original student. Experiments on three image models and one video model show increased feature diversity with little inference overhead, alongside trade-offs in some quality metrics.

Critical strengths
  1. The method is simple and practical: it improves existing students without repeating distillation or increasing sampling steps, adding only approximately 0.002%–0.004% denoiser parameters.
  2. All three image models show substantial diversity gains and improved FID. Teacher-prefix interventions, multiple evaluation metrics, and endpoint-error analysis provide meaningful support for the approach.
Critical weaknesses
  1. The mechanistic interpretation is stronger than the current evidence. Cumulative teacher-prefix replacement establishes the first step as an effective repair location but does not independently isolate individual steps’ contributions. The main diagnostic is also limited to SD3-Medium. Thus, the claim that most diversity loss occurs at the first step requires qualification or further support.
  2. The practical benefit of video diversity recovery remains insufficiently characterized. Table 8 demonstrates a favorable trade-off on SD3-Medium, but video diversity gains accompany lower text alignment and imaging quality. Reduced Dynamic Degree does not itself imply worse motion quality; nevertheless, the results do not establish how much diversity improvement remains at comparable quality and text alignment.
Questions for authors
  1. Does the current evidence support most diversity loss occurring at the first step, or the more conservative conclusion that it is an effective repair location? If single-step intervention results are available, how do they support this interpretation?
  2. Can Wan2.1 achieve a favorable trade-off similar to Table 8? Using bias scaling or existing checkpoints, do substantial diversity gains remain at approximately matched text alignment and generation quality?
Reviewer_vE6XOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the loss of sample diversity caused by diffusion model distillation. The authors observe that replacing the first denoising step with the corresponding teacher trajectory restores much of the lost diversity. Based on this observation, they propose ReSeed, a post-hoc method that learns lightweight block-wise bias adapters by aligning the student's first-step velocity with the teacher's average velocity over the same interval. ReSeed improves sample diversity without retraining the original student model or modifying the subsequent denoising steps, while introducing negligible inference overhead.

Critical strengths
  1. Practical Post-hoc Applicability: Unlike DP-DMD and GAD, which modify the distillation training process, ReSeed can be applied to already-distilled student models without repeating distillation or fine-tuning the original student weights.
  2. Negligible Inference Overhead: ReSeed introduces less than 0.004% additional trainable parameters and reports an inference latency overhead below 0.5%, while maintaining the original number of denoising steps. This makes the approach attractive for practical deployment.
  3. Consistent Empirical Improvements: ReSeed demonstrates substantial diversity improvements across SD3-Medium, SD3.5-Medium, and FLUX-based students, while also improving FID. Its extension to video generation further demonstrates the applicability of the approach beyond text-to-image models.
Critical weaknesses
  1. Limited Novelty of the First-Step Analysis: The importance of the first denoising step for diversity preservation has already been investigated in prior work. Gandikota and Bau (2025) identified premature structural commitment at the first timestep and demonstrated that replacing this step with the base model can restore sample diversity. Furthermore, DP-DMD (Wu et al., 2026) already employs teacher-derived velocity supervision for the first denoising step. Although ReSeed provides additional quantitative analyses of first-step replacement and seed-response alignment, it remains unclear what fundamentally new insights these analyses offer beyond the existing findings.

  2. Insufficient Validation of the Adapter Design: While ReSeed demonstrates substantial diversity improvements with block-wise bias adapters, it remains unclear whether this particular adaptation strategy is essential to the results. Comparisons with alternative lightweight adaptation methods under the same teacher supervision, such as parameter-matched LoRA or output-level corrections, would help establish the advantages of the proposed design.

  3. Limited Evaluation of Prompt-Consistent Diversity: Although the paper reports improvements across several feature-space diversity metrics, these improvements do not necessarily establish better coverage of prompt-consistent semantic variations. Additional evaluations of compositional prompt following or human judgments of meaningful diversity would strengthen the evidence that ReSeed restores useful diversity rather than simply increasing feature-space differences.

Questions for authors
  1. Quality–Diversity Trade-off with GAD: The paper briefly reports that applying ReSeed to an already-trained GAD student further improves diversity. Could the authors provide a more comprehensive evaluation of GAD + ReSeed, including quality metrics and comparisons with GAD, DMD + ReSeed, and DP-DMD? In particular, does combining ReSeed with GAD provide additional benefits in terms of the quality–diversity trade-off?

  2. Ablation on Adapter Design: Could the authors provide additional ablations comparing the proposed block-wise bias adapters with alternative lightweight adaptation strategies, such as parameter-matched LoRA or a simple bias applied directly to the predicted velocity, under the same teacher supervision? This would help clarify the importance of the proposed adapter design.

  3. Training and Inference Efficiency Comparison: While ReSeed demonstrates negligible inference overhead compared to the original distilled student, could the authors provide a more comprehensive efficiency comparison with DP-DMD and GAD in terms of both training cost (e.g., GPU hours or FLOPs) and inference cost (e.g., wall-clock latency and GPU memory)? It would be helpful to distinguish the additional cost of post-hoc adaptation from the total cost including the original distillation process.

  4. Novelty of the First-Step Analysis: Given that Gandikota and Bau (2025) and DP-DMD (Wu et al., 2026) have already demonstrated the importance of early denoising steps for diversity preservation, could the authors clarify what new insights the first-step replacement and seed-response analyses provide beyond these prior findings?

1 comment

Dear Reviewer,

The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.