acceptodds
Under review as a conference paper at ICLR 2027

RESTORING DIVERSITY IN DISTILLED IMAGE AND VIDEO GENERATORS BY PROBING ACTIVATIONS

Abstract

Distilling diffusion models into few-step students substantially reduces sampling latency, but induces severe diversity collapse: samples generated from different random seeds show minimal variation. In this work, we investigate the root cause of this failure by probing the internal representations of student models relative to their teachers. We find that distilled students maintain total activation variance comparable to their teachers, but concentrate it into substantially fewer directions. This directional collapse is localized to an identifiable intermediate block at the initial denoising step. Consequently, diversity collapse stems from a misallocation of activation variance rather than an inherent loss of generative capacity. Guided by this finding, we propose ReDiverse to reinstate the missing variance directions. At training time, ReDiverse-FT fine-tunes a lightweight adapter that regresses student activations onto the teacher's, reviving the teacher's feature variance characteristic. At inference time, ReDiverse-TF introduces a training-free mechanism that steers activations along precomputed teacher principal directions. Across image and video distillation benchmarks, ReDiverse consistently restores sample diversity while preserving generation quality, further supported by the user study. Anonymous project page: [https://iclr-2027-14418.github.io/](https://iclr-2027-14418.github.io/).

Primary area:
generative models
Keywords:
Generative Models, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KJfYOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Step distillation suffers from diversity collapse, in which samples generated under different seeds are highly similar. The authors compare teacher and student activations layerwisely and uncover the core mechanism: the distilled student model retains its total activation variance, yet variance is confined to a small set of feature directions. This collapse takes place in a localizable collapse block. Downstream layers remain capable of decoding the missing diversity directions, suggesting diversity collapse originates from imbalanced variance allocation instead of permanent loss of generative capacity.

Based on this observation, the authors present two complementary solutions:

  • ReDiverse-FT: a LoRA fine-tuning approach, where adaptation is only applied to the collapse block and earlier layers; LoRA is enabled exclusively at the first inference step.
  • ReDiverse-TF: a training-free inference strategy. For each prompt, precomputed principal component directions obtained from the teacher are leveraged to perturb activations for diversity recovery during inference.

The two variants may be deployed separately or jointly. The authors conduct evaluations across four pairs: SDXL and SD3 for image generation, Wan2.1 and MiniMax-H3 for video generation. Experiments include automatic metrics, human user studies and VLM-based assessment, alongside extensive ablation studies to verify design choices and hyperparameters.

Critical strengths
  1. This work provides insightful mechanistic diagnosis. Rather than stacking heuristic tricks, the authors perform layer-wise activation probing and identify that diversity collapse stems from reduced effective directions despite preserved total variance. Activation steering experiments verify that downstream networks can still utilize the missing teacher directions, justifying their repair strategy. This analysis is inspiring to the field.

  2. The solution is highly practical. It works on existing pre-distilled students without full re-distillation. ReDiverse-FT uses minimal LoRA parameters activated only in the first inference step. It supports both image and video generation.

  3. Experiments are comprehensive: comparisons over multiple image and video models, thorough ablations, automatic metrics, human studies and VLM evaluation. Ample visualizations clearly show improved diversity in composition, pose and camera motion.

Critical weaknesses
  1. ReDiverse-TF incurs upfront teacher inference for every prompt, lacks quantitative overhead analysis and mitigation strategies, hindering real-world deployment.

  2. The identification of the collapse block is empirical. No theoretical explanation is provided for why variance-direction collapse occurs at this specific location during distillation. While the phenomenon is well observed, its root cause remains underexplained.

  3. Diversity gains come with non-negligible drops in visual quality and human preference metrics. Hyperparameters K and α require manual tuning without an adaptive selection scheme, and configuration guidance for different application scenarios is absent.

Questions for authors
  1. ReDiverse-TF requires multiple forward passes of the teacher per prompt to estimate principal directions. Could the authors quantitatively report the computational overhead of this preprocessing stage? Are there strategies to reuse these directions across prompts to reduce inference cost?

  2. The collapse block is observed across multiple models, yet a theoretical justification is missing: why does variance-direction collapse concentrate within this particular block during distillation? Could the authors further analyze the underlying triggers of this phenomenon?

  3. The method exhibits a diversity-quality trade-off. Can an adaptive strategy be designed to automatically select injection strength hyperparameters K and α, instead of grid-search tuning? For different downstream use cases, would the authors recommend FT, TF, or their combination?

4 comments
AuthorsOpenReview ↗

Response [1/2]

We thank the reviewer for finding the diagnosis insightful and the method practical. In short, TF’s setup is a one-time fixed cost shared by a prompt’s samples, and its directions can be reused across prompts (W1, Q1); for reverse-KL-based students, the objective and a per-block analysis help explain the collapse (W2, Q2); and quality preservation is comparable to baselines, K and α can be set automatically, and each variant suits a different use case (W3, Q3).

New results are in Tables R2, R4 and R10 to R15 below and in Figure R1, which is in the supplementary material (rebuttal.html). Image results use a GenEval subset of 100 prompts × 20 seeds, and all timings are on one B200 GPU.

W1 and Q1 · Cost of ReDiverse-TF

W1.1 and Q1.1 · Overhead of the per-prompt setup

Could the authors quantitatively report the computational overhead of this preprocessing stage?

The setup is a one-time fixed cost of a few teacher samples per prompt, shared by its samples, so TF is cheaper than the teacher after a few samples (Table R11). Table R10 splits the setup by stage; sampling then costs the same as the student.

Table R10. Time of each stage of ReDiverse-TF on SDXL.

StageHow oftenNFETime (s)
Load the teacher and its text encodersonce–6.0
Encode the promptper prompt–0.01
Run the teacher from 25 noises to step τ = 7per prompt175 (teacher)9.0
PCA of the 25 activations at ℓ (top-8 directions)per prompt–0.09
Setup per prompt, in totalper prompt175 (teacher)9.1
Sample an image with the studentper image4 (student)0.34

Table R11. Time per image (s) on SDXL when n samples of a prompt share one ReDiverse-TF setup.

Methodn = 1n = 4n = 16n = 64n = 100
Teacher2.882.882.882.882.88
Student0.340.340.340.340.34
ReDiverse-TF9.42.60.910.480.43
ReDiverse-FT0.420.420.420.420.42
ReDiverse (FT+TF)9.52.71.000.570.52

Q1.2 · Reusing the directions across prompts

Are there strategies to reuse these directions across prompts to reduce inference cost?

Yes. The cost can be reduced in four ways:

  • Within a prompt. The directions are computed once per prompt and reused for all its samples.
  • Across prompts. They can come from another prompt with no teacher run: the nearest prompt or caption keeps the full gain, a prompt of the same category part of it and one shared set little (Table R12).
  • Fewer teacher noises. The gain does not shrink with fewer noises (Table R4a).
  • No setup with FT. ReDiverse-FT has no per-prompt cost at all.

Table R12. ReDiverse-TF with directions taken from other prompts (80 of the prompts).

Direction sourceTeacher run at test timeDINO ↑ImageReward ↑
None (student)no60.00.93
The prompt itself (paper)once per prompt60.70.92
Nearest other GenEval promptno60.80.89
A prompt of the same GenEval categoryno60.40.92
Nearest caption in a precomputed DiffusionDB bankno60.70.91
One shared set for all promptsno60.10.91

Table R4. Sensitivity of ReDiverse-TF to the teacher noises and to the prompt wording (first 50 prompts).

(a) Directions from N teacher noises (five disjoint sets)

Directions estimated fromOverlap with N = 100 ↑DINO ↑ (mean ± sd over 5 sets)ImageReward ↑
None (student)–58.60.85
N = 5 teacher noises0.1959.5 ± 0.10.84
N = 100.1959.6 ± 0.20.84
N = 25 (default)0.3259.3 ± 0.10.85
N = 500.3659.1 ± 0.10.85

(b) Directions from a paraphrase of the prompt (N = 25)

Directions estimated fromOverlap with the original wording ↑DINO ↑ImageReward ↑
None (student)–58.60.85
The original wording1 (reference)59.40.85
“a picture of …”0.7759.50.86
“an image showing …”0.7159.60.84
“…, captured in a photograph”0.7759.10.86

W2 and Q2 · Why the collapse concentrates at this block

Why does variance-direction collapse concentrate within this particular block during distillation?

We agree that the location is found empirically, and add an explanation and a per-block analysis (Figure R1).

Linearizing the first-step activation of a block bb, hb(ϵ)≈μb+Jb ϵh_b(\epsilon) \approx \mu_b + J_b\,\epsilon, gives Σb=JbJb⊤\Sigma_b = J_b J_b^\top, TV=tr Σb\mathrm{TV} = \mathrm{tr}\,\Sigma_b and PR=(tr Σb)2/tr(Σb2)\mathrm{PR} = (\mathrm{tr}\,\Sigma_b)^2 / \mathrm{tr}(\Sigma_b^2). For reverse-KL-based distillation, as in the distribution-matching students, the loss penalizes samples outside the teacher’s modes but not missed modes, so the student can lower it by turning JbJ_b toward fewer directions without shrinking it: PR falls while TV stays, the signature we measure. The objective appears to set how much diversity is lost: LCM-SDXL, whose loss ties each noise to the teacher’s trajectory, stays close to the teacher, while the distribution-matching students and rCM, which adds a score-distillation term, lose the most (Table R2).

AuthorsOpenReview ↗

Response [2/2]

W2 and Q2 (continued)

The collapse arises at the first step because all of the seed’s information enters there (Gandikota & Bau, 2026); at block ℓ the student turns from slightly less to more sensitive to the initial noise than the teacher, at a larger activation scale (Figure R1).

Table R2. PR and TV of each student at block ℓ, relative to its teacher.

(a) SDXL students

StudentFamilyPR at ℓ, ÷ teacherTV at ℓ, ÷ teacherDINO ↑
DMD2distribution matching0.391.5759.0
LCM-SDXLconsistency0.791.4665.9
PCM-SDXLphased consistency0.591.5663.7
SDXL-Turboadversarial0.513.1561.1

(b) SD3 students

StudentFamilyPR at ℓ, ÷ teacherTV at ℓ, ÷ teacherDINO ↑
TDMdistribution matching0.540.9747.9
PCM-SD3phased consistency0.570.9657.8

(c) Wan 2.1 video students (250 noises, vs 1,000 in Tab. 1)

StudentFamilyPR at ℓ, ÷ teacherTV at ℓ, ÷ teacherDINO ↑
lightx2vdistribution matching0.841.4131.1
rCMconsistency + score distillation0.861.3529.0

Figure R1 (rebuttal.html). Activation scale and noise sensitivity at each block of the first step, SDXL teacher vs DMD2 student.

W3 and Q3 · Quality, hyperparameters and configuration

W3.1 and Q3.1 · Quality drop and the diversity–quality trade-off

Diversity gains come with non-negligible drops in visual quality and human preference metrics.

  • Quality preservation is comparable to the baselines. On SDXL, DP-DMD loses a similar amount without raising diversity; FT or TF alone is at most 5% below the teacher, and the larger drop is for FT+TF, mainly on SD3 (Tab. 4).
  • Human raters prefer ReDiverse. In the paper’s user study, it is preferred to the student and the other diversity methods in realism for images and videos, to the teacher for images, and to every method in prompt following (Fig. 10).
  • The reward models are imperfect. They may favor the more saturated, higher-contrast look of the students’ images, which score above their own teachers (Tab. 4); ReDiverse moves these properties part of the way back (Table R13).

Table R13. Saturation, contrast and sharpness of the images on SDXL.

MethodSaturationContrastSharpnessImageReward ↑
Teacher0.270.2064890.69
Student0.410.2584640.88
ReDiverse-TF0.410.2544700.87
ReDiverse-FT0.380.2455020.72
ReDiverse (FT+TF)0.370.2425190.68

W3.2 and Q3.2 · Choosing K and α automatically

Can an adaptive strategy be designed to automatically select injection strength hyperparameters K and α, instead of grid-search tuning?

We tried three automatic rules, computed per prompt with no grid search (Table R14):

  • α by variance matching (works). Along each direction, α’s range adds the variance the student’s first step lacks relative to the teacher, and nothing where it already matches; it keeps most of the grid-searched gain.
  • K from the teacher’s spectrum (works). K is the smallest number of directions explaining 70% of the variance of the teacher’s 25-noise PCA (13–16 per prompt); it is more diverse than the paper’s K = 8 at the same reward.
  • α by PR matching (does not work). α is set so that the student’s PR at ℓ matches the teacher’s; as adding variance along a few directions lowers PR, it picks α ≈ 0, with no gain. What matters is which directions TF restores (App. E.7).

Table R14. Automatic choice of α and K for ReDiverse-TF (the 80 prompts of Table R12).

(a) Injection strength α (K = 8)

SettingαDINO ↑ImageReward ↑
Student (no TF)–60.00.93
Grid search (paper)U[−2, 2]60.70.92
Automatic, variance matchingper direction60.50.90
Automatic, PR matchingper prompt60.00.92

(b) Number of directions K (α ~ U[−2, 2])

SettingKDINO ↑ImageReward ↑
Student (no TF)060.00.93
Fixed (paper)860.70.92
Automatic (70% of the 25-noise PCA’s variance)13–16, per prompt61.00.92

W3.3 and Q3.3 · Which variant to use

For different downstream use cases, would the authors recommend FT, TF, or their combination?

FT when one student serves many prompts, TF when training is not possible, and both for maximum diversity (Table R15).

Table R15. Recommended ReDiverse variant for each use case.

Use caseRecommendedWhy
Serving many users and prompts (e.g. an API)ReDiverse-FTTrained once (about 8 min); no teacher at inference.
No training possible (no pipeline, data or compute)ReDiverse-TFNo training; one setup per prompt, shared by its samples.
Many samples per prompt (exploration, synthetic data)ReDiverse (FT+TF)Largest diversity gain; the setup is shared by the samples.
Quality firstReDiverse-FT, or TF with a smaller K or αKeeps teacher-level ImageReward (Tab. 4).
A resolution the teacher was not trained for (e.g. SDXL-Turbo)ReDiverse-FTThe teacher’s directions are poor at that resolution.
AuthorsOpenReview ↗

We would be happy to clarify any remaining concerns.

Dear Reviewer KJfY, we would like to kindly remind you of our response above. In the revised paper, the changes addressing your comments are highlighted in orange:

  • Section 3.1: A pointer to why the collapse concentrates at block ℓ.
  • Section 5.2: Pointers to the automatic choice of K and α and to the recommended variant for each use case.
  • Appendix A.2 (Figure R1): A linearized view of why reverse-KL distillation concentrates the seed-dependent variance into fewer directions, and the student's higher sensitivity to the initial noise at block ℓ.
  • Appendix E.10 (Table R14): K and α set per prompt without a grid search: α by variance matching, K from the teacher's spectrum.
  • Appendix E.11 (Table R15): The recommended variant for each use case.
  • Table 7 and Appendix D.4 (in blue): The overhead of the ReDiverse-TF setup, its sharing across a prompt's samples, and the reuse of its directions across prompts.
  • Appendix E.9 (in purple): The gain of ReDiverse-TF holds with fewer teacher noises.

Dear Reviewer,

As the discussion phase is approaching its end, we would greatly appreciate it if you could let us know whether our clarifications and additional experiments have adequately addressed your remaining concerns.

We would be happy to address any further questions before the discussion period ends. Thank you again for your time and constructive engagement throughout the discussion!

Best regards,

All Authors

Reviewer_x7iwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares teacher and step-distilled student activations under the same noise, finds that the student's activation variance collapses onto fewer directions at an intermediate block of the first denoising step, and proposes ReDiverse, a LoRA-based feature regression (FT) and a training-free perturbation along teacher principal directions (TF).

Critical strengths
  1. The problem is interesting and practically relevant, and locating diversity collapse at the block level by comparing teacher and student activations is a useful diagnostic.

  2. ReDiverse-FT is plug-and-play for released checkpoints, and the same recipe works across four teacher–student pairs, including large video models.

Critical weaknesses
  1. The most direct baseline is missing. [1] already reports that distilled students keep their diversity directions but leave them unused, that the collapse happens at the first step, and proposes teacher–student hybrid sampling to fix it. The paper cites this work but does not compare against it, and the claim to be the first to interpret student and teacher activations together (L142) is overstated.

  2. The computational cost is not reported. ReDiverse-TF runs the teacher for several steps on 25 noises for every new prompt, which may offset much of the speed-up of distillation, especially when only a few samples are generated per prompt. No NFE, wall-clock or memory numbers are given for any method.

  3. The method assumes that the student shares the teacher's architecture, since both variants match or perturb activations block by block. This restricts it to step distillation, and the paper does not state it as a limitation.

  4. The diversity gains come with a clear quality drop: ImageReward falls by 21% on SDXL and 15% on SD3 (Table 4).

  5. Missing citations: [2–5].

[1] Distilling Diversity and Control in Diffusion Models. WACV 2026.

[2] FitNets: Hints for Thin Deep Nets. ICLR 2015.

[3] BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion. ECCV 2024.

[4] Understanding Dimensional Collapse in Contrastive Self-supervised Learning. ICLR 2022.

[5] RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML 2023.

Questions for authors

See weaknees 1-2.

I am willing to raise my score if the authors provide a comparison with [1] and report the inference cost of ReDiverse-TF

4 comments
AuthorsOpenReview ↗

Response [1/2]

We thank the reviewer for finding the problem practically relevant and the block-level diagnostic useful. In short, we added the comparison with [1]: on images it is as diverse as ReDiverse or slightly more but mostly at a much lower quality, and on video ReDiverse is more diverse (W1, Q1); both ReDiverse variants keep the student’s number of steps and nearly its speed, with a one-time fixed training (FT) or a one-time fixed setup per prompt (TF) (W2, Q2); we will state the shared-architecture limitation and add the citations (W3, W5); and the quality preservation is comparable to baselines (W4).

New results are in Tables R6 to R13 below and in Figure R3 and Video R1, in the supplementary material (rebuttal.html). Image results use a GenEval subset of 100 prompts × 20 seeds and video results 20 VBench prompts × 10 clips, and all timings are on one B200 GPU.

W1 and Q1 · Comparison with Gandikota & Bau [1]

[1] already reports that distilled students keep their diversity directions but leave them unused, that the collapse happens at the first step, and proposes teacher–student hybrid sampling to fix it. The paper cites this work but does not compare against it, and the claim to be the first to interpret student and teacher activations together (L142) is overstated.

I am willing to raise my score if the authors provide a comparison with [1]

We reproduced [1], in which the teacher runs the student’s first step and the student finishes, on three SDXL students, SD3 and Wan 2.1. On images, [1] is as diverse as ReDiverse or slightly more, but mostly at a much lower quality (Table R6): on DMD2 its HPSv2 is lower at the same diversity and ImageReward (28.2 vs 29.6), on SD3 and LCM-SDXL its ImageReward is far lower (0.43 vs 0.86 and −0.27 vs 0.28), and only on SDXL-Turbo does it reach the ImageReward of ReDiverse (FT+TF). On video, ReDiverse is more diverse at a higher text alignment and subject consistency (Table R7). [1] also runs the teacher for every sample, while ReDiverse-FT needs no teacher at inference (Table R9).

Figure R3 (images) and Video R1 (videos) in the supplementary material (rebuttal.html) show samples of both. We will add this comparison and rephrase L142.

Table R6. ReDiverse and DDC [1] on four image students.

(a) Diversity: DINO ↑

MethodDMD2LCM-SDXLSDXL-TurboTDM (SD3)
DDC [1]64.967.067.962.4
ReDiverse-TF59.766.561.353.6
ReDiverse-FT64.766.665.752.6
ReDiverse (FT+TF)65.167.265.860.2

(b) Quality: ImageReward ↑

MethodDMD2LCM-SDXLSDXL-TurboTDM (SD3)
DDC [1]0.70−0.270.600.43
ReDiverse-TF0.870.340.830.96
ReDiverse-FT0.720.330.570.97
ReDiverse (FT+TF)0.680.280.600.86

Table R7. ReDiverse and DDC [1] on the video student lightx2v of Wan 2.1.

MethodDINO ↑Text–video similarity ↑Subject consistency ↑
DDC [1]47.219.695.6
ReDiverse-FT47.020.897.3
ReDiverse (FT+TF)53.120.496.3

Figure R3 (rebuttal.html). DDC [1] and ReDiverse (FT+TF) on DMD2 and TDM (SD3), with the same starting noise in each column.

W2 and Q2 · Computational cost

ReDiverse-TF runs the teacher for several steps on 25 noises for every new prompt, which may offset much of the speed-up of distillation, especially when only a few samples are generated per prompt. No NFE, wall-clock or memory numbers are given for any method.

and report the inference cost of ReDiverse-TF

In short, both variants keep the student’s number of steps and nearly its speed and memory (Table R9); the only extra cost is a one-time fixed training for FT or a one-time fixed setup per prompt for TF. Table R9 gives each method’s teacher runs, NFE, time and peak memory per sample: DDC [1] runs the teacher for every image, and Noise Optimization optimizes every noise at sampling time.

  • FT trains once. It costs a small fraction of a re-distillation such as DP-DMD (Table R8) and needs no teacher at inference; its unmerged LoRA adds a small per-image overhead. Table R8 lists what each method trains: DP-DMD distills a new 4-step student, while FT trains a rank-32 LoRA on the blocks up to ℓ for 200 steps.
  • TF needs no training. Its setup per prompt is a few teacher steps on a small set of noises (Table R10), which the prompt’s samples share, so TF is cheaper than sampling the teacher after a few samples per prompt (Table R11). Table R10 times each stage of this setup, and Table R11 spreads it over n samples of a prompt: the time per image falls toward the student’s as n grows.
  • The directions can be reused. They can come from another prompt, with no teacher run at test time (Table R12). Directions from the nearest GenEval prompt or from the nearest caption in a precomputed DiffusionDB bank give the same diversity as the prompt’s own, those from a prompt of the same category less, and one shared set for all prompts almost no gain.
AuthorsOpenReview ↗

Response [2/2]

W2 and Q2 (continued)

Table R8. Training cost of each method on SDXL.

MethodWhat is trainedGPU time
Noise Opt., DDC [1], ReDiverse-TFnothingN/A
DP-DMDa new 4-step student (full distillation)39 GPU-h
ReDiverse-FT, ReDiverse (FT+TF)rank-32 LoRA on the blocks up to ℓ, 200 stepsabout 8 min

Table R9. Sampling cost of each method.

(a) SDXL with DMD2

MethodTeacher runsNFE per imageTime per image (s)Peak memory (GiB)
Teacher–1002.8811.5
Studentnever40.3411.5
Noise Opt.never–11.316.2
DP-DMDnever40.3411.5
DDC [1]for every image40.3516.4
ReDiverse-TFonce per prompt (175 NFE, 9.1 s)40.3411.5
ReDiverse-FTnever40.4211.6
ReDiverse (FT+TF)once per prompt (175 NFE, 9.1 s)40.4311.6

(b) Wan 2.1 with lightx2v

MethodTeacher runsNFE per clipTime per clip (s)Peak memory (GiB)
Teacher–100171.335.9
Studentnever48.4235.9
ReDiverse-TFonce per prompt (200 NFE, 358 s)48.4338.4
ReDiverse-FTnever48.5536.1
ReDiverse (FT+TF)once per prompt (200 NFE, 358 s)48.5538.6

Table R10. Time of each stage of ReDiverse-TF on SDXL.

StageHow oftenNFETime (s)
Load the teacher and its text encodersonce–6.0
Encode the promptper prompt–0.01
Run the teacher from 25 noises to step τ = 7per prompt175 (teacher)9.0
PCA of the 25 activations at ℓ (top-8 directions)per prompt–0.09
Setup per prompt, in totalper prompt175 (teacher)9.1
Sample an image with the studentper image4 (student)0.34

Table R11. Time per image (s) on SDXL when n samples of a prompt share one ReDiverse-TF setup.

Methodn = 1n = 4n = 16n = 64n = 100
Teacher2.882.882.882.882.88
Student0.340.340.340.340.34
ReDiverse-TF9.42.60.910.480.43
ReDiverse-FT0.420.420.420.420.42
ReDiverse (FT+TF)9.52.71.000.570.52

Table R12. ReDiverse-TF with directions taken from other prompts (80 of the prompts).

Direction sourceTeacher run at test timeDINO ↑ImageReward ↑
None (student)no60.00.93
The prompt itself (paper)once per prompt60.70.92
Nearest other GenEval promptno60.80.89
A prompt of the same GenEval categoryno60.40.92
Nearest caption in a precomputed DiffusionDB bankno60.70.91
One shared set for all promptsno60.10.91

W3 · Shared architecture

The method assumes that the student shares the teacher’s architecture, since both variants match or perturb activations block by block. This restricts it to step distillation, and the paper does not state it as a limitation.

We agree and will state it as a limitation. Both variants need the teacher’s activation at block ℓ in the student’s space, which holds for step distillation: all our students and most released few-step models. For compressed students such as BK-SDM [3], a learned linear projection between matched blocks, as in FitNets [2], is a natural extension and we leave it for future work.

W4 · Quality drop

The diversity gains come with a clear quality drop: ImageReward falls by 21% on SDXL and 15% on SD3 (Table 4).

  • Quality preservation is comparable to the baselines. On SDXL, DP-DMD loses a similar amount without raising diversity (Tab. 4), DDC [1] matches the diversity and ImageReward of ReDiverse at a lower HPSv2 (28.2 vs 29.6; Table R6), and Noise Optimization keeps quality only at a much higher sampling cost (Table R9); FT or TF alone is at most 5% below the teacher, and the larger drop appears only for FT+TF, mainly on SD3 (Tab. 4).
  • Human raters prefer ReDiverse. In the user study of the paper, it is preferred to the student and to the other diversity methods in realism for both images and videos, to the teacher for images, and to every method in prompt following (Fig. 10).
  • The reward models are imperfect. They may favor the more saturated, higher-contrast look of the students’ images, which score above their own teachers (Tab. 4); ReDiverse moves these properties part of the way back (Table R13), and on SDXL most of the drop brings the score back to the teacher’s.

FT or TF alone can therefore be used when quality matters most.

Table R13. Saturation, contrast and sharpness of the images on SDXL.

MethodSaturationContrastSharpnessImageReward ↑
Teacher0.270.2064890.69
Student0.410.2584640.88
ReDiverse-TF0.410.2544700.87
ReDiverse-FT0.380.2455020.72
ReDiverse (FT+TF)0.370.2425190.68

W5 · Missing citations [2–5]

Missing citations: [2–5].

We will cite all four: FitNets [2] (feature regression, which FT applies at one block and step), BK-SDM [3] (see W3), and [4] and [5], which measure collapse through the eigenvalue spectrum; RankMe’s effective rank is closely related to our participation ratio.

Reviewer_x7iwOpenReview ↗

Thanks for the response

Thank you for the thorough response and the extensive new experiments, which address my main concerns. I ask that the paper include the comparison with DDC [1], covering both its results and its cost, and present it candidly, together with the cost of ReDiverse.

Since ICLR allows a revised PDF of up to 10 pages during the discussion, I will raise my score to 3 once the revision with these changes is uploaded.

AuthorsOpenReview ↗

We would be happy to clarify any remaining concerns.

Dear Reviewer x7iw, thank you for your reply and your constructive suggestions. We have uploaded the revised paper, which, as requested, includes the comparison with DDC, covering both its results and its cost, together with the cost of ReDiverse. The changes are highlighted in blue:

  • Section 2: The "first" claim (L142) is rephrased.
  • Sections 3.1 and 4.1: Citations of Jing et al. and RankMe (eigenvalue-spectrum measures of collapse) and of FitNets (feature regression).
  • Section 5.1: DDC is added to the baselines.
  • Section 5.2 (Table 6, Figure 9; Tables R6–R7): DDC against all variants of ReDiverse on four image students and on Wan 2.1: on images it is about as diverse, and more on some students, but mostly at a lower quality; on video, ReDiverse is more diverse.
  • Section 5.2 (Table 7): Sampling cost of every method: both ReDiverse variants keep the student's steps and nearly its time and memory, while DDC runs the teacher for every sample.
  • Section 6: The shared-architecture assumption is stated as a limitation, with BK-SDM and FitNets.
  • Appendix D.4 (Tables R8–R12): Training cost, each stage of the ReDiverse-TF setup, its cost spread over a prompt's samples, the reuse of its directions across prompts, and the cost on Wan 2.1.
  • Appendix H.5 (Figure R3): The evaluation setup of the DDC comparison, and more examples of DDC and ReDiverse on the same initial noises, including video.
  • Appendix H.4 (in purple): The quality drop, as it also answers another reviewer.
Reviewer_6X8zOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies why few-step distilled image and video models produce similar outputs across different seeds. It finds that student activations keep a similar or larger total variance, but this variance is concentrated in fewer directions. Based on this finding, it proposes two fixes: fitting student features to teacher features with LoRA, and adding noise along teacher PCA directions. Both changes are applied at the first sampling step. Tests on four image and video model pairs show improved diversity, with some loss in quality.

Critical strengths

Paper section 3 includes an interesting analysis on probing of the diversity problem. There are some interesting take aways, e.g. the paper shows that low output diversity does not necessarily mean low total activation variance. These analysis may give a useful view of the problem.

Critical weaknesses

The scope across distillation methods is unclear. The paper often discusses “distilled models” as a broad group, but does not give a controlled comparison between different distillation methods. E.g., in the probing analysis, will DMD work similarly as consistency-based distillation methods?

The proposed fixes work empirically, but the claimed mechanism and robustness are not well established. FT matches full activations, changing more than just their variance directions. Its gains could come from general feature supervision rather than the specific mechanism proposed here. It might also undistill the model, requiring more steps to generate high quality outputs. TF uses teacher PCA directions, but its sensitivity to the sampled noises and prompts is not clear.

Considering the quality drops, the current evidence does not yet make a strong case that these are reliable solutions.

Questions for authors

Do consistency-distilled students show the same probing pattern as DMD students? If not, the scope of the main claim should be limited.

Overall, the probing is interesting and the results are promising. However, the broad claim across distillation methods and the link between the proposed mechanism and the fixes need stronger evidence. I encourage the authors to explore these points further.

4 comments
AuthorsOpenReview ↗

Response [1/2]

We thank the reviewer for finding the paper clear and the probing analysis interesting. In short, students of all three distillation families show the same probing pattern, and ReDiverse raises diversity on all of them (W1, Q1); FT’s gain comes mainly from the shift it makes in the activation distribution at block ℓ, TF is robust to the teacher noises and the prompt wording, FT keeps the student’s steps, and the quality preservation is comparable to baselines (W2); and we will clarify the scope of the claim (Q2).

New results are in Tables R1 to R5 and R13 below and in Figure R2, which is in the supplementary material (rebuttal.html). Image results use a GenEval subset of 100 prompts × 20 seeds and video results 20 VBench prompts × 10 clips.

W1 and Q1 · Scope across distillation methods

Do consistency-distilled students show the same probing pattern as DMD students?

Yes. Students of all three families (distribution matching, consistency and adversarial) show the same pattern at block ℓ: a lower PR than the teacher with a TV near or above it (Table R2, Figure R2).

The severity varies: on SDXL, the less diversity a student keeps, the lower its PR at ℓ, with LCM-SDXL closest to the teacher and PCM-SDXL between it and DMD2 (Table R2a, Figure R2); PCM-SD3 and rCM lose about as much PR as the distribution-matching students of the same teachers (Table R2).

ReDiverse raises diversity on all of these students (Table R1), so it does not rely on one distillation objective.

Table R1. Diversity (DINO ↑) of ReDiverse on students from three distillation families.

(a) Image students

MethodLCM-SDXLPCM-SDXLSDXL-TurboPCM-SD3
Teacher66.666.666.661.5
Student65.963.761.157.8
ReDiverse-TF66.564.361.363.5
ReDiverse-FT66.665.765.760.3
ReDiverse (FT+TF)67.266.465.866.1

(b) Video students of Wan 2.1

Methodlightx2vrCM
Teacher53.453.4
Student31.129.0
ReDiverse-TF33.734.6
ReDiverse-FT47.047.6
ReDiverse (FT+TF)53.154.6

Table R2. PR and TV of each student at block ℓ, relative to its teacher.

(a) SDXL students

StudentFamilyPR at ℓ, ÷ teacherTV at ℓ, ÷ teacherDINO ↑
DMD2distribution matching0.391.5759.0
LCM-SDXLconsistency0.791.4665.9
PCM-SDXLphased consistency0.591.5663.7
SDXL-Turboadversarial0.513.1561.1

(b) SD3 students

StudentFamilyPR at ℓ, ÷ teacherTV at ℓ, ÷ teacherDINO ↑
TDMdistribution matching0.540.9747.9
PCM-SD3phased consistency0.570.9657.8

(c) Wan 2.1 video students (250 noises, vs 1,000 in Tab. 1)

StudentFamilyPR at ℓ, ÷ teacherTV at ℓ, ÷ teacherDINO ↑
lightx2vdistribution matching0.841.4131.1
rCMconsistency + score distillation0.861.3529.0

Figure R2 (rebuttal.html). PR and TV at each block of the first step, student ÷ teacher, for three SDXL students.

W2 · Mechanism and robustness of the fixes

W2.1 · Is ReDiverse-FT’s gain the proposed mechanism?

Its gains could come from general feature supervision rather than the specific mechanism proposed here.

FT does change more than the variance directions, but its gain needs both its mean shift and the noise-dependent change that restores the variation across seeds at ℓ, and TF, which has no feature supervision, also raises diversity alone (Table R1).

FT moves the mean and scale at ℓ toward the teacher’s and restores the variation across seeds: PR and the overlap with the teacher’s principal directions rise, while TV returns to the teacher’s level (Table R3a, b).

Applied alone at inference, the mean shift and the noise-dependent part of FT’s change each recover only part of the gain (Table R3c).

Table R3. What ReDiverse-FT changes at block ℓ, and which part of the change gives its diversity gain.

(a) Mean and scale

MethodDistance of the mean to the teacher’sAverage token norm
Teacher0.17 (noise floor)174
Student0.85216
ReDiverse-FT0.43170

(b) Variation across seeds

MethodTV ÷ teacherPR ÷ teacherOverlap of the top-8 directions with the teacher’s
Teacher1.001.000.109 (ceiling)
Student1.570.390.048
ReDiverse-FT1.080.660.087

(c) Each part of FT’s change applied alone

Change applied at block ℓDINO ↑Share of FT’s DINO gainImageReward ↑
None (student)59.00%0.88
Full change (ReDiverse-FT)64.7100%0.72
Mean shift only61.748%0.88
Noise-dependent part only60.425%0.77
AuthorsOpenReview ↗

Response [2/2]

W2.2 · Does ReDiverse-FT undistill the student?

It might also undistill the model, requiring more steps to generate high quality outputs.

We would like to clarify that the adapter is active only in the student’s first step, so FT samples with the student’s number of steps and network evaluations, and its samples keep teacher-level quality (Tab. 4).

W2.3 · Sensitivity of ReDiverse-TF to the sampled noises and prompts

TF uses teacher PCA directions, but its sensitivity to the sampled noises and prompts is not clear.

TF is robust to both (Table R4): diversity and reward stay nearly unchanged across disjoint sets of teacher noises and across prompt paraphrases, even though the estimated directions differ, so it does not need one specific PCA estimate. Which directions are injected still matters: at the same injected energy, the student’s own directions add little diversity and random directions cost much more quality (App. E.7).

Table R4. Sensitivity of ReDiverse-TF to the teacher noises and to the prompt wording (first 50 prompts).

(a) Directions from N teacher noises (five disjoint sets)

Directions estimated fromOverlap with N = 100 ↑DINO ↑ (mean ± sd over 5 sets)ImageReward ↑
None (student)–58.60.85
N = 50.1959.5 ± 0.10.84
N = 100.1959.6 ± 0.20.84
N = 25 (default)0.3259.3 ± 0.10.85
N = 500.3659.1 ± 0.10.85

(b) Directions from a paraphrase of the prompt (N = 25)

Directions estimated fromOverlap with the original wording ↑DINO ↑ImageReward ↑
None (student)–58.60.85
The original wording1 (reference)59.40.85
“a picture of …”0.7759.50.86
“an image showing …”0.7159.60.84
“…, captured in a photograph”0.7759.10.86

W2.4 · Quality drop

Considering the quality drops, the current evidence does not yet make a strong case that these are reliable solutions.

  • Quality preservation is comparable to the baselines. On SDXL, DP-DMD loses a similar amount without gaining diversity, and Noise Optimization keeps quality only by optimizing every noise at sampling time; FT or TF alone is at most 5% below the teacher, and the larger drop appears only for FT+TF, mainly on SD3 (Table R5).
  • Human raters prefer ReDiverse. In the user study of the paper, it is preferred to the student and to the other diversity methods in realism for both images and videos, to the teacher for images, and to every method in prompt following (Fig. 10).
  • The reward models are imperfect. They may favor the more saturated, higher-contrast look of the students’ images, which score above their own teachers; ReDiverse moves these properties part of the way back (Table R13), and most of the SDXL drop brings the score back to the teacher’s (Table R5).

FT or TF alone can therefore be used when quality matters most.

Table R5. Diversity and quality of each method relative to the teacher, from Tab. 4.

(a) SDXL, student DMD2

MethodDINO ↑ImageReward ↑ImageReward vs teacher
Teacher66.30.6950%
Noise Opt.67.00.714+3%
Student59.40.854+23%
DP-DMD58.30.709+2%
ReDiverse-TF60.10.836+20%
ReDiverse-FT64.60.6940%
ReDiverse (FT+TF)65.00.672−3%

(b) SD3, student TDM

MethodDINO ↑ImageReward ↑ImageReward vs teacher
Teacher61.61.0170%
Noise Opt.56.90.967−5%
Student48.41.026+1%
DP-DMD56.80.993−2%
ReDiverse-TF53.80.965−5%
ReDiverse-FT53.50.992−2%
ReDiverse (FT+TF)60.10.872−14%

Table R13. Saturation, contrast and sharpness of the images on SDXL.

MethodSaturationContrastSharpnessImageReward ↑
Teacher0.270.2064890.69
Student0.410.2584640.88
ReDiverse-TF0.410.2544700.87
ReDiverse-FT0.380.2455020.72
ReDiverse (FT+TF)0.370.2425190.68

Q2 · Overall

However, the broad claim across distillation methods and the link between the proposed mechanism and the fixes need stronger evidence.

Thank you. We added evidence for both: the probing pattern on consistency students of three teachers (W1), and what FT changes at block ℓ, which part gives the gain and how stable TF is (W2).

AuthorsOpenReview ↗

We would be happy to clarify any remaining concerns.

Dear Reviewer 6X8z, we would like to kindly remind you of our response above. We would be happy to clarify any remaining concerns. In the revised paper, the changes addressing your comments are highlighted in purple:

  • Section 3.1: The probing pattern holds for students of three distillation families, with a severity that depends on the objective, and ReDiverse raises diversity on all of them.
  • Table 1: Ratios at block ℓ for the consistency (LCM-SDXL, PCM-SDXL, PCM-SD3, rCM) and adversarial (SDXL-Turbo) students.
  • Section 5.2: Summary of what drives the gain of ReDiverse-FT and of the robustness of ReDiverse-TF.
  • Appendix A.1 (Tables R1–R2, Figure R2): Probing and ReDiverse on these students, with all diversity and quality metrics.
  • Appendix E.8 (Table R3): What ReDiverse-FT changes at block ℓ, which part gives its gain, and that it keeps the student's number of steps.
  • Appendix E.9 (Table R4): Robustness of ReDiverse-TF to the teacher noises and the prompt wording.
  • Appendix H.4 (Table R13): The quality drop: the reward models may favor the students' saturated look, and ReDiverse-FT or ReDiverse-TF alone stays within about 5% of the teacher's ImageReward.

Dear Reviewer,

We hope that our clarifications and additional experiments above and in the revised draft have adequately addressed your concerns. As the discussion period is approaching, we would greatly appreciate it if you could review our responses and reconsider your assessment if you find the new results and explanations satisfactory.

We would be happy to address any remaining concerns!

Best regards, All authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response to AC and Reviewers

We sincerely thank the AC for overseeing our submission and appreciate all reviewers for their insightful and constructive feedback.

Below, we summarize the primary highlights of our rebuttal, focusing on the important questions and our responses. Comprehensive details on these points, as well as answers to minor clarification questions, are available in our individual responses to each reviewer.


To reviewer 1 (6X8z)

Q1. The probing pattern beyond distribution-matching students

Per reviewer 6X8z’s request, we probed students of three distillation families: distribution matching (DMD2, TDM, lightx2v), consistency (LCM-SDXL, PCM-SDXL, PCM-SD3, rCM) and adversarial (SDXL-Turbo). The take-away messages are:

(i) All of them show the same pattern at block ℓ: a lower PR than the teacher with a TV near or above it (Table R2, Figure R2).

(ii) ReDiverse raises diversity on all of them (Tables R1, R5).

Q2. Whether FT’s gain comes from the proposed mechanism, and how robust TF is

We measured what FT changes at block ℓ and tested TF with other teacher noises and prompt wordings, and confirm that:

(i) FT restores the variation across seeds at ℓ, and its gain needs both its mean shift and its noise-dependent part (Table R3); FT keeps the student’s number of steps, so it does not undistill the student.

(ii) TF, which has no feature supervision, also raises diversity and is robust to the teacher noises and the prompt wording (Table R4).

Q3. Quality drop

We show that:

(i) Quality preservation is comparable to the baselines, and FT or TF alone is at most 5% below the teacher (Table R5).

(ii) In the user study of the paper, raters prefer ReDiverse to the student and to the other diversity methods in realism for images and videos, and to the teacher for images (Fig. 10).

(iii) The reward models may favor the students’ more saturated, higher-contrast look, which ReDiverse moves part of the way back, so on SDXL most of the drop brings the score back to the teacher’s (Table R13).


To reviewer 2 (x7iw)

Q4. Comparison with Gandikota & Bau [1]

Per reviewer x7iw’s request, we reproduced [1] on three SDXL students, SD3 and Wan 2.1 (Tables R6, R7; Figure R3 and Video R1 in rebuttal.html of the supplementary material). The take-away messages are:

(i) On images, [1] is as diverse as ReDiverse or slightly more, but mostly at a much lower quality; on video, ReDiverse is more diverse at higher text alignment.

(ii) [1] runs the teacher for every sample, while ReDiverse-FT needs no teacher at inference (Table R9).

We will add this comparison and rephrase L142.

Q5. Computational cost

We report the training and sampling cost of every method (Tables R8 to R11) and confirm that:

(i) Both variants keep the student’s number of steps and nearly its speed and memory (Table R9).

(ii) FT trains once, at a small fraction of the cost of a re-distillation (Table R8).

(iii) TF’s one-time fixed setup per prompt is shared by the prompt’s samples, so TF is cheaper than the teacher after a few samples (Tables R10, R11), and its directions can be reused across prompts (Table R12).

Q6. Shared architecture and missing citations

We will state the shared-architecture requirement as a limitation, with a learned projection between matched blocks as an extension, and cite [2–5].

Q7. Quality drop

We refer to Q3.


To reviewer 3 (KJfY)

Q8. Cost of the per-prompt setup of ReDiverse-TF

We refer to Q5.

Q9. Why the collapse concentrates at block ℓ

We add an explanation and a per-block analysis:

(i) For reverse-KL-based students, a linearized view suggests that the objective can lower its loss by concentrating the noise-dependent variation into fewer directions, which lowers PR while keeping TV, as we measure (Table R2).

(ii) At ℓ, the student becomes more sensitive to the initial noise than the teacher (Figure R1).

Q10. Quality drop and the diversity–quality trade-off

We refer to Q3.

Q11. Automatic choice of K and α, and which variant to use

We tried three automatic rules (Table R14) and confirm that:

(i) Variance matching for α and the teacher’s spectrum for K need no grid search and keep the diversity gain, while PR matching gives none.

(ii) We recommend FT for serving many prompts, TF when training is not possible, and both for maximum diversity (Table R15).

AuthorsOpenReview ↗

We would be happy to provide further clarification if anything remains unclear

Dear AC and Reviewers,

Thank you again for your time and constructive feedback. We have uploaded a revised paper that includes the new results and clarifications from our responses, highlighted in purple for Reviewer 6X8z, blue for Reviewer x7iw and orange for Reviewer KJfY. If anything remains unclear, we would be happy to provide further clarification.

Dear AC and Reviewers,

Thank you again for your time and the constructive discussion. If there are any remaining concerns, we would be happy to address them.

As a brief update, we also found that ReDiverse works on a recently released model: Pika H3 Flex [1], a six-step video generator post-trained on MiniMax-H3, after our submission. Seed-matched qualitative results are in the “Additional results → Pika H3 Flex” section of the anonymous supplementary page: https://iclr-2027-14418.github.io/#PikaFlex.

[1]: Pika Labs. Pika H3 Flex, October 2026. https://huggingface.co/Pika-Labs/Pika-H3-Flex

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.