- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Step distillation suffers from diversity collapse, in which samples generated under different seeds are highly similar. The authors compare teacher and student activations layerwisely and uncover the core mechanism: the distilled student model retains its total activation variance, yet variance is confined to a small set of feature directions. This collapse takes place in a localizable collapse block. Downstream layers remain capable of decoding the missing diversity directions, suggesting diversity collapse originates from imbalanced variance allocation instead of permanent loss of generative capacity.
Based on this observation, the authors present two complementary solutions:
- ReDiverse-FT: a LoRA fine-tuning approach, where adaptation is only applied to the collapse block and earlier layers; LoRA is enabled exclusively at the first inference step.
- ReDiverse-TF: a training-free inference strategy. For each prompt, precomputed principal component directions obtained from the teacher are leveraged to perturb activations for diversity recovery during inference.
The two variants may be deployed separately or jointly. The authors conduct evaluations across four pairs: SDXL and SD3 for image generation, Wan2.1 and MiniMax-H3 for video generation. Experiments include automatic metrics, human user studies and VLM-based assessment, alongside extensive ablation studies to verify design choices and hyperparameters.
Critical strengths
-
This work provides insightful mechanistic diagnosis. Rather than stacking heuristic tricks, the authors perform layer-wise activation probing and identify that diversity collapse stems from reduced effective directions despite preserved total variance. Activation steering experiments verify that downstream networks can still utilize the missing teacher directions, justifying their repair strategy. This analysis is inspiring to the field.
-
The solution is highly practical. It works on existing pre-distilled students without full re-distillation. ReDiverse-FT uses minimal LoRA parameters activated only in the first inference step. It supports both image and video generation.
-
Experiments are comprehensive: comparisons over multiple image and video models, thorough ablations, automatic metrics, human studies and VLM evaluation. Ample visualizations clearly show improved diversity in composition, pose and camera motion.
Critical weaknesses
-
ReDiverse-TF incurs upfront teacher inference for every prompt, lacks quantitative overhead analysis and mitigation strategies, hindering real-world deployment.
-
The identification of the collapse block is empirical. No theoretical explanation is provided for why variance-direction collapse occurs at this specific location during distillation. While the phenomenon is well observed, its root cause remains underexplained.
-
Diversity gains come with non-negligible drops in visual quality and human preference metrics. Hyperparameters K and α require manual tuning without an adaptive selection scheme, and configuration guidance for different application scenarios is absent.
Questions for authors
-
ReDiverse-TF requires multiple forward passes of the teacher per prompt to estimate principal directions. Could the authors quantitatively report the computational overhead of this preprocessing stage? Are there strategies to reuse these directions across prompts to reduce inference cost?
-
The collapse block is observed across multiple models, yet a theoretical justification is missing: why does variance-direction collapse concentrate within this particular block during distillation? Could the authors further analyze the underlying triggers of this phenomenon?
-
The method exhibits a diversity-quality trade-off. Can an adaptive strategy be designed to automatically select injection strength hyperparameters K and α, instead of grid-search tuning? For different downstream use cases, would the authors recommend FT, TF, or their combination?
4 comments
Response [2/2]
W2 and Q2 (continued)
The collapse arises at the first step because all of the seed’s information enters there (Gandikota & Bau, 2026); at block ℓ the student turns from slightly less to more sensitive to the initial noise than the teacher, at a larger activation scale (Figure R1).
Table R2. PR and TV of each student at block ℓ, relative to its teacher.
(a) SDXL students
| Student | Family | PR at ℓ, ÷ teacher | TV at ℓ, ÷ teacher | DINO ↑ |
|---|---|---|---|---|
| DMD2 | distribution matching | 0.39 | 1.57 | 59.0 |
| LCM-SDXL | consistency | 0.79 | 1.46 | 65.9 |
| PCM-SDXL | phased consistency | 0.59 | 1.56 | 63.7 |
| SDXL-Turbo | adversarial | 0.51 | 3.15 | 61.1 |
(b) SD3 students
| Student | Family | PR at ℓ, ÷ teacher | TV at ℓ, ÷ teacher | DINO ↑ |
|---|---|---|---|---|
| TDM | distribution matching | 0.54 | 0.97 | 47.9 |
| PCM-SD3 | phased consistency | 0.57 | 0.96 | 57.8 |
(c) Wan 2.1 video students (250 noises, vs 1,000 in Tab. 1)
| Student | Family | PR at ℓ, ÷ teacher | TV at ℓ, ÷ teacher | DINO ↑ |
|---|---|---|---|---|
| lightx2v | distribution matching | 0.84 | 1.41 | 31.1 |
| rCM | consistency + score distillation | 0.86 | 1.35 | 29.0 |
Figure R1 (rebuttal.html). Activation scale and noise sensitivity at each block of the first step, SDXL teacher vs DMD2 student.
W3 and Q3 · Quality, hyperparameters and configuration
W3.1 and Q3.1 · Quality drop and the diversity–quality trade-off
Diversity gains come with non-negligible drops in visual quality and human preference metrics.
- Quality preservation is comparable to the baselines. On SDXL, DP-DMD loses a similar amount without raising diversity; FT or TF alone is at most 5% below the teacher, and the larger drop is for FT+TF, mainly on SD3 (Tab. 4).
- Human raters prefer ReDiverse. In the paper’s user study, it is preferred to the student and the other diversity methods in realism for images and videos, to the teacher for images, and to every method in prompt following (Fig. 10).
- The reward models are imperfect. They may favor the more saturated, higher-contrast look of the students’ images, which score above their own teachers (Tab. 4); ReDiverse moves these properties part of the way back (Table R13).
Table R13. Saturation, contrast and sharpness of the images on SDXL.
| Method | Saturation | Contrast | Sharpness | ImageReward ↑ |
|---|---|---|---|---|
| Teacher | 0.27 | 0.206 | 489 | 0.69 |
| Student | 0.41 | 0.258 | 464 | 0.88 |
| ReDiverse-TF | 0.41 | 0.254 | 470 | 0.87 |
| ReDiverse-FT | 0.38 | 0.245 | 502 | 0.72 |
| ReDiverse (FT+TF) | 0.37 | 0.242 | 519 | 0.68 |
W3.2 and Q3.2 · Choosing K and α automatically
Can an adaptive strategy be designed to automatically select injection strength hyperparameters K and α, instead of grid-search tuning?
We tried three automatic rules, computed per prompt with no grid search (Table R14):
- α by variance matching (works). Along each direction, α’s range adds the variance the student’s first step lacks relative to the teacher, and nothing where it already matches; it keeps most of the grid-searched gain.
- K from the teacher’s spectrum (works). K is the smallest number of directions explaining 70% of the variance of the teacher’s 25-noise PCA (13–16 per prompt); it is more diverse than the paper’s K = 8 at the same reward.
- α by PR matching (does not work). α is set so that the student’s PR at ℓ matches the teacher’s; as adding variance along a few directions lowers PR, it picks α ≈ 0, with no gain. What matters is which directions TF restores (App. E.7).
Table R14. Automatic choice of α and K for ReDiverse-TF (the 80 prompts of Table R12).
(a) Injection strength α (K = 8)
| Setting | α | DINO ↑ | ImageReward ↑ |
|---|---|---|---|
| Student (no TF) | – | 60.0 | 0.93 |
| Grid search (paper) | U[−2, 2] | 60.7 | 0.92 |
| Automatic, variance matching | per direction | 60.5 | 0.90 |
| Automatic, PR matching | per prompt | 60.0 | 0.92 |
(b) Number of directions K (α ~ U[−2, 2])
| Setting | K | DINO ↑ | ImageReward ↑ |
|---|---|---|---|
| Student (no TF) | 0 | 60.0 | 0.93 |
| Fixed (paper) | 8 | 60.7 | 0.92 |
| Automatic (70% of the 25-noise PCA’s variance) | 13–16, per prompt | 61.0 | 0.92 |
W3.3 and Q3.3 · Which variant to use
For different downstream use cases, would the authors recommend FT, TF, or their combination?
FT when one student serves many prompts, TF when training is not possible, and both for maximum diversity (Table R15).
Table R15. Recommended ReDiverse variant for each use case.
| Use case | Recommended | Why |
|---|---|---|
| Serving many users and prompts (e.g. an API) | ReDiverse-FT | Trained once (about 8 min); no teacher at inference. |
| No training possible (no pipeline, data or compute) | ReDiverse-TF | No training; one setup per prompt, shared by its samples. |
| Many samples per prompt (exploration, synthetic data) | ReDiverse (FT+TF) | Largest diversity gain; the setup is shared by the samples. |
| Quality first | ReDiverse-FT, or TF with a smaller K or α | Keeps teacher-level ImageReward (Tab. 4). |
| A resolution the teacher was not trained for (e.g. SDXL-Turbo) | ReDiverse-FT | The teacher’s directions are poor at that resolution. |
We would be happy to clarify any remaining concerns.
Dear Reviewer KJfY, we would like to kindly remind you of our response above. In the revised paper, the changes addressing your comments are highlighted in orange:
- Section 3.1: A pointer to why the collapse concentrates at block ℓ.
- Section 5.2: Pointers to the automatic choice of K and α and to the recommended variant for each use case.
- Appendix A.2 (Figure R1): A linearized view of why reverse-KL distillation concentrates the seed-dependent variance into fewer directions, and the student's higher sensitivity to the initial noise at block ℓ.
- Appendix E.10 (Table R14): K and α set per prompt without a grid search: α by variance matching, K from the teacher's spectrum.
- Appendix E.11 (Table R15): The recommended variant for each use case.
- Table 7 and Appendix D.4 (in blue): The overhead of the ReDiverse-TF setup, its sharing across a prompt's samples, and the reuse of its directions across prompts.
- Appendix E.9 (in purple): The gain of ReDiverse-TF holds with fewer teacher noises.
Dear Reviewer,
As the discussion phase is approaching its end, we would greatly appreciate it if you could let us know whether our clarifications and additional experiments have adequately addressed your remaining concerns.
We would be happy to address any further questions before the discussion period ends. Thank you again for your time and constructive engagement throughout the discussion!
Best regards,
All Authors
Response [1/2]
We thank the reviewer for finding the diagnosis insightful and the method practical. In short, TF’s setup is a one-time fixed cost shared by a prompt’s samples, and its directions can be reused across prompts (W1, Q1); for reverse-KL-based students, the objective and a per-block analysis help explain the collapse (W2, Q2); and quality preservation is comparable to baselines, K and α can be set automatically, and each variant suits a different use case (W3, Q3).
New results are in Tables R2, R4 and R10 to R15 below and in Figure R1, which is in the supplementary material (rebuttal.html). Image results use a GenEval subset of 100 prompts × 20 seeds, and all timings are on one B200 GPU.
W1 and Q1 · Cost of ReDiverse-TF
W1.1 and Q1.1 · Overhead of the per-prompt setup
The setup is a one-time fixed cost of a few teacher samples per prompt, shared by its samples, so TF is cheaper than the teacher after a few samples (Table R11). Table R10 splits the setup by stage; sampling then costs the same as the student.
Table R10. Time of each stage of ReDiverse-TF on SDXL.
Table R11. Time per image (s) on SDXL when n samples of a prompt share one ReDiverse-TF setup.
Q1.2 · Reusing the directions across prompts
Yes. The cost can be reduced in four ways:
Table R12. ReDiverse-TF with directions taken from other prompts (80 of the prompts).
Table R4. Sensitivity of ReDiverse-TF to the teacher noises and to the prompt wording (first 50 prompts).
(a) Directions from N teacher noises (five disjoint sets)
(b) Directions from a paraphrase of the prompt (N = 25)
W2 and Q2 · Why the collapse concentrates at this block
We agree that the location is found empirically, and add an explanation and a per-block analysis (Figure R1).
Linearizing the first-step activation of a block b, hb(ϵ)≈μb+Jbϵ, gives Σb=JbJb⊤, TV=trΣb and PR=(trΣb)2/tr(Σb2). For reverse-KL-based distillation, as in the distribution-matching students, the loss penalizes samples outside the teacher’s modes but not missed modes, so the student can lower it by turning Jb toward fewer directions without shrinking it: PR falls while TV stays, the signature we measure. The objective appears to set how much diversity is lost: LCM-SDXL, whose loss ties each noise to the teacher’s trajectory, stays close to the teacher, while the distribution-matching students and rCM, which adds a score-distillation term, lose the most (Table R2).