acceptodds
Withdrawn from ICLR 2027

Breaking to Preserve: The Geometry of Few-Step Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-step distillation is commonly viewed as compression, but this perspective overlooks what the student must acquire beyond the base model. Under six-dimensional input perturbations, six directions capture 99% of response variation in the SD1.5 base model and standard LoRA, compared with 26 for LCM-LoRA and 15 for DMD2 at one perturbation scale. Across six backbones, 90.5–99.2% of the energy in teacher–base response changes lies outside the base span; we call this out-of-span component the teacher margin. Exact matching requires recovering this margin. Any base-confined student therefore faces a matching-error floor, and feature spread alone neither recovers the margin direction nor preserves prompt following. Our breaking–fidelity decomposition reveals distinct learning requirements that equal total errors can obscure and motivates diagnostics of spread, reference alignment, and concentration. For a given starting recipe, these diagnostics predict which control is missing, explaining why the preferred complementary intervention reverses across recipes. On FLUX, complete samplewise and covariance students improve alignment with the teacher's added responses over trained distribution-matching (DM) at all 12 measured pairs and improve generation, while mean base-component error falls and complete response error rises. Translating these diagnostics into training objectives, we compare two implementations; covariance optimizes the diagnosed quantities more directly and outperforms its samplewise counterpart on five of six metrics on each backbone. Across five backbones, our one-step SD1.5 student improves all six metrics over official DMD2, and one-step klein beats its four-step counterpart on Patch FID-T.

Primary area:
generative models
Keywords:
few-step distillation, one-step image generation, diffusion models, representation geometry, teacher margin, distillation diagnostics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YHmDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

If I understand correctly, the paper reports a finding about how few-step generators respond to small perturbations of the input. Compared with the original base model, the output changes of a few-step generator span more linearly independent directions. The authors then try to promote this diversity directly as an optimization objective, to improve few-step generation.

Critical strengths

I did not find any.

Critical weaknesses
  1. The paper is nearly unreadable. I have read a large body of work on few-step distillation of diffusion models and have published and practiced in this area. Even so, I could not identify the key points of this paper. It casually introduces new concepts and terminology, and its appendix is extremely long. I do not think writing of this kind should be published under any circumstances. The paper needs a complete rewrite, not to become acceptable, but simply so that its content can be reviewed seriously.

  2. A simple and trivial method is obscured by misleading framing. The method itself is not complicated. On top of DMD2, it adds a feature-covariance term that encourages diversity and a relational distillation term on features. This could have been presented clearly. Instead, the paper wraps it in misleading and unclear arguments and turns the whole manuscript into a disaster. As a method contribution, I also consider it trivial.

  3. The logic of the paper is broken. Most of the paper describes how small perturbations of the input affect the output, and relates the diversity of this response to few-step generation. What is actually introduced in training, however, is the diversity of the output features themselves, computed over samples from independent noises, not the diversity of the model's response to input perturbations. The paper offers no convincing link between the two.

Questions for authors

See weaknesses

Reviewer_dnbXOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies few-step distillation of diffusion/flow models. Its central idea is to use the one-call base model as a functional reference and decompose the teacher response into a base-span component and an orthogonal teacher margin. The authors show that most teacher–base response change lies outside the base span, derive an exact breaking/fidelity decomposition of response error, and use feature diagnostics (spread, reference alignment, concentration) to select complementary expansion/alignment controls. The final recipe combines distribution matching, covariance expansion, within-prompt covariance alignment, and adversarial refinement, and is evaluated on five backbones.

Critical strengths
  1. Interesting and useful problem framing. Treating the one-step base as a reference point rather than only an initialization is a clear conceptual contribution for few-step distillation. 2.Clean error decomposition. The breaking/fidelity split gives a principled way to distinguish “missing teacher margin” from “losing base component,” supported by spectral-cost and rank arguments.
  2. Falsifiable diagnostic methodology. The control-swap experiments on LCM and FeatNN are convincing: the predicted preference reversal is observed, which is stronger than simply adding losses and reporting gains.
Critical weaknesses
  1. Moderate algorithmic novelty. The individual losses (DM, GAN, DINO features, covariance alignment/expansion) are largely existing tools; the main novelty is the analysis/diagnosis/selection framework rather than a new distillation objective.
  2. The theoretical decomposition is exact but somewhat definitional. Orthogonal projection plus Eckart–Young is standard; “exact response matching requires the teacher margin” follows from the chosen decomposition and does not by itself establish that generation quality requires matching that margin.
  3. Main recipe is one-step. Multi-step evaluation is a fixed-weight 1/2/4-step sweep, not multi-step chain distillation. The paper’s claims are therefore narrower than “few-step distillation” in general. 4.The paper’s core insight is the base-referenced teacher-margin decomposition, but the actual training objective optimizes DINO-feature covariance statistics, DM, and GAN—not breaking/fidelity. The decomposition is only used for diagnosis and post-hoc validation.
Questions for authors
  1. How sensitive are the margin energy share, breaking/fidelity split, and student ranking to the choice of base reference (e.g., different time step or initial noise)?

2 Margin alignment improves but complete response error increases. Is there a weighted or amplitude-calibrated response metric under which the student also beats DM? If not, how do you rule out metric-induced bias?

3 Quantitatively, how well do DINO spread/alignment diagnostics correlate with teacher-margin alignment on the same checkpoints? Did you observe any diagnostic-response disagreement?

Reviewer_JwgTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper wants to understand what few step distillation is capable of. They find that many directions are needed to capture any input perturbation. Inspired by this, they proposed a host of interventions designed to solve this issue. Combining these interventions led at an algorithm with improved performance on downstream benchmarks.

Critical strengths
  1. The paper has a scientific focus on understanding the problem space. The paper has a lot of ablations and metrics tracking quantities that might be of relevance
  2. In the main comparison in Table 4, the paper has a lot of tracked benchmarks as well as baselines to compare against. In earlier ablations, there are more controlled comparisons. The numbers do seem like improvements over baselines in various locations.
Critical weaknesses

I am struggling to understand the core research question of this paper. Is it trying to provide an analysis of the function rank, is it trying to propose an algorithmic improvement, etc. From the abstract, introduction, and body order, I couldn't infer what my takeaways from the paper should be (perhaps this is reflected in a subpar summary from me above). Regardless of the results, I would have trouble accepting this paper unless it was able to clearly communicate its problem statement. For some specific callouts, I don't have great understanding of why we studied the input perturbations, what the actual algorithm is (is it defined in the Appendix?), how I should intepret many of the tables/results, etc.

Questions for authors

How much of the paper writing is AI-generated? I had trouble reading most of the paper and I'm wondering if its related to this aspect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.