acceptodds
Under review as a conference paper at ICLR 2027

StateMix: Complementary State Sources for Few-Step Diffusion Distillation

Abstract

Few-step diffusion distillation has advanced through a variety of training objectives. Beyond the objective itself, the states on which a student is trained also shape the distillation process. We introduce noise-start states, sampled directly from a fixed Gaussian prior at intermediate target times, as a new training-state source for distribution matching. Controlled toy experiments reveal complementary learning behaviors: noise-start training strengthens responses to changed conditions, whereas rollout-start training on states produced by the student’s preceding denoising steps transmits more input-state variation. Building on this complementarity, we propose StateMix, a mixed-state training strategy that combines both sources under the same DMD2 supervision without changing inference. On text-conditioned SDXL and Qwen-Image, StateMix achieves leading human preference scores among the evaluated methods, with the mixture ratio allowing a trade-off between preference scores and similarity to the teacher’s generated distribution. We further evaluate StateMix on Qwen-Image-Edit with both text and image conditions, where public benchmarks demonstrate its effectiveness. In human GSB evaluation, the student meets all eight application-specific quality criteria relative to the teacher, with no loss under the production acceptance standard and even surpassing the teacher on most evaluation dimensions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Models, Diffusion Distillation, Few-Step Generation, Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_53eEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes StateMix, a mixed-state training strategy for distribution matching diffusion distillation. StateMix combines noise-start states sampled from a fixed Gaussian prior with intermediate states generated by the student, while retaining DMD2 supervision and leaving inference unchanged. The authors evaluate the method on text-to-image generation and image editing using public benchmarks, supplemented by human GSB evaluation for image editing.

Critical strengths

The core idea is simple and easy to understand. It modifies the training procedure while preserving the inference process.

Critical weaknesses
  1. The motivation for noise-start training could be clearer.
    The paper does not convincingly explain why fixed Gaussian inputs at intermediate times should improve a sampler that uses rollout states during inference. Section 3.2 explicitly shows that matching the pooled training distribution does not ensure matching the final inference distribution.

  2. The two DMD2 baselines need a more direct comparison.
    Appendix C.1 explains that Table 4 uses a separate fixed-budget ablation model with other training variables controlled. However, DMD2 performs substantially worse than in Table 1, while StateMix’s results remain identical. A direct comparison of training budgets, configurations, and checkpoint-selection criteria would help explain this gap.

  3. Numerical reporting could be more concise.
    Appendix C.1 reports full-precision HPSv3 values such as 9.994385379046202. Although this is explicitly acknowledged, rounded values in the prose would improve readability, with full-precision records provided separately.

Questions for authors
  1. Could the authors provide a direct comparison of the training configurations, update counts, and checkpoint-selection criteria for the DMD2 models in Tables 1 and 4? Which differences could explain the performance gap?

  2. What motivates using a fixed Gaussian prior at intermediate target times, despite its difference from the states encountered during inference?

Reviewer_1GAiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies DMD2 few-step diffusion distillation and identifies that the source of intermediate states used for training strongly impacts the behavior of distilled models. Existing DMD2 is trained with rollout-start states; the authors propose noise-start states as an alternative state source.

Controlled Gaussian mixture toy experiments reveal their complementary traits: noise-start training yields strong conditional responsiveness yet tends to erase inherent input variations, while rollout-start training better preserves input variations but responds poorly to condition switches. Leveraging this complementarity, the authors propose StateMix, which mixes the two state sources with probability p during DMD2 training and keeps the inference pipeline unchanged.

On 4-step text-to-image tasks with SDXL and Qwen-Image, StateMix outperforms baseline DMD2 and competitors such as PiFlow on human-preference metrics including HPS and ImageReward, at the cost of a trade-off on FID (distribution alignment with the teacher). Evaluations are further conducted on Qwen-Image-Edit for image editing, showing improvements on public benchmarks CompBench and GEditBench v2. Large-scale human GSB evaluation demonstrates that StateMix passes all eight production-quality thresholds, whereas vanilla DMD2 passes only one. Ablations investigate the performance trade-off introduced by mixing probability p.

Critical strengths
  1. This work identifies complementary properties between the two training state sources, with toy experiments clearly demonstrating the trade-off.
  2. The method is simple to implement and adds no inference overhead, achieving metric gains on image generation and editing tasks.
Critical weaknesses
  1. The core complementary properties are mainly empirically observed from toy experiments without theoretical grounding. The generalization from toy setups to real images relies on empirical transfer and lacks causal evidence, raising risks of confounding factors.

  2. The hyperparameter p is tuned via grid search, and no practical strategy for its automatic selection is provided. Model performance is sensitive to p, and the value must be searched experimentally for each mode

Questions for authors
  1. The complementary effect is observed in toy experiments, yet its transfer to image settings remains empirical. Could the authors further analyze the underlying mechanism that enables the noise-start or rollout-start trade-off seen in toy experiments to recur in the high-dimensional image latent space? Can additional evidence be provided to rule out confounding factors?

  2. Are the key real-world experimental results reproduced across multiple random seeds? If yes, please report metric variance over seeds. If not, please discuss result robustness. Ablations of different p values on SDXL should also be supplemented.

  3. Since p currently requires grid search, is there a principled way to set p automatically based on the task and dataset instead of sweeping through values empirically?

Reviewer_eBCzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StateMix changes the training inputs used for few-step distribution matching distillation: at a sampled generation time, it selects either a detached student rollout state or fresh Gaussian noise, while retaining the DMD2 supervision and inference procedure. A controlled conditional Gaussian-mixture study finds complementary responses to changed conditions and input perturbations. The paper evaluates four-step SDXL and Qwen-Image generation, eight-step Qwen-Image-Edit, a mixture-probability ablation, training cost, and an application-specific human study. Reported automatic preference scores improve, with trade-offs in teacher-distribution similarity and diversity; the editing student passes the specified human-evaluation gates.

Critical strengths
  • The training-state intervention is simple and supported by a carefully controlled ablation study.
  • The results suggest a useful practical modification without additional inference complexity.
Critical weaknesses
  • The theoretical framing does not establish that mixing is beneficial—it actually shows pooled matching can be misleading. The paper's own Proposition shows that matching the pooled mixture Q=∑iaiQiQ = \sum_i a_i Q_i does not imply matching any component QiQ_i; the decomposition leaves a nonnegative Jensen–Shannon residual. This undercuts the mechanistic story: if the fake score only ever sees the pooled law, there is no theoretical guarantee that StateMix improves the inference distribution.
  • I suspect that the improvement actually comes from time-shift generation. As early as Diff-Instruct—the first paper to study diffusion score distillation—the authors found that, in single-step generation, it is beneficial for training not to set the timestep to TT but to a smaller value. Rather than StateMix’s improvement coming from improving the inference distribution, I think it is more likely that the actual improvement comes from the time shift.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.