- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Many-to-One self-distillation (M2O), a method for distilling the higher-quality multi-step sampling distribution of a pretrained MeanFlow model into a single-step generator. Specifically, M2O minimizes a distribution-matching loss between multi-step teacher samples and single-step student samples in a pretrained visual feature space. Experiments on ImageNet-256 demonstrate meaningful improvements in FID with less than one H200 GPU-hour of additional training, while preserving single-step inference.
Critical strengths
The paper's main strengths lie in its simplicity, computational efficiency, and empirical effectiveness. The proposed approach is conceptually simple, leveraging a pretrained MeanFlow model as its own multi-step teacher and optimizing a distribution-matching objective without requiring real training data during adaptation. Particularly impressive is its computational efficiency: the main results are achieved with less than one H200 GPU-hour of additional training after teacher selection, while preserving single-step inference. Despite this small adaptation budget, the method achieves meaningful improvements in FID on ImageNet-256. The extensive ablation studies further support the effectiveness of the proposed approach and its individual components.
Critical weaknesses
Insufficient justification for the proposed distribution-matching objective. While the authors compare their objective against several paired-regression baselines, the comparisons overlook potentially stronger alternatives, such as latent-space pseudo-Huber loss or E-LatentLPIPS [2], which has demonstrated effectiveness in diffusion distillation. More importantly, the authors do not compare against alternative distribution-matching objectives, particularly FD-loss [1], which has demonstrated strong results for one-step generation. Furthermore, paired ConvNeXt regression initially performs comparably to the proposed method (1.82 vs. 1.83 FID at 4K updates) but deteriorates with further training, raising questions about whether the baselines were sufficiently optimized.
Limited methodological novelty. The proposed method largely builds upon established techniques for generative model distillation. Scale-wise Distillation [3] already investigates MMD-based distribution matching in pretrained feature spaces, while DMD [4] and FD-loss [1] demonstrate the effectiveness of distribution-level objectives for one-step generation. Furthermore, Diffusion2GAN [2] has explored distilling multi-step diffusion samples into a one-step generator using perceptual regression. The primary novelty therefore appears to lie in using the stronger multi-step sampling distribution of the same MeanFlow checkpoint as a self-distillation target, rather than introducing a fundamentally new distillation technique. While the computational efficiency is impressive, the methodological contribution seems relatively incremental.
[1] Representation Fréchet Loss for Visual Generation, arXiv, 2026.
[2] Distilling Diffusion Models into Conditional GANs, ECCV, 2024.
[3] Scale-wise Distillation of Diffusion Models, ICLR, 2026.
[4] One-step Diffusion with Distribution Matching Distillation, CVPR, 2024.
Questions for authors
- Alternative paired-regression objectives: Could the authors compare against stronger latent-space regression losses, such as pseudo-Huber loss or E-LatentLPIPS [2]? These comparisons would help establish whether distribution matching is genuinely preferable to paired regression.
- Alternative distribution-matching objectives: Have the authors considered FD-loss [1] as an alternative to the proposed MMD-based objective? In particular, could FD-loss be applied to the same multi-step teacher and single-step student feature distributions to isolate the contribution of the specific distribution-matching loss?
- Distinction from prior work: Given that Scale-wise Distillation [3] already investigates MMD-based distribution matching in pretrained feature spaces, could the authors clarify the key methodological differences and novel contributions of M2O beyond applying existing distribution-matching techniques to a multi-step MeanFlow self-teacher?
2 comments
Thank you for the detailed rebuttal and additional experiments. The new comparisons with tuned paired regression, pseudo-Huber, E-LatentLPIPS, and FD-loss adequately address most of my concerns regarding experimental soundness.
However, my main concern regarding methodological novelty remains. The FD-loss results suggest that the specific distribution-matching objective is not essential. The authors instead emphasize the self-distillation setting, but I believe its significance beyond the practical benefit of avoiding an external teacher warrants further clarification.
In particular, what new methodological insight does the proposed self-distillation setting provide beyond applying established distribution-matching techniques to a stronger multi-step sampler? The teacher-step ablation in Table 3 establishes the importance of teacher quality, but does not isolate any distinctive property of self-distillation. Similarly, the SwD-MMD comparison uses the same checkpoint and teacher for both methods, primarily comparing distillation objectives rather than establishing the significance of the self-distillation setting. While a comparison with an external teacher could help clarify this distinction, a stronger conceptual justification would also be valuable.
I appreciate the practical efficiency and substantially strengthened empirical evidence. Nevertheless, I remain unconvinced that the work offers a sufficiently substantial methodological contribution, and therefore maintain my overall rating of weak rejection.
Response to Reviewer Rjse: baseline tuning, alternative objectives, distinction from SwD
Thank you for the detailed review. Please see the general response above for the evaluation protocol shared by all numbers below.
1.1 Baseline tuning and seeds
We re-tuned paired ConvNeXt regression from the official checkpoint over four learning rates (3×10⁻⁷ to 10⁻⁵) with checkpoints at 100, 250, 500, 750, and 1,000 updates, in addition to the longer runs in the paper. Development FID selects LR 3×10⁻⁶ at 1,000 updates; the paper's longer runs show degradation beyond this point. With three training seeds each, M2O has the lower FID in every seed and the tuned paired baseline has the lower DINOv2 FD in every seed.
Fixed CFG 5; 50K evaluation per run; SD across training seeds with the evaluation bank fixed.
1.2 Alternative objectives
We evaluated latent pseudo-Huber, the authors' E-LatentLPIPS implementation (with its default L1 term), and a population FD objective (FD-loss) on the same ConvNeXt features and the same multi-step teacher, each with its own learning-rate and stopping search and development-set CFG selection (single training seed):
Pseudo-Huber and E-LatentLPIPS select the smallest learning rate and earliest stop in their grids; every larger learning rate or longer run worsens development FID. In this setting, latent paired regression does not transfer the multi-step advantage. FD-loss does: its FID is within 0.02 of M2O and its DINOv2 FD is better. The two distribution objectives agree, and both beat every paired objective by at least 0.04 FID. This isolates what matters: matching generated distributions in a pretrained feature space against the self-teacher, rather than the particular kernel. We will present FD-loss alongside the KID-style kernel as an equally valid instantiation.
1.3 Distinction from SwD and contribution
SwD distills an external multi-step teacher into a separate scale-wise student with patch-level MMD in diffusion features across a resolution schedule. M2O keeps one released checkpoint, uses its own multi-step mode as the frozen teacher, matches global features of decoded images, and leaves the architecture and the one-step inference path unchanged. The empirical test of this distinction is in our reply to Reviewer PvhV: the SwD MMD-only loss adapted to the same checkpoint and teacher reaches 1.834, M2O reaches 1.678. The contribution is the self-distillation setting and its validation: an already one-step-capable checkpoint improves from its own stronger multi-step mode, in under one GPU-hour, without training images or added deployment cost, and the teacher-step, feature, and objective controls identify what makes this transfer work. The distillation loss is a known tool; the finding is where to point it and how little it costs.