acceptodds
Under review as a conference paper at ICLR 2027

When More Calls Help: Dataset-Free Low-Compute Self-Distillation for MeanFlow

Abstract

Distillation has become an effective way to compress multi-step generative models into fewer steps while retaining much of the teacher’s sample quality. We observe a complementary opportunity in MeanFlow: selected multi-step samplers of an already one-step-capable checkpoint produce better samples than its one-step mode. Together, these observations suggest a simple route to a stronger one-step model: distill the better multi-step mode back into one step, keeping the quality lost in distillation smaller than the quality gained by sampling. We call this recipe **Many-to-One self-distillation (M2O)**. This transfer requires only a small adaptation budget. Using only random noise, class labels, and a frozen self-teacher from the same checkpoint, M2O matches generated distributions in a frozen visual feature space **without accessing training data**. On ImageNet-256, M2O improves iMF-B/2 from 3.35 to 3.00 FID-50K in 0.60 H200-hour. On iMF-L/2, it reaches 1.72 in 0.93 H200-hour, improving on the guidance-calibrated one-step baseline of 1.88 by 0.16 FID. A fixed-guidance control improves from 2.31 to 1.72 at CFG 5.00. Both students preserve single-step, single-branch inference. Our results show that a checkpoint’s multi-step advantage can become a practical source of low-compute, dataset-free improvement for one-step generation.

Primary area:
generative models
Keywords:
MeanFlow, one-step generation, self-distillation, dataset-free adaptation, generative models, flow models, low-compute training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RjseOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Many-to-One self-distillation (M2O), a method for distilling the higher-quality multi-step sampling distribution of a pretrained MeanFlow model into a single-step generator. Specifically, M2O minimizes a distribution-matching loss between multi-step teacher samples and single-step student samples in a pretrained visual feature space. Experiments on ImageNet-256 demonstrate meaningful improvements in FID with less than one H200 GPU-hour of additional training, while preserving single-step inference.

Critical strengths

The paper's main strengths lie in its simplicity, computational efficiency, and empirical effectiveness. The proposed approach is conceptually simple, leveraging a pretrained MeanFlow model as its own multi-step teacher and optimizing a distribution-matching objective without requiring real training data during adaptation. Particularly impressive is its computational efficiency: the main results are achieved with less than one H200 GPU-hour of additional training after teacher selection, while preserving single-step inference. Despite this small adaptation budget, the method achieves meaningful improvements in FID on ImageNet-256. The extensive ablation studies further support the effectiveness of the proposed approach and its individual components.

Critical weaknesses

Insufficient justification for the proposed distribution-matching objective. While the authors compare their objective against several paired-regression baselines, the comparisons overlook potentially stronger alternatives, such as latent-space pseudo-Huber loss or E-LatentLPIPS [2], which has demonstrated effectiveness in diffusion distillation. More importantly, the authors do not compare against alternative distribution-matching objectives, particularly FD-loss [1], which has demonstrated strong results for one-step generation. Furthermore, paired ConvNeXt regression initially performs comparably to the proposed method (1.82 vs. 1.83 FID at 4K updates) but deteriorates with further training, raising questions about whether the baselines were sufficiently optimized.

Limited methodological novelty. The proposed method largely builds upon established techniques for generative model distillation. Scale-wise Distillation [3] already investigates MMD-based distribution matching in pretrained feature spaces, while DMD [4] and FD-loss [1] demonstrate the effectiveness of distribution-level objectives for one-step generation. Furthermore, Diffusion2GAN [2] has explored distilling multi-step diffusion samples into a one-step generator using perceptual regression. The primary novelty therefore appears to lie in using the stronger multi-step sampling distribution of the same MeanFlow checkpoint as a self-distillation target, rather than introducing a fundamentally new distillation technique. While the computational efficiency is impressive, the methodological contribution seems relatively incremental.

[1] Representation Fréchet Loss for Visual Generation, arXiv, 2026.

[2] Distilling Diffusion Models into Conditional GANs, ECCV, 2024.

[3] Scale-wise Distillation of Diffusion Models, ICLR, 2026.

[4] One-step Diffusion with Distribution Matching Distillation, CVPR, 2024.

Questions for authors
  • Alternative paired-regression objectives: Could the authors compare against stronger latent-space regression losses, such as pseudo-Huber loss or E-LatentLPIPS [2]? These comparisons would help establish whether distribution matching is genuinely preferable to paired regression.
  • Alternative distribution-matching objectives: Have the authors considered FD-loss [1] as an alternative to the proposed MMD-based objective? In particular, could FD-loss be applied to the same multi-step teacher and single-step student feature distributions to isolate the contribution of the specific distribution-matching loss?
  • Distinction from prior work: Given that Scale-wise Distillation [3] already investigates MMD-based distribution matching in pretrained feature spaces, could the authors clarify the key methodological differences and novel contributions of M2O beyond applying existing distribution-matching techniques to a multi-step MeanFlow self-teacher?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Rjse: baseline tuning, alternative objectives, distinction from SwD

Thank you for the detailed review. Please see the general response above for the evaluation protocol shared by all numbers below.

1.1 Baseline tuning and seeds

We re-tuned paired ConvNeXt regression from the official checkpoint over four learning rates (3×10⁻⁷ to 10⁻⁵) with checkpoints at 100, 250, 500, 750, and 1,000 updates, in addition to the longer runs in the paper. Development FID selects LR 3×10⁻⁶ at 1,000 updates; the paper's longer runs show degradation beyond this point. With three training seeds each, M2O has the lower FID in every seed and the tuned paired baseline has the lower DINOv2 FD in every seed.

MethodFID seed 42 / 43 / 44 ↓FID mean ± SD ↓DINOv2 FD mean ± SD ↓
Tuned paired ConvNeXt1.737 / 1.765 / 1.7721.758 ± 0.018105.94 ± 1.35
M2O1.678 / 1.665 / 1.7111.685 ± 0.024109.68 ± 0.79

Fixed CFG 5; 50K evaluation per run; SD across training seeds with the evaluation bank fixed.

1.2 Alternative objectives

We evaluated latent pseudo-Huber, the authors' E-LatentLPIPS implementation (with its default L1 term), and a population FD objective (FD-loss) on the same ConvNeXt features and the same multi-step teacher, each with its own learning-rate and stopping search and development-set CFG selection (single training seed):

MethodSelected CFGFID ↓DINOv2 FD ↓
Official iMF7.51.849118.80
Latent pseudo-Huber7.51.849118.81
Author E-LatentLPIPS + L17.51.850118.89
Paired ConvNeXt (tuned)5.01.737107.41
Same-feature FD-loss5.01.700104.73
M2O5.01.678109.99

Pseudo-Huber and E-LatentLPIPS select the smallest learning rate and earliest stop in their grids; every larger learning rate or longer run worsens development FID. In this setting, latent paired regression does not transfer the multi-step advantage. FD-loss does: its FID is within 0.02 of M2O and its DINOv2 FD is better. The two distribution objectives agree, and both beat every paired objective by at least 0.04 FID. This isolates what matters: matching generated distributions in a pretrained feature space against the self-teacher, rather than the particular kernel. We will present FD-loss alongside the KID-style kernel as an equally valid instantiation.

1.3 Distinction from SwD and contribution

SwD distills an external multi-step teacher into a separate scale-wise student with patch-level MMD in diffusion features across a resolution schedule. M2O keeps one released checkpoint, uses its own multi-step mode as the frozen teacher, matches global features of decoded images, and leaves the architecture and the one-step inference path unchanged. The empirical test of this distinction is in our reply to Reviewer PvhV: the SwD MMD-only loss adapted to the same checkpoint and teacher reaches 1.834, M2O reaches 1.678. The contribution is the self-distillation setting and its validation: an already one-step-capable checkpoint improves from its own stronger multi-step mode, in under one GPU-hour, without training images or added deployment cost, and the teacher-step, feature, and objective controls identify what makes this transfer work. The distillation loss is a known tool; the finding is where to point it and how little it costs.

Thank you for the detailed rebuttal and additional experiments. The new comparisons with tuned paired regression, pseudo-Huber, E-LatentLPIPS, and FD-loss adequately address most of my concerns regarding experimental soundness.

However, my main concern regarding methodological novelty remains. The FD-loss results suggest that the specific distribution-matching objective is not essential. The authors instead emphasize the self-distillation setting, but I believe its significance beyond the practical benefit of avoiding an external teacher warrants further clarification.

In particular, what new methodological insight does the proposed self-distillation setting provide beyond applying established distribution-matching techniques to a stronger multi-step sampler? The teacher-step ablation in Table 3 establishes the importance of teacher quality, but does not isolate any distinctive property of self-distillation. Similarly, the SwD-MMD comparison uses the same checkpoint and teacher for both methods, primarily comparing distillation objectives rather than establishing the significance of the self-distillation setting. While a comparison with an external teacher could help clarify this distinction, a stronger conceptual justification would also be valuable.

I appreciate the practical efficiency and substantially strengthened empirical evidence. Nevertheless, I remain unconvinced that the work offers a sufficiently substantial methodological contribution, and therefore maintain my overall rating of weak rejection.

Reviewer_Tbz4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper observes that an iMF checkpoint produces better samples with some multi-step sampling settings than with its one-step mode. It proposes Many-to-One self-distillation (M2O), which distills this multi-step mode back into the one-step mode of the same checkpoint. The adaptation uses only random noise and class labels, and matches the teacher and student sample distributions in a frozen feature space. On class-conditional ImageNet 256×256, M2O improves the one-step FID.

Critical strengths
  1. Simple and practical. The method does not change the architecture or the one-step inference cost, uses no training images during adaptation.
  2. Interesting analysis for what part attributue to the gains.
Critical weaknesses
  1. FID is the only image-quality metric, and the training objective is closely aligned with it. The loss matches the feature moments of an ImageNet-pretrained ConvNeXtV2. FID compares the mean and covariance of features from an Inception network that is also trained on ImageNet. So part of the FID gain may come from aligning these feature statistics rather than from better images.
  2. The experiments are limited to iMF. All results use two iMF sizes on class-conditional ImageNet 256×256. So it is unclear whether the recipe works for other one-step models, or for other tasks such as text-to-image generation.
Questions for authors
  1. Can you report more metrics, such as FD_DINOv2, precision/recall, and IS, for the baseline, teacher, and student?
  2. Does the student keep its multi-step capability after adaptation? If so, can this method be iterated by using the student’s multi-step mode as the next teacher?
  3. Does this recipe transfer to other one-step model families?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Tbz4: additional metrics, multi-step mode, other families and text-to-image, presentation

Thank you for the detailed review. Please see the general response above for the evaluation protocol shared by all numbers below.

2.1 Metrics beyond Inception FID

The training loss uses ConvNeXt feature moments and FID uses Inception features, both ImageNet-trained. We therefore added DINOv2 FD (self-supervised features, no ImageNet labels), precision/recall in DINOv2 and Inception spaces, and Inception Score, each on 50,000 images with per-model development-set CFG. M2O improves FID, DINOv2 FD, DINOv2 precision, and DINOv2 recall over the official model; Inception Score is 1.4% lower. The paper's independent ResNet-50 class-agreement diagnostic also improves. The gain is a property of the images, not of the Inception feature space.

MetricOfficial iMF, CFG 7.5M2O, CFG 5
FID ↓1.8491.678
DINOv2 FD ↓118.80109.99
DINOv2 precision ↑0.5950.604
DINOv2 recall ↑0.5350.559
Inception precision ↑0.7160.727
Inception recall ↑0.6190.620
Inception Score ↑266.1262.5

Three-seed FID and DINOv2 FD are in our reply to Reviewer Rjse.

2.2 Multi-step mode after adaptation

The student remains a working multi-step sampler. At fixed CFG 5, its four-step DINOv2 FD is 80.75, the best value in the study and better than the official four-step 96.74; its four-step FID is 1.760, above its own one-step 1.678. The adaptation moves the one-step mode toward the teacher and reshapes the multi-step mode rather than preserving it, so the student's multi-step mode is a better teacher by DINOv2 FD and not by FID. A second round would therefore need teacher selection by the target metric, as in the first round; we did not run one.

ModelNFECFGFID ↓DINOv2 FD ↓
Official iMF152.257141.38
M2O151.678109.99
Official iMF451.57796.74
M2O451.76080.75

2.3 Other model families and text-to-image

M2O needs three things from a checkpoint: a usable one-step mode and a better multi-step mode of the same weights, so the teacher costs no training; conditioning inputs and noise, no images; and a frozen feature extractor. Nothing in the objective is specific to class labels or ImageNet; labels enter only through the generator, so a prompt set takes their place in text-to-image. The released iMF checkpoints are class-conditional. A text-to-image study needs a one-step-capable text-to-image checkpoint of this kind, a prompt-based development metric for teacher and CFG selection, and multi-step teacher rollouts at high resolution with a billion-parameter generator, which exceeds the compute available during the discussion period.

The evidence for transfer within the family is direct: the same recipe with unchanged hyperparameters (learning rate, batch, parameter mask, loss) works on two iMF sizes that differ by 4.6× in parameters, under selected and fixed guidance, and every control (self-target vs. multi-step target, latent vs. random vs. pretrained features, paired vs. distribution matching) orders the same way at both sizes. The recipe depends on a good one-step starting point, as the randomized-initialization control (FID 346) shows, and the gain is bounded by the teacher's margin. The claim is thus: given a one-step-capable checkpoint with a measurable multi-step margin, a sub-hour, image-free adaptation transfers part of that margin. Text-to-image and other families are the natural next step; the code will be released for that purpose.

2.4 Presentation

We revised the manuscript as described in the general response: one definition of the two L/2 references, a recipe table, removal of repeated numeric prose in Sections 4–5, a compact statement of the target-versus-representation finding, and an explicit SwD contrast in Related Work. Pointers to any passage that remains unclear are welcome.

AuthorsOpenReview ↗

Follow-up on our rebuttal and revisions

As the discussion period is drawing to a close, we kindly invite you to share any remaining questions or concerns so that we can address them before the deadline. We have added a controlled iMF-L/2 study with additional baselines, including SwD-MMD and Neon-style adaptations, evaluations beyond Inception FID, and an analysis of multi-step sampling after adaptation. We have also clarified the method’s scope and simplified the presentation. We would appreciate your feedback on whether these additions address your concerns. Thank you for your time and consideration.

Reviewer_PvhVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Many-to-One self-distillation (M2O) a simple but effective method to distill a pretrained Improved MeanFlow (iMF) to a better single step generator. The paper leverages a multi-step sampler with freezed pretrained iMF as the teacher and use KID-style distribution objective with ConvNeXtV2-Tiny features without accessing real training images during distillation. On ImageNet-256, the experiments show M2O brings one-step FID of iMF-B/2 from 3.35 to 3.00.

Critical strengths
  1. Easy method but measurable improvement
  2. Easy to read and clear contribution
Critical weaknesses
  1. Insufficient baseline comparison: Table 4 presents ARD, π\pi-Flow, FACM, and MF-RAE. Are the FID score of these 4 baselines from original paper? Can you apply these method to iMF and report the FID scores?
  2. Could you also appy Neon (Neon: Negative Extrapolation From Self-Training Improves Image Generation) and SwD (Scale-wise Distillation of Diffusion Models to iMF? I think they have identical scenario with M2O, distillation from teacher without additional training data. Could you compare these method with M2O?
Questions for authors
  1. I wonder if M2O works on regular diffusion models like DDPM or flow matching models? Does it rely on a strong one step generator? Can it generalize to broader diffusion model classes?
2 comments
AuthorsOpenReview ↗

Response to Reviewer PvhV: Table 4 sources, SwD and Neon baselines on iMF, applicability beyond MeanFlow

Thank you for the detailed review. Please see the general response above for the evaluation protocol shared by all numbers below.

3.1 Table 4 and controlled baselines on iMF

The ARD, π-Flow, FACM, and MF-RAE entries in Table 4 are published numbers under their original settings, as the caption states; they place our numbers among ImageNet generators. Each of these is a training or distillation framework around a different teacher and backbone (DiT/SiT-XL teachers, RAE latents), not a procedure for adapting a given checkpoint, so porting them to iMF would measure our re-derivation rather than their method. For controlled comparison on the same checkpoint we adapted the two methods you suggested, which share M2O's setting: the SwD MMD-only loss, and a Neon-style procedure (short u-head self-training followed by weight extrapolation). Single-step inference, development-set CFG selection, and 50K final evaluation throughout.

MethodNFESelected CFGFID ↓DINOv2 FD ↓
Official iMF17.51.849118.80
SwD-MMD adaptation17.51.834120.03
Neon-u adaptation, α = −0.2517.51.837120.58
M2O15.01.678109.99

The SwD row uses its MMD loss without the multi-resolution student or sampler; the Neon row applies the extrapolation step to our u-head adaptation over three learning rates, two step counts, and α ∈ {−1, −0.25}. Both improve FID by about 0.015 and leave DINOv2 FD unchanged. M2O improves FID by 0.17 and DINOv2 FD by 8.8. The difference is the target: a verified better multi-step mode of the same checkpoint, matched in a pretrained feature space.

3.2 Positive versus negative extrapolation

We compared positive (α = +1) and negative (α = −0.25) merges of the same adapted checkpoint. Negative extrapolation has the lower FID at both matched CFG values, and stays ahead when the positive control selects its own CFG (10.5 → 2.261 vs 1.837). The extrapolation direction reported by Neon holds here; its magnitude on this checkpoint is small.

MergeαCFGFID ↓
Positive control+15.02.984
Negative extrapolation−0.255.02.197
Positive control+17.52.322
Negative extrapolation−0.257.51.837
Positive control, own selected CFG+110.52.261

3.3 DDPM, flow matching, and the one-step starting point

M2O applies to checkpoints whose single network has both a usable one-step mode and a better multi-step mode: MeanFlow, and by the same structure consistency- and shortcut-type models. A standard DDPM or flow-matching model has no one-step mode, so for those the multi-step teacher must be distilled into a separately trained one-step student, which is the regime of DMD and SiD. The recipe relies on a good one-step starting point: the randomized-initialization control reaches 346 FID under the same budget, and the attainable gain is the teacher's margin over the one-step mode. Within these conditions the result is a sub-hour, image-free improvement of a deployed one-step model; cross-family experiments are the next step and are listed as such in the revised paper.

AuthorsOpenReview ↗

Follow-up on our rebuttal and revisions

As the discussion period is drawing to a close, we kindly invite you to share any remaining questions or concerns so that we can address them before the deadline. We have added a controlled iMF-L/2 study with additional baselines, including SwD-MMD and Neon-style adaptations, evaluations beyond Inception FID, and an analysis of multi-step sampling after adaptation. We have also clarified the method’s scope and simplified the presentation. We would appreciate your feedback on whether these additions address your concerns. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: new controlled study on iMF-L/2, protocol, and revised PDF

We thank the reviewers for their careful reading. We ran a new controlled study on iMF-L/2 (class-conditional ImageNet-256, single-step inference): nine adaptation methods and controls, three training seeds for the two strongest methods, four-step evaluations of the original and adapted models, and five metrics (FID, DINOv2 FD, precision/recall in DINOv2 and Inception spaces, Inception Score), 22 final evaluations of 50,000 images each.

Protocol. Every method selects its CFG on a separate 5K development bank over {2, 3.5, 5, 7.5, 10.5, 13} and is evaluated once on a held-out 50K bank. The held-out bank is new (sample-ID offset 6.5M, disjoint from the paper's 700K bank), so absolute values shift slightly from the paper for identical checkpoints: the paper's M2O student scores 1.72 there and 1.678 here, and the official checkpoint at fixed CFG 5 scores 2.31 there and 2.257 here. All comparisons in this response share one bank, evaluator, and reference statistics. Under this protocol the official one-step baseline selects CFG 7.5 (FID 1.849), a stronger reference than the paper's 1.88.

Summary of findings. (1) After tuning the paired ConvNeXt baseline over learning rate and stopping step, M2O still has the lower FID in each of three training seeds (1.685 ± 0.024 vs 1.758 ± 0.018). (2) Latent pseudo-Huber and E-LatentLPIPS do not move away from the released model under any learning rate in their grids. (3) FD-loss on the same features is a second effective distribution objective (FID 1.700, DINOv2 FD 104.73): distribution matching in a pretrained feature space transfers the self-teacher's gain, and the choice of kernel is secondary. (4) The gains hold in DINOv2 FD and in DINOv2 precision and recall, not only in Inception FID. (5) SwD-MMD and Neon-style adaptations of the same checkpoint improve FID by about 0.015; M2O improves it by 0.17.

Revised PDF. The two L/2 references (best-tuned CFG 10.5 → 1.88; teacher's fixed CFG 5 → 2.31) are defined once in the introduction and used consistently; a recipe table replaces scattered settings; repeated numeric prose in Sections 4–5 is removed; the contrast with SwD is stated in Related Work; the new study is Appendix E. Detailed replies to each reviewer follow under the respective reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.