acceptodds
Withdrawn from ICLR 2027

DifferenceFlow: Dynamic Implicit Differences for One-Step Generative Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

MeanFlow has emerged as a powerful framework for training one-step generative models from scratch, without relying on pretrained teachers or multi-stage distillation. However, its derivative-based training target, which involves temporal derivatives evaluated through Jacobian-vector products, can be difficult to optimize early in training. Recent approaches such as AlphaFlow alleviate this difficulty through target interpolation, but their interpolation weight is heuristic rather than derived from the MeanFlow identity, which biases the target toward the instantaneous velocity. To address these limitations, we propose **DifferenceFlow**, a discrete formulation of MeanFlow that replaces the temporal derivative with a finite difference between model predictions along the same trajectory. Derived directly from the discretized MeanFlow identity, the DifferenceFlow target removes this bias and is second-order accurate in the interval length, while shrinking the interval during training yields a coarse-to-fine curriculum toward the original MeanFlow objective. On class-conditional ImageNet-1000 at resolution, our best model with fine-tuning outperforms state-of-the-art MeanFlow-based methods, achieving an FID of 2.29 with a single function evaluation.

Primary area:
generative models
Keywords:
One-step generation, MeanFlow, Flow matching, Finite difference, Curriculum learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yHDUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DifferenceFlow introduces a training curriculum for one-step generative models built on MeanFlow. It derives a target from a finite-difference approximation of the MeanFlow identity, replacing AlphaFlow’s mixing weight α\alpha with α/(1+α)\alpha/(1+\alpha). The curriculum progressively reduces α\alpha and ultimately switches back to JVP-based MeanFlow training. Its analysis shows that the proposed target cancels a leading velocity-gap bias and achieves second-order accuracy in interval length under the stated assumptions. Evaluation covers Swiss-roll and ImageNet-256×256, including training-recipe ablations and baseline comparisons. DiT-B/2 achieves a 1-NFE FID of 5.41, improving to 2.29 with additional FD-loss fine-tuning.

Critical strengths
  1. Principled target design and error analysis. The paper derives its mixing coefficients from MeanFlow identity and shows that they cancel a leading velocity-gap bias present in AlphaFlow’s target under the stated assumptions. This provides a clear technical rationale for the modification and improves understanding of target accuracy in MeanFlow-based formulation.

  2. Empirical gains before additional fine-tuning. Using DiT-B/2, the method achieves an ImageNet 1-NFE FID of 5.41, compared with 5.60 for AlphaFlow and 6.14 for MeanFlow in the authors’ reproductions, with improvements across several feature-space metrics. These results provide evidence that the proposed curriculum has practical value independently of the additional FD-loss fine-tuning.

Critical weaknesses
  1. Limited novelty and an insufficient connection between theory and final performance. DifferenceFlow primarily modifies AlphaFlow’s finite-α\alpha target weights and ultimately switches back to JVP-based MeanFlow for final optimization. The ideal-target bias analysis does not establish whether the proposed correction explains the gains after this shared MeanFlow stage, particularly with bootstrapped predictions and adaptive loss weighting. This limits the methodological advance and leaves the practical significance of the theoretical correction insufficiently demonstrated.

  2. Incomplete comparisons with relevant recent methods. The controlled experiments focus on MeanFlow and AlphaFlow with DiT-B/2, omitting relevant developments such as iMF. Although Table 3 includes broader benchmark results, it does not provide matched training and fine-tuning controls. In particular, the best FID of 2.29 uses additional FD-loss fine-tuning without corresponding MeanFlow or AlphaFlow controls. These omissions weaken both the assessment of competitiveness and the attribution of the strongest results to the proposed target.

Questions for authors
  1. For Table 4, were training budgets, time sampling, flow-matching ratios, and loss weighting matched across methods? Did AF and DF use identical α\alpha schedules and the same duration of final JVP-based MeanFlow training?
  2. How consistent is the FID improvement from 5.60 to 5.41 across random seeds?
  3. Were AF and MF evaluated with the same FD-based fine-tuning as DF? If so, what FIDs did they achieve?
  4. What practical advantages does the proposed formulation offer over Improved MeanFlow and pixel MeanFlow, and what evidence supports them?
2 comments

We thank the reviewer for the careful review and constructive feedback. We appreciate your recognition of our principled target design and error analysis, particularly the cancellation of the leading velocity-gap bias under the stated assumptions. We also appreciate your acknowledgment of the empirical gains across multiple feature-space metrics before additional FD-loss fine-tuning. We address your concerns below. All changes in the revised manuscript are highlighted in red.

Q1: Fairness of training configurations in the AF–DF comparison.

A: For comparisons within the same training setting in Table 4, training budgets, time sampling, flow-matching ratios, and loss weighting are matched. AF and DF use the same α\alpha schedule and switch to JVP-based MeanFlow training at the same stage, with identical final JVP training durations. Thus, the primary difference is the target interpolation coefficients during finite-difference training.

Q2: Multi-seed consistency of FID improvements.

A: In additional evaluations, we found an existing DifferenceFlow checkpoint achieving a lower FID than the originally reported 5.41. Keeping this checkpoint fixed and varying only the sampling seed, DifferenceFlow achieved 5.2406±0.04365.2406 \pm 0.0436 (mean ±\pm SD) across ten seeds with 50,000 samples each (Table R3-Q2). Scores ranged from 5.1851 to 5.3085, all below AlphaFlow's reported 5.60. AlphaFlow's FIDs across sampling seeds were also consistent with its reported value. These evaluations demonstrate robustness to sampling randomness, but not to independent training seeds.

Table R3-Q2: FID across different seeds.

Run01234
FID ↓5.25445.20055.30855.18515.2773
Run56789
FID ↓5.22675.28085.27075.18565.2169

Mean ± SD: 5.2406±0.04365.2406 \pm 0.0436

Q3: Matched Fréchet-distance fine-tuning for AF, MF, and DF.

A: We applied identical Fréchet-distance fine-tuning to AF, MF, and DF. As shown in Table R3-Q3, DF achieves an FID of 2.29, outperforming AF (2.49) and MF (2.67). DF also leads on 15 of 16 metrics across multiple representation spaces, confirming its advantage under matched fine-tuning.

Table R3-Q3: Comparison after Fréchet-distance fine-tuning under the B/2+ setting with CFG.

MetricDifferenceFlowAlphaFlowMeanFlow
FDr-6 ↓15.0016.9017.75
IS ↑251.44253.26246.17
FID(JiT) FD ↓2.372.582.74
FID(ADM) FD ↓2.292.492.67
FID(ADM) FDr ↓1.361.481.59
MAE cls FD ↓0.420.490.51
MAE cls FDr ↓10.6312.1412.84
MAE avg FD ↓1.201.471.63
DINOv2 cls FD ↓207.55235.63248.13
DINOv2 cls FDr ↓14.6216.6117.49
DINOv2 avg FD ↓65.1373.6676.62
CLIP cls FD ↓156.97174.87182.03
CLIP cls FDr ↓28.0331.2332.51
SigLIP cls FD ↓19.5222.1023.19
SigLIP cls FDr ↓32.5336.8338.66
SigLIP avg FD ↓39.6644.4446.12

Q4: Practical advantages over Improved MeanFlow and pixel MeanFlow.

A: Improved MeanFlow and pixel MeanFlow use larger training scales and different configurations, making fully fair comparisons challenging. Matched experiments would require retraining and tuning beyond our computational resources. Our contribution combines finite-difference targets with a dynamic curriculum that could extend to other MeanFlow-based methods. Progressively shrinking the difference interval transitions training from explicit velocity supervision toward local MeanFlow constraints, forming an easy-to-hard process intended to mitigate gradient conflicts and improve optimization. Our MF/AF comparisons and curriculum ablations support this strategy.

For a controlled comparison, we evaluated DifferenceFlow and Improved MeanFlow on unconditional CIFAR-10 with identical model sizes and training budgets (Table R3-Q4). At 1 NFE, DifferenceFlow achieves an FID of 18.84, outperforming Improved MeanFlow (19.63), providing additional evidence under matched conditions.

Table R3-Q4: Unconditional generation performance on CIFAR-10.

MethodNFEFID ↓
AlphaFlow123.45
iMeanFlow119.63
MeanFlow124.38
Ours118.84

Thank you for the clarification and additional experiments. The matched FD fine-tuning comparison addresses one of my concerns: DF retains an advantage over AF and MF when all three receive the same fine-tuning. For Q3, could you also report iMF’s 1-NFE FID on ImageNet-256 before and after applying the same FD fine-tuning protocol, ideally at a comparable model size and training budget? This would help assess whether DF’s advantage extends to a stronger baseline under the same fine-tuning conditions.

The CIFAR-10 comparison is less convincing. DF improves over iMF from 19.63 to 18.84 FID in your matched setup, but the original MeanFlow paper reports 2.92 FID for unconditional CIFAR-10 at 1 NFE (Table 3). I understand that the training setups may differ. Could you specify the backbone, training budget, and FID evaluation protocol, and clarify whether the advantage persists in a setup that reproduces a competitive MeanFlow baseline?

The matched AF–DF comparison supports the practical value of changing the early target weights. My remaining concern is whether the bias cancellation in the theoretical analysis explains the gains observed after both methods switch to the same JVP-based training schedule.

Reviewer_8sxSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a discrete formulation of MeanFlow named DifferenceFlow that replaces the temporal derivative with a finite difference between model predictions along the same trajectory. Experiments cover a Swiss-roll distribution and class-conditional ImageNet-256 generation. With a DiT-B/2 backbone, DifferenceFlow achieves an FID of 5.41, compared with 5.60 for the reproduced AlphaFlow baseline. Additional Fréchet-distance fine-tuning reduces the FID to 2.29.

Critical strengths
  1. The proposed target follows directly from a finite-difference discretization rather than introducing an independently chosen interpolation coefficient. It preserves the existing model interface and one-step inference procedure.
  2. The method section explicitly distinguishes satisfying the discretized equation from approximating the true average velocity. The conditional-path analysis also addresses the difference between sampled training velocities and marginal flow velocities.
  3. The paper evaluates multiple representation spaces, including DINOv2, CLIP, MAE, and SigLIP. Improvements in several of these spaces provide broader evidence than a single FID result.
Critical weaknesses
  1. The principal controlled comparison is 5.41 FID for DifferenceFlow versus 5.60 for AlphaFlow. The much larger improvement to 2.29 comes from the additional Fréchet-distance fine-tuning procedure adopted from prior work. Tables 3–4 do not report MeanFlow or AlphaFlow with the same fine-tuning.
  2. The fine-tuning setup also leaves important details unspecified, including the feature extractor used for optimization, the source of the reference moments, and relevant numerical stabilization choices. Reporting improvements in representations not used by the fine-tuning objective would help establish generalization beyond the optimized feature space.
  3. The theoretical advantage concerns oracle targets and does not establish lower error in the practical bootstrapped targets. The analysis is valuable as a local approximation result. It does not yet explain the claimed optimization advantage of the implemented algorithm.
  4. The experiments do not directly isolate the proposed interpolation coefficient. Table 2 studies the flow-matching ratio, residual weighting, time sampling, and curriculum schedule. These experiments establish sensitivity to implementation choices, but they do not directly test the main proposed change under a fully specified common training configuration.
  5. Several performance and convergence claims exceed the available evidence. The paper repeatedly claims improvements across model scales, but the reported DifferenceFlow configurations all use the 131M-parameter B/2 architecture; B/2+ adds fine-tuning rather than increasing model size.

Some minor issues:

  1. Line 55, "at early stage" should be "at an early stage"
  2. "use … MMD … and … Gaussian-FD … as the quantitative metric" should be "use … MMD … and … Gaussian-FD … as quantitative metrics"
  3. "A lower MMD and Gaussian-FD indicates better agreement" should be "Lower MMD and Gaussian-FD values indicate better agreement"
Questions for authors
  1. Can the authors apply the same Fréchet-distance fine-tuning procedure to the reproduced MeanFlow-B/2 and AlphaFlow-B/2 models and report the resulting FID and representation metrics?
  2. In the existing AF–DF comparison, are all training components identical except for the interpolation coefficient? Please specify the schedules, residual weights, training budgets, and random seeds used for each method.
  3. How much does the proposed coefficient improve performance across repeated B/2 runs? Does the improvement persist before Fréchet fine-tuning?
1 comment

We thank the reviewer for the careful review and constructive feedback. We appreciate your recognition of our principled finite-difference formulation, which preserves the model interface and one-step inference procedure; the explicit distinction between discrete consistency and average-velocity accuracy, together with our conditional-path analysis; and the broader empirical evidence provided by evaluations across multiple representation spaces. We address your concerns point by point below. All changes in the revised manuscript are highlighted in red.

Q1: Fréchet-distance fine-tuning comparison with MeanFlow and AlphaFlow.

A: Thank you for the suggestion. For FD fine-tuning, we use the 2048-dimensional global average-pooled features of a frozen Inception-v3. The reference mean and covariance are the precomputed statistics of real ImageNet images at 256×256256\times256 released with ADM. To stabilize optimization, we maintain an exponential moving average of the generated feature moments and apply adaptive weighting to the FD loss. We have fine-tuned MeanFlow-B/2 and AlphaFlow-B/2 using the same Fréchet-distance fine-tuning configuration as DifferenceFlow and report the complete set of 16 evaluation metrics. As shown in Table R2-Q1, DifferenceFlow achieves an FID (ADM) of 2.29, compared with 2.49 for AlphaFlow and 2.67 for MeanFlow, with corresponding FDr-6 values of 15.00, 16.90, and 17.75. DifferenceFlow achieves the best results on 15 of the 16 reported metrics, including distances across multiple representation spaces. These results show that DifferenceFlow retains its advantages in generation quality and distribution matching across representations when both baselines receive the same fine-tuning. We have added these results to Section 5.3.

Table R2-Q1: Comparison after Fréchet-distance fine-tuning under the B/2+ setting with CFG. All models are evaluated at 1 NFE. Best results are highlighted in bold.

MetricDFAFMF
FDr-6 ↓15.0016.9017.75
IS ↑251.44253.26246.17
FID(JiT) FD ↓2.372.582.74
FID(ADM) FD ↓2.292.492.67
FID(ADM) FDr ↓1.361.481.59
MAE cls FD ↓0.420.490.51
MAE cls FDr ↓10.6312.1412.84
MAE avg FD ↓1.201.471.63
DINOv2 cls FD ↓207.55235.63248.13
DINOv2 cls FDr ↓14.6216.6117.49
DINOv2 avg FD ↓65.1373.6676.62
CLIP cls FD ↓156.97174.87182.03
CLIP cls FDr ↓28.0331.2332.51
SigLIP cls FD ↓19.5222.1023.19
SigLIP cls FDr ↓32.5336.8338.66
SigLIP avg FD ↓39.6644.4446.12

Q2: Training configuration and fairness of the AlphaFlow–DifferenceFlow comparison.

A: Thank you for the question. For AlphaFlow, we directly use the officially released checkpoint. For DifferenceFlow, we align the network architecture, time embeddings, optimizer, number of training steps, and other basic training hyperparameters with the official configuration to ensure a fair comparison in terms of model configuration and training budget. The α\alpha schedule is a key hyperparameter design in our method; among the scheduling configurations we examined, the configuration adopted by AlphaFlow also achieved the best results for DifferenceFlow. Since the training seed associated with the official checkpoint was not provided, we cannot guarantee identical seeds. However, our empirical observations indicate that seed variation has a relatively small effect on training and evaluation results, suggesting that it is unlikely to change our main conclusions.

Q3: Multi-seed robustness and performance before Fréchet-distance fine-tuning.

A: Thank you for the question. We follow the single-training-run setting of the baselines used in our comparison. Given the computational cost of repeated training, we have not conducted multiple training-seed runs. Instead, we supplement the comparison with evaluations across multiple random seeds using fixed model checkpoints. The evaluation seed has a small effect on FID, supporting the stability of the results under sampling randomness. Furthermore, before Fréchet-distance fine-tuning, DifferenceFlow already achieves an FID of 5.41, compared with 5.60 for AlphaFlow and 6.14 for MeanFlow. Thus, the performance advantage is already present before fine-tuning, while the additional multi-seed evaluations further examine the stability of the evaluation results.

Reviewer_oGRLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DifferenceFlow, a finite-difference form of meanflow for one-step sampling. Instead of using jvp, the author uses the difference between two nearby trajectory points to compute the target velocity. The results seems better than alphaflow and meanflow in one-step generation.

Critical strengths

The paper is easier to read and understand the organization of paper section makes sense to me

Getting rid of jvp might be compatible with flash attention and compile module in pytorch framework and possibly leads to faster training.

Critical weaknesses
  1. The contribution is somewhat limited. Replacing the jvp by finite difference is quite popular. The same technique has been scaled up and apply on the transition models (Transition Models: Rethinking the Generative Learning Objective). Could you please provide more comparison with this technique ?

  2. It is not clear that where the performance gain compared to meanflow and alphaflow. Could you please provide the ablation of finite difference and training curriculum to see which leads to that gain ?

  3. The experiment is quite limited in small scale B size model. I would recommend author to provide more experiment with different size of model and different task like text-to-image ?

Questions for authors

Please see the weaknesses

1 comment

We thank the reviewer for the thoughtful review and constructive feedback. We appreciate your positive assessment of the paper’s clarity and organization, as well as your recognition that removing JVPs may improve compatibility with FlashAttention and PyTorch compilation, potentially enabling faster training. We address your concerns point by point below. All changes in the revised manuscript are highlighted in red.

Q1: Novelty and comparison with Transition Models (TiM).

A: Thank you for pointing out this relevant work. We agree that replacing JVP with finite differences is an established technique, and TiM demonstrates its value for large-scale generative training. Specifically, TiM approximates the network’s temporal derivative using central differences with a fixed small step size, and its objective reduces to MeanFlow under OT-FM. DifferenceFlow contributes a finite-interval target formulation and an associated training curriculum: we set the difference interval to Δt=α(t−r)\Delta t=\alpha(t-r), solve the discretized MeanFlow identity to obtain the interpolation weights α/(1+α)\alpha/(1+\alpha) and 1/(1+α)1/(1+\alpha), and progressively shrink this interval during training. Thus, α\alpha jointly controls the target composition and the temporal scale of the constraint. We further analyze how these weights eliminate a specific velocity-gap bias term in AlphaFlow’s oracle target error. Our contribution lies in this identity-derived target design and its dynamic curriculum, rather than finite differences themselves.

Q2: Ablation of finite differences and the training curriculum.

A: Thank you for the suggestion. Table 2(d) on page 7 already provides the relevant ablation to examine the role of the training curriculum within the finite-difference framework. Specifically, it includes pure finite-difference training with a fixed α\alpha, alongside linear, logarithmic, and sigmoid curriculum schedules. The fixed-α\alpha setting retains the finite-difference target while keeping the relative difference interval unchanged throughout training, providing a baseline for assessing the dynamic curriculum. With the DifferenceFlow target formulation and other experimental settings held consistent, progressively adjusting the interval improves generation quality over the tested fixed setting, while the choice of schedule also affects final performance. These ablations support the role of the dynamic curriculum in DifferenceFlow.

Table: Ablation of the α\alpha schedule in DifferenceFlow.

α\alpha scheduleFID ↓FDr-6 ↓IS ↑
Fixed (0.010.01)126.38104.6615.06
Sigmoid41.1252.7837.24
Log47.7359.1535.82
Linear59.2869.1131.35

Q3: Evaluation across model scales and generation tasks.

A: Thank you for the suggestion. We agree that evaluation across model scales and generation tasks would provide a more comprehensive assessment of the method’s applicability. Given our limited computational resources, training larger XL models is currently challenging. To address this suggestion, we include additional text-to-image generation experiments in Table R1-Q3(a), together with an unconditional-generation experiment on CIFAR-10 in Table R1-Q3(b), to examine DifferenceFlow beyond class-conditional ImageNet generation. These experiments broaden the coverage of tasks and datasets and provide complementary evidence for assessing the method across different settings. We have added these results to Section 5.3.

Table R1-Q3(a): Text-to-image generation performance.

MethodNFEFID ↓CLIP ↑
AlphaFlow1118.8027.41
MeanFlow1101.3627.49
DifferenceFlow173.2827.82

Table R1-Q3(b): Unconditional generation performance on CIFAR-10.

MethodNFEFID ↓
AlphaFlow123.45
iMeanFlow119.63
MeanFlow124.38
Ours118.84

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.