acceptodds
Under review as a conference paper at ICLR 2027

What Does One-Step Flow Matching Learn with Adaptive Weighting?

Abstract

One-step generative models replace iterative sampling with one network evaluation, but training them remains challenging. MeanFlow uses adaptive loss weighting to stabilize training, downweighting large residuals that can reflect valid target variation rather than prediction error. Through improved MeanFlow's velocity-regression formulation, we characterize the selected conditional law, giving necessary and sufficient conditions on radial weights for it to have no greater total variance than the original law at every stationary prediction, accounting for mean shifts. Population flows reveal changes in generated proportions even with convex regression and no neural fitting, while image experiments show that similar FID can conceal substantial shifts in class composition. Our prediction weighting removes target dependence, retaining adaptive scaling and the pointwise conditional mean while better matching training class proportions in our unconditional generation experiments. Comparisons on CIFAR-10 and ImageNet-LT distinguish target preservation from sample quality, establishing weighting as a generative modeling choice rather than merely a stabilization technique.

Primary area:
generative models
Keywords:
one-step generative modeling, flow matching, MeanFlow, loss weighting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_o1sSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at the adaptive loss weighting used in MeanFlow and its follow-ups, where large residuals get smaller detached weights to stabilize training. The authors propose a prediction weighting method which uses the prediction norm instead of the residual norm which was used in works like MeanFlow so the conditional mean target is preserved. The papers uses velocity-regression form of improved MeanFlow to show that this weighting change update strength and the conditional target the model learns, since it re weights only the valid targets at the same input. They give necessary and sufficient conditions on radial weights for the selected target law to have no greater total variance than the original. Experiments on Gaussian mixtures, a spiral, CIFAR-10(-LT100) and ImageNet-LT show that adaptive weighting can change class proportions while FID stays similar. Prediction weighting is best on CIFAR but clearly worse than adaptive weighting on ImageNet-LT, and the gap mostly closes only after adding perceptual losses.

Critical strengths
  1. The work is novel and interesting in producing more correctly weighted images from conditional flow matching and is relevant to the recent advances being made in the field.
  2. The diagnosis is clear and useful. The paper shows that adaptive weighting changes which target the model learns, not just optimization behavior. Population-flow experiment shows this without any neural network fitting and with a convex loss, so the effect cannot be blamed on training issues. Since this weighting is used in MeanFlow, iMF, AlphaFlow and pMF, this matters for a lot of recent one-step work.
  3. The theory is clean and sharp and mathematics is mostly correct. The variance result gives necessary and sufficient conditions, handles the mean shift properly, and comes with counterexamples showing the p ≤ 1 range cannot be extended.
  4. The paper is honest about negative results, such as the ImageNet ordering and the p = 1 boundary not being a performance threshold.
Critical weaknesses
  1. On CIFAR-10, predictive weighting improves FID and class-prior agreement, but on ImageNet-LT it underperforms adaptive weighting, including without guidance. Perceptual continuation narrows the gap, but this introduces an additional objective outside the target-preservation argument. The ImageNet recipe was also developed for adaptive weighting and not independently tuned for prediction weighting, so it remains unclear whether the gap reflects the weighting rule or the training recipe.
  2. The paper demonstrates that adaptive weighting changes the target, but does not establish that this change is harmful. On CIFAR-10, adaptive weighting at p = 0.5 has lower class-prior error than unweighted training on both long-tailed data (2.23 vs. 2.80) and balanced data (0.78 vs. 2.43). On ImageNet-LT, it also achieves better FID and tail-generation metrics. Target distortion may therefore sometimes be beneficial for one-step generation, and the experiments do not isolate whether the target shift itself causes either the improvements or the failures.
  3. Preserving the pointwise conditional mean does not guarantee better generated samples. The guarantee concerns the population regression target, not the ability of a finite network to learn it or the quality of the resulting flow. The spiral example also illustrates that correct-target regression can still produce blurry one-step samples. The paper would benefit from a clearer account of when target preservation translates into better generation.
  4. Prediction weighting is target-preserving, but it is not necessarily neutral across inputs. Its prediction-dependent weight changes the relative contribution of different training examples and potentially different noise levels and time intervals. The paper does not establish why the prediction norm is the appropriate scale, or separate its effect from simpler input-dependent weighting schemes. A time-only weighting baseline, alongside the target-independent weighting methods discussed in the related work, would help clarify what drives the results.
  5. The image-generation experiments use a single seed and do not report uncertainty across runs. This is particularly concerning given the instability of unweighted long-tailed CIFAR training: its FID rises from 23.04 at 140k updates to 84.30 at 160k. The reported differences in generated class proportions and quality could therefore be sensitive to training trajectories, and multiple seeds are needed to establish the robustness of the comparisons.
  6. The variance result does not appear to predict generation quality or class-prior fidelity consistently. The toy examples show rare-group suppression, whereas CIFAR-LT at p = 1 exhibits rare-class over generation. Since the paper also clarifies that the p = 1 boundary is not a performance threshold, it would be useful to explain what practical predictions, if any, follow from the variance guarantee beyond characterizing the selected target distribution.
  7. The empirical scope is very narrow, focusing on CIFAR-10 and ImageNet-LT with a 118M-parameter DiT-B. Evaluation on larger or more diverse settings, and comparisons with methods explicitly designed to improve rare-mode coverage, would help establish the generality of the findings.
Questions for authors
  1. Can the authors demonstrate a concrete failure attributable to the target shift, rather than merely a change in class proportions or quality metrics? Ideally, this would include multiple seeds and an evaluation at ImageNet scale.
  2. How does prediction weighting perform with a recipe independently tuned for it on ImageNet-LT, including learning rate, ε, p, and time sampling? Without this comparison, it is difficult to distinguish limitations of the weighting rule from limitations of the transferred recipe.
  3. How much of prediction weighting's effect comes from its variation across time and other inputs? Can the authors compare it against a time-only weighting baseline matched to its average weight over time, and against the target-independent weighting schemes of Kim et al. (2025) and Billera et al. (2025)?
  4. Is prediction weighting stable at MeanFlow's original learning rate of 6 × 10⁻⁴, where unweighted iMF reportedly diverged? Since stability motivates adaptive weighting, this is important for assessing whether prediction weighting can serve as a practical alternative.
  5. Can Corollary 5 predict which groups gain or lose influence in the observed experiments, particularly the contrast between rare-group suppression in the toy examples and rare-class overgeneration on CIFAR-LT at p = 1?
6 comments
AuthorsOpenReview ↗

Response to Reviewer o1sS (1/3)

Thank you for the detailed questions. Adaptive weighting changes both learning strength and the conditional targets the model learns to average. We characterize that selection and show its consequences for distribution recovery. Prediction weighting preserves the regression target while retaining adaptive scaling. The benefit of either rule depends on how well the model learns and which properties of the generated distribution matter for the application.

Target selection and generation (W2, W3, Q1). The population experiment provides a concrete failure to recover a prescribed distribution caused by weighting alone, without neural fitting error.

In Section 5.1 of the submitted paper, we integrate both fields from the same initial noise, and the pointwise regression is convex at p=0.5p=0.5. Nevertheless, weighting reduces the rarest component's generated mass from the prescribed 0.686% to 0.130%, with the reduction persisting across numerical resolutions. Here the change in proportions measures a failure to recover a known distribution, caused by the objective rather than by neural optimization. At the stationary prediction in Table 1, total variance decreases by 13.8740, while the squared error relative to the true marginal velocity is 3.4772. Both measurements were already in submitted appendix Table 4.

The new Figure 4 shows this selection on CIFAR images. At the same noisy input, adaptive weighting changes how much valid clean images contribute to the conditional mean. Appendix B.5 quantifies this change across inputs using the closed-form conditional law of the training data.

The trained adaptive models also have larger errors relative to the closed-form mean when r<tr < t (new Table 15). For the corrected velocity at p=0.5p=0.5, RMS error is 1.365, versus 0.245 with prediction weighting and 0.223 without weighting, expressed per pixel in clean-image units. The same comparison holds at p=1p=1 for r<tr < t.

A separate control computes the residual weights around the exact conditional mean itself. Even with these weights, the expected adaptive update points away from the mean (Appendix B.5). Thus even the correct conditional mean can cease to be a stationary solution of the weighted regression problem.

We agree that adaptive weighting can improve sample quality while changing the target. Our ImageNet results favor it for FID. The population experiment shows the complementary case, where target selection reduces fidelity to a known distribution without neural fitting error. Our aim is to make this choice explicit and characterize the conditional law being selected.

Recovering the intended distribution also requires learning the target accurately. In the submitted spiral experiment, unweighted iMF and prediction weighting share the true conditional-mean target, yet prediction at p=0.5p=0.5 improves one-call sliced Wasserstein distance from 0.0174 to 0.0096 under the same training budget. Here prediction weighting improves learning while retaining the intended velocity as the regression solution.

Which groups gain influence (W6, Q5). Corollary 5 predicts a group's local gain or loss from its average weight relative to competing targets. Its conditional mass increases when its average weight exceeds the overall conditional average, which depends on the arrangement of its targets relative to the prediction. In the added CIFAR analysis at p=1p=1 and t∈[0.9,1]t\in[0.9,1], airplane's average conditional probability rises from 41.63% to 44.20%, whereas automobile's falls from 23.16% to 17.89%, although both are common classes (Appendix B.5).

Total variance cannot increase at stationary predictions for 0<p≤10 < p\leq1, even after accounting for the shifted mean. This constrains the spread of the selected law around a mean that can itself move between groups. The guarantee therefore permits both gains and losses of rare-group influence. Final generated proportions also depend on the resulting flow. The population experiment follows the change in conditional targets through integration to the loss of rare-component mass. The CIFAR analysis shows that the local selection can favor one common class over another, consistent with the criterion depending on relative weights rather than class frequency alone.

AuthorsOpenReview ↗

Response to Reviewer o1sS (2/3)

Prediction weighting and stability (W4, Q3, Q4). Prediction weighting trains successfully at the original learning rate of 6×10−46\times10^{-4} without added gradient clipping. Both weighted models learn useful generators. Only one of three unweighted runs maintains useful performance. The other two collapse after initial improvement:

iMF weightingppFID
Unweighted027.70, 385.86, 421.58
Adaptive0.520.24
Prediction0.516.09

These are CIFAR-10-LT100 results after 120k updates, evaluated with 10,000 samples (new Table 12), with their learning curves in the new Figure 8. The submitted lower-LR experiment also deteriorated from 23.04 to 84.30. Together with the spiral results, these observations support the stability benefit of prediction weighting while it preserves the unweighted target.

The prediction norm supplies a scale from the model itself without consulting the sampled target. Substituting it for the residual norm retains the inverse-norm form, exponent, and offset, while leaving the squared regression error unchanged. All valid targets at the same input then receive the same weight, preserving the intended conditional mean (Section 4). This implements the preservation principle discussed by Kim et al. and Billera et al. without an auxiliary estimator or additional weighting hyperparameters. The adaptive rules at p=0.5p=0.5 and p=1p=1 correspond to pseudo-Huber and logarithmic Cauchy losses, respectively, up to a constant gradient scale (Appendix A.2).

Prediction weighting places little relative emphasis on particular times in the diagonal inputs we measured. At p=0.5p=0.5 with r=tr=t, across early, middle, and late training stages, the ratio of the largest to smallest bin mean stays between 1.12 and 1.14 on CIFAR-10-LT100 and between 1.09 and 1.13 on ImageNet-LT, compared with 1.58 to 2.67 and 3.37 to 3.85 for adaptive weighting (new Table 17 in Appendix B.6). The final CIFAR-10-LT100 coefficients of variation across inputs are 0.09 for prediction and 0.54 for adaptive weighting. Appendix B.6 includes mean curves and percentile bands for both datasets. A time-bin average would retain the nearly flat means while removing variation across states at the same time. We did not train a separate time-only baseline.

Direct FM tests prediction weighting without JVP-based training, with the same conditional-mean target as unweighted FM. Prediction weighting at p=0.5p=0.5 improves LT FID from 55.42 to 51.87 with nearly unchanged prior error (new Table 13). Its benefit therefore extends beyond stabilizing JVP-based training. Both FM objectives preserve the same conditional mean, so the benefit comes from how that regression problem is learned.

ImageNet performance and tuning (W1, Q2). We used the same pMF-based recipe for all weighting rules (Section 5.4). Prediction weighting at p=1p=1 improves Val-BAL FID over unweighted training from 62.54 to 49.79, while adaptive reaches 30.71 and also leads without guidance (Table 3 and Appendix B.7).

These settings were developed with adaptive weighting and were not retuned for prediction weighting. The requested independent search over learning rate, ϵ\epsilon, pp, and time sampling exceeds the compute available for this rebuttal.

The strength of adaptive weighting on ImageNet extends to direct FM, where FID improves from 42.45 to 18.88 (Table 3). Its practical advantage therefore extends beyond JVP-based training. These results support evaluating weighting as a modeling choice. Our analysis makes explicit which conditional law is selected along with the change in training emphasis, so that this choice can be considered alongside sample quality.

Prediction weighting is useful when the intended conditional mean should remain the learning target and unweighted training is difficult. It offers this combination without an auxiliary network. CIFAR shows that the same choice can also improve both sample quality and class-prior agreement.

Adding pMF's perceptual losses narrows the FID gap to 10.60 versus 9.64 (Section 5.4). Prediction weighting can be used with these additional losses, although the feature objectives fall outside the target-preservation guarantee, as noted in the submitted paper.

AuthorsOpenReview ↗

Response to Reviewer o1sS (3/3)

Robustness and scope (W5, W7). The repeated runs confirm the sensitivity you identified in unweighted training, with all outcomes reported in Table 12 and Figure 8. The main image comparisons use one training seed. The submitted spiral comparison reports means and standard deviations over three training seeds (Table 7). For the underlying mechanism, the population flow shows the generative consequence without neural fitting, and the exact-mean image control shows that adaptive weighting changes the target even without a learned prediction (Appendix B.5).

CIFAR and ImageNet test two substantially different uses of one-step models: unconditional generation with extensive weighting comparisons, and class-conditional high-resolution generation with guidance. Their different performance orderings help establish why target preservation and sample quality should be considered separately. Direct FM further shows that the distinction is not confined to JVP-based training.

Recovering a rare mode and assigning it the correct probability are distinct goals. The population study and image comparisons examine the probability assigned to each group, which can change even when modes remain present. Generated proportions therefore complement coverage and sample-quality metrics when evaluating whether a model reproduces its training distribution.

Reviewer_mupVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the adaptive loss weighting used in MeanFlow, w=(∥f−Y∥2+ε)−pw=(\|f-Y\|^2+\varepsilon)^{-p}, which is also used in iMF, AlphaFlow and pMF. The main point is that in flow matching a large residual at a given input is often not a model error, because the target YY is random given the input. So when the weight downweights large residuals, it changes the target that the model learns. The model learns the mean of a reweighted law Q∝wPQ \propto wP instead of the true conditional mean. The authors show that for radial weights, the variance of this selected law is never larger than the original one if and only if η(s)\eta(s) is nonincreasing and sη(s)s\eta(s) is nondecreasing, which means p≤1p \le 1 for the MeanFlow weight. They then propose to compute the weight from the norm of the prediction instead of the residual, which keeps the conditional mean. They test this on a Gaussian mixture without any network, a spiral toy, CIFAR-10, CIFAR-10-LT and ImageNet-LT. Prediction weighting works better on CIFAR, but on ImageNet-LT the original adaptive weighting is better.

Critical strengths

I like the main observation of the paper. This weighting is used in almost every MeanFlow follow-up, and people have looked at it only as a trick for stability and FID. The paper shows that it actually changes what the model learns, and it also says what the model learns instead. I went over the proofs in Appendix A (Theorem 1, Theorem 7, Proposition 8 and the examples in A.3 and A.6) and did not find a problem. The if and only if condition in Theorem 7 is a nice result, and I have not seen it before.

I also found the population flow experiment in Section 5.1 very useful. There is no network there, and for p=0.5p=0.5 the problem at each point is convex, so the shift in the generated proportions can only come from the weighting.

Critical weaknesses

My main concern is that the image experiments do not really show the effect that the theory is about. The theory is about selecting targets at a fixed input. But the adaptive weight and the prediction weight also give different weights to different inputs, for example to different (r,t)(r,t). This part does not change the target, but it can change training a lot. So the better CIFAR results of prediction weighting may partly come from a different weighting over inputs and not from removing target selection. The paper does not check how much the adaptive weight changes within an input compared to across inputs. One simple check would be to replace each adaptive weight with its average in (r,t)(r,t) bins.

My second concern is about the results of prediction weighting. On CIFAR it is better, but on ImageNet-LT at 300k it is much worse (49.79 vs 30.71 FID), and the gap is still there without guidance (83.92 vs 56.12). After the perceptual continuation the two become close (10.60 vs 9.64), but the perceptual loss is not covered by the target preservation result. Also, on CIFAR the learning rate was reduced from 6×10−46\times10^{-4} to 10−410^{-4} because the unweighted runs diverged, so we do not know if prediction weighting keeps the stability benefit of adaptive weighting at the original learning rate. On ImageNet, the recipe was tuned with adaptive weighting. Because of this, it is hard to say when one should use prediction weighting. I see this more as a limitation on the practical side than on the theory.

Minor: I think the paper should discuss the connection to mean shift and Weiszfeld iterations, and also Dao et al. (ICLR 2025), which uses the Cauchy loss in consistency training, and the paper "On Variance Reduction in Learning Mean Flows".

Questions for authors

How does prediction weighting compare with an adaptive weight averaged in (r,t)(r,t) bins, or with a weight that depends only on time? This would help to separate the two effects.

What happens at the original learning rate of 6×10−46\times10^{-4}? Is prediction weighting stable there?

Is it possible to measure how strong the target selection is on real images? For CIFAR, the conditional law given xtx_t under the training set has a closed form, so for example KL(Q∥P)\mathrm{KL}(Q\|P) or the effective sample size as a function of tt could be reported.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mupV (1/2)

Thank you for the careful reading of the theory and population study. We agree that input-dependent scaling can contribute to prediction weighting's CIFAR gains. The added measurements examine both conditional target selection and the variation of weights across inputs.

Target selection and input scaling (W1, Q1, Q3). On CIFAR, adaptive weighting shifts conditional means most at high noise, while most weight variation occurs across inputs. At a fixed input, the expected regression gradient is proportional to EP[w](f−EQY)\mathbb E_P[w](f-\mathbb E_QY) (Section 3.1). The selected mean determines what the model is asked to predict, while EP[w]\mathbb E_P[w] determines how strongly that input contributes to learning.

Thank you for suggesting this concrete analysis. It allowed us to examine the conditional target selection identified by the theory directly on images. The new Figure 4 illustrates the change among valid clean-image targets at the same noisy input. Using the closed-form conditional law you suggested, we find that selection is concentrated at high noise and that the overall mean displacement more than doubles from p=0.5p=0.5 to p=1p=1:

ppTime rangeKL(Q∥P)\mathrm{KL}(Q\Vert P)Relative ESSMean shift
0.5Overall0.002970.99410.00532
1Overall0.013770.97570.01162
0.5[0.9,1]0.033370.93320.01151
1[0.9,1]0.148180.72740.02242

Here PP is the true conditional law under the training data and QQ the selected law. Mean shift is RMS per pixel on the [−1,1][-1,1] scale. Relative ESS measures how unevenly the conditional targets are reweighted. Unchanged probabilities give KL zero and relative ESS one. Appendix B.5 reports all time bins.

The trained adaptive models also have larger errors relative to the closed-form mean when r<tr < t (new Table 15). For the corrected velocity at p=0.5p=0.5, RMS error is 1.365, versus 0.245 with prediction weighting and 0.223 without weighting, expressed per pixel in clean-image units. The same comparison holds at p=1p=1 for r<tr < t.

A separate control computes the residual weights around the exact conditional mean itself. Even with these weights, the expected adaptive update points away from the mean (Appendix B.5). Target selection therefore persists even when the prediction error is removed.

Most weight variation is across inputs. Variation among targets at the same input accounts for 0.992% and 0.0397% of total weight variance at p=0.5p=0.5 and 11. Substantial across-input scaling thus coexists with the conditional-mean shift at high noise. Prediction weighting retains scaling across inputs while giving all valid targets at the same input a common weight.

Prediction weighting places little relative emphasis on particular times in the diagonal inputs we measured. At p=0.5p=0.5 with r=tr=t, across early, middle, and late training stages, the ratio of the largest to smallest bin mean stays between 1.12 and 1.14 on CIFAR-10-LT100 and between 1.09 and 1.13 on ImageNet-LT (new Table 17). Adaptive weighting gives 1.58 to 2.67 and 3.37 to 3.85, respectively. The final CIFAR-10-LT100 coefficients of variation across inputs are 0.09 for prediction and 0.54 for adaptive weighting. Appendix B.6 also reports mean curves and 10th--90th percentile bands. A time-bin average would retain these nearly flat means while removing variation across states at the same time. We did not train a separate time-only baseline.

In iMF, a comparison with unweighted training can reflect both improved stability and a different allocation of learning across inputs. Direct FM removes the JVP-specific source of instability, with the same conditional-mean target as unweighted FM. Prediction weighting at p=0.5p=0.5 improves LT FID from 55.42 to 51.87 with nearly unchanged prior error (new Table 13). Its benefit therefore extends beyond stabilizing JVP-based training. Both FM objectives preserve the same conditional mean, so the benefit comes from how that regression problem is learned. The submitted population experiment isolates target selection without neural fitting. Table 1 brings the already-reported variance decrease and mean shift from submitted appendix Table 4 into the main text.

Taken together, these results explain why we retain input-dependent scaling while removing target dependence from the weight. Scaling can help a model learn the intended target, while target selection can redistribute generated mass even without fitting error.

AuthorsOpenReview ↗

Response to Reviewer mupV (2/2)

Practical value and stability (W2, Q2). Prediction weighting trains successfully at 6×10−46\times10^{-4} without added gradient clipping. Both weighted methods learn useful generators. Only one of three unweighted runs maintains useful performance. The other two collapse after initial improvement:

iMF weightingppFID
Unweighted027.70, 385.86, 421.58
Adaptive0.520.24
Prediction0.516.09

These are CIFAR-10-LT100 results after 120k updates, evaluated with 10,000 samples (new Table 12), with their learning curves in the new Figure 8. Together with the submitted lower-LR curves and spiral results, they support prediction weighting as a target-preserving alternative with better training stability than unweighted iMF.

These stability results give a practical reason to choose prediction weighting when preserving the intended regression target matters. It requires no auxiliary network or additional training stage. On CIFAR-10-LT100, this choice also improves FID (29.16 to 18.38) and prior error (2.23 to 0.78) over adaptive weighting at p=0.5p=0.5.

On ImageNet-LT, adaptive weighting remains the stronger choice for FID, including without guidance. Its benefit also appears in direct FM, where FID improves from 42.45 to 18.88 (Table 3). Its advantage therefore extends beyond JVP-based training. These results favor adaptive weighting when FID under this recipe is the priority.

Adding perceptual losses narrows the Val-BAL FID gap to 10.60 versus 9.64 (Table 4). We agree that these feature objectives fall outside the target-preservation guarantee, as stated in Section 5.4. This result shows that prediction weighting can also be combined with perceptual losses to improve sample quality.

Related work (Minor). Section 2 and Appendix A.2 now discuss the requested connections. The stationary equation has the weighted-centroid form of mean-shift and Weiszfeld iterations, with smoothed inverse-distance weights at p=0.5p=0.5. Dao et al. use a Cauchy loss in consistency training, and Lu and Wang study gradient variance through the JVP. Our variance result concerns the selected conditional law around its own mean, including the exact conditions for a non-increase.

Reviewer_aVwsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how adaptive weighting in one-step flow models affects generation. The authors show that such weighting changes not only the update strength but also which conditional targets the model learns to average. To address this, the author proposes prediction weighting, which replaces the residual norm with the prediction norm as the weight. The proposed method is validated on CIFAR-10 and ImageNet.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The paper provides a thorough theoretical analysis.
  3. The paper experimentally demonstrates the potential class-proportion shift, and points out that FID may fail to reveal it.
Critical weaknesses
  1. Validation is limited to CIFAR-10 and ImageNet, and the proposed method does not outperform adaptive weighting on ImageNet.
  2. The paper mainly compares against iMF, while other works using the same technique are not compared, so the generalisability of the proposed method is not fully validated.
  3. There is a lack of comparison with other weighting schemes.
Questions for authors

Please see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer aVws

Thank you for recognizing the theoretical analysis and the evidence on class composition. In iMF's regression formulation, several clean images can supply valid velocity targets at the same noisy input. Their conditional mean is the velocity to be learned. Downweighting large residuals can therefore change that average, even when those residuals reflect valid target variation. Our main contribution is to determine the structure of the law selected by this widely used training rule. We give necessary and sufficient conditions on radial weights for its total variance not to exceed that of the original law at every stationary prediction, accounting for the shift in the mean. Prediction weighting removes target dependence from the multiplier while retaining adaptive scaling of the same squared regression error.

ImageNet performance and empirical scope (W1). Prediction weighting improves over unweighted training on both image datasets. On CIFAR-10-LT100, it also outperforms adaptive weighting at p=0.5p=0.5, improving FID from 29.16 to 18.38 and prior error from 2.23 to 0.78 (Table 2). On ImageNet-LT at p=1p=1, Val-BAL FID improves from unweighted training's 62.54 to 49.79, although adaptive weighting is better at 30.71 (Table 3).

Prediction weighting keeps the conditional-mean target of unweighted regression while improving learning. It also trains at the original learning rate where unweighted iMF can collapse (Appendix B.4). On CIFAR, these benefits extend to both sample quality and class-prior agreement. Under the ImageNet recipe, adaptive weighting gives better FID. Choosing a rule therefore depends on whether one prioritizes that quality metric or also requires the regression target to remain faithful to the prescribed data distribution.

The new Figure 4 shows how adaptive weighting increases the normalized contribution of some clean targets to the conditional mean and decreases that of others, as quantified in Appendix B.5. These are valid targets at the same noisy input, so changing their relative weights changes what the model is asked to average. The submitted population experiment in Section 5.1 shows that this change can alter the generated distribution even without neural fitting error.

Other models (W2). The submitted experiments include SongUNet iMF, a pMF-based ImageNet model, and direct FM. The ImageNet model uses a pixel-space DiT with class conditioning and guidance, extending the comparison beyond the unconditional CIFAR setting.

We use iMF because its velocity-regression formulation exposes the target being averaged in MeanFlow training. MeanFlow learns average velocities through a derivative identity, so the effect of weighting is less apparent than in ordinary velocity regression. Follow-ups such as AlphaFlow and Euler Mean Flows retain the broad structure of squared regression for learning average velocities or flow maps against targets derived from trajectory consistency (Section 2). In iMF's formulation, the corrected prediction depends only on the model inputs, while the sampled velocity target remains random at those inputs. This makes it possible to identify the conditional law selected by the weight. The theorem applies to this conditional regression problem independently of the network architecture.

Direct FM removes JVPs and average-velocity learning to test how much the observations depend on these features of MeanFlow. On CIFAR-10-LT100, adaptive weighting at p=1p=1 improves FID from 55.42 to 49.44 while worsening prior error from 4.69 to 5.83 (Section 5.5). Thus the distinction between sample quality and distribution recovery also appears outside iMF.

Other weighting rules (W3). Adaptive weighting at p=0.5p=0.5 has the gradient of the pseudo-Huber (Charbonnier) loss, and p=1p=1 corresponds to a logarithmic Cauchy loss, up to a constant factor (Appendix A.2). The radial characterization in Section 3.2 extends beyond these choices and identifies the weights that share the total-variance guarantee at stationarity.

A positive time-only weight preserves the conditional mean, as do the target-independent rules discussed by Kim et al. and Billera et al. Changing time sampling also changes the relative weight of time regions in the expected objective. Prediction weighting uses the same preservation principle and obtains its scale from the existing prediction, without an auxiliary estimator.

Unweighted training fixes the conditional-mean target, adaptive weighting changes both target and scale, and prediction weighting retains adaptation without changing that target. The model and squared error stay the same. In the added direct-FM comparison, prediction weighting at p=0.5p=0.5 improves FID from 55.42 to 51.87 with nearly unchanged prior error (new Table 13). Both FM rules have the same conditional-mean target, so this comparison demonstrates a learning benefit with the regression target preserved.

Thank you for your responses. The responses have basically addressed my concerns. Given my limited familiarity with this specific subfield and considering the comments from other reviewers, I will raise my rating to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author response: summary and revisions

We thank the reviewers for their careful reading and questions. Our paper asks what adaptive weighting makes a one-step model learn, beyond its effect on optimization. Through iMF's velocity-regression formulation, we identify how it changes the conditional targets being averaged. The submitted analysis gives necessary and sufficient conditions for the selected conditional law to have no greater total variance than the original law at stationary predictions. The accompanying measurements show variance reduction together with mean displacement (Table 1, moved from submitted appendix Table 4). The population experiment shows that weighting can change the generated distribution even without neural fitting error. Prediction weighting retains adaptive scaling while preserving the regression target. The different performance orderings on CIFAR and ImageNet show why target preservation and sample quality are both relevant when choosing the objective.

The discussion raises two common questions: whether target selection can be identified directly in image models, and whether prediction weighting retains the practical benefits of adaptation. We have added focused evidence for both.

Target selection on images. Adaptive weighting shifts conditional image means, with larger shifts at high noise. The new Figure 4 illustrates how weights calculated from the learned prediction redistribute the contributions of valid clean images. Appendix B.5 measures these changes against the exact conditional law of the CIFAR training data. The trained adaptive models also have larger corrected-velocity errors against the true conditional mean than models trained with prediction weighting or without weighting for r<tr < t (new Table 15).

Stability at the original learning rate. The additional experiment at 6×10−46\times10^{-4} confirms that both prediction and adaptive weighting learn successfully. Only one of three unweighted runs maintained useful performance, while the other two collapsed after initial improvement. The new Figure 8 and Table 12 report all three outcomes. Prediction weighting combines target preservation with a stability benefit over unweighted training.

Learning benefits in FM. Direct FM removes JVP-based training while keeping the same conditional-mean target for unweighted and prediction weighting. Prediction weighting improves FID with nearly unchanged prior error in this comparison (new Table 13). Input-dependent scaling therefore remains useful when both rules preserve the target and neither uses JVPs. The added time profiles in Appendix B.6 and weight-variance measurements in Appendix B.5 further describe how the rules distribute training weight across inputs.

Loss weighting should therefore be assessed both by the generation it produces and by the regression target it selects. Prediction weighting offers a simple target-preserving choice with demonstrated learning benefits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.