- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper looks at the adaptive loss weighting used in MeanFlow and its follow-ups, where large residuals get smaller detached weights to stabilize training. The authors propose a prediction weighting method which uses the prediction norm instead of the residual norm which was used in works like MeanFlow so the conditional mean target is preserved. The papers uses velocity-regression form of improved MeanFlow to show that this weighting change update strength and the conditional target the model learns, since it re weights only the valid targets at the same input. They give necessary and sufficient conditions on radial weights for the selected target law to have no greater total variance than the original. Experiments on Gaussian mixtures, a spiral, CIFAR-10(-LT100) and ImageNet-LT show that adaptive weighting can change class proportions while FID stays similar. Prediction weighting is best on CIFAR but clearly worse than adaptive weighting on ImageNet-LT, and the gap mostly closes only after adding perceptual losses.
Critical strengths
- The work is novel and interesting in producing more correctly weighted images from conditional flow matching and is relevant to the recent advances being made in the field.
- The diagnosis is clear and useful. The paper shows that adaptive weighting changes which target the model learns, not just optimization behavior. Population-flow experiment shows this without any neural network fitting and with a convex loss, so the effect cannot be blamed on training issues. Since this weighting is used in MeanFlow, iMF, AlphaFlow and pMF, this matters for a lot of recent one-step work.
- The theory is clean and sharp and mathematics is mostly correct. The variance result gives necessary and sufficient conditions, handles the mean shift properly, and comes with counterexamples showing the p ≤ 1 range cannot be extended.
- The paper is honest about negative results, such as the ImageNet ordering and the p = 1 boundary not being a performance threshold.
Critical weaknesses
- On CIFAR-10, predictive weighting improves FID and class-prior agreement, but on ImageNet-LT it underperforms adaptive weighting, including without guidance. Perceptual continuation narrows the gap, but this introduces an additional objective outside the target-preservation argument. The ImageNet recipe was also developed for adaptive weighting and not independently tuned for prediction weighting, so it remains unclear whether the gap reflects the weighting rule or the training recipe.
- The paper demonstrates that adaptive weighting changes the target, but does not establish that this change is harmful. On CIFAR-10, adaptive weighting at p = 0.5 has lower class-prior error than unweighted training on both long-tailed data (2.23 vs. 2.80) and balanced data (0.78 vs. 2.43). On ImageNet-LT, it also achieves better FID and tail-generation metrics. Target distortion may therefore sometimes be beneficial for one-step generation, and the experiments do not isolate whether the target shift itself causes either the improvements or the failures.
- Preserving the pointwise conditional mean does not guarantee better generated samples. The guarantee concerns the population regression target, not the ability of a finite network to learn it or the quality of the resulting flow. The spiral example also illustrates that correct-target regression can still produce blurry one-step samples. The paper would benefit from a clearer account of when target preservation translates into better generation.
- Prediction weighting is target-preserving, but it is not necessarily neutral across inputs. Its prediction-dependent weight changes the relative contribution of different training examples and potentially different noise levels and time intervals. The paper does not establish why the prediction norm is the appropriate scale, or separate its effect from simpler input-dependent weighting schemes. A time-only weighting baseline, alongside the target-independent weighting methods discussed in the related work, would help clarify what drives the results.
- The image-generation experiments use a single seed and do not report uncertainty across runs. This is particularly concerning given the instability of unweighted long-tailed CIFAR training: its FID rises from 23.04 at 140k updates to 84.30 at 160k. The reported differences in generated class proportions and quality could therefore be sensitive to training trajectories, and multiple seeds are needed to establish the robustness of the comparisons.
- The variance result does not appear to predict generation quality or class-prior fidelity consistently. The toy examples show rare-group suppression, whereas CIFAR-LT at p = 1 exhibits rare-class over generation. Since the paper also clarifies that the p = 1 boundary is not a performance threshold, it would be useful to explain what practical predictions, if any, follow from the variance guarantee beyond characterizing the selected target distribution.
- The empirical scope is very narrow, focusing on CIFAR-10 and ImageNet-LT with a 118M-parameter DiT-B. Evaluation on larger or more diverse settings, and comparisons with methods explicitly designed to improve rare-mode coverage, would help establish the generality of the findings.
Questions for authors
- Can the authors demonstrate a concrete failure attributable to the target shift, rather than merely a change in class proportions or quality metrics? Ideally, this would include multiple seeds and an evaluation at ImageNet scale.
- How does prediction weighting perform with a recipe independently tuned for it on ImageNet-LT, including learning rate, ε, p, and time sampling? Without this comparison, it is difficult to distinguish limitations of the weighting rule from limitations of the transferred recipe.
- How much of prediction weighting's effect comes from its variation across time and other inputs? Can the authors compare it against a time-only weighting baseline matched to its average weight over time, and against the target-independent weighting schemes of Kim et al. (2025) and Billera et al. (2025)?
- Is prediction weighting stable at MeanFlow's original learning rate of 6 × 10⁻⁴, where unweighted iMF reportedly diverged? Since stability motivates adaptive weighting, this is important for assessing whether prediction weighting can serve as a practical alternative.
- Can Corollary 5 predict which groups gain or lose influence in the observed experiments, particularly the contrast between rare-group suppression in the toy examples and rare-class overgeneration on CIFAR-LT at p = 1?
6 comments
Response to Reviewer o1sS (2/3)
Prediction weighting and stability (W4, Q3, Q4). Prediction weighting trains successfully at the original learning rate of without added gradient clipping. Both weighted models learn useful generators. Only one of three unweighted runs maintains useful performance. The other two collapse after initial improvement:
| iMF weighting | FID | |
|---|---|---|
| Unweighted | 0 | 27.70, 385.86, 421.58 |
| Adaptive | 0.5 | 20.24 |
| Prediction | 0.5 | 16.09 |
These are CIFAR-10-LT100 results after 120k updates, evaluated with 10,000 samples (new Table 12), with their learning curves in the new Figure 8. The submitted lower-LR experiment also deteriorated from 23.04 to 84.30. Together with the spiral results, these observations support the stability benefit of prediction weighting while it preserves the unweighted target.
The prediction norm supplies a scale from the model itself without consulting the sampled target. Substituting it for the residual norm retains the inverse-norm form, exponent, and offset, while leaving the squared regression error unchanged. All valid targets at the same input then receive the same weight, preserving the intended conditional mean (Section 4). This implements the preservation principle discussed by Kim et al. and Billera et al. without an auxiliary estimator or additional weighting hyperparameters. The adaptive rules at and correspond to pseudo-Huber and logarithmic Cauchy losses, respectively, up to a constant gradient scale (Appendix A.2).
Prediction weighting places little relative emphasis on particular times in the diagonal inputs we measured. At with , across early, middle, and late training stages, the ratio of the largest to smallest bin mean stays between 1.12 and 1.14 on CIFAR-10-LT100 and between 1.09 and 1.13 on ImageNet-LT, compared with 1.58 to 2.67 and 3.37 to 3.85 for adaptive weighting (new Table 17 in Appendix B.6). The final CIFAR-10-LT100 coefficients of variation across inputs are 0.09 for prediction and 0.54 for adaptive weighting. Appendix B.6 includes mean curves and percentile bands for both datasets. A time-bin average would retain the nearly flat means while removing variation across states at the same time. We did not train a separate time-only baseline.
Direct FM tests prediction weighting without JVP-based training, with the same conditional-mean target as unweighted FM. Prediction weighting at improves LT FID from 55.42 to 51.87 with nearly unchanged prior error (new Table 13). Its benefit therefore extends beyond stabilizing JVP-based training. Both FM objectives preserve the same conditional mean, so the benefit comes from how that regression problem is learned.
ImageNet performance and tuning (W1, Q2). We used the same pMF-based recipe for all weighting rules (Section 5.4). Prediction weighting at improves Val-BAL FID over unweighted training from 62.54 to 49.79, while adaptive reaches 30.71 and also leads without guidance (Table 3 and Appendix B.7).
These settings were developed with adaptive weighting and were not retuned for prediction weighting. The requested independent search over learning rate, , , and time sampling exceeds the compute available for this rebuttal.
The strength of adaptive weighting on ImageNet extends to direct FM, where FID improves from 42.45 to 18.88 (Table 3). Its practical advantage therefore extends beyond JVP-based training. These results support evaluating weighting as a modeling choice. Our analysis makes explicit which conditional law is selected along with the change in training emphasis, so that this choice can be considered alongside sample quality.
Prediction weighting is useful when the intended conditional mean should remain the learning target and unweighted training is difficult. It offers this combination without an auxiliary network. CIFAR shows that the same choice can also improve both sample quality and class-prior agreement.
Adding pMF's perceptual losses narrows the FID gap to 10.60 versus 9.64 (Section 5.4). Prediction weighting can be used with these additional losses, although the feature objectives fall outside the target-preservation guarantee, as noted in the submitted paper.
Response to Reviewer o1sS (3/3)
Robustness and scope (W5, W7). The repeated runs confirm the sensitivity you identified in unweighted training, with all outcomes reported in Table 12 and Figure 8. The main image comparisons use one training seed. The submitted spiral comparison reports means and standard deviations over three training seeds (Table 7). For the underlying mechanism, the population flow shows the generative consequence without neural fitting, and the exact-mean image control shows that adaptive weighting changes the target even without a learned prediction (Appendix B.5).
CIFAR and ImageNet test two substantially different uses of one-step models: unconditional generation with extensive weighting comparisons, and class-conditional high-resolution generation with guidance. Their different performance orderings help establish why target preservation and sample quality should be considered separately. Direct FM further shows that the distinction is not confined to JVP-based training.
Recovering a rare mode and assigning it the correct probability are distinct goals. The population study and image comparisons examine the probability assigned to each group, which can change even when modes remain present. Generated proportions therefore complement coverage and sample-quality metrics when evaluating whether a model reproduces its training distribution.
Response to Reviewer o1sS (1/3)
Thank you for the detailed questions. Adaptive weighting changes both learning strength and the conditional targets the model learns to average. We characterize that selection and show its consequences for distribution recovery. Prediction weighting preserves the regression target while retaining adaptive scaling. The benefit of either rule depends on how well the model learns and which properties of the generated distribution matter for the application.
Target selection and generation (W2, W3, Q1). The population experiment provides a concrete failure to recover a prescribed distribution caused by weighting alone, without neural fitting error.
In Section 5.1 of the submitted paper, we integrate both fields from the same initial noise, and the pointwise regression is convex at p=0.5. Nevertheless, weighting reduces the rarest component's generated mass from the prescribed 0.686% to 0.130%, with the reduction persisting across numerical resolutions. Here the change in proportions measures a failure to recover a known distribution, caused by the objective rather than by neural optimization. At the stationary prediction in Table 1, total variance decreases by 13.8740, while the squared error relative to the true marginal velocity is 3.4772. Both measurements were already in submitted appendix Table 4.
The new Figure 4 shows this selection on CIFAR images. At the same noisy input, adaptive weighting changes how much valid clean images contribute to the conditional mean. Appendix B.5 quantifies this change across inputs using the closed-form conditional law of the training data.
The trained adaptive models also have larger errors relative to the closed-form mean when r<t (new Table 15). For the corrected velocity at p=0.5, RMS error is 1.365, versus 0.245 with prediction weighting and 0.223 without weighting, expressed per pixel in clean-image units. The same comparison holds at p=1 for r<t.
A separate control computes the residual weights around the exact conditional mean itself. Even with these weights, the expected adaptive update points away from the mean (Appendix B.5). Thus even the correct conditional mean can cease to be a stationary solution of the weighted regression problem.
We agree that adaptive weighting can improve sample quality while changing the target. Our ImageNet results favor it for FID. The population experiment shows the complementary case, where target selection reduces fidelity to a known distribution without neural fitting error. Our aim is to make this choice explicit and characterize the conditional law being selected.
Recovering the intended distribution also requires learning the target accurately. In the submitted spiral experiment, unweighted iMF and prediction weighting share the true conditional-mean target, yet prediction at p=0.5 improves one-call sliced Wasserstein distance from 0.0174 to 0.0096 under the same training budget. Here prediction weighting improves learning while retaining the intended velocity as the regression solution.
Which groups gain influence (W6, Q5). Corollary 5 predicts a group's local gain or loss from its average weight relative to competing targets. Its conditional mass increases when its average weight exceeds the overall conditional average, which depends on the arrangement of its targets relative to the prediction. In the added CIFAR analysis at p=1 and t∈[0.9,1], airplane's average conditional probability rises from 41.63% to 44.20%, whereas automobile's falls from 23.16% to 17.89%, although both are common classes (Appendix B.5).
Total variance cannot increase at stationary predictions for 0<p≤1, even after accounting for the shifted mean. This constrains the spread of the selected law around a mean that can itself move between groups. The guarantee therefore permits both gains and losses of rare-group influence. Final generated proportions also depend on the resulting flow. The population experiment follows the change in conditional targets through integration to the loss of rare-component mass. The CIFAR analysis shows that the local selection can favor one common class over another, consistent with the criterion depending on relative weights rather than class frequency alone.