acceptodds
Withdrawn from ICLR 2027

MeanFlowAdvantage: Advantage-Weighted Least Squares for Average-Velocity Generators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

MeanFlow enables efficient few-step generation by predicting interval-average velocities, but this representation creates a mismatch for reward fine-tuning: existing advantage-based objectives are typically defined on instantaneous velocities or equivalent -space predictions, whereas inference directly uses the learned average-velocity map. We introduce MeanFlowAdvantage, a signed advantage-weighted least-squares objective for average-velocity generators. Our key construction uses a shared, detached MeanFlow derivative correction to express the reward objective in prediction space while making rollout and reference regularization exact penalties on the average-velocity network deployed at inference. The resulting formulation preserves MeanFlow's native few-step sampler and provides a direct mechanism for transferring reward improvements to the deployed flow map. On SD3.5-Medium, MeanFlowAdvantage improves all eight reported metrics over the matched four-step MeanFlowNFT baseline and, with only four NFEs, matches or exceeds the 40-step DiffusionNFT baseline on six of eight metrics. The same objective also transfers to DNA promoter design, where it supports both teacher-free on-policy RL for a generator defined on a manifold and teacher-guided reward-graded distillation, with the latter yielding the lowest one-step Sei profile MSE among the compared configurations.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Post Training, Distillation, MeanFlow

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7aNFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MeanFlowAdvantage (MFA), a signed advantage-weighted least-squares objective for reward fine-tuning of average-velocity generators. Unlike conventional flow-matching models, MeanFlow directly predicts the interval-average velocity uθ(xt,s,t)u_\theta(x_t,s,t), which is used for few-step generation. However, existing reward fine-tuning methods typically operate in instantaneous-velocity or x0x_0-prediction space, creating a mismatch between the training objective and the representation deployed at inference time.

To address this issue, the authors construct an induced prediction using a shared, stop-gradient MeanFlow derivative correction. This formulation enables signed advantage-based reward optimization in prediction space while ensuring that rollout and reference regularization directly constrain the deployed average-velocity maps. The paper further provides a theoretical analysis of the resulting objective and evaluates the method on text-to-image generation and DNA promoter design.

Critical strengths

1. A practically meaningful problem with a representation-aware formulation.

The paper addresses an important challenge in reward fine-tuning of few-step generative models. Since MeanFlow directly uses interval-average velocities for inference, applying conventional prediction-space objectives without accounting for this representation may introduce a mismatch between optimization and generation. The proposed formulation addresses this issue by applying reward supervision in prediction space while explicitly regularizing the underlying average-velocity maps. This is a reasonable and well-motivated design choice that preserves the native few-step sampling mechanism of MeanFlow.

2. Promising empirical results across two different application domains.

The experiments cover both text-to-image generation and DNA promoter design, demonstrating the applicability of the proposed objective beyond a single domain.

On SD3.5-Medium, MFA improves all eight reported metrics over MeanFlowNFT under matched evaluation settings. In particular, Aesthetic Score increases from 5.928 to 6.300 (+0.372), while HPSv3 improves from 13.676 to 13.933 (+0.257). Notably, neither of these metrics is directly optimized during training, suggesting that the improvements are not restricted to the training rewards.

The DNA promoter experiments further demonstrate the flexibility of the framework in both teacher-free on-policy RL and teacher-guided reward-graded distillation. In the distillation setting, MFA achieves a Sei profile MSE of 0.0467, compared with 0.0570 for MeanFlowNFT, corresponding to approximately an 18% reduction. These results strengthen the empirical support for the proposed formulation.

Critical weaknesses

1. Limited methodological novelty relative to AdvantageFlow and MeanFlowNFT.

The methodological novelty appears somewhat limited, as the proposed framework primarily combines existing techniques from AdvantageFlow and MeanFlowNFT.

Specifically, AdvantageFlow has already introduced the signed advantage-weighted least-squares objective with rollout and reference regularization. Meanwhile, MeanFlowNFT has established how to apply reward-based fine-tuning to average-velocity generators through an induced instantaneous-velocity predictor derived from the MeanFlow identity.

The proposed MFA objective largely follows the AdvantageFlow formulation (Eq. 8), while adopting the MeanFlow-based prediction transformation already explored by MeanFlowNFT. The main technical distinction lies in using a shared, detached derivative correction so that the regularization terms reduce to direct differences between average-velocity predictions (Eqs. 7–10).

Although this construction provides a useful representation-aligned formulation, the core optimization objective and prediction-space transformation are both closely related to prior work. Consequently, the contribution appears more incremental than fundamentally novel, and the paper could more clearly articulate the methodological advances beyond integrating these existing components.

2. A substantial gap between the theoretical assumptions and practical implementation.

The theoretical guarantees rely on strong assumptions that may not hold in practical training. In particular, Assumption A3 requires

Eπold[uˉ∣Z]=uold(Z),\mathbb{E}_{\pi_{\mathrm{old}}}[\bar{u}\mid Z]=u_{\mathrm{old}}(Z),

which assumes that the rollout model is already a calibrated MeanFlow fixed point under its own sampling distribution.

This assumption is central to the population-level interpretation in Theorem 1 and the associated analysis. Theorem 2 further characterizes exact sampling under an idealized self-consistent fixed-point condition.

However, the practical implementation involves finite-step sampling, EMA-based rollout models, LoRA parameterization, finite-difference derivative estimation, and sample-dependent adaptive scaling. These factors introduce discrepancies relative to the assumptions of exact corrections, calibrated rollouts, and unrestricted function optimization used in the theoretical analysis.

The experimental results also indicate that the learned model does not fully satisfy the idealized flow-map consistency condition. As reported in Section 5.1 and Figure 4, the relative semigroup residual increases from 0.40 at two sampling steps to 0.76 at 32 steps, suggesting that the theoretical fixed point is only approximately attained.

Although the analysis offers useful intuition, the gap between its idealized assumptions and the actual training procedure limits the extent to which the theoretical guarantees can explain the observed practical behavior.

3. Relatively modest improvements on several evaluation metrics.

Although MFA outperforms MeanFlowNFT on all eight reported metrics under matched evaluation settings, the improvements on several metrics are relatively small.

For example, ImageReward increases only from 1.446 to 1.451 (+0.005), CLIPScore from 0.296 to 0.297 (+0.001), and OCR from 0.627 to 0.631 (+0.004). Similarly, HPSv2 improves only from 0.327 to 0.330 (+0.003).

These gains are substantially smaller than those observed for Aesthetic Score and HPSv3. Therefore, while the consistent numerical improvements are encouraging, their practical significance is less convincing for several metrics. The reported results support improvements over the baseline, but the overall empirical advantage appears uneven across different evaluation criteria.

Questions for authors

See weaknesses

Reviewer_w3qsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MFA, a signed advantage-weighted least-squares objective for reward fine-tuning of average-velocity generators. It applies an AdvantageFlow-style objective to induced MeanFlow predictions, and evaluate on image generation and DNA promoter design.

Critical strengths

Although the training framework largely follows MeanFlowNFT, the paper provides useful receips for performance improvement. Moreover, the authors do lots of math works to provide an unified framework. The modification of loss function is simple but might be useful. I regard this as a substantive strength, regardless the contribution is incremental. My main reservations concern the empirical support for its benefits rather than an absence of methodological novelty.

Critical weaknesses

The emperical evidencea are insufficient. Compared to matched MeanFlowNFT, most image gains in Table 1 are modest, including 0.005 in ImageReward, 0.001 in CLIPScore, and 0.004 in OCR. The ratios of improvement are low. Aesthetic Score is a more substantial positive result than any other metrics. It remains unclear whether the larger Aesthetic gain comes from the objective structure, effective update strength, reference anchoring, or normalization. Controlled empirical attribution would address this concern. I believe repeated experimental can be helpful, but the attribution analysis ismore improtant for me.

The image evidence is concentrated on SD3.5-Medium with one training reward combination. The four-seed DNA distillation result is encouraging. DNA on-policy RL does not establish an advantage over NFT on the optimized reward. Thus both reproducibility and coverage limit my confidence in the practical benefit of the proposed objective. Stable replication and a matched on-policy comparison on another image backbone or a distinct reward configuration will improve my assessment. I suggest the authors put some effort on DNA promoter generation tasks, it will provide strong evidence on supporting MFA.

The ablations do not sufficiently isolate the objective's contribution from hyperparameters and implementation choices. Table S1 matches many important settings, but the methods use different objective coefficients and reference penalties, and the implementations also differ in normalization. Different objectives need not use identical numerical coefficients, what is missing is a clear isolation of those items. Figures 5 and S2 provide useful learning curves and stability diagnostics, but primarily establish the importance of components within the MFA recipe.

Questions for authors

See weakness part.

Reviewer_tECvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MeanFlowAdvantage (MFA), which adapts AdvantageFlow’s signed advantage-weighted least-squares objective to MeanFlow generators. A shared, stop-gradient derivative correction makes rollout and reference regularization reduce exactly to differences between average-velocity predictions. The paper analyzes the resulting reward-reweighted target distribution and characterizes self-consistent fixed points that induce exact flow maps. Experiments cover SD3.5-Medium text-to-image generation and DNA promoter generation. The approach is practically relevant, and the central algebraic derivations appear sound under the stated assumptions. However, the novelty relative to existing methods is overstated.

Critical strengths
  • Applying advantage-weighted regression to finite-interval average-velocity models is a reasonable research direction. Equation (10) clearly exposes the relationship between reward fitting and average-velocity regularization. Theorem 1 provides a useful interpretation of the objective as MeanFlow regression under a reward-reweighted distribution with a frozen rollout derivative.

  • Promising results on both text-to-image generation and DNA promoter generation

Critical weaknesses
  • Novelty is overstated: MeanFlowNFT already uses shared derivatives and their exact cancellation into average-velocity differences.
  • Image improvements lack uncertainty estimates: Several gains are small, with no reported training-seed variability or paired confidence intervals.
  • The strongest simple DNA baseline is underemphasized: Constant-advantage NFT achieves 0.0474 MSE versus MFA’s 0.0467, substantially narrowing the headline advantage.
Questions for authors
  • What is genuinely new beyond combining MeanFlowNFT’s construction with AdvantageFlow’s loss?
  • Do gains survive matched tuning budgets and multiple image-training seeds?
  • Does graded MFA significantly outperform constant-advantage distillation under the same multi-seed protocol?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.