acceptodds
Withdrawn from ICLR 2027

MEND: RL VIA PROXIMAL TARGET MATCHING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reward-based post-training of flow models often relies on KL penalties, refer- ence models, or thousands of updates. Direct reward backpropagation provides a direction but does not check whether each proposed change justifies its magni- tude. We introduce MEND, a reinforcement learning method based on proximal target matching. A reward cap assigns unchanged targets to high-scoring samples. Below the cap, MEND proposes moves along the reward gradient and accepts a move only when its capped reward gain exceeds a quadratic displacement penalty. The model learns the accepted displacements through regression, without a KL term, frozen reference model, or advantage weights. MEND scores higher than the Flow-GRPO adapter on five of six evaluators at the same reported distance to base-model images. Under an equal-budget protocol, it exceeds the reported ReFL and DiffusionNFT curves at every reported update across four training rewards, reaching PickScore 24.03 versus 23.92 and 23.43, respectively. A three-reward run also exceeds the released DiffusionNFT checkpoint on all three training re- wards. The method transfers to another flow backbone and a distilled model. Code and checkpoints will be released.

Primary area:
generative models
Keywords:
diffusion models, flow matching, reward fine-tuning, reinforcement learning from human feedback, reinforcement learning, information projection, exponential tilting, text-to-image generation, sample diversity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_g1G7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper generates multiple samples per prompt, uses reward gradients to construct candidate moves, and selects a candidate based on capped reward minus a quadratic displacement penalty. It then uses the selected displacement to construct a regression loss for training. Under a 100-update budget, the method reports better results on the training reward metrics than ReFL and DiffusionNFT.

Critical strengths
  1. Under the 100-update setting, the method outperforms ReFL and DiffusionNFT on the training reward metrics. The training curves also show faster improvement in the early stages.
Critical weaknesses
  1. Overall, the method lacks a natural derivation from first principles and includes many design choices that seem ad hoc to me. I find it difficult to understand the motivation and mathematical rationale behind these choices. The method also has too many hyperparameters that need to be set or tuned, such as the reward-cap quantile (0.75), the global cap floor, the proposal step sizes (0.1, 0.2, 0.4), the acceptance-rate target used to adjust the displacement penalty, the keep-loss weight, and the intermediate time step (t=0.278t=0.278).
  2. The paper provides limited evidence that the method achieves better final performance than ReFL or DiffusionNFT. Table 3 compares methods under a fixed 100-update budget rather than after convergence. Table 2 compares MEND with released checkpoints trained with different reward objectives and numbers of updates.
  3. The paper constructs candidates yjy_j using reward gradients and then uses the selected displacement to define a regression loss. This extra step feels indirect and insufficiently motivated.
Questions for authors
  1. The regression loss is applied at a stored state near t=0.278t=0.278. Why is only one intermediate time step used? How does this supervision affect the full sampling trajectory, and could the choice of time step limit the final reward improvement?
Reviewer_wBZGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MEND transforms the reward tuning of the flow model into sample-by-sample proximal target selection: proposals are made along the reward gradient, validated by subtracting the cost of the second displacement from the capped reward gain, and only the accepted displacements are regressed, thus eliminating the need for KL, frozen references, or advantage weights, outperforming Flow-GRPO and DiffusionNFT in very few updates.

Critical strengths

The paper is well organized and clearly written. The authors also provide detailed experimental analyses to validate the proposed method.

Critical weaknesses
  1. The proposed method requires differentiating the reward with respect to the generated sample, whereas conventional RL methods such as Flow-GRPO and ReFL only require scalar reward evaluations. This restricts MEND to differentiable rewards and excludes black-box, rule-based, or human-preference rewards unless a surrogate reward model is trained.

  2. The paper should more explicitly discuss reward hacking. MEND removes the KL term and relies on capped reward gains, a quadratic displacement penalty, and a lagged behavior adapter to limit per-step movement. However, these mechanisms only constrain individual target displacements and do not prevent the training reward itself from being over-optimized. The LAION aesthetic experiment in Table 10 already illustrates this risk.

  3. The cap and price are heuristic hyperparameters that require manual tuning. It remains unclear whether the same settings are stable across different backbones, such as Flux and Qwen-Image. The paper should evaluate hyperparameter robustness on more backbones and a broader set of rewards to demonstrate that the method is not overly sensitive to these choices.

  4. Since MEND requires reward backpropagation and multiple proposal evaluations per round, it is important to report wall-clock time, GPU memory, and total GPU-hours under matched computational budgets to substantiate the claimed update efficiency.

Questions for authors

Please see the weaknesses.

Reviewer_kfR1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the post-training method for flow-matching-based text-to-image generation. The proposed method, MEND, I assumed it belongs to the type of "Gradient ascent". The main novelty comes from 2 points in my opinion: (1) set a cap so that top-scored samples don't move during the optimization, and (2) even for none-top-scored samples, set a selection process that only worthwhile gradient moves are allowed. The experiment results are good, mostly from the efficiency view.

Critical strengths
  1. the experiment results is impressive, especially from the efficiency view.
  2. the topic is critical and has a good application (T2I) impact in real life.
Critical weaknesses

The main problem is, the paper is bit hard to read, for me it's more like giving many method details without much motivation clarification or proof. For example, regarding the motivation, I have two questions:

(1) in Figure 2(b), why these RL methods like flow-GRPO will result in a hard-mode loss? For my understanding, even for hard prompts, as long as you try group-based sampling and there is a high-score trajectory, then after training this high-score trajectory is rewarded leading to the final performance boost. Therefore why is there a hard-mode loss and if it exists, what will happen? Because from so many open-sourced works like dance-grpo, flow-grpo, they indeed improve the performance.

(2) for equation(3), why is it designed in this form, I mean the subtracted term? I understand the motivation is to select if the gradient move is worthwhile or not, however it seems like there are many terms that can evaluate this and why the current one is chosen? Also, the selection also applies to every sample below the cap, while it is verified for the current sample, how to make sure that this verified move is also reliable to other samples? In the worst case, one move will lead to a modest improvement in the current sample but result in a degradation in the global view.

Besides these, I have an additional question regarding the experiment:

(3) for flow-grpo there is a accelerated version, please see https://github.com/yifan123/flow_grpo/blob/main/scripts/multi_node/sd3_fast.sh. Since the main success in the experiment is largely from the efficiency, could you compare MEND with this variant?

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.