acceptodds
Withdrawn from ICLR 2027

Reinforcing Diffusion Models via Value-Implicit Reward Matching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Online Reinforcement Learning (RL) has proven effective for aligning diffusion and flow models with task-specific rewards. However, existing methods rely heavily on newly generated on-policy samples for stable policy improvement, leading to poor sample efficiency. We introduce **VIRM**, a value-implicit reward matching framework that casts policy improvement as regression on replayed samples. VIRM builds on the optimality condition of KL-regularized RL, where the optimal policy exponentially tilts the reference toward high reward. Inverting this identity reveals that each sample's reward is implicitly encoded by the policy–reference log-density ratio and a per-prompt log-partition term. Using an exponential moving average (EMA) policy as the reference, VIRM optimizes the current policy by matching this policy-implied reward to external feedback. Concretely, the log-density ratio is estimated via an ELBO difference that acts as an implicit value, while the intractable log-partition term is predicted by a token-wise value head. VIRM learns from replayed data without importance weights or stored denoising trajectories. To focus replay on correctable errors, samples are prioritized by reducible loss, a reference-calibrated measure of learnability rather than intrinsic difficulty. On GenEval2, VIRM improves FLUX.2-klein-base-9B from to , outperforming the strongest baseline by more than 40%, with concurrent gains in CLIP score, PickScore, and HPSv2. These results establish value-implicit reward matching as a sample-efficient post-training method for diffusion and flow models.

Primary area:
generative models
Keywords:
Reward Matching, Diffusion Models, Flow Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_evrkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the high rollout and reward-evaluation costs from low update-to-data ratio of diffusion and flow model alignment by proposing Value-Implicit Reward Matching (VIRM). VIRM reformulates KL-regularized optimal tilted distribution to derive implicit reward of a given sample from the current policy, then apply regression between the implicit and actual reward to induce the reward-tilted distribution. This enabled to justify re-using old off-policy samples to achieve high update-to-data ratio and data efficiency. Experiments on SD3.5 and FLUX.2 report improved compositional alignment and out-of-domain performance, especially improving GenEval2 for FLUX.2 to 65.4(baseline 55.2).

Critical strengths
  • The paper presents a distinct approach to diffusion/flow model fine-tuning through implicit reward matching, apart from many RL-inspired(e.g., flowGRPO) or RWR-based(e.g., diffusionNFT) methods.
  • The primary result suggests faster and more effective reward alignment against other state-of-the-art baselines.
Critical weaknesses

I'd like to revise the score if the following are resolved in the rebuttal.

  • Insufficient justification of zϕz_{\phi}: The theoretical normalizer shall be a scalar, but the paper didn't adequately justify the design of zϕz_{\phi} to have the same size as the data, or its prediction through an auxiliary model head.
  • Several unprovided results about the following limit the soundness of the proposed method:
    • Diversity is assessed with several sample images only, without quantitative metrics such as Vendi[1] or DreamSim[2] diversity score.
    • Diffusion fine-tuning methods are sensitive to many factors(e.g., #samples per prompt, #prompts per iteration, batch size, and #update steps per iteration), where these details are not reported; it is unclear whether they were matched with the baselines, and hard to reproduce the results.

[1] The Vendi Score: A Diversity Evaluation Metric for Machine Learning, TMLR 2023

[2] DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data, NeruIPS 2023

Questions for authors
  • Since z(c)z(c) is a reward-specific quantity independent of the generative model, is it necessary to predict zϕz_{\phi} by an additional head on the generative model?
  • Could the authors provide an ablation result where zϕz_{\phi} is learned as a model-independent scalar?
  • Since methods that use a forward process(e.g., DiffusionNFT) utilize multiple training samples xtx_t from a single sampled clean image x0x_0, isn't it misleading to characterize such methods as having UTD = 1?
Reviewer_a3nQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Value-Implicit Reward Matching (VIRM), a framework for improving the sample efficiency of reinforcement learning for diffusion and flow models. It combines ELBO-based reward matching, a learned token-wise value head, and prioritized replay to enable high-UTD training without importance weighting. Experiments on SD3.5-Medium and FLUX.2-klein-base-9B demonstrate promising improvements.

Critical strengths
  1. The paper addresses the sample efficiency of reinforcement learning for diffusion and flow models, which is an important problem given the high computational cost of RL.
  2. The combination of a learned value head and prioritized replay is interesting and potentially useful for improving data utilization.
  3. The method achieves competitive performance across different model architectures and evaluation metrics on Geneval2.
Critical weaknesses

1. Novelty over closely related work.

The overall novelty of the proposed method appears somewhat limited, particularly in comparison with Centered Reward Distillation (CRD). Several key ingredients of VIRM, including the formulation based on the Gibbs optimal policy, the ELBO-based approximation of the policy-reference log-density ratio, and the MSE-based matching between implicit and external rewards, have already been explored in CRD.

More importantly, CRD also explicitly allows replay-buffer-based data sampling. Therefore, the general idea of combining reward matching with experience replay is not entirely new. The main distinction of VIRM appears to lie in enabling independent per-sample replay rather than group-coupled reward matching (CRD).

The contribution statements in the Introduction do not sufficiently distinguish VIRM from prior work. In particular, the first two contributions emphasize reward matching, ELBO-based density-ratio estimation, and reward centering, although these techniques have already been explored in CRD. The authors should more clearly identify which components are genuinely novel.

Additionally, the empirical claim that VIRM "consistently" improves generation quality and diversity appears stronger than the supporting evidence. VIRM does not outperform the strongest baseline on every evaluation metric. The authors should qualify these claims and provide stronger empirical evidence, particularly for diversity preservation.

2. Comparison with CRD.

Although CRD is currently available as a preprint rather than a formally accepted conference publication, it is highly relevant to the proposed method. I wonder whether the authors have considered a direct comparison with CRD. Given the substantial methodological overlap between CRD and VIRM, such a comparison is essential for understanding to what extent the reported improvements over existing baselines can be attributed to VIRM's specific contributions, rather than to the framework already explored in CRD.

Furthermore, since high UTD efficiency is a central claim of VIRM, a comparison between CRD with group-level replay and VIRM with per-sample replay, under matched generation budgets, would help clarify whether the proposed method provides a meaningful advantage in sample efficiency and training stability.

Given that CRD has publicly available code, such a comparison would substantially strengthen the empirical justification for the proposed contributions.

3. Figure 1 could be more informative.

Figure 1 appears more like a promotional illustration than a technical overview of the proposed method.

Questions for authors

1. Gap between the theoretical formulation and the practical value head.

In Theorem 1, the normalization term z(c)z(c) is defined as a prompt-dependent scalar, whereas the practical objective employs an image-conditioned, token-wise value head z(xt,t,c)z(x_t, t, c). This discrepancy raises a potential issue: a sufficiently expressive value head may absorb the reward-matching residual without inducing meaningful improvements in the diffusion policy.

Although Theorem 2 introduces an approximation error for the learned value head, the paper does not establish that this error is sufficiently small in practice.

The authors should provide stronger theoretical or empirical justification for the token-wise design.

2. Question regarding potential overfitting of the value head.

I wonder whether the authors have considered an experiment where the diffusion model is frozen and only the token-wise value head is trained. It would be interesting to examine how much the reward-matching MSE can be reduced by training only the value head, and compare this with the normal joint training setting. If the value head alone can fit the reward-matching objective to a very low loss, this may suggest a potential shortcut solution, where minimizing the training objective does not necessarily correspond to meaningful policy improvement. Such an experiment would help clarify the role and necessity of the proposed token-wise value head.

Reviewer_WM15OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VIRM, a value-implicit reward matching framework that casts policy improvement as regression on replayed samples. It combines an ELBO-based log-density-ratio surrogate, a learned token-wise normalizer, and prioritized replay to enable repeated updates on rewarded outputs without importance weighting or stored denoising trajectories. Experiments on SD3.5-Medium and FLUX.2-klein-base-9B show improved performance and training efficiency over the evaluated baselines.

Critical strengths
  • The paper addresses a practical challenge in online diffusion RL: the high cost of sampling and reward evaluation, coupled with limited reuse of generated samples. This provides a compelling motivation for replay-based policy optimization.

  • Empirical results on GenEval2 are strong, showing substantial improvements on FLUX.2-klein-base-9B.

Critical weaknesses
  • The identification guarantee relies on a prompt-only normalizer and an exact density ratio, whereas the practical loss uses an image-conditioned token-wise head and a denoising-error surrogate. It remains unclear whether minimizing this loss preserves the theoretical guarantee.

  • The experimental scope is limited to multi-reward training on GenEval2-style prompts. Additional in-domain experiments on distinct tasks, such as text rendering, are needed to establish the method’s generality beyond compositional generation.

Questions for authors
  1. What prevents the image-conditioned token-wise value head from absorbing sample-dependent reward variation? How does the practical loss retain the identification property established for a prompt-only normalizer and an exact density ratio?

  2. Does VIRM remain effective when trained on tasks beyond GenEval2-style compositional generation, such as text rendering? Can you provide additional in-domain results to substantiate its generality?

  3. How does performance vary with the UTD ratio? Since greater data reuse may increase policy drift and overfitting, how is UTD selected, and could reward improvement or saturation guide when to refresh the anchor and collect new samples?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.