acceptodds
Withdrawn from ICLR 2027

GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A commonly used family of RL algorithms for diffusion policies conducts softmax reweighting over samples from the behavior policy, which often induces an over-greedy policy and fails to utilize feedback from negative samples. In this work, we introduce GeMPO, a simple and unified framework that generalizes reweighting scheme in diffusion RL from softmax to general monotonic functions. GeMPO revisits diffusion RL via a measure matching perspective: First, we construct a virtual target policy measure via solving a regularized policy optimization objective; Second, we minimize the divergence between the current policy and this target measure through reweighted flow matching. This formulation offers two key advantages: i) It extends weight design beyond traditional exponential reweighting, allowing it to be tailored to diverse reward landscapes; and ii) by relaxing the non-negativity constraint on the target measure, our framework provides a principled justification for negative reweighting. We provide interpretations of how negative reweighting actively repels the policy from suboptimal actions and thus facilitates exploration. Extensive empirical evaluations demonstrate that GeMPO achieves competitive or superior performance by leveraging these flexible weighting schemes, and we provide practical guidelines for selecting reweighting methods in practice.

Primary area:
reinforcement learning
Keywords:
Diffusion Policy, Reinforcement Learning, Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dPE8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

GeMPO proposes a analytical framework for diffusion RL that generalizes reward reweighting beyond conventional exponential weighting, including linear, square, and negative weighting to improve policy optimization and exploration.

Critical strengths

Theoretical Insight and Generality.

The paper provides a principled and insightful theoretical framework that unifies existing reweighting methods under a general measure-matching formulation, offering deeper insights into the design of diffusion RL algorithms and broad applicability across diverse tasks.

Critical weaknesses
  1. Although negative reweighting yields promising improvements in some settings, its benefits are not consistent across tasks, with standard GeMPO variants achieving stronger overall performance in certain domains. This raises questions about how broadly the proposed negative-reweighting mechanism improves exploration and whether its effectiveness depends heavily on the task and reward landscape.

  2. The text-to-image experiments primarily use model-based rewards, including PickScore, HPS v2.1, and CLIPScore. It remains unclear whether the observed benefits of different reweighting strategies generalize to detector-based rewards, such as GenEval or OCR, which may exhibit different reward landscapes.

Questions for authors

Can the authors establish sufficient conditions for stable optimization with negative reweighting, and provide principled guidance for selecting the truncation parameter across different tasks?

Reviewer_nJV5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeMPO, a generalized measure-matching framework for online diffusion reinforcement learning. It unifies existing reweighted flow-matching methods through (f)-divergence regularization and extends the framework to signed measures, enabling negative reweighting. Experiments across continuous control, image generation, and DNA sequence generation demonstrate promising results.

Critical strengths
  1. The proposed framework provides a unified interpretation of existing methods, including QVPO [1] and DPMD [2].

  2. The paper provides meaningful theoretical analysis, including policy improvement for signed measures and the optimal weighted flow-matching solution.

  3. The experiments cover diverse domains and include extensive ablations.

Critical weaknesses
  1. Novelty. The framework mainly generalizes existing weighted flow-matching methods [1, 2], while negative reweighting has been explored in WD1 [3]. The authors should clarify the contributions beyond prior work.

  2. Theory and stability. The policy improvement guarantee for signed target measures does not directly extend to the learned diffusion policy. Moreover, negative weights may lead to an unbounded objective, as acknowledged in Appendix J. Stronger clarification of the stability conditions is needed.

  3. Experimental analysis. Although extensive ablations are provided, direct evidence of the exploration benefits and optimization stability of negative reweighting in complex control tasks remains limited.

  4. Related work. The paper could better discuss recent diffusion and flow RL methods, including CGPO [4], GenPO [5], GenPO++ [6], FPO [7], and FPO++ [8]. In particular, comparisons with FPO and FPO++ would help clarify the differences in optimization objectives and stability mechanisms.

  5. Presentation. The paper lacks an overview figure and an explicit summary of its main contributions in the Introduction. Adding both would improve clarity.

[1] Ding et al. (2024). Diffusion-Based Reinforcement Learning via Q-Weighted Variational Policy Optimization.

[2] Ma et al. (2025). Efficient Online Reinforcement Learning for Diffusion Policy.

[3] Tang et al. (2025). wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models.

[4] Ding et al. (2026). Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance.

[5] Ding et al. (2025). GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning.

[6] Hu et al. (2026). GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios.

[7] McAllister et al. (2026). Flow Matching Policy Gradients.

[8] Yi et al. (2026). Flow Policy Gradients for Robot Control.

Questions for authors
  1. Under what conditions does negative reweighting guarantee stable optimization and a valid probability flow?

  2. Can the authors provide direct evidence of improved exploration in complex control tasks?

  3. What are the key advantages of GeMPO over existing diffusion and flow RL methods?

Reviewer_h25POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The main contribution of this paper is an extension of AWR-style policy optimization for diffusion-based online RL. The extension mainly comes from replacing the KL term in the regularized policy optimization objective with a general ff-divergence. This leads to different reweighting functions and provides a unified interpretation of the weighting schemes used in methods such as DPMD and QVPO. The paper further relaxes the non-negativity constraint on the target measure to allow negative weights, aiming to better utilize low-value samples and improve exploration. In the locomotion setting, the method operates within an actor–critic framework, where a learned Q-function scores candidate actions and the actor is updated through reweighted flow matching. The method is evaluated on bandit, locomotion, image generation, and DNA sequence generation tasks.

Critical strengths
  1. The paper provides a unified view of AWR-style methods for diffusion-based RL. By replacing the KL term with a general ff-divergence, it derives different reweighting functions and connects the weighting schemes used in DPMD and QVPO. I find this useful for understanding the relation between these methods and how other weighting functions can be designed.

  2. The method is evaluated on several types of tasks, including locomotion, image generation, and DNA sequence generation. In particular, the best negative-weight variant improves the predicted DNA activity by 14.3% over the strongest reported baseline. These results show that the proposed extension can bring practical benefits beyond the toy example, which supports the usefulness of the method.

Critical weaknesses

The main concern I have is whether the experiments sufficiently support the motivation for the proposed extensions. Negative reweighting makes sense as a way to push the model away from low-value actions, but it is not clear when this provides a meaningful benefit over simply assigning these actions smaller or zero weights. In locomotion, the negative variant is only compared with GeMPO-Linear on two tasks, with modest gains, and is not included in the main comparison with Exp and Square. Similarly, although the paper argues that exponential reweighting can be overly greedy, Table 2 shows very similar normalized average scores for Square and Exp, 117.0 and 116.7, with 4 out of 6 tasks having better performance using Exp. This leaves the practical benefit of the main contributions insufficiently established in the online control setting.

Questions for authors
  1. The paper argues that exponential reweighting can be overly greedy, but GeMPO-Square and GeMPO-Exp achieve very similar mean scores in Table 2. Can the authors clarify under what conditions the alternative weighting functions provide a clear benefit?
  2. Is there direct evidence that negative reweighting improves exploration outside the toy bandit setting?
  3. Can the authors provide the full critic-training procedure and explain how it is combined with Algorithm 1? In particular, are the critic architecture, update rule, and training budget kept the same across the GeMPO variants and the relevant baselines? This would help determine how much of the improvement comes from the proposed reweighting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.