acceptodds
Under review as a conference paper at ICLR 2027

Policy Optimization via Drifting

10 views

Abstract

Reinforcement learning (RL) of diffusion and flow-based policies has become a popular paradigm in robot learning, where actions are continuous and multimodal. Sampling from these generative policies, however, costs many sequential network evaluations and incurs latency, making them a poor fit for dynamic tasks with tight feedback loops. Drifting models offer an appealing alternative. Empirically, they are as expressive as diffusion and flow models, but they can produce a sample with a single network evaluation. In this work, we propose *reward drifting*, an on-policy policy gradient optimization method. Starting from the theory of Wasserstein gradient flows, we derive a single fixed-point objective and justify that optimizing it is equivalent to increasing the expected discounted reward. Our method does not require access to action-likelihoods or surrogates. Empirically, we evaluate our algorithm across a range of manipulation tasks, including Push-T, dexterous in-hand reorientation, and long horizon bimanual manipulation. Our method offers substantially faster inference than diffusion and flow-matching baselines fine-tuned with existing RL methods.

Primary area:
reinforcement learning
Keywords:
Generative modeling, Reinforcement learning, Transport, Variational Calculus

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_N6DbOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Reward Drifting, an on-policy method for fine-tuning one-step generative policies without evaluating action likelihoods. The method estimates a reward-driven attraction and entropy-related repulsion field from sampled actions and their returns, then trains the generator toward the displaced samples. Experiments on three simulated manipulation tasks show high success rates comparable to the evaluated diffusion and flow baselines, together with much lower policy inference latency.

Critical strengths
  1. The method addresses a practical problem: combining fast action generation with reinforcement learning updates that do not require tractable action likelihoods. (I do like the solid mathematical part but I am not familiar with it and have notified the AC upon this)
  2. The experiments demonstrate substantial reductions in inference latency across three manipulation tasks, in both eager and compiled execution modes. These results highlight the method's potential for applications where fast policy inference is important.
Critical weaknesses
  1. Reward Drifting substantially reduces inference latency, but ReinFlow appears to learn faster and better on Push-T in Figure 4, while their final success rates are similar. The three tasks provide encouraging evidence, but the near-saturated final success rates make it difficult to judge whether a performance gap would emerge on a broader or more challenging benchmark suite. Additional evaluations would strengthen the claim that the method retains performance comparable to ReinFlow while improving inference efficiency.

  2. As I understand it, the main results rely on BC initialization (please correct me if I am wrong). As acknowledged in Section 7, learning from scratch does not recover all behavior modes in the reported experiment. This limits the evidence for learning diverse behaviors without demonstrations, although the results support the intended fine-tuning setting.

Questions for authors

See Weaknesses.

1 comment

We thank the reviewer for their questions and address them below.

Q.1. Reward Drifting substantially reduces inference latency, but ReinFlow appears to learn faster and better on Push-T in Figure 4, while their final success rates are similar. The three tasks provide encouraging evidence, but the near-saturated final success rates make it difficult to judge whether a performance gap would emerge on a broader or more challenging benchmark suite. Additional evaluations would strengthen the claim that the method retains performance comparable to ReinFlow while improving inference efficiency.

In addition to the 3 tasks in the original submission, we include 8 new robomimic tasks as well as 5 new MuJoCo locomotion benchmarks giving us a total of 16 tasks across different simulators and robot morphologies to further strengthen our claim. Across all new tasks we observe that Reward Drifting has superior or comparable performance to Reinflow while being faster at run time. Upon making changes to the way credit assignment (details in Section 5 of the revised draft) is done in Reward Drifting we found that we are able to reliably beat Reinflow on wall-clock time on most tasks.

Q.2. As I understand it, the main results rely on BC initialization (please correct me if I am wrong). As acknowledged in Section 7, learning from scratch does not recover all behavior modes in the reported experiment. This limits the evidence for learning diverse behaviors without demonstrations, although the results support the intended fine-tuning setting.

As correctly pointed out by the reviewer, reward drifting is intended for the post-training setting. As shown in Figure 3, if the original behavior cloning policy is multimodal, then reward drifting retains both modes during training and there is no collapse.

We included the learning from scratch experiments in our original submission for the sake of completeness. Learning generative policies from scratch that reliably learn all possible modes remains an open problem and we do not claim that as a contribution of our work. If these additions resolve the concerns behind your assessment, we would be grateful if you would reconsider your score.

Reviewer_QJMuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Reward Drifting, an on-policy method for fine-tuning one-step generative policies without evaluating action likelihoods. It estimates a reward-driven Wasserstein gradient field from samples and uses this field to construct policy updates. Experiments on three robotic manipulation tasks show success rates broadly comparable to DPPO and ReinFlow, with lower measured inference latency.

Critical strengths
  1. The paper addresses a useful practical problem: fine-tuning a one-step generative policy without requiring tractable action likelihoods. Figure 4 (p. 8) shows competitive final success rates relative to DPPO and ReinFlow on all three evaluated tasks.

  2. Table 1 (p. 9) provides concrete evidence of low inference latency for the proposed implementation. The exploration study in Figure 5 (p. 8) also provides useful evidence about the effect of different noise schemes on learning

Critical weaknesses
  1. The evidence does not sufficiently isolate the benefit of the proposed policy-update mechanism. Figure 4 (p. 8) evaluates complete systems, while Table 3 (p. 17) shows that Reward Drifting and DBPO differ in both backbone architecture and exploration mechanism. Figure 5 (p. 8) further demonstrates that exploration design substantially affects learning. Consequently, the current results do not clearly establish how much of the observed advantage comes from the proposed reward-drifting update rather than these other design choices. This matters because the update mechanism is a central methodological contribution.

  2. Some empirical comparisons support narrower conclusions than the overall presentation suggests. MFPO uses substantially smaller interaction budgets and is evaluated without critic-guided candidate selection (Table 6, p. 18), whereas the main on-policy methods use the budgets in Table 2 (p. 17). These differences are disclosed, but MFPO’s lower curve in Figure 4 cannot cleanly establish an algorithm-level advantage. Similarly, the latency results in Table 1 establish the speed of the tested implementations, while the architectural differences in Table 3 limit attribution of the speedup to the proposed update.

Questions for authors
  1. Could you clarify which existing results isolate the contribution of the reward-drifting update from backbone and exploration choices? If this is not isolated by the current experiments, a focused comparison with matched backbone and exploration settings on one representative task would help assess the methodological contribution (Figure 4, p. 8; Table 3, p. 17).

  2. Since Theorem 1 is acknowledged as an existing result (p. 5), could you state precisely which aspects of the field estimator and update procedure constitute the new technical contribution? In particular, please clarify the distinction from Koo et al. (2026), already discussed on p. 3, including its discussion of exponential value weighting. This is a request to clarify the contribution, rather than a requirement for an additional comparison with contemporaneous or unpublished work.

2 comments

We thank the reviewers for their comments and address their questions below.

Q.1. Could you clarify which existing results isolate the contribution of the reward-drifting update from backbone and exploration choices? If this is not isolated by the current experiments, a focused comparison with matched backbone and exploration settings on one representative task would help assess the methodological contribution (Figure 4, p. 8; Table 3, p. 17).

The original experiments used the code implementation from the original DBPO code provided by Gao et. al. In order to isolate the contribution of the reward drifting update and the exploration choice, we run a controlled experiment where both methods use the same architecture and use gaussian noise of the same magnitude.

The results can be found in Figure 11 of the revised draft. Across all 6 tasks (3 seeds per method) we observe that reward drifting with gaussian noise continues to converge faster and more reliably than DBPO.

Q.2. Since Theorem 1 is acknowledged as an existing result (p. 5), could you state precisely which aspects of the field estimator and update procedure constitute the new technical contribution? In particular, please clarify the distinction from Koo et al. (2026), already discussed on p. 3, including its discussion of exponential value weighting. This is a request to clarify the contribution, rather than a requirement for an additional comparison with contemporaneous or unpublished work.

Thank you for raising this. Theorem 1 only provides a clear theoretical motivation and a starting point for our algorithm, it has not been explored in the context of a practical algorithm before, and our main technical contribution is combining that view with the Wasserstein field estimator and update procedure built from it. We estimate the reward’s Wasserstein gradient directly from sampled actions and return estimates, without action likelihoods or critic action gradients, and connect the resulting regression update to a parameter-space JKO step. The closely related, concurrent work of Koo et al. (DFP) also develops a Wasserstein interpretation and discusses exponential value weighting. Moreover, their implemented update uses top-K critic-selected actions, whereas ours retains soft value weighting to estimate the reward field, with an explicit entropy coefficient. Our JKO formulation additionally explains the proximal update and its inner optimization steps in a more rigorous manner. We have clarified this connection in Appendix A.

Finally, Koo et al. developed their algorithm for the off-policy setting, whereas ours is specifically designed for on-policy RL. To that end, we also adapted DFP’s implementation to our on-policy setup. On the eight added manipulation tasks, our method attains 100% held-out success, compared with DFP’s scores in Figure 7. These empirical results complement the methodological distinction

Q.3. MFPO uses substantially smaller interaction budgets and is evaluated without critic-guided candidate selection (Table 6, p. 18), whereas the main on-policy methods use the budgets in Table 2 (p. 17). These differences are disclosed, but MFPO’s lower curve in Figure 4 cannot cleanly establish an algorithm-level advantage.

We agree that MFPO uses fewer environment interactions in these experiments. Since Reward Drifting is an on-policy method, comparing with other on-policy methods would be most fair; we nevertheless include off-policy baselines for completeness and report their interaction budgets separately. We include all critic-guided baselines with best-of-K selection in Figure 4 and Figure 7 of the revised draft. We use K = 16 for all critic-guided methods. Candidate selection improves their performance but adds inference cost.

That being said, it is important to emphasize that our practical efficiency claim concerns wall-clock performance, not fewer samples than MFPO. With the updated implementation, Reward Drifting takes approximately 0.2 hours versus ReinFlow’s 1.5 hours on PushT, 0.5 versus 1.8 hours on D’Claw, and 1.2 versus 1.7 hours on Transport, at comparable success rates. These measured pipeline times, detailed in the revised paper, show that Reward Drifting can offer faster training as well as faster inference.

We hope the matched-backbone and exploration comparisons, expanded evaluation, and clearer reporting of training and inference costs address your concerns. If these additions resolve the concerns behind your assessment, we would be grateful if you would reconsider your score.

Thank you to the authors for the thoughtful and detailed rebuttal. I appreciate the additional clarifications, which have addressed several of my concerns. Based on the responses, I have updated my score to 3 (Weak Accept).

Reviewer_Yh1FOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  1. The proposed reward-driven drifting objective is novel and theoretically well-motivated.

  2. The method achieves competitive performance while significantly reducing inference latency.

  3. The paper provides comparisons with recent baselines and useful analyses of exploration strategies.

Critical strengths

Incomplete related work. Several relevant generative RL methods are missing, including QVPO [1], DIME [2], GenPO [3], and FPO [4]. These methods explore different approaches to reward-driven generative policy optimization, entropy regularization, and likelihood-related challenges. Discussing their connections to Reward Drifting would better contextualize the contribution.

High interaction cost. The method requires extensive on-policy sampling and return estimation for multiple actions per state, which may limit its applicability to real-world robotics.

Insufficient ablations. Although DBPO is included as a baseline, the comparison does not fully isolate the proposed optimization objective from the antithetic latent exploration strategy.

Critical weaknesses
  1. Incomplete related work. Several relevant generative RL methods are missing, including QVPO [1], DIME [2], GenPO [3], and FPO [4]. These methods explore different approaches to reward-driven generative policy optimization, entropy regularization, and likelihood-related challenges. Discussing their connections to Reward Drifting would better contextualize the contribution.

  2. High interaction cost. The method requires extensive on-policy sampling and return estimation for multiple actions per state, which may limit its applicability to real-world robotics.

  3. Insufficient ablations. Although DBPO is included as a baseline, the comparison does not fully isolate the proposed optimization objective from the antithetic latent exploration strategy.

[1] Ding, S., et al. Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization. NeurIPS, 2024.

[2] Celik, O., et al. DIME: Diffusion-Based Maximum Entropy Reinforcement Learning. ICML, 2025.

[3] Ding, S., et al. GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning. NeurIPS, 2025.

[4] McAllister, D., et al. Flow Matching Policy Gradients. ICLR, 2026.

Questions for authors
  1. Could the authors further discuss the connections and differences between Reward Drifting and QVPO [1], DIME [2], GenPO [3], and FPO [4]?

  2. How does the overall wall-clock training cost compare with existing methods?

  3. Can the authors provide a comparison with DBPO under matched exploration settings?

  4. Could the authors extend the evaluation to standard MuJoCo locomotion benchmarks and additional IsaacLab tasks to further demonstrate the generality and robustness of the proposed method?

1 comment

We thank the reviewer for their feedback. We address the questions and concerns below.

Q.1. Could the authors further discuss the connections and differences between Reward Drifting and QVPO [1], DIME [2], GenPO [3], and FPO [4]?

Like Reward Drifting, all four methods train expressive generative policies (diffusion or flow models) with reinforcement learning. QVPO, DIME, GenPO and FPO were designed and evaluated primarily for learning from scratch. Reward Drifting targets the post-training setup where we refine a behavior-cloned generative policy without collapsing the behavior it already has. We added these 4 methods to our related work section alongside discussions:

  1. QVPO weights the diffusion model's denoising loss on the policy's own samples by their Q-values. Being diffusion-based, they incur latency at inference-time; however, both QVPO and reward drifting improve the policy from value-weighted samples of the current policy, without evaluating a likelihood. Reward Drifting, however, does not reweight a generative training loss. Instead, it displaces each sampled action along a field computed from the actions sampled at the same state, and trains the one-step generator by regression to map the same latent to the displaced action.
  2. DIME maximizes a maximum-entropy objective for a diffusion policy by deriving a lower bound on the policy's entropy and backpropagating the critic's gradient through the full denoising chain. Reward Drifting's update also contains a repulsive term that spreads the policy's samples apart, playing the role of the entropy bonus. This term is computed directly from the actions sampled at each state, so it requires no entropy bound. Since Reward Drifting uses only the critic's values for sampled actions, never their gradients, it also requires no backpropagation through a sampler.
  3. GenPO makes the flow exactly invertible, by augmenting the action space and using invertible coupling updates, so that PPO's likelihood ratio can be computed exactly. Reward Drifting needs no likelihood and places no invertibility constraint on the generator.
  4. FPO replaces PPO's likelihood ratio with a surrogate built from conditional flow-matching losses. Reward Drifting instead follows a JKO step in the space of action distributions, which in practice moves each sampled action a small step along the field. Limiting how far the actions move plays the role of PPO's trust region, and needs neither a likelihood nor a surrogate.

Action generation cost (Table 1). All four methods generate an action through a multi-step denoising or flow integration, and QVPO additionally samples several candidate actions and keeps the one its critic ranks highest. Reward Drifting's generator produces an action in a single forward pass.

Experiments. We include additional experiments with the 4 new methods on all the 16 tasks in total. These can be found in Figures 7, 8, 9, and 10 of the revised draft. Since all four methods were designed for learning from scratch, we adapt each to the post-training setting for the manipulation tasks. Locomotion policies continue to be trained from scratch. Each method's actor is first behavior-cloned on the same demonstrations with its respective objective. Each method then runs its own online algorithm with a freshly initialized critic warmed up for a few iterations. We generally find that on the BC-initialized manipulation setups, they tend to perform less favorably.

Q.2. How does the overall wall-clock training cost compare with existing methods?

We refer the reviewer to Figure 7 and Figure 9 in the revised draft. In summary, we find that the latency gains that we see at inference time also hold at training time with the reward drifting method training faster than baselines such as Reinflow on many tasks.

Q.3. Can the authors provide a comparison with DBPO under matched exploration settings?

We refer the reviewer to Figure 11 in the revised draft. In order to isolate the benefit of the drifting update, we make the neural network architecture (backbone) identical for both Reward Drifting and DBPO. Moreover, we use gaussian noise with reward drifting at the same magnitude in order to ensure that the exploration settings are identical. Across all 6 tasks (3 seeds per method) we observe that reward drifting with gaussian noise continues to converge faster and more reliably than DBPO.

Q.4. Could the authors extend the evaluation to standard MuJoCo locomotion benchmarks?

We refer the reviewer to Figures 9 and 10 for newer experiments on the MuJoCo locomotion benchmarks. We also include 8 new robomimic tasks in Figures 7 and 8.

We chose additional Robomimic tasks due to the ease of data generation for BC training using MimicGen or DexMimicGen. We observe that Reward Drifting continues to be comparable or superior to other methods on wallclock training time.

If these resolve your concerns, we’d be grateful if you’d consider updating your score.

Comments on the paper

1 comment

We thank all three reviewers for their thoughtful feedback and encouraging assessment of our work. We appreciate Reviewer Yh1F’s description of the objective as “novel and theoretically well-motivated,” and Reviewers QJMu and N6Db’s recognizing the “usefulness” and practicality of our method in “substantial latency reduction” in robot learning.

Their comments helped us identify where the explanation and experimental evidence needed strengthening. As a result, we made changes to the paper (colored in blue) and our response now includes the following:

  1. Broader baseline comparisons. We add seven baselines, bringing the total to eleven: the four methods suggested by Reviewer Yh1F (QVPO, DIME, GenPO, and FPO), together with three off-policy methods (FQL, DriftQL, and DFP). We also expand the related-work discussion to explain their connections to reward drifting and clarify the contribution relative to closely related concurrent work.

  2. A substantially larger evaluation. We evaluate our method, its variants, and the baselines on sixteen tasks: the three original tasks, eight additional Robomimic tasks, and five MuJoCo locomotion tasks (Figures 7, 8, 9, 10 of the new draft). This directly addresses the request for evidence beyond the original, near-saturated manipulation benchmarks.

  3. Clearer theory, algorithm, and more controlled ablations. We now present one simplified algorithm that consolidates the explicit and implicit variants of drifting into one single pseudo-code. We also reorganized the background and Appendix A to make the connection from Wasserstein gradient flow through the JKO formulation to the practical drifting update explicit, alongside its connections with similar single-step policy methods. Importantly, we also add ablations of the number of inner steps and the entropy weight.

  4. Improved wall-clock training time. Motivated by Reviewer Yh1F’s concerns about interaction cost and question about wall-clock training time that Reviewer N6Db’s brought up, we introduced a small change to the Reward Drifting algorithm for the rebuttal. Rather than estimating the Wasserstein gradient field at every decision point, we branch only at selected points, try alternative actions in parallel, and compare their rewards over short continuations, using a value function to estimate the remaining return. The revised appendix describes this procedure in detail. With this implementation, Reward Drifting takes approximately 0.2 hours versus ReinFlow’s 1.5 hours on PushT, 0.5 versus 1.8 hours on D’Claw, and 1.2 versus 1.7 hours on Transport, at comparable success rates. These measured pipeline times show that Reward Drifting can offer faster training as well as faster inference.

We are once again grateful for the reviewers’ suggestions, which have helped us make both the presentation and the evaluation stronger. We address each reviewer’s remaining questions individually below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.