- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Reward Drifting, an on-policy method for fine-tuning one-step generative policies without evaluating action likelihoods. The method estimates a reward-driven attraction and entropy-related repulsion field from sampled actions and their returns, then trains the generator toward the displaced samples. Experiments on three simulated manipulation tasks show high success rates comparable to the evaluated diffusion and flow baselines, together with much lower policy inference latency.
Critical strengths
- The method addresses a practical problem: combining fast action generation with reinforcement learning updates that do not require tractable action likelihoods. (I do like the solid mathematical part but I am not familiar with it and have notified the AC upon this)
- The experiments demonstrate substantial reductions in inference latency across three manipulation tasks, in both eager and compiled execution modes. These results highlight the method's potential for applications where fast policy inference is important.
Critical weaknesses
-
Reward Drifting substantially reduces inference latency, but ReinFlow appears to learn faster and better on Push-T in Figure 4, while their final success rates are similar. The three tasks provide encouraging evidence, but the near-saturated final success rates make it difficult to judge whether a performance gap would emerge on a broader or more challenging benchmark suite. Additional evaluations would strengthen the claim that the method retains performance comparable to ReinFlow while improving inference efficiency.
-
As I understand it, the main results rely on BC initialization (please correct me if I am wrong). As acknowledged in Section 7, learning from scratch does not recover all behavior modes in the reported experiment. This limits the evidence for learning diverse behaviors without demonstrations, although the results support the intended fine-tuning setting.
Questions for authors
See Weaknesses.
We thank the reviewer for their questions and address them below.
Q.1. Reward Drifting substantially reduces inference latency, but ReinFlow appears to learn faster and better on Push-T in Figure 4, while their final success rates are similar. The three tasks provide encouraging evidence, but the near-saturated final success rates make it difficult to judge whether a performance gap would emerge on a broader or more challenging benchmark suite. Additional evaluations would strengthen the claim that the method retains performance comparable to ReinFlow while improving inference efficiency.
In addition to the 3 tasks in the original submission, we include 8 new robomimic tasks as well as 5 new MuJoCo locomotion benchmarks giving us a total of 16 tasks across different simulators and robot morphologies to further strengthen our claim. Across all new tasks we observe that Reward Drifting has superior or comparable performance to Reinflow while being faster at run time. Upon making changes to the way credit assignment (details in Section 5 of the revised draft) is done in Reward Drifting we found that we are able to reliably beat Reinflow on wall-clock time on most tasks.
Q.2. As I understand it, the main results rely on BC initialization (please correct me if I am wrong). As acknowledged in Section 7, learning from scratch does not recover all behavior modes in the reported experiment. This limits the evidence for learning diverse behaviors without demonstrations, although the results support the intended fine-tuning setting.
As correctly pointed out by the reviewer, reward drifting is intended for the post-training setting. As shown in Figure 3, if the original behavior cloning policy is multimodal, then reward drifting retains both modes during training and there is no collapse.
We included the learning from scratch experiments in our original submission for the sake of completeness. Learning generative policies from scratch that reliably learn all possible modes remains an open problem and we do not claim that as a contribution of our work. If these additions resolve the concerns behind your assessment, we would be grateful if you would reconsider your score.