- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Self-improvement via reinforcement learning (RL) has been extensively studied in single-task settings and has demonstrated effectiveness in adaptation. This paper investigates the multi-task setting, asking whether online post-training can simultaneously improve performance across multiple tasks and generalize to novel tasks unseen during online training. To this end, the paper decouples policy training from policy evaluation. Experiments on MolmoSpaces and LIBERO demonstrate that critic-guided refinement using Best-of-N action sampling is crucial for enabling multi-task learning and generalization. In contrast, critic-free direct policy adaptation, instantiated through filtered supervised fine-tuning (FSFT), fails to generalize effectively.
Critical strengths
This paper conducts many experiments to evaluate direct policy adaptation and critic-guided refinement methods for generalization beyond training tasks, including unseen scenes and objects, as well as the influence of the critic architecture, loss target, and action filtering at test time.
Critical weaknesses
-
This paper does not provide a unique motivation. The two questions this paper aims to answer, namely policy improvement across multiple tasks and generalization to new test tasks, are exactly the learning objectives of multi-task RL and meta-RL. Moreover, there is also a lack of in-depth analysis on the introduction of a value network and on why critic-guided refinement is key for generalization.
-
It appears that this paper does not propose new methods and instead only evaluates combinations of existing methods. For example, weighted policy updates have been studied in prior work [1-2], and the critic-guided refinement seemingly inherits from the soft actor-critic algorithm [3].
[1] ARM: Advantage Reward Modeling for Long-Horizon Manipulation
[2] Latent-Variable Advantage-Weighted Policy Optimization for Offline RL (NeurIPS 2022)
[3] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (ICML 2018)
Questions for authors
-
In Equation 5, why is the optimization objective of the state-value network to fit the Q-value network?
-
In Figure 7, why does FSFT+BoN perform worse than BoN on the in-distribution Molmo?
5 comments
W1/Q1: There is also a lack of in-depth analysis on the introduction of a value network [...] In Equation 5, why is the optimization objective of the state-value network to fit the Q-value network?
We purposefully choose a widely adopted recipe for training a value network, relying directly on temporal-difference learning, target networks and twin critics, as done in SAC [6] or TD3 [7]. We additionally fit a state-value network to the expected action-value: this is done to ensure that this network computes the value according to its definition: , as minimizing the MSE regresses the output of the critic towards this expectation. This state-value function can then be used as a temporal-difference target for the Q-value network, as done in SAC (see Equations 5 and 8 in [6]). In principle, this is done to reduce the variance of the target. In practice, we found this to be effective, although not necessary.
We have added an ablation of this mechanism in Appendix C.8. On 4-task Molmo with BoN, we evaluate standard SARSA () and compare it with our use of a state-value function as the target. In this experiment, SARSA underperforms our default recipe, as shown by the following results:
| MolmoSpaces | Q-only SARSA | Default (Q+V) |
|---|---|---|
| Training tasks | 0.55 | 0.76 |
| Unseen scenes | 0.25 | 0.37 |
| Unseen objects | 0.18 | 0.25 |
| We have referred to this ablation in the main text (line 228), which will hopefully clarify the objective. |
W2: It appears that this paper does not propose new methods and instead only evaluates combinations of existing methods.
We acknowledge that the paper does not propose new methods. Given that the setting we tackle - multi-task post-training of generalist policies - is largely unstudied, we believe that studying how fundamental, existing mechanisms behave is arguably the most valuable contribution. Moreover, while the mechanisms are not novel, their implementation requires particular care given the novelty of the setting, involving, for instance, the reliance on a supervised loss for stable policy updates (Equation 3), and important choices in resource allocation (Appendix C.3). We thus believe that open sourcing our recipes can provide additional value and inform future research.
The main goal of this work is thus not to propose a method, but rather to provide new knowledge, as encouraged by the ICLR reviewer guidelines.
Q2: In Figure 7, why does FSFT+BoN perform worse than BoN on the in-distribution Molmo?
The confidence intervals for the performance of FSFT+BoN and BoN in in-distribution MolmoSpaces tasks overlap entirely. The difference in performance is thus not significant, and may be explained by noise; nevertheless, we believe that the addition of a supervised loss can, in principle, lead to loss of diversity in action samples and thus affect performance.
While improving our OGPO recipe as discussed above, we found that the performance of FSFT+BoN can also be generally improved using the same stable critic recipe. With this updated recipe, FSFT+BoN is as effective as BoN (see Figure 9).
We appreciate your encouragement to clarify our motivation and contributions. We hope we were able to present both with clarity. Please let us know whether our rebuttal addressed your concerns sufficiently, and whether your evaluation of our submission has changed. We are happy to continue this conversation through the rebuttal phase; please do not hesitate to ask any questions or share your thoughts.
References
[1] Wei et al., Finetuned Language Models are Zero-Shot Learners, ICLR 2022
[2] Chung et al., Scaling Instruction-Finetuned Language Models, JMLR 2024
[2] Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training, COLM 2025
Dear reviewer,
We wanted to follow up before the discussion period ends on November 18. Given our clarifications, we would highly appreciate it if you would consider increasing your score. Please let us know if you have any remaining questions -- we would be happy to provide more details. Thank you again for taking the time to carefully review our paper and for the helpful suggestions!
Thank you for the feedback. We understand that the main concerns surround the motivation of this work, and the fact that this submission is not centered on designing a novel method from scratch. We will address each individually, and answer the remaining questions.
The motivation for this work is to find an improvement routine that leads generalist policies to better performance on tasks, both seen and unseen. More generally, the purpose of this work is not to derive a novel algorithm from scratch, but rather to precisely study fundamental post-training mechanisms in a new setting: multi-task manipulation with generalist policies. To the best of our knowledge, this is the first work to do so.
Similar studies have been important in other domains, for instance finding that well-designed large-scale post-training is crucial for LLMs [1,2,3]. We explore whether this is also true in robotics, and pinpoint the conditions that enable large-scale post-training of robot policies, particularly in terms of generalization. We believe that demonstrating this form of generalization at this scale in the context of robotics, alongside a clear open-source recipe that enables these results, are valuable contributions that would be useful to practitioners and researchers alike.
To further clarify, in deep RL terms, we operate in a multi-task setting (in a contextual MDP as described in Section 3.1). This work does not involve an adaptation phase, and we do not claim to develop algorithms that “learn to adapt”; we focus on RL fine-tuning rather than meta-learning. Based on your suggestion, we extended the discussion of related works and specifically the connections and differences with multi-task and meta-learning in Section A of the Appendix in the new revision.
In response to your question, we have also introduced a finer-grained analysis of generalization trends in Section 4.3, particularly through Figures 6 and 7. We find that the critic appears to be the main driver of generalization to unseen objects: BoN, which keeps the policy frozen and only re-ranks sampled actions using the learned critic, improves on all six unseen objects as the number of training tasks grows, whereas FSFT improves clearly only on bowls. The gains also appear to depend on similarity to the training objects: training on similarly shaped objects transfers better (e.g., spatulas to knives, boxes to bowls).
In addition, we found that methods such as OGPO and AWR can be substantially strengthened by strongly stabilizing value estimation (see Section D.2). We thus extended our submission and evaluation to include OGPO as a representative of a third mechanism for policy improvement, namely, critic-guided policy extraction. We find that OGPO displays the same scaling properties as BoN, while reaching an even higher success rate. This suggests that the mechanism enabling generalization is likely to lie within the ability of the critic itself to generalize, rather than in how the critic’s signals are leveraged (test-time action refinement or policy updates). We highlight that the goal of this work is to demonstrate that transfer and scaling are even possible in the first place; an important finding is that value learning plays a key role for these results. We hope that future research could provide additional analysis to explain why value learning enables transfer.