acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement

Abstract

The introduction of generalist robot policies has enabled a single, large model to learn a variety of tasks from a vast pre-training corpus. While these models often display powerful zero-shot generalization capabilities, they nonetheless typically require an adaptation phase to reach sufficient performance in deployment. Iterated self-improvement via reinforcement learning (RL) has demonstrated promise in enabling this adaptation, yet such approaches have been largely explored in single-task settings – where the goal is to simply improve on a narrow deployment task – which lies in stark contrast to the multi-task capabilities of the base policy. In this work we ask whether generalist robot policies are capable of multi-task, generalist improvement in deployment – targeting many diverse tasks simultaneously – and what algorithmic recipe enables this. To answer this question, we first develop a set of structured evaluations to test generalist improvement capabilities, and systematically evaluate algorithmic and practical design choices regarding value estimation, policy extraction and resource allocation. We show that not only generalist policies can be improved online across diverse tasks, but that the right recipe can enable generalization and increase performance on tasks not considered during online training. We characterize when this generalization occurs, and show that the training task distribution as well as the use of value-based RL are both critical. To the best of our knowledge, this is the first evidence for online generalist improvement for robot foundation models.

Primary area:
reinforcement learning
Keywords:
vision-language-action models, reinforcement learning, post-training, multi-task learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1wosOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Self-improvement via reinforcement learning (RL) has been extensively studied in single-task settings and has demonstrated effectiveness in adaptation. This paper investigates the multi-task setting, asking whether online post-training can simultaneously improve performance across multiple tasks and generalize to novel tasks unseen during online training. To this end, the paper decouples policy training from policy evaluation. Experiments on MolmoSpaces and LIBERO demonstrate that critic-guided refinement using Best-of-N action sampling is crucial for enabling multi-task learning and generalization. In contrast, critic-free direct policy adaptation, instantiated through filtered supervised fine-tuning (FSFT), fails to generalize effectively.

Critical strengths

This paper conducts many experiments to evaluate direct policy adaptation and critic-guided refinement methods for generalization beyond training tasks, including unseen scenes and objects, as well as the influence of the critic architecture, loss target, and action filtering at test time.

Critical weaknesses
  1. This paper does not provide a unique motivation. The two questions this paper aims to answer, namely policy improvement across multiple tasks and generalization to new test tasks, are exactly the learning objectives of multi-task RL and meta-RL. Moreover, there is also a lack of in-depth analysis on the introduction of a value network and on why critic-guided refinement is key for generalization.

  2. It appears that this paper does not propose new methods and instead only evaluates combinations of existing methods. For example, weighted policy updates have been studied in prior work [1-2], and the critic-guided refinement seemingly inherits from the soft actor-critic algorithm [3].

[1] ARM: Advantage Reward Modeling for Long-Horizon Manipulation

[2] Latent-Variable Advantage-Weighted Policy Optimization for Offline RL (NeurIPS 2022)

[3] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (ICML 2018)

Questions for authors
  1. In Equation 5, why is the optimization objective of the state-value network to fit the Q-value network?

  2. In Figure 7, why does FSFT+BoN perform worse than BoN on the in-distribution Molmo?

5 comments

Thank you for the feedback. We understand that the main concerns surround the motivation of this work, and the fact that this submission is not centered on designing a novel method from scratch. We will address each individually, and answer the remaining questions.

W1: This paper does not provide a unique motivation.

The motivation for this work is to find an improvement routine that leads generalist policies to better performance on tasks, both seen and unseen. More generally, the purpose of this work is not to derive a novel algorithm from scratch, but rather to precisely study fundamental post-training mechanisms in a new setting: multi-task manipulation with generalist policies. To the best of our knowledge, this is the first work to do so.

Similar studies have been important in other domains, for instance finding that well-designed large-scale post-training is crucial for LLMs [1,2,3]. We explore whether this is also true in robotics, and pinpoint the conditions that enable large-scale post-training of robot policies, particularly in terms of generalization. We believe that demonstrating this form of generalization at this scale in the context of robotics, alongside a clear open-source recipe that enables these results, are valuable contributions that would be useful to practitioners and researchers alike.

To further clarify, in deep RL terms, we operate in a multi-task setting (in a contextual MDP as described in Section 3.1). This work does not involve an adaptation phase, and we do not claim to develop algorithms that “learn to adapt”; we focus on RL fine-tuning rather than meta-learning. Based on your suggestion, we extended the discussion of related works and specifically the connections and differences with multi-task and meta-learning in Section A of the Appendix in the new revision.

W1: There is also a lack of in-depth analysis [...] on why critic-guided refinement is key for generalization.

In response to your question, we have also introduced a finer-grained analysis of generalization trends in Section 4.3, particularly through Figures 6 and 7. We find that the critic appears to be the main driver of generalization to unseen objects: BoN, which keeps the policy frozen and only re-ranks sampled actions using the learned critic, improves on all six unseen objects as the number of training tasks grows, whereas FSFT improves clearly only on bowls. The gains also appear to depend on similarity to the training objects: training on similarly shaped objects transfers better (e.g., spatulas to knives, boxes to bowls).

In addition, we found that methods such as OGPO and AWR can be substantially strengthened by strongly stabilizing value estimation (see Section D.2). We thus extended our submission and evaluation to include OGPO as a representative of a third mechanism for policy improvement, namely, critic-guided policy extraction. We find that OGPO displays the same scaling properties as BoN, while reaching an even higher success rate. This suggests that the mechanism enabling generalization is likely to lie within the ability of the critic itself to generalize, rather than in how the critic’s signals are leveraged (test-time action refinement or policy updates). We highlight that the goal of this work is to demonstrate that transfer and scaling are even possible in the first place; an important finding is that value learning plays a key role for these results. We hope that future research could provide additional analysis to explain why value learning enables transfer.

W1/Q1: There is also a lack of in-depth analysis on the introduction of a value network [...] In Equation 5, why is the optimization objective of the state-value network to fit the Q-value network?

We purposefully choose a widely adopted recipe for training a value network, relying directly on temporal-difference learning, target networks and twin critics, as done in SAC [6] or TD3 [7]. We additionally fit a state-value network to the expected action-value: this is done to ensure that this network computes the value according to its definition: Vπ(s)=Ea∼πQπ(s,a)V^\pi(s) = \mathbb{E}_{a \sim \pi} Q^\pi(s,a), as minimizing the MSE regresses the output of the critic towards this expectation. This state-value function can then be used as a temporal-difference target for the Q-value network, as done in SAC (see Equations 5 and 8 in [6]). In principle, this is done to reduce the variance of the target. In practice, we found this to be effective, although not necessary.

We have added an ablation of this mechanism in Appendix C.8. On 4-task Molmo with BoN, we evaluate standard SARSA (Qπ(s,a)←r+γQπ(s’,a’)Q^\pi(s,a) \leftarrow r + \gamma Q^\pi(s’,a’)) and compare it with our use of a state-value function as the target. In this experiment, SARSA underperforms our default recipe, as shown by the following results:

MolmoSpacesQ-only SARSADefault (Q+V)
Training tasks0.550.76
Unseen scenes0.250.37
Unseen objects0.180.25
We have referred to this ablation in the main text (line 228), which will hopefully clarify the objective.

W2: It appears that this paper does not propose new methods and instead only evaluates combinations of existing methods.

We acknowledge that the paper does not propose new methods. Given that the setting we tackle - multi-task post-training of generalist policies - is largely unstudied, we believe that studying how fundamental, existing mechanisms behave is arguably the most valuable contribution. Moreover, while the mechanisms are not novel, their implementation requires particular care given the novelty of the setting, involving, for instance, the reliance on a supervised loss for stable policy updates (Equation 3), and important choices in resource allocation (Appendix C.3). We thus believe that open sourcing our recipes can provide additional value and inform future research.

The main goal of this work is thus not to propose a method, but rather to provide new knowledge, as encouraged by the ICLR reviewer guidelines.

Q2: In Figure 7, why does FSFT+BoN perform worse than BoN on the in-distribution Molmo?

The confidence intervals for the performance of FSFT+BoN and BoN in in-distribution MolmoSpaces tasks overlap entirely. The difference in performance is thus not significant, and may be explained by noise; nevertheless, we believe that the addition of a supervised loss can, in principle, lead to loss of diversity in action samples and thus affect performance.

While improving our OGPO recipe as discussed above, we found that the performance of FSFT+BoN can also be generally improved using the same stable critic recipe. With this updated recipe, FSFT+BoN is as effective as BoN (see Figure 9).


We appreciate your encouragement to clarify our motivation and contributions. We hope we were able to present both with clarity. Please let us know whether our rebuttal addressed your concerns sufficiently, and whether your evaluation of our submission has changed. We are happy to continue this conversation through the rebuttal phase; please do not hesitate to ask any questions or share your thoughts.

References

[1] Wei et al., Finetuned Language Models are Zero-Shot Learners, ICLR 2022

[2] Chung et al., Scaling Instruction-Finetuned Language Models, JMLR 2024

[2] Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training, COLM 2025

Dear reviewer,

We wanted to follow up before the discussion period ends on November 18. Given our clarifications, we would highly appreciate it if you would consider increasing your score. Please let us know if you have any remaining questions -- we would be happy to provide more details. Thank you again for taking the time to carefully review our paper and for the helpful suggestions!

Reviewer_H2JaOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how increasing the number and diversity of in-distribution training tasks affects the performance of general reinforcement learning policies on unseen task combinations. It primarily examines two mechanisms—direct policy adaptation and evaluation-guided refinement—across diverse combinations of objects and scenes. The results show that evaluation-guided refinement achieves the strongest performance on held-out tasks, while also delivering consistent improvements across a broad range of training tasks. These findings highlight the importance of appropriately designed refinement strategies for improving policy performance across diverse task settings.

Critical strengths

(1)This paper is the first to demonstrate that online post-training of generalist policies can generalize to novel robot-manipulation tasks that have not been encountered before. This shows that moving beyond the single-task paradigm is not only feasible, but also highly important for achieving broader improvement. (2)This paper systematically evaluates the key design choices required for generalist policy improvement, including algorithm design, data curriculum, and resource allocation. It also compares the effects of different value-learning methods, such as Monte Carlo methods and temporal-difference methods, as well as different network architectures and objective functions, in multi-task settings. In addition, it investigates how to make the most effective use of these value estimates to guide policy extraction and improvement.

Critical weaknesses

(1)The paper does not provide sufficiently detailed explanations of the specific procedures and contents of the training tasks and unseen test tasks. The definition of a task and the way task expansion enables generalization are also not explained in sufficient detail or clarity. (2)The figures based on the experimental data are not very intuitive. The visual representations of multiple experimental categories, such as seen tasks, unseen scenes, and unseen objects, together with their corresponding strategies and comparison methods, are overly simplistic. The descriptions are also not sufficiently concise or clear, making them somewhat confusing and difficult to understand.

Questions for authors

(1)When testing the FSFT+BoN combination, did the authors try more LIBERO tasks, such as 16 or 64 tasks? Did they also try more possible combinations? (2)Since the performance of direct policy adaptation, as well as its combination with critic-guided refinement, is inferior to that of critic-guided refinement alone, does this mean that direct policy adaptation has no value at all? Or could some useful applications be found for it and incorporated into policy updates? (3)In Section 4.2, when answering Q2, the paper states that as long as a VLA undergoes post-training on a sufficiently diverse set of tasks, it can achieve improvements beyond the training distribution. Is this improvement inevitable? Or does it only work for certain types of tasks? Furthermore, can it achieve significant improvements on more complex tasks?

4 comments

Thank you for the supportive and constructive review. We are happy to comment on each point in detail.

W1: The paper does not provide sufficiently detailed explanations of the specific procedures and contents of the training tasks and unseen test tasks

Our main evaluation (Figure 4) heavily builds on MolmoSpaces, and defines each task directly as the combination of the object that needs to be picked, and of the scene in which this action is commanded. Unseen tasks may involve seen objects in unseen scenes, or also objects not observed during training. Due to the large scale of the evaluation, task splits are defined in detail in Section D.1 of the Appendix, and accompanied by the procedure we followed for designing the splits and by renders of each task we consider. Please let us know if any part of this section would be needed in the main text; we would be happy to make this change.

Following your suggestion, we also expanded our analysis of why task expansion enables generalization in the new Section 4.3. We find that the benefit of adding tasks also depends on what the added tasks cover: gains on unseen objects appear mainly for methods that use a critic (BoN), and they seem to follow the similarity of the unseen objects to the training objects (e.g., training on spatulas transfers to knives and forks), whereas fine-tuning the policy without a critic (FSFT) does not reliably transfer.

We further refer to Section C.2 of the Appendix for an ablation in which we scale the number of scenes of a single object, or the number of objects with one scene each. We find that each axis transfers best along itself: scaling scenes mainly improves performance on unseen scenes, while scaling objects is what eventually transfers to unseen objects. It is therefore beneficial to scale along the axis of desired generalization.

If anything remains unclear, please let us know. We are happy to further elaborate in this rebuttal phase.

W2: The figures based on the experimental data are not very intuitive

On top of fine-grained results in Appendix E, we have added results over each unseen object in the main body of the paper (Figure 7). This allows us to clarify that the critic appears to be the main driver of generalization, that generalization seems to follow object similarity to the training set, and that policy updates can degrade performance on some objects (e.g., wine bottles). We hope that this increased level of detail, and balances the aggregated statistics in the main plot (Figure 4), which is meant to present overall trends instead. We are happy to implement further suggestions on how to make figures more intuitive and clearer.

Q1: When testing the FSFT+BoN combination, did the authors try more LIBERO tasks, such as 16 or 64 tasks? Did they also try more possible combinations?

Yes: while testing the FSFT+BoN combination, we evaluated it on more task sets, spanning from 1 to 64 tasks. The performance reported in Figure 7 was simply aggregated over the smaller and largest scale (1 and 64 tasks). Other combinations did not appear to convincingly outperform FSFT+BoN, and were thus largely tested in smaller-scale settings, on LIBERO (Figure 8).

We have since further explored design choices that enable a method to combine both policy adaptation and critic guidance. We found that expressive policy extraction via OGPO or AWR can be enabled through several implementation choices, including further ensembling for critic stabilization, as detailed in Section D.2 of the Appendix. We have thus updated our manuscript to include this third mechanism (critic-guided policy extraction), and extended our main evaluation to include OGPO results across all task splits in Figure 4. We can report that this method displays the same scaling properties as BoN, while achieving an even higher success rate, confirming the necessity of value estimation for generalization as the diversity in tasks increases. We believe this strengthens our results, and hope it provides the analysis your question suggests.

Q2: Does this mean that direct policy adaptation has no value at all?

We believe that direct policy adaptation remains valuable if in-distribution performance is the goal: it is effective at increasing the success rate on training tasks, and simple to implement.

Q3: Is this improvement inevitable? Or does it only work for certain types of tasks? Furthermore, can it achieve significant improvements on more complex tasks?

Thank you for this suggestion. We refer to Figure 7, which disaggregates results over objects, and thus shows performance on individual tasks, as we mentioned above. The fine-grained analysis (Section 4.3) shows that generalization to unseen objects is clearest when they resemble the training objects. For example, BoN transfers better from spatulas to knives than to bowls (Figure 6). We note that we do not claim that improvements can be induced over tasks of arbitrary complexity; in fact, we report the limits of generalization trends on unrelated tasks in Appendix C.1.


We hope that our responses and our updates to the submission may fully address your remaining comments. We remain thankful for your assessment of our work, and open to answer any further questions.

Dear Reviewer,

As the end of the rebuttal phase on November 18 approaches, we would like to ask whether our response has fully addressed your comments, and whether the new results might lead you to increase your score. We are happy to answer any questions that may remain. Thank you again for your insightful review and your support!

Thank you for the detailed and constructive response. I am pleased to see that the authors have made substantial additions addressing my concerns regarding task expansion, generalization mechanisms, and figure clarity (especially the addition of Figure 7 and the exploration of the new OGPO mechanism). The authors' candid discussion on task complexity and the limitations of generalization is also valuable. Based on the current revisions and clarifications, my initial concerns have been largely addressed. However, considering the overall contribution, the scope of the experiments, and the limitations regarding generalization on complex or unrelated tasks (as acknowledged by the authors in Q3), I believe a score of 3 (Weak Accept) remains the most objective reflection of this paper's specific situation. Therefore, I will maintain my current score (3), leaning towards acceptance.

Reviewer_kZQzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates whether online reinforcement learning can improve generalist robot policies across multiple tasks while preserving or improving performance on unseen tasks. The authors compare direct policy adaptation through filtered supervised fine-tuning (FSFT) with critic-guided action selection using Best-of-N (BoN), and study how their performance changes with increasing training task diversity. The main finding is that BoN achieves stronger generalization than FSFT on held-out scenes and objects in MolmoSpaces, while improvements are considerably more limited on LIBERO.

Critical strengths
  • The paper studies an important problem in generalist robot learning: whether online post-training can improve performance beyond the specific tasks used for adaptation.
  • I appreciate the systematic empirical investigation of different training regimes, critic architectures, and resource allocation strategies. The controlled scene/object variations in MolmoSpaces are also useful for studying different axes of generalization. The observed improvements on unseen objects with critic-guided action selection are interesting.
Critical weaknesses
  1. The effect of task diversity is confounded with the amount of online training data.
    • A central claim is that generalist improvement scales with training task diversity. However, the number of collected episodes also increases linearly with the number of tasks, from 200 episodes for one task to 12,800 for 64 tasks. The optimization budget also differs between single-task and multi-task experiments. Consequently, it is difficult to determine how much of the improvement comes from increased task diversity rather than simply having substantially more interaction data.
    • I would like to see experiments with a fixed total interaction budget across different numbers of training tasks.
  2. The main unseen-scene scaling experiment does not maintain a consistent generalization setting.
    • In Figure 4, unseen-scene performance is evaluated on a fixed set of eight scenes corresponding to the eight object categories in the largest training split. However, smaller training splits only cover a subset of these objects. Therefore, as the number of training tasks increases, the evaluation changes from a mixture of unseen-object and unseen-scene generalization to purely unseen-scene generalization. While the additional controlled experiments in Figure 5 partially address this issue, the main result would be more convincing with evaluation sets that maintain comparable train-test relationships across training scales.
  3. The comparison does not sufficiently isolate the benefit of value-based RL.
    • The main performance gains come from BoN, which keeps the pretrained VLA frozen and selects among 32 candidate action chunks using a learned critic. In contrast, FSFT directly modifies the pretrained policy, making it more susceptible to forgetting. Moreover, the critic receives proprioceptive information that is not provided to the VLA policy, and the methods collect data using different action-selection mechanisms.
  4. The evidence for generalist improvement remains limited to closely related tasks.
    • The strongest generalization results are obtained on MolmoSpaces, where tasks largely involve picking up different objects in different scenes. While these results demonstrate useful transfer, the underlying manipulation objective remains essentially unchanged. In contrast, the LIBERO experiments show no improvement over the base policy on held-out tasks, even as the number of training tasks increases.
Questions for authors
  1. Could the authors provide fixed-interaction-budget task diversity experiments and clarify how much of the Figure 4 improvement remains under a consistent held-out evaluation protocol?
  2. Additionally, how does a supervised action-quality predictor perform under the same BoN inference setup, and how much of the OOD improvement persists with fewer action candidates?
4 comments

Thank you for the insightful comments and the detailed review, which prompted us to adjust our evaluation settings and run additional ablations. We are happy to answer each question directly.

W1/Q1: The effect of task diversity is confounded with the amount of online training data.

We understand the suggestion, and have run an additional experiment to validate our claim that the improved performance is due to task scaling rather than simply collecting more data. To disambiguate whether generalization stems from task diversity or interaction budget, we have repeated single-task and 4-task BoN runs in MolmoSpaces while adopting the same interaction and compute budgets as the 16-task run. We report detailed results in Appendix C.7, and confirm that generalization is largely driven by the increase in task diversity: simply collecting more data increases performance on unseen tasks only modestly, and scaling trends remain visible at matched budget. This new evidence aligns with the existing ablation in Figure 5, which shows that performance depends on the diversity in objects and scenes even with the same interaction budget. Thank you for suggesting this ablation, which we believe clarifies the necessity of task diversity.

W2: The main unseen-scene scaling experiment does not maintain a consistent generalization setting.

Thank you for spotting this inconsistency! We resolved it by updating our evaluation protocol throughout the submission. For unseen scenes evaluations, each training split is now evaluated on unseen scenes of the objects it was trained on, with 8 held-out scenes per training object (new split lists in Table 3). For unseen objects evaluations, every point on the scaling curve (Figure 4, right) is evaluated on the exact same set of tasks (6 unseen objects, 8 scenes each; see Table 3). To evaluate generalization at an even larger scale, we have also extended the training-task splits to 144. The resulting trends remain unchanged: performance on unseen scenes and unseen objects grows with the number of training tasks (see Figure 4).

W3: The comparison does not sufficiently isolate the benefit of value-based RL.

We appreciate this suggestion, and address each of the three concerns precisely.

Access to proprioception In this work, policy networks ignore proprioception in order to match the standard π0.5\pi_{0.5} architecture and design choices. We have ablated whether critics benefit from their access to proprioception by removing it from their inputs. We report performance for BoN on single MolmoSpaces tasks.

MolmoSpaces (1 training task)BoNBoN, critic w/o proprioceptionBase policy
Seen tasks0.727 [0.659, 0.794]0.724 [0.667, 0.773]0.302
Unseen scenes0.388 [0.364, 0.413]0.394 [0.361, 0.428]0.309
Unseen objects0.197 [0.186, 0.207]0.194 [0.185, 0.204]0.289

We find that, in practice, the availability of proprioception does not improve performance on either seen or unseen tasks: the backbone is capable of extracting sufficient information from multiple camera views.

Action-selection The difference in action selection between BoN and FSFT is instead harder to ablate, as action selection is enabled by access to a critic, and is precisely the simplest mechanism through which value estimates enter data collection. Running BoN without action selection would reduce to evaluating the base policy. We believe that an informative experiment is not straightforward to design, but we are happy to evaluate any ablation you may suggest.

Updates to the pre-trained policy To further isolate the benefits of the critic as suggested, we have added a refined instantiation of OGPO (details in Appendix B) to our main scaling curves. As shown across Figures 1, 4 and 7, this recipe demonstrates the same scaling properties of BoN, while reaching an even higher success rate. As they are both critic-based methods, these results further corroborate our previous results. We also note that these new results provide the comparison you suggest, and demonstrate that updates to the pre-trained policy do not necessarily result in forgetting. While the goal of this work is to demonstrate that value-based learning is a critical ingredient in transfer and scaling, we believe these results will be important for future research to understand why exactly value learning plays its current role. We would like to thank you again for noting this point, and we hope our new results help address this concern.

W4: The evidence for generalist improvement remains limited to closely related tasks.

To better understand what drives generalization and its limits, we have added a fine-grained analysis to Section 4.3, which suggests that generalization comes mainly from critic learning and appears to depend on similarity between the unseen objects and the training objects (e.g., spatulas to forks and knives), while policy updates alone do not transfer and can even degrade performance (e.g., on wine bottles), as depicted in Figure 7 in the revised manuscript.

This analysis is consistent with the lack of generalization in LIBERO, which we directly acknowledge: arbitrary generalization is generally not to be expected unless the benchmark involves sufficient object similarity and a shared structure. We believe that this contributes to providing a more informative and nuanced study.

Q2: how does a supervised action-quality predictor perform under the same BoN inference setup?

We would kindly ask the reviewer to clarify what would constitute a “supervised action-quality predictor”. We are happy to run additional ablations, but unsure of what this suggestion means.

Q2: how much of the OOD improvement persists with fewer action candidates?

To measure this, we have evaluated the BoN checkpoint that achieves maximal generalization (trained on 144 tasks) and re-evaluated its performance on unseen scenes and objects as NN decreases.

MolmoSpacesBoN, N=32BoN, N=16BoN, N=8BoN, N=4BoN, N=2Base Policy (N=1)
Training tasks0.5990.5910.5570.5140.4320.290
Unseen scenes0.6040.6110.5850.5030.4470.297
Unseen objects0.5230.4390.4540.4190.3790.289

We find that OOD improvements gradually decrease, but persist for all N≥2N \geq 2; this suggests that a critic may induce generalization even with limited test-time compute.


We are thankful for your feedback, which was helpful in isolating and confirming the benefits of value-based RL. We would like to ask whether our responses addressed your concerns in full, and we are happy to continue this discussion through the rebuttal.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the matched-budget experiments and the revised evaluation protocol, which largely address my first two concerns. The new OGPO results and ablations also strengthen the empirical evidence. I still have some reservations about the precise contribution of value learning and the relatively narrow scope of demonstrated generalization, but I acknowledge that the rebuttal has substantially improved the paper. I will raise my score accordingly.

Dear Reviewer,

Thank you for suggesting those experiments, considering our rebuttal, and raising your score! We remain available in case you may have any additional question.

Comments on the paper

1 comment

We would like to thank all reviewers for the constructive feedback, and summarize how our submission was revised in response.

  • We included a third mechanism for policy improvement, namely critic-guided policy extraction, represented by a more stable OGPO recipe. We updated Figures 1, 4, 9 and Appendices accordingly, finding that this mechanism also displays scaling in generalization abilities as the task diversity increases, thus supporting the importance of critic-based learning. (kZQz, H2Ja, 1wos)

  • We introduced a new experimental section answering Q3: “What is the nature of the observed generalization, and in which cases is it most effective?”. Figures 6 and 7 now report a fine-grained analysis of generalization trends. (kZQZ, H2Ja, 1wos)

  • We expanded our main evaluation in Figure 4 by training all methods on the largest task set yet, spanning 144 tasks, confirming that the scaling trends reported up to 64 tasks persist. (kZQZ)

  • We updated and expanded the set of evaluation tasks: unseen scenes evaluation now covers only objects seen in training (8 held-out scenes per training object), and unseen objects evaluation uses the same 6 unseen objects (8 scenes per unseen object) for every training split (Details in Table 3) (kZQZ)

  • We added two additional ablations as Appendices C.7 and C.8, studying performances at matched data collection budget, and in absence of a state-value network, respectively. (kZQZ, 1wos)

We believe that these updates strengthen this work, and thank reviewers for prompting them. We marked them in green for ease of parsing. We look forward to further discussions in the rebuttal phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.