acceptodds
Under review as a conference paper at ICLR 2027

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies

Abstract

Reinforcement learning (RL) enables generalist robot policies to improve through trial-and-error interaction, yet its effectiveness is fundamentally constrained by sparse task rewards. Existing general-purpose reward models typically alleviate this issue by learning task progress from expert demonstrations, but introduce a distribution mismatch with the mixed-quality rollouts encountered during policy optimization, making their estimates unreliable on suboptimal and failed behaviors from which the policy must learn. In this work, we introduce a demonstration-free reward learning paradigm where dense reward feedback can be learned directly from sparse task outcomes and policy experience. We theoretically show that terminal task outcomes implicitly define dense success-probability feedback at intermediate timesteps, which can be recursively learned through bootstrapping. Based on this insight, we introduce eVTA, which learns success probabilities from mixed-quality policy rollouts through temporal-difference-style bootstrapping, without expert demonstrations or intermediate annotations. We further introduce RL with Evolving Rewards (RLER), a closed-loop framework that adapts eVTA using newly collected rollouts as the policy evolves. Experiments show that eVTA provides more informative rewards than state-of-the-art reward models and achieves the best average policy performance across all LIBERO task suites under the same RL training budget, improving success rates by 5.4%-13.8% over the initial policy. In real-world manipulation, RLER further improves overall success rates by 20%-26%, with 35%-36% gains under out-of-distribution conditions. These results demonstrate the effectiveness of demonstration-free reward learning and adapting rewards as the policy evolves.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Reinforcement Learning, Sparse Rewards, Reward Learning, Generalist Robot Policies, Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x2MwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a dense reward learning paradigm without requiring expert demonstrations. The proposed method is based on the insight that the expected task success probability as a reward signal satisfies the martingale property and can be learned via one-step bootstrapping.

Critical strengths
  1. I find the idea quite neat and clever, and the resulting approach is principled, simple yet practically effective. That said, I am not too familiar with the literature on reward learning in RL, so I leave the evaluation of the novelty compared to the existing literature to other reviewers.
  2. The evaluation has a broad coverage on tasks, baselines, metrics, with one caveat I state in the weakness section.
  3. The presentation is well-structured and clear overall.
Critical weaknesses
  1. What theorem 2 supports is a “sequential optimization” scheme that optimize from the terminal step backward. The introduced TD learning is more of a convenient approximation. While commonly used in settings such as off-policy RL, it is not justified sufficiently here. More experimental or analytical results could address this point.
  2. Eq. (11) - the justification of this transformation is only valid in certain specific settings. For example, for fixed-horizon tasks, adding or subtracting a constant doesn’t affect the optimization since the constant is policy-independent. This point needs clarification.
  3. My understanding is that all the training runs, including the TD reward function learning and RL with the learned reward, only uses one trial (random seed). And all the claims and interpretation are based on a single trial. As a consequence, no randomness across seeds (and therefore the stability of the proposed approach and the baselines) is evaluated anywhere. The multiple evaluation trials in each setting only characterizes the performance of a single training trial. Since it is known that RL tends to have a high variance, I am not convinced by the existing evidence that the claims still hold true with more trials for each method. I prefer a narrower evaluation coverage but with more rigorous statistical evidence over broader coverage with a single trial for each setting.
Questions for authors

The last weakness doesn't appear to be easy to address within the rebuttal period. But if I misunderstood the number of training runs used for evaluation, I am happy to raise my score and give a more positive recommendation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer x2Mw - Part 1 (W1&W2)

Thank you for the thoughtful and constructive feedback. We are encouraged that you find our core idea neat and clever, and the proposed approach principled, simple, yet practically effective. We also appreciate your recognition of our broad experimental coverage across tasks, baselines, and metrics, as well as the clear and well-structured presentation. We have revised the manuscript to address your concerns, with major changes highlighted in blue. Below, we respond to each comment in detail. We hope our revisions and responses resolve your concerns and that you will consider raising your score.

W1. Justification of the update order of TD bootstrapping.

Thank you for your thoughtful comment. We would like to clarify that although Theorem 2 uses backward induction in its proof, explicit backward sequential optimization is not required. In the newly added Appendix E.3, we prove that, under exact conditional-expectation updates and terminal anchoring, the success-probability solution is unique and can be recovered regardless of the update order. To further validate this finding in practice, we compare randomly shuffled minibatch training (Random, our default implementation) with reverse-temporal-order training (Reverse, from terminal to initial timesteps) on LIBERO-Spatial under identical training configurations. The results are summarized below.

Training OrderVOC ↑VROC ↑MSE ↓Kendall's τa\tau_a ↑
Random0.870.820.100.99
Reverse0.800.840.090.98

Random achieves reward quality comparable to Reverse, providing empirical evidence that explicit backward ordering is not necessary for effective TD-based reward learning. The complete theoretical analysis and experimental results are provided in Appendix E.3 of the revised manuscript.

We also provide further justification for our TD design. Appendix E.4 shows that ideal TD bootstrapping and Monte Carlo (MC) regression share the same population target, while TD provides lower-variance supervision. Experiments across all four LIBERO suites further demonstrate that TD provides more informative intermediate feedback and better captures task evolution. Additionally, Appendix D.1 examines the effects of target-network update intervals and context window sizes, providing empirical support for our training configurations. Together, these analyses provide complementary theoretical and empirical evidence for our TD-based reward learning approach.

W2. Justification of the reward transformation in Eq. (11).

Thank you for this comment. All our RL experiments, including both simulation and real-world settings, adopt variable-length episodes, a common practice in embodied RL (e.g., RLinf, Yu et al., 2026; πRL\pi_{\mathrm{RL}}, Chen et al., 2025). Specifically, episodes terminate upon task success, with a maximum step limit to prevent indefinite execution. Consequently, the cumulative constant shift introduced by Eq. (11) depends on the actual episode length and is therefore policy-dependent. This transformation prevents policies from accumulating additional positive rewards simply by prolonging task execution.

AuthorsOpenReview ↗

Response to Reviewer x2Mw - Part 2 (W3)

W3. Statistical reliability across training runs.

Thank you for raising this concern. To directly assess training stability, we conduct two additional independent training runs (seeds 43 and 44) for both reward learning and RL post-training, yielding three seeds in total.

For reward learning, we select LIBERO-Spatial as a representative suite to evaluate the training stability of eVTA$_0$ across different random seeds. The results show consistently high reward quality with low variance across all four metrics, as summarized below.

MetricSeed 42Seed 43Seed 44Mean ± Std
VOC ↑0.870.840.850.853 ± 0.015
VROC ↑0.820.830.820.823 ± 0.006
MSE ↓0.100.080.080.087 ± 0.012
Kendall's τa\tau_a ↑0.990.990.970.983 ± 0.012

For RL post-training, we select LIBERO-Long, the most challenging suite featuring long-horizon manipulation tasks, to assess the stability of policy improvements enabled by eVTA$_0$. As shown below, eVTA$_0$ consistently achieves the highest success rate across all three independent training runs, with an average of 58.7% ± 4.5%, outperforming all baselines.

MethodSeed 42Seed 43Seed 44Mean ± Std
Binary41.841.243.842.3 ± 1.4
Handcrafted45.450.842.646.3 ± 4.2
Robometer29.427.428.828.5 ± 1.0
eVTA₀54.063.059.058.7 ± 4.5

These additional experiments demonstrate the stability of eVTA$_0$ reward learning across random seeds and further confirm that its advantages in downstream RL performance are consistently maintained across independent training runs. These new results and analyses have been added to Appendix D.1 of the revised manuscript.

Finally, we would also like to clarify the motivation behind our original evaluation protocol. Our evaluation follows a common practice in large-scale robotic reward learning (e.g., Robo-Dopamine, Tan et al., 2026; Robometer, Liang et al., 2026) and VLA RL (e.g., πRL\pi_{\mathrm{RL}}, Chen et al., 2025; LaST-R1, Chen et al., 2026), where extensive benchmark evaluations are often reported without multi-seed training statistics for every setting, given the substantial cost of training large vision-language models and conducting extensive environment interactions. Importantly, our experiments involve multi-task training: eVTA$_0$ is jointly trained on rollouts from all 40 LIBERO tasks, while each RL policy is jointly optimized across 10 tasks within a suite. This provides broad evidence across diverse tasks. Our additional multi-seed experiments complement this evaluation by directly assessing training stability.

Reviewer_x2MwOpenReview ↗

Response to Authors’ Rebuttal

I thank the authors for their responses and additional experiments. The responses to W1 and W2 address my concerns.

Regarding W3, however, if I understand correctly, 3-seed results (still a small number of seeds) are now available only for RL on LIBERO-Long and reward learning on LIBERO-Spatial, while all other results, including all real-world experiments, remain single run. As a minor point, a standard deviation estimated from only three runs is itself highly uncertain and thus not very informative on its own. I encourage the authors to report proper measures of statistical significance, such as confidence intervals or p-values.

I maintain my original evaluation as my final recommendation.

Thanks for your follow-up comments. We have directly addressed your original concern by conducting additional independent training runs for both reward learning and RL post-training, and we now further report confidence intervals and significance tests. On LIBERO-Long, eVTA$_0$ improves the average success rate over the strongest baseline, Handcrafted, by 12.4%, with an unadjusted 95% confidence interval of [2.5%, 22.3%]. Two-sided Welch tests yield Holm-adjusted p-values of 0.035, 0.035, and 0.016 against Binary, Handcrafted, and Robometer, respectively, providing further statistical evidence for the performance advantages of eVTA$_0$. Crucially, eVTA$_0$ ranks first under every tested seed, and even its lowest success rate (54.0%) exceeds the highest result of any baseline (50.8%). Such a substantial and consistently observed performance gap provides evidence against attributing the reported advantage merely to a favorable choice of training seed.

For the real-world experiments, we evaluate the learned policies through repeated trials across ID and OOD conditions, including variations in object placement and robotic execution. Such repeated physical evaluations are also widely adopted in leading VLA studies, including Physical Intelligence's π0.5\pi_{0.5} and π0.6∗\pi_{0.6}^{*}. Importantly, multi-seed training in real-world settings is less straightforward to interpret than in simulation, as physical environments cannot be precisely reset to identical conditions, and performance variations may arise from both training randomness and unavoidable environmental differences. In contrast, repeated evaluations across diverse physical conditions directly assess the practical reliability and generalization of the learned policies. We therefore believe that the absence of multi-seed real-world training should not, by itself, undermine the validity of our real-world findings, particularly when repeated physical evaluations are standard practice in leading embodied VLA research (π0.5\pi_{0.5}, π0.6⋆\pi_{0.6}^{\star}, etc.).

Your latest response confirms that our replies to W1 and W2 have resolved your concerns. We have also substantially strengthened the evidence for W3 through additional independent training runs and the statistical analyses you requested. In light of these efforts and new evidence, we sincerely hope you will reconsider raising your score. We believe that the remaining concerns regarding random seeds will not undermine the central conclusions and contributions of our paper, particularly given the consistent improvements demonstrated by our additional experiments. We hope you can reconsider the severity of this remaining issue and whether it truly justifies rejecting a paper whose central findings are supported by substantial theoretical and empirical evidence.

Reviewer_QfyaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

To address the challenge of sparse task rewards in reinforcement learning, this paper proposes eVTA₀, a demonstration-free reward learning approach that learns success-probability rewards from mixed-quality policy rollouts. Furthermore, the authors present RL with Evolving Rewards (RLER), a closed-loop reinforcement learning framework that dynamically updates the reward model using newly collected trajectories during policy optimization. Experiments show that the method delivers competitive reward estimation and distinguishes successful and failed trajectories more effectively. Real-robot experiments further verify that RLER improves policy performance.

Critical strengths

Demonstration-free reward learning carries substantial practical value. Sparse reward constitutes a major bottleneck for reinforcement learning of general robotic policies. By adopting TD-style bootstrapping to learn success probabilities, this work learns informative dense rewards from sparse binary outcome feedback without manual intermediate progress annotations. The proposed closed-loop framework enables simultaneous optimization of policy learning and reward learning. It is practically meaningful for improving the scalability of general robotic policy learning. Both simulated and real-world experiments validate the efficacy of the approach.

Critical weaknesses

The transformation from success probability to RL reward lacks theoretical justification. In Equation 11, the authors convert the predicted success probability into stepwise rewards. Although this transformation preserves the relative ordering of single-step reward predictions, it does not necessarily maintain the ordering of cumulative returns across trajectories of different lengths. The accumulated penalty from failure probability may introduce an implicit bias toward shorter trajectories, which can misalign the policy optimization objective with the original goal of maximizing task success rate.

Questions for authors
  1. Could the authors provide further justification for the rationality of this reward transformation in Eq. (11)?
  2. The paper emphasizes that eVTA₀ performs reward learning using only sparse binary feedback, yet the appendix mentions additional augmentation. An ablation study on this component would be appreciated.
  3. eVTA₀ is capable of estimating success probabilities for intermediate timesteps. Could the authors report an evaluation on the accuracy of these success probability estimates?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Qfya - Part 1 (W1&Q1)

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of the practical value of demonstration-free reward learning, the effectiveness of learning dense rewards from sparse binary outcomes, and the potential of our closed-loop RLER framework. We are also encouraged by your positive assessment of our simulation and real-world experiments. We have revised the manuscript to address your concerns, with major changes highlighted in blue. Below, we respond to each comment in detail. We hope our revisions resolve your concerns and that you will consider maintaining or raising your score.

W1/Q1. Justification of reward transformation.

Thank you for your comment. We agree that subtracting a constant from stepwise rewards does not necessarily preserve cumulative-return ordering across trajectories of different lengths. However, the purpose of Eq. (11) is not to preserve trajectory-level returns, but to prevent the accumulation of additional positive rewards merely by prolonging task execution. In particular, robotic manipulation tasks generally allow policies to recover from intermediate failures through repeated attempts or corrective actions. In this context, episodes terminate early upon successful completion, whereas unsuccessful rollouts continue until the maximum horizon. Thus, policies cannot reduce accumulated penalties by prematurely terminating failed attempts.

To further justify this transformation, we provide a new theoretical analysis in Appendix E.2. We prove that, for ideal success-probability predictions, the expected transformed return can be expressed as a failure-probability penalty weighted by episode duration. When unsuccessful episodes continue until the maximum horizon, this expected return becomes a positive affine function of the task success probability. Consequently, maximizing the ideal expected transformed return is mathematically equivalent to maximizing the task success probability. Therefore, although Eq. (11) does not preserve individual trajectory-return ordering, it remains aligned with the task-success objective in expectation under the stated conditions. The complete theoretical derivation and proof are provided in Appendix E.2 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Qfya - Part 2 (Q2&Q3)

Q2. Ablation of terminal-target augmentation.

Thank you for this suggestion. We clarify that the core TD-based reward learning procedure of eVTA$_0$ relies only on binary terminal outcomes, while the soft terminal-target augmentation introduced in Appendix B serves as an auxiliary training strategy. To assess its contribution, we conduct an additional ablation on LIBERO-Spatial, comparing eVTA$_0$ trained with binary terminal targets against the full model with augmentation under identical training configurations.

VariantVOC ↑VROC ↑MSE ↓Kendall's τa\tau_a ↑
Binary Only0.850.790.130.99
With Augmentation0.870.820.100.99

Overall, both variants achieve comparable reward quality, demonstrating that eVTA$_0$ can learn effective reward signals from binary outcomes alone. The most notable benefit of augmentation is the reduction in MSE from 0.13 to 0.10. This may be because binary supervision assigns the same zero terminal target to all unsuccessful trajectories, including those close to task completion. In contrast, soft terminal targets distinguish near-success attempts from less successful ones, providing more informative supervision that may improve the accuracy of terminal success predictions, consistent with the observed reduction in MSE. We have added this ablation study in Appendix D.1 of the revised version.

Q3. Accuracy of intermediate success-probability estimates.

Thank you for your thoughtful comment. Directly evaluating the accuracy of intermediate success probabilities is challenging because each rollout provides only a binary terminal outcome, rather than ground-truth probabilities at individual timesteps. Such probabilities could be estimated through repeated policy continuations from the same intermediate states, but obtaining reliable estimates requires substantial sampling. For example, achieving a margin of error of ±0.05 at approximately 95% confidence would require around 400 independent continuations per state in the worst case. Extending this procedure across diverse tasks and intermediate states would incur considerable computational cost. Moreover, accurately reproducing identical intermediate states is particularly challenging in real-world robotic settings.

This challenge is also encountered by existing progress reward models. Given the difficulty of obtaining reliable ground-truth values at intermediate timesteps, prior works such as GVL (Ma et al., 2025), VLAC (Zhai et al., 2025), and Robometer (Liang et al., 2026) have instead relied on indirect metrics, including temporal consistency and trajectory ranking, to assess intermediate reward quality. Following this practice, we adopt complementary evaluation metrics. VOC and VROC assess how intermediate predictions reflect task progression and deviations from successful execution, respectively. MSE measures prediction accuracy at terminal states, where ground-truth outcomes are available, while Kendall's τa\tau_a evaluates the separation between successful and failed trajectories. Together with the qualitative analyses in Figures 3, 9, and 10, these evaluations provide complementary evidence of intermediate reward quality and its alignment with task evolution.

Thank you for the response. My main concerns have been addressed, and I will maintain my original rating.

Reviewer_SpamOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes eVTA0 which is a a VLM-based reward model train on Qwen3-VL-4B with a sigmoid head on a special [REWARD] token that predicts the probability of task success from a task description and a short window of recent observations. The model is trained only on terminal success or failure labels from mixed quality policy rollouts.

Critical strengths

The main strength is that this is a simple recipe and it works on long horizon tasks. The simple recipe could be expected to scale across tasks and settings. This is important because RL finetuning is a practical setting grounded in real world experiments.

Critical weaknesses

The main weakness is that there is limited novelty. The method is not new and the target is also a standard value function. The experimental rigor also does not support its claims. There are limited real world results, RLER versus fixed eVTA0 is 42/50 vs 36/50 and 43/50 vs 39/50. This is important because the approach is needs to work in the real world for generalist robot policies.

Questions for authors

What is the result of online RL like with more recent and sample efficient online RL algorithms compared to IQL?

What is the inference cost of the reward model compared to not using the reward model?

How does the proposed method prevent reward hacking?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Spam - Weaknesses (Part 1)

Thank you for your comments. We are encouraged by your recognition of the simplicity, effectiveness, and scalability of our approach, as well as its practical relevance to real-world robotic RL. We have revised the manuscript to address your concerns, with major changes highlighted in blue. Below, we respond to each comment in detail. We hope our revisions and responses resolve your concerns and that you will consider raising your score.

W1. Novelty and relationship to value functions.

Thank you for your comment. We would like to clarify that our contribution lies in establishing a new paradigm for learning and adapting dense rewards for generalist robot policy optimization, rather than proposing a new TD update rule or value-function architecture. To the best of our knowledge, eVTA$_0$ is the first general-purpose reward model to learn dense success-probability feedback directly from mixed-quality policy rollouts and terminal task outcomes. Unlike representative progress reward models such as VLAC and Robometer, this approach requires neither expert demonstrations nor intermediate progress annotations. Building on this capability, RLER further enables reward learning to evolve alongside policy optimization by incorporating newly collected rollouts.

To realize this demonstration-free reward learning paradigm, TD bootstrapping and the target network are employed to train eVTA$_0$, rather than to optimize the robot policy. Specifically, TD bootstrapping propagates terminal outcome supervision to intermediate timesteps through next-step success-probability predictions, with the slowly updated target network providing stable learning targets. The resulting eVTA$_0$ supplies dense rewards for downstream RL.

We have also thoroughly investigated the relationship between eVTA$_0$ and conventional value functions, both theoretically and empirically, in Appendix E.1 of the original manuscript. In standard RL, a critic estimates a value function, i.e., the expected cumulative reward, to assist policy optimization. In contrast, eVTA$_0$ learns to predict eventual task success and provides dense rewards to the RL algorithm, which can still maintain its own critic. We theoretically show that success probability coincides with a value function under terminal-only binary rewards and discount factor γ=1\gamma=1, but this mathematical equivalence does not imply that eVTA$_0$ and an RL critic serve the same role. To further examine this distinction, we deliberately compare eVTA$_0$ with a PPO critic under this setting in Figure 14. Results show that eVTA$_0$ provides more accurate and reliable success-probability estimates, while its dense rewards enable more effective and sustained policy improvement, even when PPO already maintains its own critic. These findings demonstrate the practical benefits of explicitly learning a standalone reward model.

W2. Experimental rigor and support for our claims.

Thank you for raising this concern. We would like to clarify that our experiments are systematically designed to evaluate three core claims through well-controlled comparisons. First, reward quality (Section 4.1): we evaluate eVTA$_0$ against representative state-of-the-art reward models across 70 tasks on LIBERO and MetaWorld, using complementary metrics to assess temporal consistency and the ability to distinguish successful from failed rollouts. Second, policy learning (Section 4.2): we evaluate downstream RL performance across all four LIBERO suites (40 tasks), ensuring identical initial policies, RL configurations, and interaction budgets across reward methods to isolate the effect of reward quality. Third, reward evolution (Section 4.3): we compare RLER with a fixed-reward baseline on three real-world manipulation tasks, using the same accumulated rollout data and matched training budgets to isolate the benefit of adapting rewards as the policy evolves. Together, these experiments demonstrate that eVTA$_0$ provides more reliable reward feedback than existing methods, consistently improves downstream policy learning, and enables further performance gains through RLER, particularly under out-of-distribution conditions.

We further strengthen the experimental evidence with additional analyses and experiments in the revised manuscript. These include three independent training runs for reward learning and downstream RL (Appendix D.1, Tables 8–9), ablation studies (Appendix D.1), qualitative reward analyses (Appendix D.2), detailed real-world results (Appendix D.3), and theoretical and empirical investigations of reward transformation, TD update order, and TD versus MC regression (Appendices E.2–E.4). Together, these supplementary studies provide further evidence of the stability and validity of our approach.

AuthorsOpenReview ↗

Response to Reviewer Spam - Weaknesses (Part 2)

W3. Scope and reliability of real-world evaluations.

Thank you for emphasizing the importance of real-world validation. We would like to clarify that our original experiments evaluate each policy over 50 physical trials per task, covering both in-distribution (ID) and out-of-distribution (OOD) conditions. This evaluation scale is comparable to recent real-world robotic reward-learning studies, such as ROBOMETER (RSS 2026, 20 evaluation trials per task) and Robo-Dopamine (CVPR 2026, 20 trials per ID/OOD condition).

To further strengthen real-world validation, we have extended our experiments to a new bimanual manipulation task on an AgileX dual-arm robotic platform. This task requires the left arm to steadily grasp a plate while the right arm picks and places a carrot (ID) or a chili pepper (OOD) onto it. The results are summarized below, with RLER (Round 2) achieving the highest success rates. Together with the original dynamic grasping (pick carrot on the moving conveyor) and precise-control (put shuttlecock onto another) manipulation tasks, our evaluation now covers three representative tasks across two robotic platforms, encompassing distinct manipulation challenges and distribution shifts. Our real-world evaluation design is consistent with recent practices in robotic reward learning, as exemplified by Robo-Dopamine (CVPR 2026), which reports ID/OOD evaluations on three representative real-world tasks (Table 4). The consistent improvements across all three tasks further demonstrate the effectiveness of RLER across different manipulation challenges, robotic platforms, and distribution shifts. We have accordingly updated Section 4.3, Appendix C.3 and Appendix D.3 in the revised manuscript.

Table R.1. Additional real-world evaluation on the bimanual manipulation task "use the left arm to grasp and hold the plate while the right arm picks up the vegetable from the table and places it on the plate" (success rate, %).

PolicyIDOODTotal
SFT68.0%32.0%50.0%
eVTA$_0$-RLER (Round 1)76.0%52.0%64.0%
eVTA$_0$-RLER (Round 2)92.0%68.0%80.0%
Fixed eVTA$_0$88.0%56.0%72.0%
AuthorsOpenReview ↗

Response to Reviewer Spam - Questions

Q1: What is the result of online RL like with more recent and sample efficient online RL algorithms compared to IQL?

Thank you for your question. Our experiments already include online RL, alongside IQL. Specifically, in Section 4.2, we employ online GRPO (Line 424) to post-train π0.5\pi_{0.5} across all four LIBERO task suites (40 tasks), where eVTA$_0$ achieves the best average performance among the compared reward methods (Figure 4). We further evaluate online PPO (Line 1500) in Appendix E.1 (Figure 14), showing that eVTA$_0$ rewards enable sustained policy improvement compared with sparse binary rewards. These results demonstrate that eVTA$_0$ can effectively support both online RL (GRPO and PPO) and offline RL (IQL) without being tied to a specific RL algorithm. Note that our focus is on learning effective reward feedback, rather than comparing the sample efficiency of different RL algorithms.

Q2: What is the inference cost of the reward model compared to not using reward model?

Thank you for your question. We have additionally measured the inference latency and GPU memory consumption of eVTA$_0$ and representative reward models, as summarized in Table R.2. eVTA$_0$ requires only 0.212 ± 0.004 s per reward prediction, with a peak allocated GPU memory of 8.76 GiB, achieving lower inference latency than most of the compared reward models. In contrast, binary and handcrafted rewards incur negligible additional inference overhead. Importantly, eVTA$_0$ computes rewards during RL training and is not needed at deployment, adding no deployment overhead.

Table R.2. Reward model inference latency and GPU memory consumption.

MethodLatency (s) ↓Peak GPU Memory (GiB) ↓
Binary / HandcraftedNegligibleNo reward model
eVTA$_0$ (4B)0.212 ± 0.0048.76
RoboMeter (4B)0.420 ± 0.0328.97
GVL (4B)0.364 ± 0.0408.32
VLAC (2B)0.285 ± 0.0046.49
VLAC (8B)0.530 ± 0.02018.08
TOPReward (4B)0.144 ± 0.0038.69

Despite the additional inference overhead, eVTA$_0$ enables more efficient policy learning. As shown in Table R.3, eVTA$_0$ exceeds 45% success with only 640 interaction episodes and 4.9 hours, compared with 1,280 episodes and 7.8 hours for handcrafted rewards. Similarly, eVTA$_0$ surpasses 50% success with fewer episodes (4,480 vs. 6,400) and less time (32.9 vs. 37.5 hours). Moreover, after 6,400 interaction episodes, eVTA$_0$ achieves a substantially higher success rate (63.0% vs. 50.8%). Thus, eVTA$_0$ reaches target success rates with fewer interactions and less time, while achieving substantially higher peak performance. This analysis is included in Appendix D.1 of the revised manuscript.

Table R.3. Comparison of policy learning efficiency and peak performance between eVTA$_0$ and handcrafted rewards.

ComparisonMethodSuccess RateEpisodesCompute Time
Reaching >45% successeVTA$_0$49.40%6404.9 h
Handcrafted47.20%1,2807.8 h
Reaching >50% successeVTA$_0$55.40%4,48032.9 h
Handcrafted50.80%6,40037.5 h
After full trainingeVTA$_0$63.00%6,40046.2 h
Handcrafted50.80%6,40037.5 h

Q3: How does the proposed method prevent reward hacking?

Thank you for your question. Our method naturally mitigates the risk of reward hacking by grounding reward learning in actual task success rather than intermediate progress or heuristic signals. Specifically, eVTA$_0$ learns to predict eventual task success probability from policy rollouts, so its prediction directly reflects whether the observed behavior is likely to lead to successful completion. This helps distinguish genuinely beneficial actions from behaviors that appear to make progress but ultimately fail. For example, in Figure 3, eVTA$_0$ lowers its prediction when a successful grasp is followed by an unsuccessful placement, rather than maintaining a high prediction score like Robometer. Additional visualizations in Appendix D.2 (Figures 9 and 10) further support this observation.

We further mitigate reward hacking through reward transformation and adaptive reward learning. The reward transformation in Eq. (11) prevents policies from accumulating additional positive rewards merely by prolonging task execution, with its theoretical justification provided in Appendix E.2. Moreover, as the policy evolves, the trajectories it generates may differ from those used to train eVTA$_0$, potentially reducing the reliability of eVTA$_0$’s predictions. RLER addresses this by continually updating eVTA$_0$ with rollouts from the evolving policy and their actual task outcomes, adapting reward learning to the current policy distribution. Importantly, our experiments directly evaluate policy improvements using actual task success rates rather than reward model predictions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.