- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a dense reward learning paradigm without requiring expert demonstrations. The proposed method is based on the insight that the expected task success probability as a reward signal satisfies the martingale property and can be learned via one-step bootstrapping.
Critical strengths
- I find the idea quite neat and clever, and the resulting approach is principled, simple yet practically effective. That said, I am not too familiar with the literature on reward learning in RL, so I leave the evaluation of the novelty compared to the existing literature to other reviewers.
- The evaluation has a broad coverage on tasks, baselines, metrics, with one caveat I state in the weakness section.
- The presentation is well-structured and clear overall.
Critical weaknesses
- What theorem 2 supports is a “sequential optimization” scheme that optimize from the terminal step backward. The introduced TD learning is more of a convenient approximation. While commonly used in settings such as off-policy RL, it is not justified sufficiently here. More experimental or analytical results could address this point.
- Eq. (11) - the justification of this transformation is only valid in certain specific settings. For example, for fixed-horizon tasks, adding or subtracting a constant doesn’t affect the optimization since the constant is policy-independent. This point needs clarification.
- My understanding is that all the training runs, including the TD reward function learning and RL with the learned reward, only uses one trial (random seed). And all the claims and interpretation are based on a single trial. As a consequence, no randomness across seeds (and therefore the stability of the proposed approach and the baselines) is evaluated anywhere. The multiple evaluation trials in each setting only characterizes the performance of a single training trial. Since it is known that RL tends to have a high variance, I am not convinced by the existing evidence that the claims still hold true with more trials for each method. I prefer a narrower evaluation coverage but with more rigorous statistical evidence over broader coverage with a single trial for each setting.
Questions for authors
The last weakness doesn't appear to be easy to address within the rebuttal period. But if I misunderstood the number of training runs used for evaluation, I am happy to raise my score and give a more positive recommendation.
4 comments
Response to Reviewer x2Mw - Part 2 (W3)
W3. Statistical reliability across training runs.
Thank you for raising this concern. To directly assess training stability, we conduct two additional independent training runs (seeds 43 and 44) for both reward learning and RL post-training, yielding three seeds in total.
For reward learning, we select LIBERO-Spatial as a representative suite to evaluate the training stability of eVTA$_0$ across different random seeds. The results show consistently high reward quality with low variance across all four metrics, as summarized below.
| Metric | Seed 42 | Seed 43 | Seed 44 | Mean ± Std |
|---|---|---|---|---|
| VOC ↑ | 0.87 | 0.84 | 0.85 | 0.853 ± 0.015 |
| VROC ↑ | 0.82 | 0.83 | 0.82 | 0.823 ± 0.006 |
| MSE ↓ | 0.10 | 0.08 | 0.08 | 0.087 ± 0.012 |
| Kendall's ↑ | 0.99 | 0.99 | 0.97 | 0.983 ± 0.012 |
For RL post-training, we select LIBERO-Long, the most challenging suite featuring long-horizon manipulation tasks, to assess the stability of policy improvements enabled by eVTA$_0$. As shown below, eVTA$_0$ consistently achieves the highest success rate across all three independent training runs, with an average of 58.7% ± 4.5%, outperforming all baselines.
| Method | Seed 42 | Seed 43 | Seed 44 | Mean ± Std |
|---|---|---|---|---|
| Binary | 41.8 | 41.2 | 43.8 | 42.3 ± 1.4 |
| Handcrafted | 45.4 | 50.8 | 42.6 | 46.3 ± 4.2 |
| Robometer | 29.4 | 27.4 | 28.8 | 28.5 ± 1.0 |
| eVTA₀ | 54.0 | 63.0 | 59.0 | 58.7 ± 4.5 |
These additional experiments demonstrate the stability of eVTA$_0$ reward learning across random seeds and further confirm that its advantages in downstream RL performance are consistently maintained across independent training runs. These new results and analyses have been added to Appendix D.1 of the revised manuscript.
Finally, we would also like to clarify the motivation behind our original evaluation protocol. Our evaluation follows a common practice in large-scale robotic reward learning (e.g., Robo-Dopamine, Tan et al., 2026; Robometer, Liang et al., 2026) and VLA RL (e.g., , Chen et al., 2025; LaST-R1, Chen et al., 2026), where extensive benchmark evaluations are often reported without multi-seed training statistics for every setting, given the substantial cost of training large vision-language models and conducting extensive environment interactions. Importantly, our experiments involve multi-task training: eVTA$_0$ is jointly trained on rollouts from all 40 LIBERO tasks, while each RL policy is jointly optimized across 10 tasks within a suite. This provides broad evidence across diverse tasks. Our additional multi-seed experiments complement this evaluation by directly assessing training stability.
Response to Authors’ Rebuttal
I thank the authors for their responses and additional experiments. The responses to W1 and W2 address my concerns.
Regarding W3, however, if I understand correctly, 3-seed results (still a small number of seeds) are now available only for RL on LIBERO-Long and reward learning on LIBERO-Spatial, while all other results, including all real-world experiments, remain single run. As a minor point, a standard deviation estimated from only three runs is itself highly uncertain and thus not very informative on its own. I encourage the authors to report proper measures of statistical significance, such as confidence intervals or p-values.
I maintain my original evaluation as my final recommendation.
Thanks for your follow-up comments. We have directly addressed your original concern by conducting additional independent training runs for both reward learning and RL post-training, and we now further report confidence intervals and significance tests. On LIBERO-Long, eVTA$_0$ improves the average success rate over the strongest baseline, Handcrafted, by 12.4%, with an unadjusted 95% confidence interval of [2.5%, 22.3%]. Two-sided Welch tests yield Holm-adjusted p-values of 0.035, 0.035, and 0.016 against Binary, Handcrafted, and Robometer, respectively, providing further statistical evidence for the performance advantages of eVTA$_0$. Crucially, eVTA$_0$ ranks first under every tested seed, and even its lowest success rate (54.0%) exceeds the highest result of any baseline (50.8%). Such a substantial and consistently observed performance gap provides evidence against attributing the reported advantage merely to a favorable choice of training seed.
For the real-world experiments, we evaluate the learned policies through repeated trials across ID and OOD conditions, including variations in object placement and robotic execution. Such repeated physical evaluations are also widely adopted in leading VLA studies, including Physical Intelligence's and . Importantly, multi-seed training in real-world settings is less straightforward to interpret than in simulation, as physical environments cannot be precisely reset to identical conditions, and performance variations may arise from both training randomness and unavoidable environmental differences. In contrast, repeated evaluations across diverse physical conditions directly assess the practical reliability and generalization of the learned policies. We therefore believe that the absence of multi-seed real-world training should not, by itself, undermine the validity of our real-world findings, particularly when repeated physical evaluations are standard practice in leading embodied VLA research (, , etc.).
Your latest response confirms that our replies to W1 and W2 have resolved your concerns. We have also substantially strengthened the evidence for W3 through additional independent training runs and the statistical analyses you requested. In light of these efforts and new evidence, we sincerely hope you will reconsider raising your score. We believe that the remaining concerns regarding random seeds will not undermine the central conclusions and contributions of our paper, particularly given the consistent improvements demonstrated by our additional experiments. We hope you can reconsider the severity of this remaining issue and whether it truly justifies rejecting a paper whose central findings are supported by substantial theoretical and empirical evidence.
Response to Reviewer x2Mw - Part 1 (W1&W2)
Thank you for the thoughtful and constructive feedback. We are encouraged that you find our core idea neat and clever, and the proposed approach principled, simple, yet practically effective. We also appreciate your recognition of our broad experimental coverage across tasks, baselines, and metrics, as well as the clear and well-structured presentation. We have revised the manuscript to address your concerns, with major changes highlighted in blue. Below, we respond to each comment in detail. We hope our revisions and responses resolve your concerns and that you will consider raising your score.
W1. Justification of the update order of TD bootstrapping.
Thank you for your thoughtful comment. We would like to clarify that although Theorem 2 uses backward induction in its proof, explicit backward sequential optimization is not required. In the newly added Appendix E.3, we prove that, under exact conditional-expectation updates and terminal anchoring, the success-probability solution is unique and can be recovered regardless of the update order. To further validate this finding in practice, we compare randomly shuffled minibatch training (Random, our default implementation) with reverse-temporal-order training (Reverse, from terminal to initial timesteps) on LIBERO-Spatial under identical training configurations. The results are summarized below.
Random achieves reward quality comparable to Reverse, providing empirical evidence that explicit backward ordering is not necessary for effective TD-based reward learning. The complete theoretical analysis and experimental results are provided in Appendix E.3 of the revised manuscript.
We also provide further justification for our TD design. Appendix E.4 shows that ideal TD bootstrapping and Monte Carlo (MC) regression share the same population target, while TD provides lower-variance supervision. Experiments across all four LIBERO suites further demonstrate that TD provides more informative intermediate feedback and better captures task evolution. Additionally, Appendix D.1 examines the effects of target-network update intervals and context window sizes, providing empirical support for our training configurations. Together, these analyses provide complementary theoretical and empirical evidence for our TD-based reward learning approach.
W2. Justification of the reward transformation in Eq. (11).
Thank you for this comment. All our RL experiments, including both simulation and real-world settings, adopt variable-length episodes, a common practice in embodied RL (e.g., RLinf, Yu et al., 2026; πRL, Chen et al., 2025). Specifically, episodes terminate upon task success, with a maximum step limit to prevent indefinite execution. Consequently, the cumulative constant shift introduced by Eq. (11) depends on the actual episode length and is therefore policy-dependent. This transformation prevents policies from accumulating additional positive rewards simply by prolonging task execution.