- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents GRVM, a visual-language reward model that separately predicts task progress, goal-relative state quality, and trajectory outcome. Its main contribution is a supervision pipeline that combines stage-local end-effector goals with failure and recovery annotations. The resulting model is evaluated through offline trajectory scoring and downstream policy learning in simulation and on real robots. The distinction between temporal advancement and execution quality is intuitive and well motivated, particularly for recognizing deterioration and recovery. However, I am not yet convinced that the experiments sufficiently establish the practical benefit of the proposed supervision relative to its annotation requirements and existing reward models. My current recommendation is weak rejection, but I am open to raising my score if these concerns are adequately addressed during the rebuttal phase.
Critical strengths
The proposed approach is well motivated. Predicting state quality alongside task progress is intuitive and promising for reward modeling, especially during failure and recovery, when temporal advancement may not reflect actual improvement. Using proximity to successful stage-end end-effector poses, together with failure-aware relabeling, provides a reasonable supervision signal for staged manipulation tasks.
Empirical evidence for failure-sensitive value estimation. The offline results show substantial improvements in failure–recovery ordering, including triplet accuracy of 0.556 compared with the strongest baseline’s 0.352. The ablations also indicate that pose supervision and failure data contribute to these improvements. The controlled LIBERO experiments provide additional evidence that frozen GRVM feedback can improve policy learning over sparse rewards.
Critical weaknesses
The annotation requirements are substantial but insufficiently quantified. The pipeline requires semantic stage boundaries, failure onset, rollback-stage assignments, recovery relationships, and synchronized end-effector poses. Although GPT generates the semantic annotations, task-specific examples, human auditing, and iterative corrections remain necessary. Without quantifying this effort, it is difficult to assess the practical cost of constructing the dataset or scaling the approach to additional data and tasks.
The policy post-training experiments include limited baseline comparisons. The LIBERO experiments include RoboMeter as the only learned reward-model baseline, alongside sparse rewards, while the real-world experiments compare only against the SFT baseline. These comparisons provide limited evidence for the specific benefit of GRVM over alternative reward models, particularly in real-world manipulation.
Questions for authors
- Was GRVM trained on any LIBERO data or trajectories from the real-robot evaluation tasks?
- How much human annotation and auditing did the dataset require? In particular, how many trajectories needed manual correction, and how much effort would be needed to add a new task?
- Can you provide a real-robot comparison against sparse-reward RL or another learned reward model under the same policy initialization and interaction budget?
- Can you strengthen the LIBERO comparison with another learned reward baseline or a progress-only GRVM variant that retains failure supervision? This would help isolate the benefit of the state-quality component.
6 comments
Response to Reviewer W4L6 (2/3): Additional Real-Robot and LIBERO Baselines
W2, Q3 & Q4: The policy post-training experiments include limited baseline comparisons.
We add all of the requested comparisons.
(1) Real robot (Q3). We add sparse-reward RL (0-1) and two learned value models, Robometer and Robo-Dopamine, on all four real-robot tasks. All methods use the same SFT initialization, RL algorithm, interaction budget, and evaluation protocol (15 trials per task), and all learned value models are given the same three trajectories per task, each using them through its own objective (see Q1). GRVM achieves the best success rate on every task and improves the average from 85.0% (sparse reward) to 95.0%, while the two learned rewards reach 73.3% and 76.7%.
| Task | SFT | 0-1 | Robometer | Robo-Dopamine | GRVM |
|---|---|---|---|---|---|
| Cup place | 60.0 ± 0.0 | 93.3 ± 11.5 | 86.7 ± 11.5 | 93.3 ± 11.5 | 100.0 ± 0.0 |
| Bag unzip | 60.0 ± 0.0 | 93.3 ± 11.5 | 73.3 ± 11.5 | 80.0 ± 0.0 | 100.0 ± 0.0 |
| Block insert | 53.3 ± 11.5 | 80.0 ± 0.0 | 80.0 ± 0.0 | 73.3 ± 11.5 | 93.3 ± 11.5 |
| Drawer assemble | 46.7 ± 11.5 | 73.3 ± 11.5 | 53.3 ± 11.5 | 60.0 ± 0.0 | 86.7 ± 11.5 |
| Avg. | 55.0 ± 4.1 | 85.0 ± 5.0 | 73.3 ± 5.0 | 76.7 ± 4.1 | 95.0 ± 4.1 |
Success rate (%) over 15 trials per task; std is computed over three groups of five trials, and the std of the average assumes independent groups across tasks.
(2) LIBERO: an additional learned reward and a progress-only GRVM variant (Q4). We add Robo-Dopamine as a second learned-reward baseline, and GRVM w/o Pose, which is trained on the same data with failure-aware supervision but without pose-closeness supervision, using its progress prediction p̂ directly as Φ in Eq. 8 (Appendix C). This is exactly the progress-only variant with failure supervision suggested by the reviewer. All methods share the same policy initialization, PPO configuration, interaction budget, and evaluation protocol (Sec. 4.2).
| Method | Object | Spatial | Goal | LIBERO-10 | Avg. |
|---|---|---|---|---|---|
| Sparse reward (RLinf) | 96.4 | 94.2 | 86.1 | 82.7 | 89.8 |
| Robometer | 91.1 | 87.2 | 86.2 | 77.9 | 85.6 |
| Robo-Dopamine | 92.3 | 85.7 | 86.1 | 79.4 | 85.9 |
| GRVM w/o Pose (progress + failure) | 96.8 | 94.7 | 88.3 | 83.7 | 90.9 |
| GRVM | 97.8 | 95.8 | 94.6 | 89.0 | 94.3 |
Final success rate (%) during PPO training, smoothed with factor 0.6.
(3) The state-quality component accounts for most of the gain.
- GRVM achieves the best result on every suite, outperforming both learned-reward baselines.
- Failure supervision alone (w/o Pose) improves over sparse rewards on every suite, but only modestly (89.8% → 90.9% on average).
- Adding pose-closeness raises the average to 94.3%, with the largest gains on LIBERO-Goal (88.3% → 94.6%) and LIBERO-10 (83.7% → 89.0%), the suites where GRVM's gains over sparse rewards are also largest.
- Of GRVM's 4.5-point average gain over sparse rewards, 3.4 points therefore come from the state-quality component. Offline, removing pose-closeness likewise lowers Triplet Accuracy from 0.556 to 0.245 (Table 3).
We will add these comparisons to Sec. 4 of the revised paper.
Response to Reviewer W4L6 (3/3): Training Data and Few-Shot Adaptation
Q1: Was GRVM trained on any LIBERO data or real-robot evaluation trajectories?
Pretraining: no. None of the nine pretraining sources (Table 5) contains LIBERO data or trajectories from our real-robot platform or tasks.
Downstream: few-shot adaptation with three trajectories per task. Prior value models commonly adapt to a new task with a single demonstration, via fine-tuning in Robo-Dopamine and an in-context reference in VLAC. Following this practice, we adapt GRVM to each LIBERO and real-robot task with only three trajectories: one expert demonstration, and one successful and one failed rollout from the evaluation of the SFT policy. These trajectories are naturally produced by the standard policy-training pipeline and require no additional data collection. After adaptation, GRVM is kept frozen during online RL.
Baselines. On LIBERO, both learned-reward baselines have already seen LIBERO during pretraining: Robometer's training data include LIBERO, and Robo-Dopamine's include all four LIBERO suites. Both are therefore used as released. On the real robot, all learned value models are given the same three trajectories per task: Robometer uses all three through its progress and preference losses, and Robo-Dopamine is fine-tuned with its official hop-regression objective on the two successful ones, since its progress labels assume a successful trajectory. GRVM thus outperforms baselines pretrained on LIBERO while using only three LIBERO trajectories per task.
We will add the details of the adaptation procedure to the revised paper.
To Reviewer W4L6. We thank the reviewer again for the constructive and specific suggestions. We have added all of the requested comparisons, namely sparse-reward and learned-reward baselines on the real robot, a second learned reward on LIBERO, and the progress-only variant with failure supervision, and we have quantified the annotation effort. We hope these additions resolve the concerns raised, and we would be grateful if the reviewer would consider raising the score toward acceptance.
Response to Reviewer W4L6 (1/3): Quantified Annotation Effort
We thank the reviewer for the constructive and specific suggestions, and for finding the approach well motivated and the evidence for failure-sensitive value estimation convincing. We address each concern below.
W1 & Q2: The annotation requirements are substantial but not quantified. How much human annotation and auditing was required, how many trajectories needed manual correction, and how much effort is needed to add a new task?
Short answers to Q2 first, followed by details.
(1) Source of each annotation.
(2) Measured cost. We measured both procedures on a DROID failure task (pack-cloth, 96 trajectories): fully manual annotation from scratch takes ~128 min, while our audit procedure takes ~16 min. For fully manual annotation, we extrapolated to the full training set by video duration; since 96.1% of the data (by duration) are successful executions requiring only stage annotation, we separately measured success-only annotation at ~50% of the per-second time of failure annotation. For our pipeline, we conservatively assume that every task is audited separately (~16 min × 1,168 tasks), so the reported audit time is an upper bound. Labor is priced at a publicly reported annotator wage of $25/h.
API costs are taken from our billing records for annotating the full training set with GPT-5.4:
Compared with fully manual annotation, our pipeline reduces human time by at least ~94% and total cost by ~73% (~$ \36.2\\text{k}\ vs. ~$ \134\\text{k}\).
(3) Cost of scaling.
We will add this cost analysis to the appendix of the revised paper.