acceptodds
Under review as a conference paper at ICLR 2027

GRVM: Goal-Relative Value Modeling Beyond Temporal Progress for Robot Manipulation

Abstract

Robot value models increasingly provide feedback for execution evaluation and policy improvement. However, supervision based on task advancement does not necessarily reflect whether the current physical state has become more favorable for task completion. This distinction is particularly important around failures and recoveries, where temporal progress and physical state quality can diverge. We introduce Goal-Relative Value Model (GRVM), which separately predicts task progress, goal-relative state quality, and trajectory outcome from visual observations and language. A human-audited annotation pipeline constructs dense goal-relative supervision using end-effector poses at successful stage completion as local goals and extends this supervision to failed and recovered executions through failure-aware relabeling. We pretrain GRVM on 286,316 annotated trajectories (2,612 hours), including 17,286 failure/recovery trajectories. We also construct a balanced, multi-embodiment benchmark of 1,000 held-out trajectories to evaluate continuous-label agreement and local state ordering around failures and recoveries. GRVM improves triplet accuracy by 58.0% and recovery-direction accuracy by 39.7% relative to the strongest competing baseline for each metric. With GRVM kept frozen, its feedback increases mean success across four LIBERO suites from 89.8% to 94.3% over sparse-reward RL while accelerating learning. GRVM-guided online RL raises mean success from 55.0% (SFT) to 95.0% across four real-robot tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Goal-Relative Value Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W4L6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GRVM, a visual-language reward model that separately predicts task progress, goal-relative state quality, and trajectory outcome. Its main contribution is a supervision pipeline that combines stage-local end-effector goals with failure and recovery annotations. The resulting model is evaluated through offline trajectory scoring and downstream policy learning in simulation and on real robots. The distinction between temporal advancement and execution quality is intuitive and well motivated, particularly for recognizing deterioration and recovery. However, I am not yet convinced that the experiments sufficiently establish the practical benefit of the proposed supervision relative to its annotation requirements and existing reward models. My current recommendation is weak rejection, but I am open to raising my score if these concerns are adequately addressed during the rebuttal phase.

Critical strengths

The proposed approach is well motivated. Predicting state quality alongside task progress is intuitive and promising for reward modeling, especially during failure and recovery, when temporal advancement may not reflect actual improvement. Using proximity to successful stage-end end-effector poses, together with failure-aware relabeling, provides a reasonable supervision signal for staged manipulation tasks.

Empirical evidence for failure-sensitive value estimation. The offline results show substantial improvements in failure–recovery ordering, including triplet accuracy of 0.556 compared with the strongest baseline’s 0.352. The ablations also indicate that pose supervision and failure data contribute to these improvements. The controlled LIBERO experiments provide additional evidence that frozen GRVM feedback can improve policy learning over sparse rewards.

Critical weaknesses

The annotation requirements are substantial but insufficiently quantified. The pipeline requires semantic stage boundaries, failure onset, rollback-stage assignments, recovery relationships, and synchronized end-effector poses. Although GPT generates the semantic annotations, task-specific examples, human auditing, and iterative corrections remain necessary. Without quantifying this effort, it is difficult to assess the practical cost of constructing the dataset or scaling the approach to additional data and tasks.

The policy post-training experiments include limited baseline comparisons. The LIBERO experiments include RoboMeter as the only learned reward-model baseline, alongside sparse rewards, while the real-world experiments compare only against the SFT baseline. These comparisons provide limited evidence for the specific benefit of GRVM over alternative reward models, particularly in real-world manipulation.

Questions for authors
  • Was GRVM trained on any LIBERO data or trajectories from the real-robot evaluation tasks?
  • How much human annotation and auditing did the dataset require? In particular, how many trajectories needed manual correction, and how much effort would be needed to add a new task?
  • Can you provide a real-robot comparison against sparse-reward RL or another learned reward model under the same policy initialization and interaction budget?
  • Can you strengthen the LIBERO comparison with another learned reward baseline or a progress-only GRVM variant that retains failure supervision? This would help isolate the benefit of the state-quality component.
6 comments
AuthorsOpenReview ↗

Response to Reviewer W4L6 (1/3): Quantified Annotation Effort

We thank the reviewer for the constructive and specific suggestions, and for finding the approach well motivated and the evidence for failure-sensitive value estimation convincing. We address each concern below.

W1 & Q2: The annotation requirements are substantial but not quantified. How much human annotation and auditing was required, how many trajectories needed manual correction, and how much effort is needed to add a new task?

Short answers to Q2 first, followed by details.

  • Total human effort: at most ~311 h of auditing for the full training set (286,316 trajectories, 2,612 h of video), compared with an estimated ~5,350 h for fully manual annotation.
  • Trajectories manually corrected: only audited samples that fail the audit (at most 10 per round). These corrections serve as examples for the refined prompt, and the corrected samples are then re-annotated by GPT together with all other trajectories; all final training labels are produced by GPT.
  • Adding a new task: a few reference examples plus 3–5 audit rounds (~16 min of human time); adding trajectories to an existing task requires no additional human effort.

(1) Source of each annotation.

  • End-effector poses are read directly from the robot logs provided by all nine data sources, at no annotation cost.
  • Stage boundaries, failure onsets, and rollback stages are generated by GPT; recovery is derived from them and requires no separate annotation.
  • Human effort consists of a few reference examples and auditing 10 trajectories per round (~4.5 min per round, 3–5 rounds). In practice, prompts and audits are shared across tasks of the same dataset and task category.

(2) Measured cost. We measured both procedures on a DROID failure task (pack-cloth, 96 trajectories): fully manual annotation from scratch takes ~128 min, while our audit procedure takes ~16 min. For fully manual annotation, we extrapolated to the full training set by video duration; since 96.1% of the data (by duration) are successful executions requiring only stage annotation, we separately measured success-only annotation at ~50% of the per-second time of failure annotation. For our pipeline, we conservatively assume that every task is audited separately (~16 min × 1,168 tasks), so the reported audit time is an upper bound. Labor is priced at a publicly reported annotator wage of $25/h.

SettingFully manualOurs (human audit)
One task (pack-cloth, measured)~128 min (~$53)~16 min
Full training set (est.)~5,350 h (~$134k)≤ ~311 h (~$7.8k)

API costs are taken from our billing records for annotating the full training set with GPT-5.4:

DataDurationAPI cost per hour of videoTotal API cost
Non-success (failure / recovery)101.2 h~$20.93~$2.1k
Successful2,510.4 h~$10.48~$26.3k
Total2,611.6 h–~$28.4k

Compared with fully manual annotation, our pipeline reduces human time by at least ~94% and total cost by ~73% (~$ \36.2\\text{k}\ vs. ~$ \134\\text{k}\).

(3) Cost of scaling.

  • More trajectories for an existing task category: only API cost (~$\10.48 and ~$\\20.93 per hour of successful and non-success video, respectively), with no additional human effort.
  • A new task category: a few reference examples and 3–5 audit rounds (~16 min of human time).
  • A new downstream task for policy learning: three trajectories (one expert demonstration, and one successful and one failed policy rollout).

We will add this cost analysis to the appendix of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer W4L6 (2/3): Additional Real-Robot and LIBERO Baselines

W2, Q3 & Q4: The policy post-training experiments include limited baseline comparisons.

We add all of the requested comparisons.

(1) Real robot (Q3). We add sparse-reward RL (0-1) and two learned value models, Robometer and Robo-Dopamine, on all four real-robot tasks. All methods use the same SFT initialization, RL algorithm, interaction budget, and evaluation protocol (15 trials per task), and all learned value models are given the same three trajectories per task, each using them through its own objective (see Q1). GRVM achieves the best success rate on every task and improves the average from 85.0% (sparse reward) to 95.0%, while the two learned rewards reach 73.3% and 76.7%.

TaskSFT0-1RobometerRobo-DopamineGRVM
Cup place60.0 ± 0.093.3 ± 11.586.7 ± 11.593.3 ± 11.5100.0 ± 0.0
Bag unzip60.0 ± 0.093.3 ± 11.573.3 ± 11.580.0 ± 0.0100.0 ± 0.0
Block insert53.3 ± 11.580.0 ± 0.080.0 ± 0.073.3 ± 11.593.3 ± 11.5
Drawer assemble46.7 ± 11.573.3 ± 11.553.3 ± 11.560.0 ± 0.086.7 ± 11.5
Avg.55.0 ± 4.185.0 ± 5.073.3 ± 5.076.7 ± 4.195.0 ± 4.1

Success rate (%) over 15 trials per task; std is computed over three groups of five trials, and the std of the average assumes independent groups across tasks.

(2) LIBERO: an additional learned reward and a progress-only GRVM variant (Q4). We add Robo-Dopamine as a second learned-reward baseline, and GRVM w/o Pose, which is trained on the same data with failure-aware supervision but without pose-closeness supervision, using its progress prediction p̂ directly as Φ in Eq. 8 (Appendix C). This is exactly the progress-only variant with failure supervision suggested by the reviewer. All methods share the same policy initialization, PPO configuration, interaction budget, and evaluation protocol (Sec. 4.2).

MethodObjectSpatialGoalLIBERO-10Avg.
Sparse reward (RLinf)96.494.286.182.789.8
Robometer91.187.286.277.985.6
Robo-Dopamine92.385.786.179.485.9
GRVM w/o Pose (progress + failure)96.894.788.383.790.9
GRVM97.895.894.689.094.3

Final success rate (%) during PPO training, smoothed with factor 0.6.

(3) The state-quality component accounts for most of the gain.

  • GRVM achieves the best result on every suite, outperforming both learned-reward baselines.
  • Failure supervision alone (w/o Pose) improves over sparse rewards on every suite, but only modestly (89.8% → 90.9% on average).
  • Adding pose-closeness raises the average to 94.3%, with the largest gains on LIBERO-Goal (88.3% → 94.6%) and LIBERO-10 (83.7% → 89.0%), the suites where GRVM's gains over sparse rewards are also largest.
  • Of GRVM's 4.5-point average gain over sparse rewards, 3.4 points therefore come from the state-quality component. Offline, removing pose-closeness likewise lowers Triplet Accuracy from 0.556 to 0.245 (Table 3).

We will add these comparisons to Sec. 4 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer W4L6 (3/3): Training Data and Few-Shot Adaptation

Q1: Was GRVM trained on any LIBERO data or real-robot evaluation trajectories?

Pretraining: no. None of the nine pretraining sources (Table 5) contains LIBERO data or trajectories from our real-robot platform or tasks.

Downstream: few-shot adaptation with three trajectories per task. Prior value models commonly adapt to a new task with a single demonstration, via fine-tuning in Robo-Dopamine and an in-context reference in VLAC. Following this practice, we adapt GRVM to each LIBERO and real-robot task with only three trajectories: one expert demonstration, and one successful and one failed rollout from the evaluation of the SFT policy. These trajectories are naturally produced by the standard policy-training pipeline and require no additional data collection. After adaptation, GRVM is kept frozen during online RL.

Baselines. On LIBERO, both learned-reward baselines have already seen LIBERO during pretraining: Robometer's training data include LIBERO, and Robo-Dopamine's include all four LIBERO suites. Both are therefore used as released. On the real robot, all learned value models are given the same three trajectories per task: Robometer uses all three through its progress and preference losses, and Robo-Dopamine is fine-tuned with its official hop-regression objective on the two successful ones, since its progress labels assume a successful trajectory. GRVM thus outperforms baselines pretrained on LIBERO while using only three LIBERO trajectories per task.

We will add the details of the adaptation procedure to the revised paper.


To Reviewer W4L6. We thank the reviewer again for the constructive and specific suggestions. We have added all of the requested comparisons, namely sparse-reward and learned-reward baselines on the real robot, a second learned reward on LIBERO, and the progress-only variant with failure supervision, and we have quantified the annotation effort. We hope these additions resolve the concerns raised, and we would be grateful if the reviewer would consider raising the score toward acceptance.

Reviewer_JdZ2OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a VLM-based progress model. In addition to predicting the task progress, which is common in these models as the name suggests, the model in this paper also predicts goal-relative state quality, and trajectory outcome. It does so by using supervision from stage-local end-effector goals and failure/recovery annotations.

Critical strengths
  • I like the overall motivation of predicting state quality.
Critical weaknesses
  • The main motivation and the goal for generalist reward/value models has been training them in a scalable way. That's why prior work has tried to come up with ideas that can generate supervision with no human labels. Rewinding videos is one famous example of this. This paper, on the other hand, relies on additional human annotations. It also uses GPT annotations, which does not hurt scalability, so that one is okay. But given the extra human supervision, it is not really fair to say the method developed by this paper outperforms the other generalist reward/value models. The method in this paper has extra information the other methods do not have. It is also not as scalable as the others. I can see the value of this model, but the claims about the comparisons against the baselines should be tuned down.
  • While I like the motivation of predicting the quality of states toward manipulation tasks, end-effector proximity is just a heuristic. It does not necessarily capture grasp validity, object configuration, or contact quality. Constant post-failure labels further simplify the physical state.
  • Unless I am missing something, the real robot experiments compare the method only against SFT. Without sparse-reward RL or another learned reward, the gains cannot be attributed specifically to the method of this paper.
  • Reference labels follow the proposed supervision design, which would be a bias in evaluation. Instead, independent human judgments would better establish whether predictions give reliable state quality values.
  • Offline frame selection uses annotated event boundaries. But this will be unavailable during deployment. How is this going to be solved in deployment? Perhaps a uniform distribution would be a good ablation.
  • Overall I find the results unsurprising. With extra supervision, the model will of course perform better. Perhaps one nice insight is that the end-effector proximity, despite being a heuristic, gives useful supervision.
  • Minor: Robometer should be typed with a lower-case m.
Questions for authors

See weaknesses section.

9 comments
AuthorsOpenReview ↗

Response to Reviewer JdZ2 (1/6): Scalability and Fairness of Comparison

We thank the reviewer for the thoughtful review and for appreciating the motivation of predicting state quality. We address each concern below.

W1: GRVM relies on additional human annotations, which makes the comparison with other generalist value models unfair and the method less scalable.

(1) Human effort does not scale with data. All per-trajectory training labels (stage boundaries, failure onsets, and rollback stages) are generated by GPT, which the reviewer agrees does not hurt scalability, and end-effector poses are read from existing robot logs. Human involvement is limited to writing a few reference examples per task category and auditing 10 GPT outputs per round (3–5 rounds). Samples that fail an audit are corrected by humans only to serve as examples in the refined prompt, and are then re-annotated by GPT like all other trajectories, so all final training labels are produced by GPT. For the full training set (286,316 trajectories), human effort totals at most ~311 h, versus an estimated ~5,350 h for fully manual annotation, and adding trajectories requires only GPT annotation.

(2) Supervision beyond temporal progress is standard, and GRVM uses less data. Prior generalist value models also rely on additional supervision: Robometer uses trajectory preferences and success classification, RoboReward uses human-verified negative samples, VLAC predicts pairwise progress, and Robo-Dopamine uses human-annotated keyframes. Meanwhile, GRVM is trained on less data: 286K trajectories versus ~1.17M for Robometer, and 2,612 h of video versus ~3,400 h for Robo-Dopamine.

(3) The gains come from the supervision design. With the same backbone and training data, removing pose-closeness while keeping failure supervision lowers the average LIBERO success from 94.3% to 90.9%, close to sparse rewards (89.8%); further ablations are given in W6. Extracting this supervision from signals already present in robot data, yet unused by prior value models, is our contribution.

(4) Revised claims. We will state explicitly in the abstract, introduction, and experiments that GRVM is trained with a different supervision design.

AuthorsOpenReview ↗

Response to Reviewer JdZ2 (2/6): End-Effector Proximity and Post-Failure Labels

W2: End-effector proximity is a heuristic that may not capture grasp validity, object configuration, or contact quality, and constant post-failure labels oversimplify the physical state.

We agree that end-effector proximity is a heuristic, but so is temporal progress, which prior value models rely on; its value lies in being available at scale. Temporal progress only reflects how far an execution has advanced in time and cannot distinguish a state approaching the goal from one drifting away at the same time step. GRVM adds this goal-relative information from end-effector poses that are routinely recorded in robot data, without sacrificing scalability. As the reviewer also notes (W6), this heuristic provides useful supervision. Importantly, pose-closeness is never used in isolation from the object state:

  • Grasp validity and object configuration are covered by visually judged failure labels. Failure onsets and rollback stages are annotated from visual observations, so missed grasps, drops, and misplaced objects are labeled as failures. After such a failure, the value does not increase even if the end-effector moves close to the goal, until the failure is recovered. Stage goals are defined by the end-effector pose at a visually valid stage completion, so they correspond to valid object states.
  • Post-failure labels are not constant, and they correct the heuristic. After a failure, the progress target decreases over time, so the composite value keeps decreasing. Pose-closeness is held at the level of the rollback stage precisely because the end-effector pose no longer reflects the task state after a failure (e.g., the gripper may remain near the goal after the object has dropped).
  • Empirical evidence. On the deformable-object trajectories in our benchmark, removing pose-closeness lowers SD-F from 0.644 to 0.278 and Triplet Accuracy from 0.523 to 0.205. On the real robot, GRVM-guided RL improves the object-state-dominated tasks Bag unzip (60.0% → 100.0%) and Drawer assemble (46.7% → 86.7%).

Contact quality typically requires force or tactile sensing, which is rarely recorded in large-scale robot datasets; supervision based on it would not scale, which is exactly why we build on recorded end-effector poses.

AuthorsOpenReview ↗

Response to Reviewer JdZ2 (3/6): Real-Robot Baselines

W3: The real-robot experiments compare only against SFT.

We add sparse-reward RL (0-1) and two learned value models, Robometer and Robo-Dopamine, on all four real-robot tasks. All methods use the same SFT initialization, RL algorithm, interaction budget, and evaluation protocol (15 trials per task). All learned value models, including GRVM, are given the same three trajectories per task (one expert demonstration, and one successful and one failed SFT rollout), and each uses them through its own training objective. Robometer uses the demonstration for its progress loss and the successful/failed rollout pair for its preference loss. Robo-Dopamine is fine-tuned with its official hop-regression objective on the two successful trajectories; the failed rollout is not used, since its progress labels (i/M for the i-th of M frames) assume a successful trajectory. GRVM achieves the best success rate on every task and improves the average from 85.0% (sparse reward) to 95.0%, while the two learned rewards reach 73.3% and 76.7%.

TaskSFT0-1RobometerRobo-DopamineGRVM
Cup place60.0 ± 0.093.3 ± 11.586.7 ± 11.593.3 ± 11.5100.0 ± 0.0
Bag unzip60.0 ± 0.093.3 ± 11.573.3 ± 11.580.0 ± 0.0100.0 ± 0.0
Block insert53.3 ± 11.580.0 ± 0.080.0 ± 0.073.3 ± 11.593.3 ± 11.5
Drawer assemble46.7 ± 11.573.3 ± 11.553.3 ± 11.560.0 ± 0.086.7 ± 11.5
Avg.55.0 ± 4.185.0 ± 5.073.3 ± 5.076.7 ± 4.195.0 ± 4.1

Success rate (%) over 15 trials per task; std is computed over three groups of five trials, and the std of the average assumes independent groups across tasks.

On LIBERO, GRVM likewise outperforms sparse-reward RL, Robometer, and a newly added Robo-Dopamine baseline on all four suites (average 94.3% vs. 89.8%, 85.6%, and 85.9%). We will add these comparisons to Sec. 4 of the revised paper.

Reviewer_Cw8kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GRVM, a robot value model built on Qwen3-VL-4B that separately predicts task progress, goal-relative pose-closeness, and trajectory outcome. A LLM-based pipeline annotates stage boundaries, failure frames, and rollback stages. These annotations are combined with end-effector poses at stage completion, which serve as local goals, to construct dense supervision, including failure-aware relabeling for failed and recovered executions. GRVM shows clear gains in offline value evaluation, LIBERO simulation, and real-robot RL. The authors also construct a 1,000-trajectory benchmark for evaluating value models around failures and recoveries.

Critical strengths

Strengths  Clear motivation. The paper explicitly distinguishes task advancement from physical state quality, which matters most around failures and recoveries.  Well-designed method. Using end-effector poses at stage completion as local goals provides dense, physically grounded supervision, and the failure/recovery relabeling is carefully designed.  Comprehensive validation, covering offline evaluation, simulation (LIBERO), and real-robot experiments.

Critical weaknesses

Weaknesses  The offline benchmark is self-constructed and drawn from the same sources as the training data, and its reference labels come from the same annotation pipeline. This may unfairly favor GRVM over the baselines.  The method requires additional annotation and synchronized end-effector poses. This increases data cost and limits applicability to data without robot-state records.  The human audit is relatively weak: only 10 trajectories are sampled per task category per round, and no quantitative measure of annotation quality is reported.  The reliance on end-effector poses and several heuristic design choices may limit generalization.

Questions for authors

Questions  What is the rationale behind the definition of Φ in Eq. 8? How were the exponents 0.4/0.6 chosen, and how sensitive are the results to them?  Is pose-closeness effective for tasks involving deformable objects or dominated by object state? How is the pose defined for bimanual tasks?  In Table 3, removing failure data improves MAE and SD-R. What explains this?

10 comments
AuthorsOpenReview ↗

Response to Reviewer Cw8k (1/7): Benchmark Construction and Third-Party Evaluation

We thank the reviewer for the careful review and for recognizing the clear motivation, the well-designed method, and the comprehensive validation of our work. We address each concern below.

W1: The offline benchmark is self-constructed from the same sources and annotation pipeline as the training data, which may favor GRVM.

(1) Task-level held-out split with human-annotated references. The benchmark is held out at the task level: each benchmark task category is treated as a leakage group, and all of its episodes, including related views, temporal crops, and derived slices, are excluded from GRVM training. All benchmark reference annotations (stage boundaries, failure onsets, and rollback stages) are made manually by humans from scratch, independently of the GPT-based pipeline used to construct training targets.

(2) Evaluation on a third-party benchmark. We further evaluate GRVM on ContextProgress-Bench[1], whose tasks, episodes, and human progress annotations are constructed entirely by a third party. Following its official protocol, we evaluate on all 120 episodes under the three context settings defined by the benchmark. GRVM achieves the highest Spearman correlation in every setting, and the second-lowest MAE in the no-context setting, which matches GRVM's inference protocol.

ModelNo-context MAE ↓No-context Spearman ↑Oracle MAE ↓Oracle Spearman ↑Self-chained MAE ↓Self-chained Spearman ↑
ProgressLM (demo=5)†17.950.744.180.9916.700.99
Robo-Dopamine15.590.743.430.984.660.97
Robometer25.420.534.910.9311.390.92
TOPReward30.950.566.700.909.420.89
VLAC24.430.724.500.9717.130.94
RynnValue-4B30.300.11––––
GRVM (ours)18.070.816.520.9912.350.98

†ProgressLM uses five demonstrations as additional context; it is shown for reference and excluded from the comparison, following ProgressCompass. RynnValue-4B's official implementation does not support context inputs, so only the no-context setting is reported. MAE is on a 0–100 scale. Best results among compared models are in bold.

These results indicate that GRVM's advantage is not an artifact of our benchmark construction. We will clarify the held-out split and the human annotation of the benchmark in Sec. 4.1 and Appendix D, and add the third-party evaluation to the appendix.

[1] Zhang J, Wu K, Qian C, et al. ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context[J]. arXiv preprint arXiv:2609.36684, 2026.

AuthorsOpenReview ↗

Response to Reviewer Cw8k (2/7): Annotation Cost and Pose Availability

W2: The method requires additional annotation and synchronized end-effector poses, increasing data cost and limiting applicability.

(1) Annotation cost is a one-time pretraining cost with small human effort. Annotations and poses are used only to construct training targets; at inference, GRVM takes only multi-view RGB observations and the task instruction. Human effort consists of a few reference examples plus 10 audited trajectories per refinement round (~4.5 min per round, 3–5 rounds), with prompts and audits shared at the (dataset, task-category) level. Because audit time does not grow with the number of trajectories, adding trajectories to an annotated task requires no additional human effort.

Human time. We measured both procedures on a DROID failure task (pack-cloth, 96 trajectories; also used in our response to W3): fully manual annotation from scratch takes ~128 min, while our audit procedure takes ~16 min. We then extrapolated to the full training set (286,316 trajectories, 1,168 tasks). For manual annotation, we scale by video duration; since 96.1% of our training data (by duration) are successful executions, which require only stage annotation, we separately measured success-only annotation at ~50% of the per-second time of failure annotation. For our pipeline, we conservatively assume that every task is audited separately (~16 min × 1,168 tasks), whereas in practice audits are shared across tasks of the same category, so the reported audit time is an upper bound. Labor is priced at a publicly reported annotator wage of $25/h.

SettingFully manualOurs (human audit)
One task (pack-cloth, measured)~128 min (~$53)~16 min
Full training set (est.)~5,350 h (~$134k)≤ ~311 h (~$7.8k)

API cost, taken from our billing records for annotating the full training set with GPT-5.4:

DataDurationAPI cost per hour of videoTotal API cost
Non-success (failure / recovery)101.2 h~$20.93~$2.1k
Successful2,510.4 h~$10.48~$26.3k
Total2,611.6 h–~$28.4k

Compared with fully manual annotation, our pipeline reduces human time by at least ~94% and total cost by ~73% (~$ \36.2\\text{k}\ vs. ~$ \134\\text{k}\). Adapting GRVM to a new downstream task requires only three trajectories: one expert demonstration, and one successful and one failed policy rollout.

(2) Most generalist value models also rely on curated supervision. Robo-Dopamine segments trajectories using human-annotated multi-view keyframes. RoboReward uses human progress scores (RoboArena) and VLM/LLM-based counterfactual relabeling. Robometer relies on curated failure datasets and manually inspects 10 trajectories per data source to determine task endpoints. Robo-Dopamine 2.0 synthesizes failure states with an image-editing API. RynnValue uses the native subtask annotations of its source datasets together with VLM-generated captions. GRVM's GPT-based annotation with lightweight human auditing follows the same practice, while additionally providing failure- and recovery-aware supervision.

(3) Pose dependence is limited to training-time target construction.

  • No pose at inference. In all LIBERO and real-robot experiments, GRVM receives only RGB observations and the instruction.
  • EE poses are widely recorded but unused by prior work. All nine data sources we use (286,316 trajectories, 2,612 hours) provide synchronized end-effector poses. Several of them, including AgiBot World, DROID, and OXE, are also used to train Robometer, RoboReward, VLAC, and RynnValue, yet none of these models exploits this signal. Removing pose-closeness supervision lowers Triplet Accuracy from 0.556 to 0.245 (Table 3).
  • Data without poses. Frames without a valid synchronized pose are excluded only from pose-closeness supervision (Appendix A.2) and still contribute progress and outcome supervision, so pose-free data can be mixed into training. With the rapid growth of UMI-style and egocentric data that provide gripper or hand poses, this supervision is also becoming increasingly easy to obtain.

We will add the cost analysis and the discussion of pose availability to the appendix of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer Cw8k (3/7): Quantitative Annotation Quality

W3: The human audit is weak (10 trajectories per round), and annotation quality is not quantified.

We add two quantitative evaluations of annotation quality and explain the design of the audit.

(1) Large-scale comparison with human annotations. On the 500 non-successful trajectories of our benchmark (125 per source), which were annotated from scratch by humans, we compare the GPT annotations produced by our pipeline with the human annotations trajectory by trajectory:

SourceFailure onset agreement (within ±1 s)Rollback stage agreement (≤1 stage)
ViFailback96.8%97.6%
AgiBot85.6%86.4%
VLAC-Cut100.0%92.8%
DROID94.4%98.4%
Overall94.2%93.8%

(2) Held-out validation of the audit procedure. On a DROID failure task (pack-cloth, placing cloth into a container; also used in our cost analysis in W2), we randomly split the trajectories into an audit pool and a held-out set. Humans first annotated the held-out set from scratch. We then ran the iterative procedure of the paper: in each round, GPT annotates all unaudited trajectories, and humans audit 10 of them and refine the prompt. GPT outputs on the held-out set were not inspected until the iteration finished. Under the same criterion used in each audit round, the final annotations are correct on 44 of the 45 held-out trajectories (97.8%, 95% CI: 88.2%–99.9%).

(3) Why 10 trajectories per round. Each audit round is designed to detect systematic prompt errors. A flawed prompt mislabels many trajectories at once, so such errors are reliably detected with a small sample: with 10 samples, an error rate of 20% (30%) is detected with probability 89% (97%). In our experiments, one prompt revision introduced an error, and only 2 of the 10 sampled annotations passed the audit of that round; the error was corrected in the next prompt version. Each task category goes through 3–5 rounds, so 30–50 distinct trajectories are inspected by humans. The residual error that a 10-sample audit may miss is directly quantified by (1) and (2).

We will add both evaluations to the appendix.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of New Results and Clarifications

We thank all reviewers for their constructive feedback. We summarize the new results and key clarifications below; detailed responses are given to each reviewer.

New results.

  • Third-party benchmark. On ContextProgress-Bench[1], whose data and human labels are constructed entirely by a third party, GRVM achieves the highest Spearman correlation under all three context settings.
  • Real-robot baselines. Under the same initialization and interaction budget, GRVM reaches 95.0% average success, compared with 85.0% for sparse-reward RL, 73.3% for Robometer, and 76.7% for Robo-Dopamine.
  • LIBERO. GRVM achieves 94.3% average success over four suites, compared with 89.8% (sparse reward), 85.6% (Robometer), 85.9% (Robo-Dopamine), and 90.9% for a progress-only GRVM variant that retains failure supervision. This isolates the contribution of the state-quality component.
  • Annotation cost and quality. Human effort for the full training set is at most ~311 h of auditing, compared with an estimated ~5,350 h for fully manual annotation, and all final training labels are produced by GPT. GPT annotations agree with independent human annotations on 94.2% of failure onsets (within ±1 s).

Clarifications.

  • All final training labels are generated by GPT; human corrections of audited samples are used only to refine the prompt. End-effector poses come from existing robot logs. All benchmark references are annotated by humans from scratch.
  • GRVM is not pretrained on LIBERO or on any real-robot evaluation data. It is adapted to each downstream task with three trajectories and kept frozen during RL.
AuthorsOpenReview ↗

Summary of Revisions in the Uploaded PDF

We thank all reviewers for their feedback. We have uploaded a revised PDF; all changes made during the discussion are marked in blue. The main revisions are:

  • Evaluation (Cw8k, JdZ2): the task-level held-out split, human-annotated benchmark references, and uniform input sampling in offline evaluation are clarified (Sec. 4.1, App. D). Results on the third-party ContextProgress-Bench are added to the main text (Table 1b).
  • Policy-learning baselines (JdZ2, W4L6): sparse-reward RL, Robometer, and Robo-Dopamine on all four real-robot tasks (Table 2); Robo-Dopamine on LIBERO (Fig. 5); and a progress-only GRVM variant with failure supervision (Sec. 4.4, Table 13). The three-trajectory adaptation and annotation-free online scoring are described in Secs. 4.2–4.3 and App. E–F.
  • Annotation (Cw8k, JdZ2, W4L6): all final labels are generated by GPT, with human corrections used only to refine prompts (Sec. 3.1, App. A.6); agreement with independent human annotations, held-out validation of the audit, and the human and API cost are added (App. A.9).
  • Additional analyses (Cw8k): the rationale and sensitivity of the composite value Φ (App. D.1), the deformable-object analysis (App. D.2), the bimanual pose definition (App. B), and the composition of the w/o Failure variant (Sec. 4.4, App. C).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.