- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
RoboValue is a proposed benchmark to evaluate diverse reward/value models for robotic manipulation tasks. It provides teleoperated trajectories from both simulation and real world tasks with different embodiments as the evaluation dataset covering both successful ones and failed ones with recovery. The benchmark also proposes additional evaluation metrics to better asses different reward/value models about their task-state understanding, temporal progress, failure/recovery reasoning, and value consistency, performance under environmental and embodiment shifts.
Critical strengths
-
The paper is clear and easy to follow with details in the appendix. Incorporating the real failures and recovery into the value/reward model evaluation shows the benefit to evaluate those models more comprehensively.
-
The analysis of the benchmark results across different models are thorough and shows the pros and cons of different models over different metrics.
Critical weaknesses
- The failed trajectories in the benchmark com from the teleoperation and not from actual policy rollout. Unlike human teleoperation which is instructed to fail in certain ways, failures from policy rollouts are more diverse or sometimes more nuanced. Therefore, evaluating only on the teleoperation trajectories may not provide a good proxy to the scenarios where the value function is actually used. Moreover, the paper doesn't evaluate any downstream applications of different reward/value models. For example, for finetuning with RL, it would be interesting to see if any of those metrics can show the correlation between reward quality vs the performance of downstream RL. This may provide more useful signals for the community to know which reward model to choose.
Questions for authors
- The episode-level score v is not defined in the paper. This is also critical for the comparison between trajectories. Can the authors clarify whether you are using max, sum, or discounted sum of the per-step value to determine the episode-level score?
- Would the authors also be able to evaluate some other simple heuristic baselines in the benchmark to help readers understand better what each metric is evaluating. For example, a random baseline and a baseline where reward is proportional to the timestep can help clarify why some metrics can better evaluate the models that learn a short-cut from the timesteps.
- It would be easier for reader to understand if the authors can provide a table to explicitly highlight the difference between the proposed benchmark vs prior benchmark. Also among a list of metrics proposed in the paper, some of them are borrowing from prior work (e.g., VOC). Therefore, it is also better to have a table to clarify which are the novel metrics different from prior work.
3 comments
Q2. What simple heuristic baselines reveal
Q2. How do random and timestep baselines score, and which metrics expose a temporal shortcut?
We added both baselines you requested, together with a constant baseline and an annotation-based progress oracle (Appendix G.3). Random draws each value independently from a uniform distribution on [0, 1]. Constant always returns 0.5. The linear clock increases with position throughout playback, including when actions are reversed. The oracle uses the annotated subtasks, failures, and recoveries.
| Predictor | VOC | Memory-VOC | Cycle-VOC | VS | TRR | TGA-CF |
|---|---|---|---|---|---|---|
| Random | 0.21 | 1.54 | −0.38 | 0.51 | 16.83 | 21.45 |
| Constant | 0 | 0 | 0 | 0 | 0 | 0 |
| Linear clock | 100 | 100 | 0 | 100 | 0 | 0 |
| Annotation oracle | 100 | 100 | 100 | 100 | 100 | 100 |
These are in-domain scores ×100 (Table 10). On a fluent forward demonstration, elapsed time is correlated with completion. A clock can therefore order the states correctly even when the same scene recurs. It also updates at every step and never reverses, earning perfect VS. Constant gets zero on VS because it provides no updates.
Cycle-VOC supplies a controlled test of this shortcut. The video reverses after a shared turning point, with the forward history still available. The clock has rank correlation +1 on the forward half and −1 against progress on the reverse half, giving 0 overall. Its TRR and TGA-CF scores are also zero because it cannot react to failure or distinguish instructions for the same video. Table 11 shows the same 100/100/0 pattern for the clock on VOC, Memory-VOC, and Cycle-VOC under both generalization shifts. This is why we read a high Memory-VOC or VS score together with reversal and grounding results.
We will also manually inspect sample inputs and their corresponding predictions for each model to check for unintended timestamp cues.
Q3. Benchmark comparison, metric origins and downstream use
Q3. Which benchmarks already cover these capabilities, and which metric constructions does RoboValue add?
Appendix B now compares 23 prior benchmarks and model-specific evaluation suites with RoboValue. Table 4 maps their coverage across the metrics, simulation and real data, and the two generalization shifts. For example, RBM-EVAL already evaluates outcome ranking, cross-task grounding, and forward progress, while ContextProgress-Bench evaluates history-dependent progress. We summarize the metric origins below.
| Metrics | Prior work or contribution in RoboValue |
|---|---|
| SA, TGA-CT, VOC | Established outcome-ranking, cross-task grounding, and forward-ordering tests. VOC follows Ma et al. (2025). |
| Memory-VOC, SIA | Memory-VOC applies VOC to repeated visual states. SIA scores a model-generated subtask description against the annotated stage using a separate judge. |
| TGA-CF | Tests the original instruction against four controlled edits to object, action, placement, or constraint on the same successful video. All four comparisons must pass. |
| Cycle-VOC | Uses one continuous forward–reverse sequence and retains the forward history at the turn. RoboBrain 2.5 reported the two playback directions separately. |
| FPL, TRR | ProcVLM used the earliest global maximum for failure timing. FPL uses the peak before the largest decline and covers recoverable errors. TRR compares three matched branches and scores the corrective attempt and its observable outcome separately. |
| VS, CSVC | VS combines Kaufman’s efficiency ratio with non-flat time coverage. CSVC aligns the same semantic subtask across valid orders and compares gain dispersion with the magnitude of the shared gain. |
The last four rows describe the six constructions we introduce. Sections 3.2–3.4 and Appendix B explain how we apply them across simulation and real trajectories with separate embodiment and environment shifts.
Your concern about teleoperation and downstream RL. The benchmark was collected by teleoperation (Section 3.2 and Appendix N.1). This let us match the initial error scenario across trajectories while varying the outcome. For instance, after dropping the teapot lid the robot can leave it, retrieve and seat it, or retrieve and drop it again (Appendix F.2). A model that increases its reward for both retries but keeps rewarding the second after the lid falls has missed a distinction that matters to a recovery policy. The controlled branches give us examples of each outcome even when a particular policy rarely produces one of them. In the next benchmark release, we plan to add failures from autonomous policy rollouts so we can also examine the errors that arise under learned behavior. Failure and recovery protocols.
The metrics address different decisions. SA and TGA test whether a sparse reward distinguishes success and instruction fulfillment. For data screening, VS helps flag expert trajectories whose predicted values oscillate or stay flat. GPT-6.1 Sol scores 96.23 on in-domain Cycle-VOC, a relevant check for dense rewards when progress is undone. Flora has a smaller FPL error (25.46 versus 29.72 for GPT-6.1 Sol), which matters for an execution monitor that must identify when the failure began, or for a curator retaining the valid episode prefix. Such boundaries are also relevant to intervention data of the kind collected in HG-DAgger. We have not measured whether a higher benchmark score predicts higher downstream RL return. Testing that relationship would require training otherwise matched policies with different reward models and measuring success in autonomous rollouts. Kelly et al., HG-DAgger.
Q1. How the episode score is computed
Thank you for asking how we turn frame-level values into an episode score and how the benchmark can inform reward-model selection. We revised both the main paper and the supplementary material and combined them in a single PDF.
We added evaluations of four general-purpose frontier models (GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, and DeepSeek-V4.1-Flash), all with eight-frame inputs. Section 4.1 describes the 24 model configurations. Table 1 reports in-domain scores and Table 2 reports results under embodiment and environment shifts. We clarified episode scoring in Section 3.4 and Appendix K.1, added random, constant, time-only, and annotation-based reference curves in Appendix G.3 (Tables 10–11), and compared RoboValue with 23 prior benchmarks and evaluation suites in Appendix B (Table 4).
Q1. How do per-step values determine the episode-level score?
For Success Accuracy (SA), the episode score is the arithmetic mean of the sampled values over the final 2% of the episode duration. This small terminal window smooths fluctuations around the final state. Earlier values have zero weight. For example, a large peak when the robot grasps the lid cannot raise the score if the model assigns low values throughout the terminal window after the lid is dropped.
Within each task and evaluation setting, we compare every successful episode with every partially successful or failed episode. SA is the fraction of these pairs for which the successful episode has a strictly higher terminal mean. A tie earns zero credit.
Task Grounding Accuracy (TGA) uses a different comparison because its question is whether the same video fulfills a given instruction. We average pairwise value gains between observations in the initial and final 2% of that video, once per candidate instruction. The correct instruction must have a larger mean gain than every alternative. Section 3.4 states both rules, and Appendix K.1 gives their equations.