- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces RoboRef, a visual reward model designed to improve downstream robot reinforcement learning by reducing reward overprediction on unsuccessful behaviors. Building on Robometer, it introduces dense progress and success supervision for failed executions using simulator state, a two-stage VLM–LLM annotation pipeline, and partial-execution heuristics. It also introduces an asymmetric training objective that penalizes overprediction more strongly than underprediction.
Validation includes offline reward benchmarks, four simulated RL tasks across three suites, and two real robot tasks. A matched symmetric-loss control isolates the effect of the asymmetric objective, while on-policy analyses and an oracle intervention investigate reward hacking. The results show improved downstream policy learning despite weaker performance on some conventional offline metrics, supporting the importance of evaluating reward models under policy optimization.
Critical strengths
Useful empirical evidence identifying a consequential reward-model failure under policy optimization.
The combination of on-policy reward analysis and a targeted oracle intervention is a strength of the paper. The authors observe that unsuccessful policy trajectories can receive substantial predicted reward, then investigate whether correcting these errors can restore learning. In Appendix G.3, they retain Robometer and the SAC training setup but use simulator ground truth to suppress success-level rewards at genuinely unsuccessful states, leaving other predictions unchanged. This intervention raises true task success on PullCube from approximately 0% to 92%, close to the 94% obtained with the environment’s ground-truth dense reward.
This result provides concrete evidence that correcting a particular class of reward overestimation can be sufficient to recover effective policy learning in the evaluated setting. It also helps explain why reward-model accuracy on a fixed offline dataset may not adequately characterize the model’s usefulness when a policy actively optimizes its predictions. Although the intervention requires privileged success labels and does not establish a deployable correction method, it strengthens the paper’s explanation of the observed RL failures and motivates evaluating reward models on the behaviors encountered during policy optimization.
Critical weaknesses
-
Limited methodological novelty and insufficient evidence for the distinctive value of the failure-curation pipeline.
The importance of incorporating failed executions into reward-model training is already recognized in the cited literature, and RoboRef largely inherits Robometer’s architecture. Its more specific contribution is therefore the direct, dense supervision of progress on failed trajectories. However, the annotation pipeline mainly combines existing vision-language and language models with rubric-based scoring, and the paper does not sufficiently establish why this particular approach represents a substantial methodological advance over simpler failure-supervision alternatives, of which some of their cited baselines also do.This concern is compounded by incomplete isolation of the curation contribution. Although the matched symmetric-loss control appropriately isolates the asymmetric objective, the comparison with original Robometer also changes the frame budget and removes the preference head (§3.4). Comparisons using the same failed trajectories, architecture, and training budget but different supervision schemes would better establish the value of the detailed progress annotations. Furthermore, the annotation audit is limited to 50 DROID clips and is also used to select the labeling model (Appendix C.3), leaving uncertainty about reliability across the wider corpus. Given the incremental modeling changes, stronger evidence for the necessity and reliability of the proposed supervision is important to assessing the contribution.
-
Insufficient justification of the conservative objective and an overly broad claim about false-negative costs.
The asymmetric objective is motivated by the assertion that reward underprediction “only slows learning” (§3.3). The oracle intervention provides meaningful evidence that false-positive rewards can prevent learning, but it does not establish the comparative claim that false negatives are consistently less consequential. Systematic missed successes can remove the only positive learning signal, while missed or delayed detections can prevent a required transition or allow continued actions to invalidate completion. The current tasks largely involve observable manipulation outcomes and do not systematically test these situations.The implementation of the proposed tradeoff is also insufficiently characterized. The paper uses a shared λ = 0.3 for the progress and success objectives, compares it with λ = 1, and provides no broader sensitivity analysis or explanation of the selected value. Although offline error decompositions and model-specific threshold calibration are reported, the consequences of missed successes at the actual online operating points remain unclear. The paper therefore demonstrates that its chosen tradeoff works well in the evaluated settings, but does not adequately establish when conservative scoring remains beneficial across tasks with different costs of missed or delayed completion. Because this error-cost assumption is central to the method’s motivation, its limited validation weakens the broader argument for the proposed objective.
Questions for authors
1. Scope and consequences of false-negative success detection
The paper motivates its asymmetric objective by stating that reward underprediction "only slows learning" (Section 3.3). Under what assumptions is this claim expected to hold? In tasks where success is transient, cannot be reattempted, or requires a timely termination or transition, missed detections could eliminate the only positive learning signal or cause an otherwise successful episode to fail.
Can the authors provide a targeted evaluation involving such tasks, particularly where success cannot be recovered or recognized from later observations? How does RoboRef compare with the baselines in terms of missed detections, detection latency, and downstream task success in these settings? Such evidence would help establish whether the proposed preference for avoiding false positives generalizes beyond the currently evaluated task structures.
2. Selection and robustness of the asymmetric loss weight
How was the asymmetric loss weight of λ = 0.3 selected, and why is the same value used for both the progress and success objectives? The paper compares λ = 0.3 against the symmetric setting λ = 1, but does not report a broader sensitivity analysis.
Have the authors experimented with different loss weights, and how do these affect the joint false-positive/false-negative tradeoff and downstream RL performance? In particular, does λ = 0.3 consistently provide a favorable tradeoff across different tasks, or would task-specific or head-specific weights perform better? An evaluation across multiple values of λ, using a consistent threshold-calibration procedure and reporting error rates at the actual deployed operating points, would help establish the robustness and generality of the proposed objective.
3. Reliability and validation of the automatic failure-annotation pipeline
Appendix C.3 evaluates the selected VLM–LLM annotation pipeline using only 50 DROID failure clips (400 sampled frames), with the same benchmark also used for model selection. However, Appendix A (Tables 5 and 7) indicates that the pipeline generates dense progress labels for 74,426 failure clips spanning DROID, RACER, RoboArena, SOAR, LIBERO, AutoEval, and other sources.
What evidence supports the reliability of this pipeline across these heterogeneous datasets? Is there an independent, source-stratified human evaluation beyond the DROID benchmark? Furthermore, how does the proposed two-stage annotation pipeline compare quantitatively with simpler direct VLM scoring or suitable annotation approaches from prior work when evaluated on the same manually labeled examples?
Given that the authors acknowledge hallucination-induced labeling errors, it would also be useful to understand whether these errors systematically overestimate or underestimate progress, particularly near completion, and whether the downstream improvements are robust to such annotation noise.