acceptodds
Withdrawn from ICLR 2027

RoboRef: A Foundation Reward Model Built for Robot RL, Not Offline Metrics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) offers a general route for robots to learn to solve tasks from their own experience. However, designing the reward that guides this learning remains a bottleneck. Recent work therefore replaces hand-crafted reward functions with reward models fine-tuned from vision-language models (VLMs) on large robotic datasets, reporting strong prediction accuracy on offline benchmarks. Yet whether this accuracy carries over to successful RL training remains underexplored. We identify two gaps shared by most VLM reward models. First, the annotated robot data these models train on mostly contain successful behavior, which leaves them poorly equipped to judge the failures produced by a policy being optimized. Second, most models penalize over- and under-prediction of the reward equally, although false positives are the main cause of reward hacking, which poses a severe problem for downstream RL. We introduce RoboRef to close these gaps. To train RoboRef, we re-curate RBM-1M, a large-scale robot reward dataset, densely annotating its failures and adding new ones so that the model learns to score the failures it might encounter during RL. Furthermore, we introduce an asymmetric loss that penalizes over-prediction more harshly than under-prediction, yielding conservative rewards that are harder for a policy to exploit. We evaluate on offline metrics and downstream RL across three simulated benchmarks and two real-world manipulation tasks. We observe that the models that lead on offline benchmarks can fail to train a policy at all. RoboRef trains competent policies where these baselines never learn and delivers the strongest downstream performance, making it, to our knowledge, the state-of-the-art VLM reward model for robot RL.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Reinforcement Learning, Reward Modeling, Vision-Language Models, Reward Hacking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WDNmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces RoboRef, a visual reward model designed to improve downstream robot reinforcement learning by reducing reward overprediction on unsuccessful behaviors. Building on Robometer, it introduces dense progress and success supervision for failed executions using simulator state, a two-stage VLM–LLM annotation pipeline, and partial-execution heuristics. It also introduces an asymmetric training objective that penalizes overprediction more strongly than underprediction.

Validation includes offline reward benchmarks, four simulated RL tasks across three suites, and two real robot tasks. A matched symmetric-loss control isolates the effect of the asymmetric objective, while on-policy analyses and an oracle intervention investigate reward hacking. The results show improved downstream policy learning despite weaker performance on some conventional offline metrics, supporting the importance of evaluating reward models under policy optimization.

Critical strengths

Useful empirical evidence identifying a consequential reward-model failure under policy optimization.

The combination of on-policy reward analysis and a targeted oracle intervention is a strength of the paper. The authors observe that unsuccessful policy trajectories can receive substantial predicted reward, then investigate whether correcting these errors can restore learning. In Appendix G.3, they retain Robometer and the SAC training setup but use simulator ground truth to suppress success-level rewards at genuinely unsuccessful states, leaving other predictions unchanged. This intervention raises true task success on PullCube from approximately 0% to 92%, close to the 94% obtained with the environment’s ground-truth dense reward.

This result provides concrete evidence that correcting a particular class of reward overestimation can be sufficient to recover effective policy learning in the evaluated setting. It also helps explain why reward-model accuracy on a fixed offline dataset may not adequately characterize the model’s usefulness when a policy actively optimizes its predictions. Although the intervention requires privileged success labels and does not establish a deployable correction method, it strengthens the paper’s explanation of the observed RL failures and motivates evaluating reward models on the behaviors encountered during policy optimization.

Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for the distinctive value of the failure-curation pipeline.
    The importance of incorporating failed executions into reward-model training is already recognized in the cited literature, and RoboRef largely inherits Robometer’s architecture. Its more specific contribution is therefore the direct, dense supervision of progress on failed trajectories. However, the annotation pipeline mainly combines existing vision-language and language models with rubric-based scoring, and the paper does not sufficiently establish why this particular approach represents a substantial methodological advance over simpler failure-supervision alternatives, of which some of their cited baselines also do.

    This concern is compounded by incomplete isolation of the curation contribution. Although the matched symmetric-loss control appropriately isolates the asymmetric objective, the comparison with original Robometer also changes the frame budget and removes the preference head (§3.4). Comparisons using the same failed trajectories, architecture, and training budget but different supervision schemes would better establish the value of the detailed progress annotations. Furthermore, the annotation audit is limited to 50 DROID clips and is also used to select the labeling model (Appendix C.3), leaving uncertainty about reliability across the wider corpus. Given the incremental modeling changes, stronger evidence for the necessity and reliability of the proposed supervision is important to assessing the contribution.

  2. Insufficient justification of the conservative objective and an overly broad claim about false-negative costs.
    The asymmetric objective is motivated by the assertion that reward underprediction “only slows learning” (§3.3). The oracle intervention provides meaningful evidence that false-positive rewards can prevent learning, but it does not establish the comparative claim that false negatives are consistently less consequential. Systematic missed successes can remove the only positive learning signal, while missed or delayed detections can prevent a required transition or allow continued actions to invalidate completion. The current tasks largely involve observable manipulation outcomes and do not systematically test these situations.

    The implementation of the proposed tradeoff is also insufficiently characterized. The paper uses a shared λ = 0.3 for the progress and success objectives, compares it with λ = 1, and provides no broader sensitivity analysis or explanation of the selected value. Although offline error decompositions and model-specific threshold calibration are reported, the consequences of missed successes at the actual online operating points remain unclear. The paper therefore demonstrates that its chosen tradeoff works well in the evaluated settings, but does not adequately establish when conservative scoring remains beneficial across tasks with different costs of missed or delayed completion. Because this error-cost assumption is central to the method’s motivation, its limited validation weakens the broader argument for the proposed objective.

Questions for authors

1. Scope and consequences of false-negative success detection

The paper motivates its asymmetric objective by stating that reward underprediction "only slows learning" (Section 3.3). Under what assumptions is this claim expected to hold? In tasks where success is transient, cannot be reattempted, or requires a timely termination or transition, missed detections could eliminate the only positive learning signal or cause an otherwise successful episode to fail.

Can the authors provide a targeted evaluation involving such tasks, particularly where success cannot be recovered or recognized from later observations? How does RoboRef compare with the baselines in terms of missed detections, detection latency, and downstream task success in these settings? Such evidence would help establish whether the proposed preference for avoiding false positives generalizes beyond the currently evaluated task structures.

2. Selection and robustness of the asymmetric loss weight

How was the asymmetric loss weight of λ = 0.3 selected, and why is the same value used for both the progress and success objectives? The paper compares λ = 0.3 against the symmetric setting λ = 1, but does not report a broader sensitivity analysis.

Have the authors experimented with different loss weights, and how do these affect the joint false-positive/false-negative tradeoff and downstream RL performance? In particular, does λ = 0.3 consistently provide a favorable tradeoff across different tasks, or would task-specific or head-specific weights perform better? An evaluation across multiple values of λ, using a consistent threshold-calibration procedure and reporting error rates at the actual deployed operating points, would help establish the robustness and generality of the proposed objective.

3. Reliability and validation of the automatic failure-annotation pipeline

Appendix C.3 evaluates the selected VLM–LLM annotation pipeline using only 50 DROID failure clips (400 sampled frames), with the same benchmark also used for model selection. However, Appendix A (Tables 5 and 7) indicates that the pipeline generates dense progress labels for 74,426 failure clips spanning DROID, RACER, RoboArena, SOAR, LIBERO, AutoEval, and other sources.

What evidence supports the reliability of this pipeline across these heterogeneous datasets? Is there an independent, source-stratified human evaluation beyond the DROID benchmark? Furthermore, how does the proposed two-stage annotation pipeline compare quantitatively with simpler direct VLM scoring or suitable annotation approaches from prior work when evaluated on the same manually labeled examples?

Given that the authors acknowledge hallucination-induced labeling errors, it would also be useful to understand whether these errors systematically overestimate or underestimate progress, particularly near completion, and whether the downstream improvements are robust to such annotation noise.

Reviewer_xQG2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboRef fine-tunes Robometer using dense failure supervision and asymmetric losses that penalize reward overestimation more strongly. Evaluation covers four simulated RL tasks and two real-robot tasks, together with offline metrics and reward-hacking diagnostics. The paper reports improved downstream learning and argues that conventional offline accuracy does not reliably predict reward-model utility under policy optimization.

Critical strengths
  1. Direct downstream RL evaluation provides practically meaningful evidence beyond offline prediction metrics.
  2. The matched symmetric-loss control demonstrates a substantial benefit from asymmetric training on the augmented dataset.
Critical weaknesses
  1. The necessity of both ingredients is not established. The same-data control supports asymmetric training, but comparison with the released Robometer also changes training budget, frame count, and heads. Without a matched asymmetric model lacking the added failure supervision, the claim that both ingredients are necessary remains insufficiently supported.
  2. Important evaluation details remain unresolved. Appendices E.3 and G.2 describe different ManiSkill protocols for RoboReward, while calibration-set sizes, false-positive budgets, and per-model thresholds are unspecified. Since false detections and termination are central to the paper’s explanation, these details are necessary to interpret the comparisons.
Questions for authors

Additional Limitations

  1. The oracle experiment supports a narrower causal conclusion. Clipping rewards at unsolved states introduces ground-truth success separation, so the resulting improvement does not independently establish that the original progress estimates are sufficient. A constant-unsolved-reward or matched oracle sparse-reward control would distinguish these explanations.
  2. On-policy comparisons use different state distributions. Each reward model is scored on trajectories generated by its own trained policy, so AUROC differences reflect both reward predictions and visited states. Evaluating all models on a shared trajectory pool would better isolate reward-model quality.
  3. The offline–online correlation has limited scope. The analysis involves only six models, including three sharing Robometer initialization, and partly mismatched offline and online input protocols. It supports limited predictive value in the tested settings, rather than a general inverse relationship between offline accuracy and RL performance.
  4. Failure-label quality needs stronger validation. Annotation accuracy is measured on the same 50 DROID clips used to select the labeling model, without an independent cross-source audit. Clearer treatment of temporary completion, later failure, and progress interpolation is also needed to assess target consistency and possible labeling artifacts.
  5. Evidence for broader deployment remains limited. The real-robot study covers two tasks with one training run per model and lacks the matched symmetric-loss control. Different frame budgets, scripted gripper actions, and the absence of the advertised held-out-robot results further limit broader deployment claims.

Questions For Authors

  1. Can matched data/objective ablations and sensitivity to the asymmetric weight establish which components drive the gains?
  2. What exact ManiSkill protocol and calibration procedure were used for each reward model?
  3. Does the oracle result persist against a constant-unsolved-reward control, and do reward-quality differences persist on shared trajectories?
Reviewer_fxW3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper attempts to improve generalist reward models. It does two things: (1) annotate robot failure data and use it for training, and (2) penalize overestimation of reward more heavily than underestimation to avoid reward hacking.

Critical strengths
  • It would be a very nice dataset contribution.
Critical weaknesses

I don't see a scientific contribution in the paper. It would be a nice dataset contribution because apparently authors annotated a lot of failure data, which is definitely good and useful for the community, but it is not written as a dataset paper. On the method side, we are looking at an earlier method, namely Robometer, and the method of this paper, which is the same method except they annotated more data and use it. It is true that Robometer does not utilize failure data to train progress head, but how to do it is obvious. So the conclusion is: when you have more annotations/supervision, the model works better. This is obvious.

The other contribution is that the model is trained in a way overestimation of rewards is penalized more heavily than underestimation. Again, this is not a novel technique. If one cares more about avoiding overestimation, this is how training would change.

Other minor comments:

  • Introduction claims "[Robometer] supervises its reward predictions only on successful trajectories." This is not true. Later parts of the paper makes the statement clearer and correct, but this sentence is incorrect. Indeed, failure trajectories do not train the progress head, but it doesn't mean they don't train reward predictions, because they train the backbone of the model through preference loss.
  • The paper has two citations for "...manufacture negatives by modifying successful videos (Lee et al., 2026; Xu et al., 2026)." While these citations are not wrong, as far as I know, the original idea came from Zhang et al.'s ReWiND paper.
Questions for authors

What is a novel scientific insight one should gain from this paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.