- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Overall, this paper's main contribution is an evaluation protocol for reward-based robot learning. This paper first highlights that purely visual rewards from off-the-shelf Robometer occasionally gives reward to the wrong action. The main contribution of the paper as I understand it is a more specific evaluation benchmark that better captures the issues with visual rewards. The experiments are validated in sim on a series of drawer opening and block stacking tasks, where failure largely comes from opening the wrong drawer (out of a stack of 3), or stacking the wrong block.
Critical strengths
To my understanding, the paper's biggest finding is that there exists a set of cases where reward hacking is not directly correlated with task success. This by itself is interesting, albeit weak as a core contribution (please see below), as it has been highlighted in previous work already. Nevertheless, I think the paper does a good job designing an evaluation protocol to elicit such behavior. There is some degree of multi-facetted analysis on these environments, looking down at the specific failure cases, which is good.
Critical weaknesses
My biggest concern about this paper is it's central contribution or novelty. As I understand it, the paper's core finding is that vision-based reward models alone are not sufficient, and set forth an evaluation protocol to highlight this fact. However, I do not see a concrete proposal nor actionable takeaway from this finding. I don't believe this paper presents a general insight that can be taken towards the broader community. It is unclear to me how much additional finding there is in this paper beyond what is already documented in RoboReward, which the authors concede in lines 137-138.
Continuing the last point, I have concerns to the generality of these results. Indeed, the simulation benchmarks are sparse, and real world reproductions themselves are not convincing. I highly suggest more thoroughness and rigor associated with the real robot results, and these should all be described in more detail. RoboReward in particular should be tested in the real world, as that is where a lot of the training data comes from. There are several scientific questions along this line left unanswered.
Questions for authors
- How general are these findings? Have you evaluated on general, public benchmarks? Have you evaluated on real-world setups?
- Why do the authors believe there is this degree of reward hacking? Is it a function of the training data, or of the architecture, or of the training objective? I would love to hear a more concrete takeaway that authors have towards how they might suggest training such reward models in the future.
4 comments
Response to Reviewer t9Fx (2/2)
W2 and Q1. Generality, public benchmarks and real-world evaluation. Coverage in the revision is as follows. The full-denoiser experiments cover two tasks (drawer and stacking), two reward models on the drawer task (Robometer and RoboReward-8B), two initializations and two samplers. The constrained experiments cover the same two tasks with two critics (Robometer and LIV) and two optimizers (PPO and GRPO), and 11 of 20 training histories amplify wrong-object failures in at least one stratum (Section 6, Appendix F). The tilt model accounts for which settings amplify and which do not (Spearman .89 across 26 settings, with the strength matched to each setting's reward gain), and it predicted the stacking outcomes before training.
We have not evaluated on a public benchmark. The protocol needs a distractor that affords the same action as the target, and outcome labels for both objects from simulator state. Most public manipulation benchmarks score task success only, so we built the two tasks to label wrong-object outcomes from simulator state. The check itself needs only rollouts, outcome labels and reward scores, so it runs on any benchmark whose evaluator labels wrong-object outcomes (Section 7).
On the real world: the revision adds a hardware study whose comparison was fixed before the first trial (Section 4, Table 2, Appendix E.1). The supervised initialization, three Robometer training runs and three completion training runs, all fine-tuned only in simulation, ran 120 trials per condition on a Franka Research 3. Robometer training raised wrong-object failures from 8.3% to 24.2% of trials and completion training left them at 10.0% (difference 14.2 [4.2, 25.0] points); success rose from 10.8% to 17.5% and 20.0%.
RoboReward in particular should be tested in the real world, as that is where a lot of the training data comes from.
The three RoboReward-8B training runs of Table 1 ran 120 trials on the same setup and reached 20.8% wrong-object failures against 10.0% after completion training (10.8 [1.7, 20.0] points, the comparison fixed before their first trial; Appendix E.3). The wrong-object behavior that optimization produced in simulation therefore carries over to the robot for both reward models. Videos of every hardware condition: https://huggingface.co/datasets/ICLR2027-Anonymous-42152/drawer-hardware-demos (also in the supplementary material). We agree that reward models trained mostly on real video should also be audited on real video, and the hardware study now does this for Robometer. On video of the 120 hardware trials of the initialization, Robometer scores wrong-object failures at .59 against a mean of .10 over all trials (difference .50 [.31, .67]), so the condition under which optimization amplifies them also holds on real frames, and the check predicts the amplification that the hardware trials then show.
Q2. Why does the hacking occur, and what follows for training reward models? We separate two levels.
At the level of the objective, the amplification depends only on how the reward scores the outcome classes on the initial policy's rollouts. It does not depend on the reward model's architecture. Any reward under which a coherent failure class scores above the rollouts it competes with will have that class amplified.
At the level of the model, Robometer reads the task category but does not ground which object the instruction names. We rescored 3,584 stored executions with only the instruction changed (Appendix G.5). An unrelated instruction ("pick up the cup") collapses every score to about .01. Naming the distractor drawer instead of the target raises the scores of executions that opened the requested drawer (.68 to .75) and barely changes those that opened the distractor (.62 to .60), so the score difference between the two instructions ranks successes below wrong-object failures (AUROC .19 [.17, .22]). Under the instruction "open the top drawer", RoboReward-8B still scores rollouts that opened the middle drawer above those that opened the top drawer (4.91 vs 4.83 on its 1 to 5 scale, Appendix G.5), so neither reward model, despite their different architectures, reliably grounds the referenced object. We read this as pointing to the training data more than to the architecture: two models with different architectures have learned what the action looks like, and neither has learned which object it should be applied to.
For training reward models we would suggest three things; the revision adds the first to Section 8. Include hard negatives that show the same scene and the same action on a different object, paired with the instruction. Evaluate on rollouts of the policies the model will train, and report the mean score of each coherent failure class next to the aggregate AUROC. Do not rely on contrasting instructions at scoring time as a repair: in our data it ranks the outcomes backwards.
I appreciate the care and attention authors took to rebut, especially the breadth and depth of additional experiments conducted. I appreciate the added real-robot experiments, and the clarifications on the takeaway. It may be the consequence of the language, writing, and figures, but the takeaways presented, and the style in which they are presented still do not convince this paper is ready for conference acceptance.
For example, the frequently referenced Appendix H describes the weighting and selection process in a very task specific manner, at least from what I understand (ln 1747-ln1748). In many ways, this feels like retroactive fitting or environment specific hyperparameter search paralleled in sim. Furthermore, the real-world experiments added, as I understand, test separate issues -- it's policies were optimized (and as my preconception suggests) heavily searched over in simulation, then distill to a policy in real. This is less of a demonstration of the policy working in real, and more of a separate "We can search in sim" and "We can do sim2real" argument.
My question about the implications for reward-model training also remains only partially addressed. The instruction interventions provide evidence of inadequate object grounding, but do not establish whether this arises from the training data, architecture, or objective. In particular, RoboReward already incorporates counterfactual instruction relabeling to generate negative examples. What specific deficiency in that supervision do the authors identify, and what evidence supports the proposed same-scene hard negatives as addressing it? I view this recommendation as a plausible hypothesis rather than an experimentally established takeaway.
Overall, I recognize the substantial improvements in the revision, but will maintain recommendation. I think more care should be given to the writing, figures, and presentation, and would strongly encourage the authors to check carefully the outputs of the agents used in their work. I still feel that the current idea is too environment and modeling specific, and thus their studied setting lacks convincing arguments beyond the tasks searched for in .
Thank you for reading the revision closely and for recognizing the new experiments and the real-robot evaluation. We agree on the presentation. On rereading, Appendices A to D read like engineering notes, so we rewrote them in plain prose. The updated PDF also rewrites the abstract, introduction and conclusion as one line of argument and rewords the sentences in Sections 3 to 7 that were hard to follow. Our answers to your other concerns follow.
Hardware. Nothing was searched over to choose what runs on the robot, and there is no distillation step. Teacher-student distillation in sim-to-real work turns a policy trained on privileged state into one that acts from images. Our policies act from images throughout: two camera views and the robot's proprioceptive state, and neither the policy nor the value network receives simulator object state during fine-tuning (Appendices A and B.1). The robot therefore runs the same weights that were fine-tuned in simulation, with no training on the robot: the first three training runs of each reward in Table 7, taken by index, each at iteration 40. The comparison was fixed before the first trial, and all 360 trials are counted (Appendix E.1, Table 10). The submitted version had a single-run hardware study in which the run was chosen after the simulated evaluation, and it resolved no difference. The replicated study replaces it as the main hardware evidence for that reason, and the single-run study remains in Appendix E.
Absolute rates differ between simulation and hardware, as expected for policies transferred without fine-tuning: the initialization succeeds in 10.8% of hardware trials against 17.2% in simulation, and its wrong-object rate is 8.3% against 14.1%. What carries over is the direction of the difference between the two rewards, in a physical scene built to match the simulated one at 1:1 scale. We agree that this tests how policies trained in simulation transfer, and the paper makes no claim of reinforcement learning on the robot. The check itself also runs on real frames: over the 120 hardware trials of the initialization, Robometer scores wrong-object failures at .59, against .34 for successes and .10 over all trials.
Appendix H. You are right that β in that table is set after training: its caption says β is chosen to match the reward gain each family reached. β enters only the magnitude predictions of Figure 4 and Table 20, and the text presents the full-denoiser ones as a first-order description. The predictions the paper relies on do not use β. The check compares the mean reward of wrong-object executions with that of the executions they compete with, all on the initial policy's rollouts (Section 7), so it has no fitted quantity. In stacking, each prediction and the comparison that tests it were committed before that placement's training runs started (Appendix I). Seven of the nine predictions in Table 3 were committed before the training runs began, and all nine held, including the settings where the check predicted no amplification.
Reward-model training. We agree that the recommendation is a hypothesis, so let us separate what is measured from what is suggested. What is measured: when the instruction names the top drawer, RoboReward-8B still scores executions that opened the middle drawer above executions that opened the top drawer, 4.91 against 4.83 on its 1 to 5 scale, and Robometer does the same (Appendix G.5). This holds with RoboReward's counterfactual relabeling in place. Its relabeling prompt lists handling a different object as the example of a no-success command, so supervision of this kind exists. The RoboReward paper does not report how often a relabeled command names another instance of the same category in the same scene, such as a second drawer of one cabinet, which is the case we measure. Our results do not establish whether the cause lies in the data, the architecture or the objective. Two models that differ in architecture and objective both fail to reliably follow which drawer the instruction names, and that is why Section 8 suggests looking at the data first. Same-scene hard negatives are our suggestion for closing this gap, and we have not trained a reward model with them.
AI tools. We used AI tools for code, analysis, figures and editing, as the paper's AI Assistance Statement says, and we are responsible for every claim. Every number in the tables is generated by scripts from stored result files, and the numbers in the text were checked against them. If any passage, figure or number still reads poorly or seems inconsistent, please let us know and we will gladly fix it.
Response to Reviewer t9Fx (1/2)
Thank you for the review. We answer the two weaknesses and the two questions; W2 and Q1 are answered together.
W1. Central contribution, actionable takeaway, and the increment over prior work. The Robometer paper documented that RoboReward gives high scores to wrong-object executions and that these false positives reinforce incorrect behavior. Our paper adds four things.
The actionable takeaways follow. Before optimizing a policy against a visual reward, score the initial policy's rollouts and compare each semantically coherent failure class with the alternative the policy can trade it against; Section 7 now states this as a four-step check. It needs outcome labels for one evaluation pass of the initial policy only, 120 to 512 executions in our studies. Without a simulator a person can label that many from video; a single training run consumes 5,040 executions, and labeling those is what a learned reward exists to replace. Across the seven full-denoiser studies of Sections 4 and 6 whose reward has no outcome supervision, this check flagged all five in which wrong-object failures rose and neither of the two in which they did not; five of the predictions were committed before training (table in the general response). When evaluating a reward model, report the mean score of coherent failure classes on rollouts of the policies it will train, not only an aggregate AUROC. During optimization, monitor outcomes with a measurement that does not come from the reward. When training in simulation, a flagged reward can be repaired without labels: put back the objects the instruction does not name, re-render each execution, and score it again. The check clears this reward, and training on it removes the amplification: no wrong-object stack in the placement where Robometer raised them from 20% to 69%, wrong-object failures on the drawer task no different from the completion reward (0.0 [−2.9, 2.9] points), and on the robot 13.3 [3.3, 23.3] points fewer than the Robometer policies with the same success (Section 7, Tables 2 and 3, Appendices E.2 and K).