acceptodds
Under review as a conference paper at ICLR 2027

CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization

Abstract

Learned visual rewards guide robot policy optimization, but can assign high scores to executions that act on the wrong object without completing the requested task. We present CriticHack, a closed-loop study of visual reward optimization in simulated drawer manipulation and identity-sensitive stacking. Our central finding is that optimizing a learned visual reward can increase wrong-object failure rates even as reward and task success improve. In a supervised-initialization study, three full-denoiser fine-tuning runs of a diffusion policy with no prior reward exposure, optimized against Robometer on a drawer task, raise simulator-measured success and wrong-object failure rates by 9.6 and 11.8 percentage points (95% CIs [6.2, 13.0] and [8.1, 15.6]) on 512 evaluation seeds, alongside higher reward. A warm-start study shows the same pattern from a policy previously optimized against learned rewards: across five runs, success and wrong-object failures rise by 8.4 and 3.8 points on average. Constrained-policy experiments, which optimize only a selector over fixed diffusion-noise sequences, reproduce reward gain and wrong-object amplification across two task–reward pairings and two optimizers. Reward-ranking analysis further shows that strong aggregate success–failure discrimination can conceal semantic misranking. On collected trajectories, Robometer assigns higher average scores to wrong-object failures than to successes; strong discrimination against other failures masks this scoring error. As a diagnostic intervention, reweighting the critic reward with a frozen verifier trained on task-specific outcome labels improves task completion and reduces wrong-object failures relative to critic-only training under the fine-tuning sampler. Verifier-only controls identify task-specific outcome supervision as a substantial source of recovery. These results show that gains in reward and task success can conceal the amplification of specific semantic failures.

Primary area:
applications to robotics, autonomy, planning
Keywords:
reward hacking, learned visual rewards, reward model evaluation, robot learning, policy optimization, diffusion policies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_t9FxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Overall, this paper's main contribution is an evaluation protocol for reward-based robot learning. This paper first highlights that purely visual rewards from off-the-shelf Robometer occasionally gives reward to the wrong action. The main contribution of the paper as I understand it is a more specific evaluation benchmark that better captures the issues with visual rewards. The experiments are validated in sim on a series of drawer opening and block stacking tasks, where failure largely comes from opening the wrong drawer (out of a stack of 3), or stacking the wrong block.

Critical strengths

To my understanding, the paper's biggest finding is that there exists a set of cases where reward hacking is not directly correlated with task success. This by itself is interesting, albeit weak as a core contribution (please see below), as it has been highlighted in previous work already. Nevertheless, I think the paper does a good job designing an evaluation protocol to elicit such behavior. There is some degree of multi-facetted analysis on these environments, looking down at the specific failure cases, which is good.

Critical weaknesses

My biggest concern about this paper is it's central contribution or novelty. As I understand it, the paper's core finding is that vision-based reward models alone are not sufficient, and set forth an evaluation protocol to highlight this fact. However, I do not see a concrete proposal nor actionable takeaway from this finding. I don't believe this paper presents a general insight that can be taken towards the broader community. It is unclear to me how much additional finding there is in this paper beyond what is already documented in RoboReward, which the authors concede in lines 137-138.

Continuing the last point, I have concerns to the generality of these results. Indeed, the simulation benchmarks are sparse, and real world reproductions themselves are not convincing. I highly suggest more thoroughness and rigor associated with the real robot results, and these should all be described in more detail. RoboReward in particular should be tested in the real world, as that is where a lot of the training data comes from. There are several scientific questions along this line left unanswered.

Questions for authors
  1. How general are these findings? Have you evaluated on general, public benchmarks? Have you evaluated on real-world setups?
  2. Why do the authors believe there is this degree of reward hacking? Is it a function of the training data, or of the architecture, or of the training objective? I would love to hear a more concrete takeaway that authors have towards how they might suggest training such reward models in the future.
4 comments
AuthorsOpenReview ↗

Response to Reviewer t9Fx (1/2)

Thank you for the review. We answer the two weaknesses and the two questions; W2 and Q1 are answered together.

W1. Central contribution, actionable takeaway, and the increment over prior work. The Robometer paper documented that RoboReward gives high scores to wrong-object executions and that these false positives reinforce incorrect behavior. Our paper adds four things.

  1. The model built to improve on RoboReward has the same error, and standard metrics hide it. Over 20,992 executions Robometer separates successes from all failures with AUROC .813 [.789, .836] and ranks wrong-object failures above successes (AUROC .372 [.333, .410]). Other failures are three quarters of the negatives and are separated almost perfectly, so the aggregate number looks strong (Section 5, Appendix B.5).
  2. Optimization amplifies the error while task success rises, so the success rate after optimization does not reveal it. From a supervised initialization, five Robometer training runs raise success by 10.2 [6.9, 13.6] points and wrong-object failures by 10.9 [7.3, 14.4]. Five training runs on the simulator's completion signal raise success by 13.9 [9.5, 18.3] and wrong-object failures by 1.6 [−1.9, 5.2]. Three RoboReward-8B training runs show the same pattern (Section 4, Table 1).
  3. A model of when this happens, tested before training. An outcome grows relative to the rollouts it competes with when its mean reward on the initial policy's rollouts exceeds theirs. On a second task, the model named in advance the one object placement of three in which Robometer amplifies wrong-object stacks (20% to 69%); in the other two they fell to 0% (Section 6, Figure 3).
  4. Robometer's published online-RL recipe, a per-step penalty plus predicted progress with termination when the success probability exceeds .6, inherits the error. On 4,464 executed episodes its success detector fires on 89% of wrong-object failures and 73% of successes (Section 4), and the recipe's return ranks wrong-object failures above successes more strongly than the terminal probability does (AUROC .125 against .195; Appendix J, Table 24). We also trained with the recipe as the reward. Three full-denoiser training runs from the supervised initialization raise wrong-object failures by 11.1 [7.7, 14.5] points, 9.2 [5.6, 12.8] more than the eight completion training runs, and the check had predicted amplification before training (Section 4, Tables 1 and 3).

The actionable takeaways follow. Before optimizing a policy against a visual reward, score the initial policy's rollouts and compare each semantically coherent failure class with the alternative the policy can trade it against; Section 7 now states this as a four-step check. It needs outcome labels for one evaluation pass of the initial policy only, 120 to 512 executions in our studies. Without a simulator a person can label that many from video; a single training run consumes 5,040 executions, and labeling those is what a learned reward exists to replace. Across the seven full-denoiser studies of Sections 4 and 6 whose reward has no outcome supervision, this check flagged all five in which wrong-object failures rose and neither of the two in which they did not; five of the predictions were committed before training (table in the general response). When evaluating a reward model, report the mean score of coherent failure classes on rollouts of the policies it will train, not only an aggregate AUROC. During optimization, monitor outcomes with a measurement that does not come from the reward. When training in simulation, a flagged reward can be repaired without labels: put back the objects the instruction does not name, re-render each execution, and score it again. The check clears this reward, and training on it removes the amplification: no wrong-object stack in the placement where Robometer raised them from 20% to 69%, wrong-object failures on the drawer task no different from the completion reward (0.0 [−2.9, 2.9] points), and on the robot 13.3 [3.3, 23.3] points fewer than the Robometer policies with the same success (Section 7, Tables 2 and 3, Appendices E.2 and K).

AuthorsOpenReview ↗

Response to Reviewer t9Fx (2/2)

W2 and Q1. Generality, public benchmarks and real-world evaluation. Coverage in the revision is as follows. The full-denoiser experiments cover two tasks (drawer and stacking), two reward models on the drawer task (Robometer and RoboReward-8B), two initializations and two samplers. The constrained experiments cover the same two tasks with two critics (Robometer and LIV) and two optimizers (PPO and GRPO), and 11 of 20 training histories amplify wrong-object failures in at least one stratum (Section 6, Appendix F). The tilt model accounts for which settings amplify and which do not (Spearman .89 across 26 settings, with the strength matched to each setting's reward gain), and it predicted the stacking outcomes before training.

We have not evaluated on a public benchmark. The protocol needs a distractor that affords the same action as the target, and outcome labels for both objects from simulator state. Most public manipulation benchmarks score task success only, so we built the two tasks to label wrong-object outcomes from simulator state. The check itself needs only rollouts, outcome labels and reward scores, so it runs on any benchmark whose evaluator labels wrong-object outcomes (Section 7).

On the real world: the revision adds a hardware study whose comparison was fixed before the first trial (Section 4, Table 2, Appendix E.1). The supervised initialization, three Robometer training runs and three completion training runs, all fine-tuned only in simulation, ran 120 trials per condition on a Franka Research 3. Robometer training raised wrong-object failures from 8.3% to 24.2% of trials and completion training left them at 10.0% (difference 14.2 [4.2, 25.0] points); success rose from 10.8% to 17.5% and 20.0%.

RoboReward in particular should be tested in the real world, as that is where a lot of the training data comes from.

The three RoboReward-8B training runs of Table 1 ran 120 trials on the same setup and reached 20.8% wrong-object failures against 10.0% after completion training (10.8 [1.7, 20.0] points, the comparison fixed before their first trial; Appendix E.3). The wrong-object behavior that optimization produced in simulation therefore carries over to the robot for both reward models. Videos of every hardware condition: https://huggingface.co/datasets/ICLR2027-Anonymous-42152/drawer-hardware-demos (also in the supplementary material). We agree that reward models trained mostly on real video should also be audited on real video, and the hardware study now does this for Robometer. On video of the 120 hardware trials of the initialization, Robometer scores wrong-object failures at .59 against a mean of .10 over all trials (difference .50 [.31, .67]), so the condition under which optimization amplifies them also holds on real frames, and the check predicts the amplification that the hardware trials then show.

Q2. Why does the hacking occur, and what follows for training reward models? We separate two levels.

At the level of the objective, the amplification depends only on how the reward scores the outcome classes on the initial policy's rollouts. It does not depend on the reward model's architecture. Any reward under which a coherent failure class scores above the rollouts it competes with will have that class amplified.

At the level of the model, Robometer reads the task category but does not ground which object the instruction names. We rescored 3,584 stored executions with only the instruction changed (Appendix G.5). An unrelated instruction ("pick up the cup") collapses every score to about .01. Naming the distractor drawer instead of the target raises the scores of executions that opened the requested drawer (.68 to .75) and barely changes those that opened the distractor (.62 to .60), so the score difference between the two instructions ranks successes below wrong-object failures (AUROC .19 [.17, .22]). Under the instruction "open the top drawer", RoboReward-8B still scores rollouts that opened the middle drawer above those that opened the top drawer (4.91 vs 4.83 on its 1 to 5 scale, Appendix G.5), so neither reward model, despite their different architectures, reliably grounds the referenced object. We read this as pointing to the training data more than to the architecture: two models with different architectures have learned what the action looks like, and neither has learned which object it should be applied to.

For training reward models we would suggest three things; the revision adds the first to Section 8. Include hard negatives that show the same scene and the same action on a different object, paired with the instruction. Evaluate on rollouts of the policies the model will train, and report the mean score of each coherent failure class next to the aggregate AUROC. Do not rely on contrasting instructions at scoring time as a repair: in our data it ranks the outcomes backwards.

I appreciate the care and attention authors took to rebut, especially the breadth and depth of additional experiments conducted. I appreciate the added real-robot experiments, and the clarifications on the takeaway. It may be the consequence of the language, writing, and figures, but the takeaways presented, and the style in which they are presented still do not convince this paper is ready for conference acceptance.

For example, the frequently referenced Appendix H describes the weighting and selection process in a very task specific manner, at least from what I understand (ln 1747-ln1748). In many ways, this feels like retroactive fitting or environment specific hyperparameter search paralleled in sim. Furthermore, the real-world experiments added, as I understand, test separate issues -- it's policies were optimized (and as my preconception suggests) heavily searched over in simulation, then distill to a policy in real. This is less of a demonstration of the policy working in real, and more of a separate "We can search in sim" and "We can do sim2real" argument.

My question about the implications for reward-model training also remains only partially addressed. The instruction interventions provide evidence of inadequate object grounding, but do not establish whether this arises from the training data, architecture, or objective. In particular, RoboReward already incorporates counterfactual instruction relabeling to generate negative examples. What specific deficiency in that supervision do the authors identify, and what evidence supports the proposed same-scene hard negatives as addressing it? I view this recommendation as a plausible hypothesis rather than an experimentally established takeaway.

Overall, I recognize the substantial improvements in the revision, but will maintain recommendation. I think more care should be given to the writing, figures, and presentation, and would strongly encourage the authors to check carefully the outputs of the agents used in their work. I still feel that the current idea is too environment and modeling specific, and thus their studied setting lacks convincing arguments beyond the tasks searched for in .

Thank you for reading the revision closely and for recognizing the new experiments and the real-robot evaluation. We agree on the presentation. On rereading, Appendices A to D read like engineering notes, so we rewrote them in plain prose. The updated PDF also rewrites the abstract, introduction and conclusion as one line of argument and rewords the sentences in Sections 3 to 7 that were hard to follow. Our answers to your other concerns follow.

Hardware. Nothing was searched over to choose what runs on the robot, and there is no distillation step. Teacher-student distillation in sim-to-real work turns a policy trained on privileged state into one that acts from images. Our policies act from images throughout: two camera views and the robot's proprioceptive state, and neither the policy nor the value network receives simulator object state during fine-tuning (Appendices A and B.1). The robot therefore runs the same weights that were fine-tuned in simulation, with no training on the robot: the first three training runs of each reward in Table 7, taken by index, each at iteration 40. The comparison was fixed before the first trial, and all 360 trials are counted (Appendix E.1, Table 10). The submitted version had a single-run hardware study in which the run was chosen after the simulated evaluation, and it resolved no difference. The replicated study replaces it as the main hardware evidence for that reason, and the single-run study remains in Appendix E.

Absolute rates differ between simulation and hardware, as expected for policies transferred without fine-tuning: the initialization succeeds in 10.8% of hardware trials against 17.2% in simulation, and its wrong-object rate is 8.3% against 14.1%. What carries over is the direction of the difference between the two rewards, in a physical scene built to match the simulated one at 1:1 scale. We agree that this tests how policies trained in simulation transfer, and the paper makes no claim of reinforcement learning on the robot. The check itself also runs on real frames: over the 120 hardware trials of the initialization, Robometer scores wrong-object failures at .59, against .34 for successes and .10 over all trials.

Appendix H. You are right that β in that table is set after training: its caption says β is chosen to match the reward gain each family reached. β enters only the magnitude predictions of Figure 4 and Table 20, and the text presents the full-denoiser ones as a first-order description. The predictions the paper relies on do not use β. The check compares the mean reward of wrong-object executions with that of the executions they compete with, all on the initial policy's rollouts (Section 7), so it has no fitted quantity. In stacking, each prediction and the comparison that tests it were committed before that placement's training runs started (Appendix I). Seven of the nine predictions in Table 3 were committed before the training runs began, and all nine held, including the settings where the check predicted no amplification.

Reward-model training. We agree that the recommendation is a hypothesis, so let us separate what is measured from what is suggested. What is measured: when the instruction names the top drawer, RoboReward-8B still scores executions that opened the middle drawer above executions that opened the top drawer, 4.91 against 4.83 on its 1 to 5 scale, and Robometer does the same (Appendix G.5). This holds with RoboReward's counterfactual relabeling in place. Its relabeling prompt lists handling a different object as the example of a no-success command, so supervision of this kind exists. The RoboReward paper does not report how often a relabeled command names another instance of the same category in the same scene, such as a second drawer of one cabinet, which is the case we measure. Our results do not establish whether the cause lies in the data, the architecture or the objective. Two models that differ in architecture and objective both fail to reliably follow which drawer the instruction names, and that is why Section 8 suggests looking at the data first. Same-scene hard negatives are our suggestion for closing this gap, and we have not trained a reward model with them.

AI tools. We used AI tools for code, analysis, figures and editing, as the paper's AI Assistance Statement says, and we are responsible for every claim. Every number in the tables is generated by scripts from stored result files, and the numbers in the text were checked against them. If any passage, figure or number still reads poorly or seems inconsistent, please let us know and we will gladly fix it.

Reviewer_AbTAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper looks at the problem of critic hacking for robot policies using general reward models (critic). The paper focuses on the problem of wrong semantic execution, where the policy executes a task that is semantically incorrect (e.g. if the task was to open the top most drawer but the policy ends up opening the middle drawer). General vision based critic models such as RoboReward are often blinded by such semantic failures. Hence, the paper focuses on the question of what happens when we optimize with such a visual critic reward. The paper finds that while overall task success improves with RL using visual critic reward, so does semantic failure rates. Both of them improve by converting complete task failures into either task success or semantic task failure. Alternatively, the paper also verifies what happens when the reward is a product of general visual critic and a learned verifier. In this case the learned verifier uses low-dimensional state and is directly trained on success/failure for the task and is thus task specific. In this setting, the task success improves while semantic failures do not increase much. In the appendix the paper further shows that the advantage of a critic + verifier reward vs only a verifier reward depends upon the task domain.

Critical strengths

The paper is mostly well written and focuses on an important problem. The overall experiments seem to be rigourous (with many seeds) which makes the results strong.

Critical weaknesses

The paper focuses on a strong problem but the technical contributions are really basic and not at all deep. It is not at all surprising (and almost well known) that RL with learned critics (where critics have many different failures models) will lead to the learned policy exploiting these failures. In this paper, the critic has semantic failure as a blind spot and optimizing with this critic improves both task success as well as such blind spots. The paper shows this clearly but I am quite unsure about the technical or useful insight from this.

The paper further shows that adding a learned verifier and using a product model can reduce such wrong semantic failures, which while interesting, still seems like a straight forward empirical study.

In my view, this is the biggest drawback of this paper, its really unclear to me what new insights did I gain from this paper except from some empirical proof in a few domains regarding issues that we already know about (in different settings such as LLMs).

Many parts of the paper feel clearly written by an AI model. In general this is not an issue, but this seems to be the case for large parts of the paper which is probably not ideal. The writing also feels weirdly artificial and not very interesting.

Questions for authors

see above

1 comment
AuthorsOpenReview ↗

Response to Reviewer AbTA

Thank you for the frank review, and for recognizing the rigor of the experiments. Your main question is what a reader learns beyond the known fact that policies exploit flawed critics, so we start there.

On what is new. We agree that a policy exploiting a flawed critic is expected, and that the closed form we use, the exponential tilt of KL-regularized optimization, is textbook. The same identity was recently used to explain how RLHF amplifies sycophancy, with its direction confirmed under best-of-N selection (Shapira et al., ICML 2026; now cited in the revision). What is less obvious is what it implies for visual rewards, and that it predicts the outcome of full RL fine-tuning before training. Three things follow.

First, no inversion is needed. From the supervised initialization Robometer barely separates successes from wrong-object failures (AUROC .539). Still, five training runs raise wrong-object failures by 10.9 [7.3, 14.4] points while success rises by 10.2 [6.9, 13.6], and five training runs on the simulator's completion signal do not (difference 9.2 [5.5, 13.0]). It is enough that wrong-object failures score above the average rollout (.61 against .32). A reward model can therefore pass a success-versus-failure audit and still produce the effect, and the success rate after optimization does not reveal it either.

Second, the usual diagnostic does not say where it happens. Inside the drawer and Robometer family, the initial success versus wrong-object AUROC correlates with the measured amplification at .10. The class-mean quantity from the tilt, which needs no fitting, correlates at .80.

Third, the model made predictions that could have failed. On a second task we committed a prediction for each of three object placements before fine-tuning on it: whether Robometer would amplify wrong-object stacks there. It did so only in the predicted one, from 20% to 69%, which is 68.8 [60.9, 76.8] points more than the ground-truth reward, and it removed them in the other two, including a placement that only swaps the two cubes. For the full denoiser the model is a first-order description (Section 6).

On the verifier and the product reward. We agree that they are a direct control. The revision moves them to Appendix C.1 and no longer lists them as a contribution. What takes their place in the main text follows from the tilt model, which says why a remedy has to act on the reward. Which outcomes gain share is set by how the reward scores them on the initial policy's rollouts. Optimizing more gently only slows the shift: wrong-object failures grow with distance from the initial policy at every checkpoint we measured (Appendix G.2). The same rule therefore gives a check that can be run before optimization. Across the seven full-denoiser studies of Sections 4 and 6 whose reward lacks outcome supervision it predicts amplification in five, and in those five the learned reward produced 5.3 to 68.8 points more wrong-object executions than the ground-truth reward. In the two it does not flag, both rewards end without a wrong-object execution. Five of the seven predictions were committed before the training runs began (Section 7, Table 3; the table is also in the general response).

The same rule also points to a repair without a verifier. A wrong-object execution earns its score by moving an object the instruction does not name, so we score Robometer again on frames re-rendered with those objects put back. The check clears this reward before training, and training on it needs no outcome labels. In the stacking placement where Robometer raised wrong-object stacks from 20% to 69%, three training runs on it end with no wrong-object stack. On the drawer task three training runs end with 8.3 [4.7, 12.0] points fewer wrong-object failures than Robometer, no different from the completion reward. On the robot the three policies end with 13.3 [3.3, 23.3] points fewer than the Robometer policies and the same success (Section 7, Table 2, Appendices E.2 and K).

On the writing. Thank you for saying this plainly; we took it seriously. We have reorganized the main text around one line of argument: the phenomenon, why the standard checks miss it, what the explanation predicts, and the check it gives before optimization. The text is rewritten in plainer prose, with the stacked qualifiers removed and most numbers moved from the running text into tables. The latest PDF goes a step further: the abstract, introduction and conclusion are rewritten once more, and Appendices A to D are rewritten in the same plain style. Blue headings and caption titles mark the added paragraphs, figures and tables; rewritten and moved text is not marked.

Reviewer_vu9HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CriticHack investigates manipulation of incorrect objects when robots optimize visual rewards. Experiments show that failures such as opening the wrong drawer or stacking the wrong block can increase even as reward scores and actual task success rates improve. This occurs because the reward model can recognize ordinary failures yet may assign higher scores to actions involving the wrong objects, allowing aggregate evaluation results to obscure semantic errors.

Critical strengths

This paper highlights a phenomenon that success rates alone can easily miss: robots may complete the target task more often while also manipulating the wrong objects more frequently. By treating “other failures” as a separate category, the paper shows how task successes and wrong-object failures can increase simultaneously, rather than grouping all failures together. This makes the behavioral changes induced by reward optimization easier to interpret.

The analysis of the reward model examines specific failure categories. Robometer achieves an AUROC of 0.782 when distinguishing successes from all failures, but only 0.398 when distinguishing successes from wrong-object failures. Through an explicit weighted decomposition, the authors show how strong discrimination of ordinary failures can mask reversed rankings for wrong-object failures. The explanation is direct, and the analysis is readily reusable in other studies.

Critical weaknesses

Full-policy training evidence is limited to one task and one reward model. Full-denoiser fine-tuning is tested only on the drawer task with Robometer’s terminal success probability. Stacking and LIV experiments mainly freeze the policy and optimize noise-selection probabilities. The findings therefore do not establish how broadly the phenomenon occurs across tasks or policy architectures.

Fewer wrong-object failures do not necessarily mean fewer wrong-object actions. Distractor manipulation counts as a wrong-object failure only if the target task fails; manipulating both objects and completing the target counts as success. The supplementary unconditional distractor-opening metric addresses this distinction, but Product’s change relative to initialization remains inconclusive.

The verifier’s ability to detect unseen errors remains unclear. Appendix A.1 states that its training data include trajectories from early Critic, Oracle, and Product policies. This is not evidence of evaluation-data leakage, but it means the verifier has encountered relevant behaviors. The results support correcting a known issue using development data, rather than detecting new errors introduced by later optimization.

Real-robot results do not reproduce the simultaneous increase in success and wrong-object failures. Across 40 trials per policy, Critic training reduces success from 30% to 25% while increasing wrong-object failures from 22.5% to 37.5%; no differences are statistically significant. Only one trained policy per condition is deployed, and the Critic checkpoint is selected after inspecting simulation results without a prespecified quantitative rule.

The full-policy verifier-only comparison does not establish the value of retaining the critic. Critic and Product each have five training runs and 512 shared evaluation seeds, versus one verifier-only run on a separate set of 128 seeds. This imbalance limits direct comparison. Their often similar performance in restricted experiments further motivates a stronger test of whether Product benefits from the original critic.

Questions for authors

see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer vu9H (1/2)

Thank you for the careful review, and for pointing out that separating wrong-object failures from other failures is what makes the behavioral change readable. Your five points shaped the new experiments. We answer them in the order of the review; the general response summarizes the changes.

W1. Full-policy evidence is limited to one task and one reward model. You are right that the submission tested this on one task with one reward model. The revision adds a second reward model and a second task under full-denoiser training, and repeats the original result with more training runs.

  • A second reward model shows the same effect. Three training runs against RoboReward-8B from the same supervised initialization, with the same optimizer and budget, raise wrong-object failures by 11.6 [8.0, 15.4] points and success by 5.9 [1.2, 10.4]. That is 9.7 [6.3, 13.2] points more wrong-object failures than the eight completion training runs, and each of the three training runs amplifies them (Section 4, Table 1, Appendix G.6).
  • A second task shows it where the model predicted, and only there. On stacking, Robometer raised wrong-object stacks from 20% to 69% in the one object placement that the tilt model flagged before training (68.8 [60.9, 76.8] points more than the target-success reward) and removed them in the other two (Section 6, Figure 3, Appendix I).
  • The original Robometer result holds with more training runs. From the supervised initialization, Robometer produces 9.2 [5.5, 13.0] points more wrong-object failures than the completion reward with five training runs per reward (permutation p = .008) and 8.3 [5.0, 11.5] with eight (p = .0002), all on the same 512 seeds (Section 4, Table 1, Appendix C.2).
  • The effect does not depend on the sampler, the distance from the initial policy or the threshold. Under the policy's native 100-step sampler the five warm-start critic policies still raise success by 10.9 [6.6, 15.2] points and wrong-object failures by 5.0 [1.0, 8.9]. Critic policies end 2.15 nats per decision from the initial policy, against 2.38 for Product and 2.45 for verifier-only policies, which do not amplify, so distance does not explain the difference. The effect holds for distractor thresholds from 5 to 40 mm (Appendices G.1 to G.3).

Some limits remain: full-denoiser training covers two simulated tasks and one policy architecture, RoboReward-8B is used on one drawer layout, and each objective has one to eight training runs (Section 8 lists the coverage). Breadth across critics and optimizers comes from the constrained experiments (two tasks, Robometer and LIV, PPO and GRPO), where 11 of 20 training histories amplify wrong-object failures in at least one stratum and the tilt model accounts for which ones (Section 6).

W2. Fewer wrong-object failures need not mean fewer wrong-object actions. We agree with the distinction, and the answer is that wrong-object actions rise too: the rewards that amplify wrong-object failures also open the distractor more often regardless of task outcome. We now report this directly.

  • Distractor opening regardless of outcome, by more than 20 mm, is now a column of Table 1. From the warm start, Robometer training raises it by 6.6 [3.3, 10.1] points, 6.4 [3.2, 9.6] more than dense completion training. From the supervised initialization, Robometer raises it 10.7 [6.4, 15.1] points more than completion training; RoboReward-8B and Robometer's recipe raise it by 14.7 and 15.0 points, against 5.0 for completion.
  • Product. Relative to the initialization the change is +2.1 [−1.2, 5.4] points (Appendix C.1, Table 6), which is inconclusive, as you say. The comparison we rely on is with the critic: critic training opens the distractor 4.5 [1.6, 7.7] points more often than Product training.
  • Where the outcomes go (Appendix G.4, Table 17). Every execution is a clean success, a success that also opens the distractor, a wrong-object failure or another failure. Successes that also open the distractor rise by 1.8 to 2.9 points under every warm-start reward of Table 17, Dense completion included, and critic and Product differ on them by +0.2 [−1.5, 1.8]. Product's lower wrong-object rate is therefore not a transfer into such successes: its extra success is clean success (+12.7 [9.0, 16.5] points against +5.5 [1.9, 8.9] for the critic). Critic training and the dense completion oracle remove the same share of other failures (12.1 and 11.7 points). They differ in where it goes: the critic sends 5.3 [2.6, 8.1] more points into wrong-object failures and 6.0 [2.9, 9.1] fewer into clean success.
AuthorsOpenReview ↗

Response to Reviewer vu9H (2/2)

This second part covers the verifier, the hardware study and the verifier-only comparison.

W3. The verifier has seen related behaviors. We agree, and the revision now presents verification as a diagnostic control only: it is no longer listed as a contribution and is reported in Appendix C.1. The drawer verifier is trained on 384 simulator-labeled development trajectories that include executions of critic-optimized development policies (Appendices A.1 and C.1). What the control shows is that a reward which scores wrong-object failures below the mean (.05 against .12) redirects the same optimizer. It does not show detection of errors that later optimization introduces. Appendix C.1 states that the verifier is task-specific and trained on simulator labels, and Section 7 says that it changes the reward with outcome labels.

W4. The real-robot results do not reproduce the simultaneous increase. We agree for the submitted study. It deployed one policy per condition, the critic checkpoint was chosen without a prespecified rule, and with 40 trials per policy nothing was statistically resolved; it is now Appendix E.4. For the revision we ran a replicated hardware study whose comparison was fixed before the first trial (Section 4, Table 2, Appendix E.1). We ran the supervised initialization, three of its Robometer training runs and three of its completion training runs, all fine-tuned only in simulation, on a Franka Research 3. Each of 120 rounds ran one policy from each condition in random order, which gives 120 trials per condition and 40 per training run.

  • Wrong-object failures rise from 8.3% of trials at initialization to 24.2% after Robometer training (+15.8 [6.7, 25.8] points) and stay at 10.0% after completion training.
  • The prespecified statistic, Robometer minus completion, is +14.2 [4.2, 25.0] points. Its interval resamples training runs and trials. Paired within rounds, 26 rounds have a wrong-object failure only under the Robometer policy and 9 only under the completion policy (exact McNemar p = .006).
  • Every Robometer training run has more wrong-object failures (9, 11 and 9 of 40 trials) than any completion training run (3, 5 and 4).
  • Success rises from 10.8% to 17.5% after Robometer training (+6.7 [−2.5, 15.8]) and to 20.0% after completion training (+9.2 [0.0, 18.3]).
  • Three RoboReward-8B training runs, run afterwards on the same setup, reach 20.8% wrong-object failures, 10.8 [1.7, 20.0] points above completion (Appendix E.3).

Videos of a success, a wrong-object failure and another failure for every hardware condition are at https://huggingface.co/datasets/ICLR2027-Anonymous-42152/drawer-hardware-demos and in the supplementary material. The videos of all trials will be released with the code.

W5. The verifier-only comparison is unbalanced. It is now balanced, and the answer to your question is that keeping the critic brings no behavioral benefit in this setting. Six verifier-only training runs are evaluated on the same 512 seeds as the critic and Product training runs. They raise success by 15.8 [12.2, 19.4] points and change wrong-object failures by −1.2 [−3.8, 1.4]. Product minus verifier-only is −0.3 [−2.5, 1.8] points of success and +0.5 [−1.3, 2.5] points of wrong-object failures (Appendix C.1, Tables 5 and 6). The paper does not claim that Product is better than the verifier alone. What the control shows is that, under the same optimizer and budget, outcome supervision that separates successes from wrong-object failures removes the amplification.

The revision also adds a way to keep the critic that needs no outcome labels (Section 7, Tables 1 and 3, Appendix K). After each execution we put the objects the instruction does not name back in their initial poses in a copy of the simulator state, render the reward frames again and score them with the same Robometer. This re-rendered reward passes the check before training. In the stacking placement where Robometer raised wrong-object stacks from 20% to 69%, three training runs on this reward end with no wrong-object stack and with 102 to 106 successes on 128 seeds. On the drawer task three training runs raise success by 14.1 [10.0, 18.2] points and wrong-object failures by 1.9 [−1.8, 5.6], which is 8.3 [4.7, 12.0] fewer than Robometer and no different from the completion reward (0.0 [−2.9, 2.9]). On the robot, the three re-rendered policies end with wrong-object failures in 10.8% of 120 trials against 24.2% for the Robometer policies (difference 13.3 [3.3, 23.3], the comparison fixed before the first trial) and with the same success (Table 2, Appendix E.2).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): what is new, and what the revision adds

Thank you to all three reviewers for the careful and constructive reviews. It is well known that a policy can exploit a flawed reward. Our finding is that a learned visual reward can pass both checks practitioners rely on while this happens: Robometer separates successes from failures well (AUROC .81) and task success rises during optimization, yet the policy acts on the wrong object more often. From a supervised initialization, five Robometer training runs raise task success by 10.2 points and wrong-object failures by 10.9, which is 9.2 [5.5, 13.0] points more wrong-object failures than five training runs on the simulator's completion signal, which raise success alone. You asked why and when this happens and how to prevent it. The revision explains the effect with one model, tests predictions committed before training on a second task, and turns the rule into a check that runs before any optimization.

An explanation that says when. Under KL-regularized optimization the optimal policy reweights the initial policy's rollouts by the exponentiated reward. An outcome therefore becomes more frequent relative to the rollouts it competes with when its mean reward exceeds theirs. Wrong-object failures do not need to outscore successes for this. On the drawer task they compete with all rollouts, most of which make little progress and score low: from the supervised initialization Robometer hardly separates successes from wrong-object failures (AUROC .539), yet wrong-object failures score .61 against a mean of .32 over all rollouts, so both outcomes grow together. With its strength set from each setting's reward gain, the same model tracks the outcome shifts of 26 constrained settings, including those where success falls (Spearman .89 for wrong-object failures, .79 for success; Sections 5 and 6, Appendix H).

Predictions committed before training. We tested the rule on a second task, stacking a target cube with a decoy cube in the scene. The initial policy picks a cube by its position, and Robometer scores a stack in the second position above one in the first, whichever cube is stacked, so the model predicts amplification only when the decoy sits in the second position. We committed the predictions and tests for three placements to version control before fine-tuning the full denoiser on each; the timestamps can be provided to the AC. The predicted direction held in all three. With the target first and a near-red decoy, Robometer raised wrong-object stacks from 20% to 69%, which is 68.8 [60.9, 76.8] points more than the target-success reward. Swapping only the two cubes removed the wrong-object stacks (18% to 0%; contrast between the two placements 66.4 [55.7, 77.6]). With near-red and blue decoys mixed, they also fell to 0% (Section 6, Figure 3, Appendix I).

A check before optimization. Because the rule uses only the initial policy's rollouts, it gives a check to run before training: score those rollouts and compare the wrong-object executions (W) with what the policy can trade them against. On the drawer task that is all executions; in stacking, where the policy chooses a position, it is the successes (S). The table lists the seven full-denoiser studies whose reward has no outcome supervision (Section 7; Table 3 adds the two re-rendered-reward studies). The check flags five, and in those five the learned reward produced 5.3 to 68.8 points more wrong-object executions than the ground-truth reward. In the two it does not flag, both rewards end without a wrong-object stack. Five of the seven predictions were committed before training. The last column is the learned reward's wrong-object change minus the ground-truth reward's, with crossed 95% intervals.

StudyRewardInitial mean reward: W / S / allCheck predictsCommittedWrong-object rateLearned minus true (points)
Drawer, supervised init.Robometer.61 / .64 / .32amplificationafter14% → 25%+9.2 [5.5, 13.0]
Drawer, supervised init.RoboReward-8B.95 / .95 / .79amplificationbefore14% → 26%+9.7 [6.3, 13.2]
Drawer, supervised init.Robometer recipe.70 / .71 / .63amplificationbefore14% → 25%+9.2 [5.6, 12.8]
Drawer, warm startRobometer.70 / .63 / .34amplificationafter17% → 20%+5.3 [2.6, 8.1]
Stacking, target first; near-red decoyRobometer.67 / .55 / .33amplificationbefore20% → 69%+68.8 [60.9, 76.8]
Stacking, target second; near-red decoyRobometer.52 / .73 / .34nonebefore18% → 0%0.0 [0.0, 0.0]
Stacking, target first; near-red or blue decoyRobometer.45 / .67 / .30nonebefore18% → 0%0.0 [0.0, 0.0]
AuthorsOpenReview ↗

General response (2/2): changes in the revision

The revision adds experiments across tasks, reward models and evaluation seeds, a hardware study, and a repair that needs no labels. The main text is reorganized around one line of argument: the phenomenon, why the standard checks miss it, what the explanation predicts, and the check it gives. The latest PDF also rewrites the abstract, introduction and conclusion, and rewrites Appendices A to D in plainer prose. Blue headings and caption titles mark the paragraphs, figures and tables the revision adds; rewritten and moved text is not marked.

On hardware. The revision adds a hardware study whose comparison was fixed before the first trial. The supervised initialization ran 120 trials on a Franka Research 3; three Robometer and three completion training runs ran 40 each. Robometer training raised wrong-object failures from 8.3% to 24.2% of trials and completion training left them at 10.0%, a difference of 14.2 [4.2, 25.0] points (Section 4, Table 2, Appendix E.1). Three RoboReward-8B training runs reached 20.8%, 10.8 [1.7, 20.0] points above completion (Appendix E.3).

A repair. A wrong-object execution earns its high score by moving an object the instruction does not name, so the explanation also says what to remove. We put back the objects the instruction does not name and score Robometer again on re-rendered frames. This needs no labels, passes the check before training, and removes the amplification on both tasks and on the robot (Section 7, Tables 1 to 3, Appendices E.2 and K).

ChangeResponds toWhere
A tilt model that explains why success and wrong-object failures rise together, and when success fallsAbTA W1, t9Fx W1 and Q2, vu9H W1Sections 5 and 6, Figures 2 and 4, Appendix H
A second task under full-denoiser RL, predicted before training: three stacking placementsvu9H W1, t9Fx W2, AbTA W1Section 6, Figure 3, Appendix I
A four-step check before optimization, its cost, and its prediction for every full-denoiser studyAbTA W2, t9Fx W1Section 7, Table 3
A second reward model under full-denoiser RL: three RoboReward-8B runsvu9H W1, t9Fx W2Section 4, Table 1, Appendix G.6
Supervised-initialization study extended from three to five runs per reward, eight with the repetition, each with a completion control on the same 512 seedsvu9H W1Section 4, Table 1, Appendix C.2
Warm-start controls re-evaluated on the shared 512 seeds; verifier-only extended from one to six runsvu9H W5Appendix C.1, Tables 5 and 6
Distractor opening measured regardless of task outcome, and a four-way outcome compositionvu9H W2Section 4, Table 1, Appendix G.4
Native 100-step sampler, distance from the initial policy, thresholds from 5 to 40 mmvu9H W1Section 4, Appendices G.1 to G.3
Rescoring under counterfactual instructionst9Fx Q2Section 5, Appendix G.5
Robometer's published online-RL recipe evaluated on our executions and used as the reward in three full-denoiser runst9Fx W1Section 4, Tables 1 and 3, Appendix J
A label-free repair: Robometer re-scored on frames re-rendered with the unnamed objects put backAbTA W2, t9Fx W1, vu9H W5Section 7, Tables 1 and 3, Appendix K
A replicated hardware study with its comparison fixed before the first trial: the supervised initialization on 120 trials, three Robometer, three completion and three RoboReward-8B training runs on 40 eachvu9H W4, t9Fx W2 and Q1Section 4, Table 2, Appendices E.1 and E.3
Main text and Appendices A to D rewritten in plainer prose, with fewer numbers in the running textAbTA W3, t9FxAbstract, Sections 1 to 9, Appendices A to D
Verifier and Product controls and the submitted hardware case study moved to the appendixfocus of the main textAppendices C.1 and E.4

Five numbers differ from the submission because more runs and evaluation seeds now stand behind them:

QuantitySubmissionRevisionWhy
Supervised init., Robometer: success / wrong-object change+9.6 / +11.8 (three runs)+10.2 / +10.9 (five runs)Two more runs per reward; the submitted three are runs 1 to 3 of Appendix C.2
Robometer minus completion, wrong-object failuresthree runs per reward+9.2 [5.5, 13.0] with five; +8.3 [5.0, 11.5] with eightAs above; the repetition is evaluated on the same 512 seeds
Robometer AUROC, success vs. all failures / vs. wrong-object.782 / .398.813 / .372All 20,992 warm-start executions on 512 seeds instead of the 128-seed checkpoint panel
Constrained grid11 of 29 strata11 of 20 training histories; the tilt model is fitted on 26 strataSame experiments, counted by training history; the tilt is undefined in three stacking strata (Appendix H)
Verifier-only controlone run, 128 seedssix runs, 512 seedsvu9H W5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.