acceptodds
Under review as a conference paper at ICLR 2027

Vision-Language Judges Prevent Reward Hacking in Robot Manipulation RL

Abstract

Reinforcement learning (RL) is a major part of the training pipeline of the most powerful models today. However, designing good RL environments is hard, and models often find creative ways to achieve high reward without fulfilling the designer's intent, known as reward hacking. This poses a major risk to the safety of AI systems. A natural but little-studied countermeasure is to let a trusted model judge each rollout during training and intervene, e.g., by penalizing or discarding flagged rollouts. We introduce three manipulation environments in which Vision-Language-Action models (VLAs) learn to reward hack during RL, and test such interventions with Vision-Language Models (VLMs) as judges. First, using oracle judges that read the true environment state, we find that small penalties suppress hacking nearly as well as strong ones while costing far less task performance when the judge falsely flags honest rollouts. We then build judges from VLMs spanning small to frontier models and improve them by letting them output scores instead of binary verdicts and giving them context and tools. Finally, we match the oversight performance of the strongest VLM judges at over lower cost by training small task-specific judges, open-weight VLMs and probes, on the frontier judge's verdicts. Beyond our setting, VLMs overseeing VLAs is a case study of a judge supervising a model that differs from it in training and capabilities, and a starting point for building such oversight more broadly.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reward hacking, Reward misspecification, AI safety, Vision-language-action models, Robot manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r1W2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains OpenVLA with PPO in three ManiSkill tasks, each with a hand-written flawed reward that leaves out one condition of the real goal. The authors claimed that policies that collect this reward without finishing the task are called reward hackers. The authors test a simple fix: a judge reviews every rewarded rollout, and flagged rollouts get a reduced reward, zero reward, a penalty, or are dropped. With an oracle judge that reads the true state, mild reductions around zero reward can suppress hacking about as well as heavy penalties are much less hurt when the judge falsely flags honest rollouts.

They then replace the oracle with VLM judges, VLM models from small to frontier. Judges do better with graded scores instead of yes/no verdicts. Runs with a VLM judge end up close to runs with a noisy oracle of matching error rates, although the judge's catch rate drops during training, which authors read as partial policy adaptation. Finally, a fine-tuned small VLM and a lightweight probe, trained on the frontier judge's verdicts, match its quality at a small fraction of the cost. The authors present this as showing that VLM judges prevent reward hacking in robot RL, and as a case study of a judge supervising a model that differs from it in capability.

Critical strengths

The oracle-judge ablations are careful. The authors vary the cut size, add symmetric and biased noise, and vary how often the judge reviews. The findings that penalties break under false flags while zero reward holds up is clear and useful. The twin runs (Figure 4) are a good way to check that real VLM judges behave like the oracle results. These experiments design and findings are insightful.

The judge analysis and the cheap judges are useful. The paper shows where the VLMs fail, which bring some lessons for future research. Score thresholds, examples and tools help. A small fine-tuned model or a probe gets close to a frontier judge at a tiny fraction of the cost. The paper also reports openly that the students' catch rates drift during training.

Critical weaknesses

The loopholes are partial success, so this does not test reward hacking. In all three tasks the "hack" is a task left unfinished: the cube is pushed past the goal, the peg overhangs a rail, the box sits on the rim or is wedged in the opening. The policy makes no exploitative choice. The end state also looks clearly different from an honest success, and the judge sees those final frames. The paper's own results also back this up. A probe with about 300k parameters on frozen SigLIP2 frame embeddings reaches near top J with 10 to 25 clips per class, as written in Section 6.3. If a tiny classifier separates the cases that easily, catching them is not a hard test for a frontier VLM. The SFT prior also supplies part of the hacking: 11% to 42% before RL, plus the scripted hack demos on PushCube. This says little about the case the paper's motivation points to, like the Hugging Face incident (OpenAI, 2026), where a model finds an unintended route. The claim in the title and the Figure 1 caption goes further than three hand-built partial-success loopholes support.

The framing is overstated, and the evidence is about judge quality and cost. The abstract calls judge-based intervention "little-studied". The related work already cites VLM success detectors and critics for robotics, VLM reward models, and judge interventions in LLM RL. What is new is the combination in a simulated manipulation testbed, which is also not reasonable enough to be called reward hacking as well.

The judges get information a real setup would not have. The score threshold is fitted on 250 hacks and 250 honest successes labeled by the ground-truth predicates. Both come from the SFT policy, which already hacks at a high rate (Appendix D). The in-context examples include a labeled hack clip of the same task. The judge only ever sees rollouts where the loophole has already fired. A real user does not have labeled hacks before RL, and does not know what hacks will appear later. The judges are never evaluated on hacks that show up during training, which are the ones that matter. The paper also does not compare against the obvious alternative: if you know the loophole well enough to label hacks, fix the reward.

Questions for authors
  1. All three loopholes leave the task unfinished and the end state is visibly different. A small probe separates them with 10 to 25 clips per class. Can you show a task where the hack is a different behavior, or where its end state looks the same as an honest success?
  2. On the sparse tasks, an oracle judge with c = 1 is close to the honest reward. What does Section 5 show beyond robustness to noisy labels?
  3. Your calibration set, evaluation set and student training data all come from the SFT prior, where hacks already occur. How would you build them when the hacks appear only during RL? How well do the judges do on hacks from late in training, or did you find such behavior?
  4. Do you have any setting where the judge is weaker than or equal to the policy on the task? Without it, what supports the claim that this is relevant to overseeing more capable models?
2 comments

We thank the reviewer for their comments and valuable feedback on our paper. Below, we address the concerns and questions raised and state the changes we have made in response.

  1. Partial success versus reward hacking

We would argue that our hacks still count as reward hacking. We use the term as in Amodei et al. (2016) and Skalse et al. (2022), i.e., the policy achieves high reward without fulfilling the designer's intent, which does not require the policy to exploit the loophole deliberately.

At the same time, we agree that the hacks are simple and that RL amplifies them and does not discover them (we now also state the latter in Section 3). Note that they are not all easy to catch, e.g., on LayPeg, a probe trained on ground-truth labels reaches J = 0.69 against 0.91 for the fine-tuned VLM at 250 clips per class (Figure 30, now also reported in Section 6.3).

  1. Framing as "little-studied"

By "little-studied" we mean intervening in training based on a judge's verdict, i.e., lowering the reward of flagged rollouts or dropping them. To our knowledge, this has so far only been studied for language models (especially Wong et al. (2026)). Previous robotics work uses VLMs to detect success or undesired behavior or to produce the reward, but does not intervene on flagged rollouts during RL. We have reformulated this sentence in the abstract to be more precise.

  1. Information a real setup would not have

Thanks for pointing this out. We agree that labeled hacks might not be available to a real user before RL (we now state in Section 6.1 where the judges use them). One way to collect such data could be to have a person label a few rollouts by hand.

Furthermore, we would like to clarify three details. Reviewing only the rollouts where the loophole fired needs no knowledge of it, since these are simply all rollouts that collected reward (now stated in Section 4). The judges are also evaluated on the rollouts of their own training runs (Figures 25 to 27). Finally, fixing the reward is part of every training experiment as the honest-reward baseline (now stated in Section 3).

  1. A different behavior or the same end state

So far, we do not have a task where the end state of a hack looks the same as an honest success. We take this as feedback for future work and in the updated version mention this in the limitations.

  1. What Section 5 shows beyond noisy labels

We agree that a perfect oracle with c = 1 on the sparse tasks is close to the honest reward. Beyond this, Section 5 also shows which reductions in reward have which effect (we found that reductions around zero reward work well on all tasks, while dropping performs worse), that false flags lead to a collapse of true success under penalty while under misses the three reductions behave similarly, and what the effects of reviewing only every k-th rollout are. We now start Section 5 by spelling out its questions.

  1. A judge weaker than the policy

Yes, in the sense that the judges are worse at the task than the policy they oversee. When the VLMs drive the arm themselves (Appendix E.1), the best one reaches 35% true success pooled over tasks and prompts, and even their best runs stay below the VLA trained with the honest reward (now stated in Section 6.1).

Thank you for the clarifications and revisions. My main concern is how well the results generalize beyond the three handcrafted loopholes, especially when hacks and genuine successes look similar at the end of a rollout. It also remains unclear how well the approach works when representative labeled examples are hard to obtain before RL. I will keep my original rating.

Reviewer_vQ5bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors build a suite of three manipulation tasks (PushCube, LayPeg, BoxInBox) which intentionally induce reward hacking during RL training of VLAs. They show using oracle judges to moderately reduce reward from hacked successes can suppress RL reward hacking while simultaneously preventing a decrease of task performance from overly harsh rewards or false flags. The authors then investigate whether a VLM can be used as a suitable judge. They note that more powerful models correlate with more accurate judges. They also show how outputting confidence scores with a calibrated threshold instead of binary labels, in addition to example rollouts and zoom tools provide significant improvement to VLM judges. After this, they show how oracle performance can translate to VLM performance by comparing a noisy oracle’s results with a VLM as a judge, and find that they are comparable. They then show the feasibility of distilling strong judge verdicts into smaller VLMs or probes by showing that a Qwen3-VL model trained on Fable 5’s judge outputs matches the accuracy of Fable 5 at over 100x lower cost.

Critical strengths

For the most part, the authors do a good job of justifying their approach. The strong oracle performance, such as the BoxInBox results showing large increases in success with plummeted hacking rates, is a good indicator that a VLM judge would be useful. The paper also shows how the method can be made more practical via the distillation approach, which can be seen as a step forward in improving RL methods in robotics. The authors are also careful to avoid overstating.

Critical weaknesses

The primary weakness in the paper is that there is no convincing justification for why one would prefer a judge over fixing the reward function. Several procedures in the pipeline the authors propose involve inside knowledge on what the hacks are. For instance, the calibrated thresholds are fit on ground-truth-labeled hacks and distilling smaller models means that the larger model has already identified hacks to train on. It seems that if one knows of the existence of a set of hacks and they are coming up enough that it is a problem, then one would also have enough information to correct the reward, reducing rollout cost and avoiding any potential slowdowns in training. While the authors argue that rewards can become difficult to write as tasks generalize, it would be natural to expect judging to become harder as well in these cases, begging the question of if the distilled models can maintain their performance.

Questions for authors
  1. Is it possible to show performance on an in-the-wild reward hacking scenario? As it stands, the three induced scenarios are nice toy examples but seem a bit contrived (PushCube even required seeded demonstrations).
  2. Do the distilled models generalize beyond the hacks that they were trained on? For example, if a distilled model is trained on identifications of rim-perched in BoxInBox, can it also detect the wedged hacks?
  3. How effective is the VLM pipeline without ground-truth labels? It seems difficult to have calibrated rollouts on non-simulator RL tasks.
  4. Can we use the VLM as a reward signal? Does that improve training?
2 comments

We thank the reviewer for their comments and valuable feedback on our paper. Below, we address the concerns and questions raised and state the changes we have made in response.

  1. Why use a judge instead of fixing the reward?

We agree that for our three tasks, fixing the reward by hand is easy, and nobody would need a judge there. This is by design, since we need the true success condition to measure hacking exactly. We have added this explanation to Section 3.

Regarding how much knowledge the judge gets, the plain judge only sees the frames and gets the same vague instruction as the VLA, and is never told the loophole. Labels enter in two places, the score threshold and the example clips (see point 4, now also stated in Section 6.1). Furthermore, the corrected reward is tested as part of every training experiment as the honest-reward baseline (we now state this in Section 3). With an oracle judge, the best reduction performs as well as it (Figure 2), and the VLM-judged runs land close to an oracle with the same initial error rates (Figure 4).

We also agree that judging gets harder as tasks get more general. While we find that judge quality rises with the general capability of the VLM in most conditions and tasks (Figure 3), whether the students keep up under more task variation remains open (see our fourth limitation).

  1. Reward hacking in the wild

We do not have such a scenario (see our first limitation). We only used seeded demonstrations in PushCube, while on the other two tasks, RL amplifies hacks that the prior already shows. We agree that testing the judges on naturally arising hacks is the most important next step, and take this as feedback for future work.

  1. Generalization of the distilled judges to other hacks

That is a fair point, and we did not test this. The students are task-specific by design, so for a new kind of hack the fallback is the general teacher, which is prompted rather than trained. We agree that holding out one kind of hack would be an interesting experiment, and we take this as feedback for future work.

  1. Performance without ground-truth labels

Without labels, what remains are the judges without examples (plain or with tools) and with a binary verdict, which only use the frames and the instruction. As can be seen in Figure 20, they are already relatively strong on PushCube, but weaker on LayPeg and BoxInBox, where the calibrated threshold is the most relevant (for the plain Fable 5 judge, it adds roughly 0.3 in J on both). We did not run training with them. Outside a simulator, the threshold could be fit on a small set of rollouts labeled by a person (so far, we used 250 per class and did not test whether fewer suffice).

  1. The VLM as a reward signal

Yes, that would be possible. We have not run this, but note that the zero-reward intervention is similar to this (i.e., a rollout only keeps its reward if the environment rewards it and the judge passes it, so the judge acts as a second success condition, which we now state in Section 4). Regarding whether this improves training, reductions around zero reward with an oracle judge raise true success on BoxInBox from 18% to 73% (Figure 2), and the VLM judges perform close to an oracle with the same initial error rates (Figure 4).

Thanks to the authors for the clarifications added to the paper and the response. Several points are clearer, and the fact that the plain judge does not receive anything more than the VLA is noted. However, my main concern remains, and I will keep my score.

My central question was why one would need a judge rather than fixing the reward, and I do not think that the response resolves this. The paper is motivated by the idea that increasing task generality makes reward harder to write, which could lead to hacks that were not previously anticipated. However, the authors' primary contributions hinge on the fact that the hacks are known---for instance, the suggestion to use a human to calibrate a threshold implies that the hacks are already known (as for calibration to occur, there must be labeled examples of hacks). The distilled students (which make the method practical during training) share this limitation, since they are trained on the teacher's calibrated verdicts for known hacks, and there is little reason to believe that they would generalize to unknown hacks.

However, I do believe that the authors have shown that the judge is effective for known hacks. I see value in the paper as a controlled study of how judge-based interventions behave, particularly the robustness of moderate reward reductions to false flags. I encourage the authors to address some of the points they have left for future work, such as showing that the method, particularly the distilled students, still works on an in-the-wild hack.

Reviewer_FwtkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to prevent reward hacking during RL by having a trusted model judge each training rollout and intervene on the ones it flags, either by lowering their reward or dropping them from training entirely. Prior work has shown that such judge-based interventions can reduce hacking in language models, including in production coding RL, but they have not been studied in robotics. This paper claims to be the first to apply them there to penalize and prevent reward hacking. The authors design three robotic manipulation environments in which hacking emerges during RL and use them to test judge-based interventions on VLA policies. Using oracle judges they find that small reward reductions suppress hacking nearly as well as harsh penalties while losing less performance to false flags. They build VLM judges from small to frontier models and improve them with calibrated scores, example rollouts, and tools for inspecting frames. And they distill the frontier judge verdicts into small open-weight VLMs and probes that match its oversight quality at much lower cost.

Critical strengths

(1). The paper proposes a novel, well-motivated problem setting with a reusable testbed. As far as I know it is the first to study judge-based interventions against reward hacking in robotics RL, using VLMs to oversee VLAs. And it contributes three environments with unambiguous success conditions that others can build on. The judging vs. acting gap makes this a strong instance of a weaker-at-acting model supervising the policy they oversee, which could be relevant beyond robotics.

(2). The experimental design is clean, as it isolates each factor. Oracle judges separate the effect of the intervention from judge quality, controlled symmetric and biased noise separates false flags from misses, and the twin-run method shows that the oracle results largely transfer to real VLM judges.

(3). The findings are well-supported. Zero reward provides a robust default, while the penalty’s fragility to false flags is clearly described. The judge improvements, including calibrated scores, examples, and tools, are simple and seem likely to transfer.

(4). The paper provides a practical path to cheap oversight. Distilling the frontier judge into small open-weight VLMs and probes provides some good results, where the fine-tuned VLM matches and the probes come close in oversight quality at a much lower cost. This makes the approach potentially usable in regular training.

Critical weaknesses

(1). The environments are simple. The paper establishes controlled benchmarks for reward hacking by creating deliberate loophole success situations. however, these loophole criteria are too simple and can be easily fixed via better reward design. A more realistic setting would be one where the correct reward is difficult to specify or measure automatically. The paper does not demonstrate that difficulty. I would like to see situations where its genuinely difficult to specify the rewards. This is acknowledged in the limitations section, but nonetheless the trade-off between hand-designed loopholes that make hacking unambiguous to measure versus more natural hacks is important and is not explored in the paper.

(2). Threshold calibration and novel hacks. How would the threshold be chosen when new unseen failures arise? The calibration set would not have similar examples. The experiments do not show that the VLM judges can generalize to novel reward-hacking strategies. The calibration set is labeled with ground-truth simulator labels and the examples condition supplies a labeled hack clip, so both presuppose knowing the hack in advance. In the setting where the reward is hard to specify neither would be available. This is partially acknowledged in the limitations, where the paper states that the VLA might not explore alternative hacks, so the testbed cannot produce a novel hack to test generalization on.

(3). Retraining judges and catch-rate decay. Retraining a judge for each new task or newly discovered failure mode could be expensive, retraining on a new failure mode requires the teacher to have caught it first, which the calibrated threshold from does not guarantee. The need for this is especially possible as experiments show that as RL progresses the catch-rate diminishes, especially for the probes. The twins are matched to initial error rates and training is somewhat limited in steps, so the study does not rule out harm under longer training or stronger optimization pressure.

(4). Hacking is partly seeded, not discovered. All three SFT priors already hack, and on PushCube the authors had to inject scripted hack demonstrations into the SFT trajectories because the policy otherwise rarely hacks. The claim that reward hacking emerges during RL is overstated, as the paper never shows RL discovering a hack that had zero probability under the prior, a case that is critical for safety.

(5). Single policy and single judge family. Every policy is OpenVLA-7B and every frontier judge is a Claude model. No other frontier VLMs or zero-shot open-weight judges are tested, so it is unclear whether the judge improvements and the scaling trend are properties of VLM judges in general or of this family. The closed-model dependence also limits reproducibility.

Questions for authors

(1). PushCube uses a per-step reward with no early termination while LayPeg and BoxInBox use a terminal reward with auto-termination on loophole success. Is this difference required for PushCube’s loophole? With auto-termination on loophole success, on the target at some point and on the target at the end are the same, so the loophole exists only because the episode keeps running. This difference makes it impossible to tell whether PushCube’s behavior is due to the task or the reward structure.

(2). The 11% hack rate versus “rarely hacks.” Line 171 says 12 of PushCube’s 50 SFT demonstrations are scripted hacks because the policy otherwise rarely hacks, while line 187 reports an 11% post-SFT hack rate on PushCube. Is the 11% measured on the SFT policy that already includes the 12 hack demos? It would also help to report the hack rate when the PushCube SFT operates on the honest demos only, both after SFT and after loophole-reward RL, so we can evaluate how much hacking RL discovers on its own rather than amplifies.

(3) PPO description. PPO is only briefly mentioned in Section 3, and then Section 4 refers to PPO updates and to dropped rollouts contributing no gradient before it has been described in any detail. A short description of the training loop at the start of Section 4, covering how rollouts are collected, where the reward modification enters, and whether dropped rollouts are removed before the batch statistics are computed, would make the interventions easier to follow.

(4). Precise instructions for the VLA. The task instructions are deliberately vague. To separate hacking caused by reward misspecification from hacking caused by instruction underspecification, could the authors run an ablation that gives the VLA the precise honest success condition in its instruction while still training against the loophole reward?

(5). Are the judges errors perceptual rather than about intent? On PushCube and LayPeg, telling the judge the exact loophole performs worse than tools + examples, and the error analysis attributes misses to slight overhangs and false flags to low gripper endings. This suggests judges’ errors are mainly perceptual, not failures to infer intent from a vague instruction. Can a customized perception-check judge do just as well as the frontier VLM? A related question concerns why knowing the loophole helps so little.

1 comment

We thank the reviewer for their comments and valuable feedback on our paper. Below, we address the concerns and questions raised and state the changes we have made in response.

  1. Simple environments and seeded hacking

We agree that for our three tasks, fixing the reward by hand is easy. This is intentional, since to measure hacking exactly, we need the true success condition (we added this as an explanation to Section 3). Importantly, the plain judge is never told about the loophole and receives only the frames and the same vague instruction as the VLA. We also agree that RL amplifies hacks that already exist in the prior and does not discover them (we now state this in the contributions and in Section 3). We take creating additional environments where the reward is hard to specify, and hacks that the prior does not produce, as feedback for future work.

  1. Threshold calibration and new kinds of hacks

It is correct that labeled hacks enter the judges both in the calibrated threshold and the example clips (we now also explicitly state this in Section 6.1). The plain judge with a binary verdict uses no labeled hacks, but also performs worse (see Figure 20). So far, we have not tested whether the judges generalize to unseen hacks, and we take this as feedback for future work.

  1. Retraining judges and catch-rate decay

Note that the students mainly save cost since the frontier judge needs no task-specific training. Furthermore, regarding decay, the probes' catch rate falls significantly over training on PushCube and LayPeg, while the fine-tuned VLMs stay close to their teacher's level (see Figure 33). We agree that a student likely only learns what its teacher catches.

  1. Single policy and judge family

We agree that using a single policy and judge family limits how far our results generalize (and now also state this in the limitations), and we would like to add additional VLAs and judge families as feedback in future work. Since the cheap judges are open-weight (Qwen3-VL-8B and SigLIP2), they are reproducible, and they generally achieve high J when trained on ground-truth labels, i.e., without any closed model (Figure 30).

  1. PushCube and the PPO description

Yes, the PushCube loophole exists since the episode keeps running once the block is on the goal. Thus, it does need the missing early termination rather than the per-step reward. In this case, we measure the 11% based on the SFT policy trained on all 50 demonstrations, which also includes the 12 scripted hacks. When we say that it "rarely hacks", we refer to RL from a policy fine-tuned on honest demonstrations (we now clarify this in Section 3).

As a small change in the paper, we have added a short description of the training loop to Section 4. Note that a dropped rollout does not contribute to a gradient and is not replaced (see footnote 2).

  1. More precise instructions and perceptual errors

We agree with the reviewer that giving the VLA the precise success condition would be interesting but leave this to future work. Furthermore, we believe that the errors of the judges are mostly perceptual (i.e., the judges often miss the slight overhangs and wedged boxes). This would also explain why, when we give the models a hint regarding the loophole, they perform worse than tools + examples on PushCube and LayPeg.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.