- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper trains OpenVLA with PPO in three ManiSkill tasks, each with a hand-written flawed reward that leaves out one condition of the real goal. The authors claimed that policies that collect this reward without finishing the task are called reward hackers. The authors test a simple fix: a judge reviews every rewarded rollout, and flagged rollouts get a reduced reward, zero reward, a penalty, or are dropped. With an oracle judge that reads the true state, mild reductions around zero reward can suppress hacking about as well as heavy penalties are much less hurt when the judge falsely flags honest rollouts.
They then replace the oracle with VLM judges, VLM models from small to frontier. Judges do better with graded scores instead of yes/no verdicts. Runs with a VLM judge end up close to runs with a noisy oracle of matching error rates, although the judge's catch rate drops during training, which authors read as partial policy adaptation. Finally, a fine-tuned small VLM and a lightweight probe, trained on the frontier judge's verdicts, match its quality at a small fraction of the cost. The authors present this as showing that VLM judges prevent reward hacking in robot RL, and as a case study of a judge supervising a model that differs from it in capability.
Critical strengths
The oracle-judge ablations are careful. The authors vary the cut size, add symmetric and biased noise, and vary how often the judge reviews. The findings that penalties break under false flags while zero reward holds up is clear and useful. The twin runs (Figure 4) are a good way to check that real VLM judges behave like the oracle results. These experiments design and findings are insightful.
The judge analysis and the cheap judges are useful. The paper shows where the VLMs fail, which bring some lessons for future research. Score thresholds, examples and tools help. A small fine-tuned model or a probe gets close to a frontier judge at a tiny fraction of the cost. The paper also reports openly that the students' catch rates drift during training.
Critical weaknesses
The loopholes are partial success, so this does not test reward hacking. In all three tasks the "hack" is a task left unfinished: the cube is pushed past the goal, the peg overhangs a rail, the box sits on the rim or is wedged in the opening. The policy makes no exploitative choice. The end state also looks clearly different from an honest success, and the judge sees those final frames. The paper's own results also back this up. A probe with about 300k parameters on frozen SigLIP2 frame embeddings reaches near top J with 10 to 25 clips per class, as written in Section 6.3. If a tiny classifier separates the cases that easily, catching them is not a hard test for a frontier VLM. The SFT prior also supplies part of the hacking: 11% to 42% before RL, plus the scripted hack demos on PushCube. This says little about the case the paper's motivation points to, like the Hugging Face incident (OpenAI, 2026), where a model finds an unintended route. The claim in the title and the Figure 1 caption goes further than three hand-built partial-success loopholes support.
The framing is overstated, and the evidence is about judge quality and cost. The abstract calls judge-based intervention "little-studied". The related work already cites VLM success detectors and critics for robotics, VLM reward models, and judge interventions in LLM RL. What is new is the combination in a simulated manipulation testbed, which is also not reasonable enough to be called reward hacking as well.
The judges get information a real setup would not have. The score threshold is fitted on 250 hacks and 250 honest successes labeled by the ground-truth predicates. Both come from the SFT policy, which already hacks at a high rate (Appendix D). The in-context examples include a labeled hack clip of the same task. The judge only ever sees rollouts where the loophole has already fired. A real user does not have labeled hacks before RL, and does not know what hacks will appear later. The judges are never evaluated on hacks that show up during training, which are the ones that matter. The paper also does not compare against the obvious alternative: if you know the loophole well enough to label hacks, fix the reward.
Questions for authors
- All three loopholes leave the task unfinished and the end state is visibly different. A small probe separates them with 10 to 25 clips per class. Can you show a task where the hack is a different behavior, or where its end state looks the same as an honest success?
- On the sparse tasks, an oracle judge with c = 1 is close to the honest reward. What does Section 5 show beyond robustness to noisy labels?
- Your calibration set, evaluation set and student training data all come from the SFT prior, where hacks already occur. How would you build them when the hacks appear only during RL? How well do the judges do on hacks from late in training, or did you find such behavior?
- Do you have any setting where the judge is weaker than or equal to the policy on the task? Without it, what supports the claim that this is relevant to overseeing more capable models?
2 comments
Thank you for the clarifications and revisions. My main concern is how well the results generalize beyond the three handcrafted loopholes, especially when hacks and genuine successes look similar at the end of a rollout. It also remains unclear how well the approach works when representative labeled examples are hard to obtain before RL. I will keep my original rating.
We thank the reviewer for their comments and valuable feedback on our paper. Below, we address the concerns and questions raised and state the changes we have made in response.
We would argue that our hacks still count as reward hacking. We use the term as in Amodei et al. (2016) and Skalse et al. (2022), i.e., the policy achieves high reward without fulfilling the designer's intent, which does not require the policy to exploit the loophole deliberately.
At the same time, we agree that the hacks are simple and that RL amplifies them and does not discover them (we now also state the latter in Section 3). Note that they are not all easy to catch, e.g., on LayPeg, a probe trained on ground-truth labels reaches J = 0.69 against 0.91 for the fine-tuned VLM at 250 clips per class (Figure 30, now also reported in Section 6.3).
By "little-studied" we mean intervening in training based on a judge's verdict, i.e., lowering the reward of flagged rollouts or dropping them. To our knowledge, this has so far only been studied for language models (especially Wong et al. (2026)). Previous robotics work uses VLMs to detect success or undesired behavior or to produce the reward, but does not intervene on flagged rollouts during RL. We have reformulated this sentence in the abstract to be more precise.
Thanks for pointing this out. We agree that labeled hacks might not be available to a real user before RL (we now state in Section 6.1 where the judges use them). One way to collect such data could be to have a person label a few rollouts by hand.
Furthermore, we would like to clarify three details. Reviewing only the rollouts where the loophole fired needs no knowledge of it, since these are simply all rollouts that collected reward (now stated in Section 4). The judges are also evaluated on the rollouts of their own training runs (Figures 25 to 27). Finally, fixing the reward is part of every training experiment as the honest-reward baseline (now stated in Section 3).
So far, we do not have a task where the end state of a hack looks the same as an honest success. We take this as feedback for future work and in the updated version mention this in the limitations.
We agree that a perfect oracle with c = 1 on the sparse tasks is close to the honest reward. Beyond this, Section 5 also shows which reductions in reward have which effect (we found that reductions around zero reward work well on all tasks, while dropping performs worse), that false flags lead to a collapse of true success under penalty while under misses the three reductions behave similarly, and what the effects of reviewing only every k-th rollout are. We now start Section 5 by spelling out its questions.
Yes, in the sense that the judges are worse at the task than the policy they oversee. When the VLMs drive the arm themselves (Appendix E.1), the best one reaches 35% true success pooled over tasks and prompts, and even their best runs stay below the VLA trained with the honest reward (now stated in Section 6.1).