- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors study the question of whether VLM reward models can tell robot maniulation successes from failures in near-miss situations. To this end, they introduce a boundary bisection, which uses LIBERO's success predicates to identify the smallest "push" that makes each demonstration fail. Replaying at multiple different pushes produces a simulator label of a near-miss, bad failures, and perturbed successes. The authors use 9 different judge configurations and four suites, finding that every judge does worse on near-misses. The paper also reports that judges similarly penalise perturbed successes, which suggests they track displacement rather than task completion. An important finding is that near-miss accuracy depends largely on the task than on the judge. These findings are validated on OpenVLA-7B's own failures in MetaWorld.
Critical strengths
-
I think the basic premise and the idea are well motivated. Connecting the near-misses to the
-
The paper is well-written and easy to follow.
-
The experimental design is overall sound. I find the perturbed-success control a particularly good design choice.
-
The result on the pre-trained encoders matching purpose-trained reward models on the harder tasks is quite interesting.
Critical weaknesses
-
After some thought, I am not sure the displacement finding in Section 6 could have been any different, since the success condition is latent. So unless the prompt explicitly states this threshold, the judge has no way of figuring out where it lies. This is particularly relevant to the graded prompts, where ranking 0.4cm above 2.3cm seems reasonable. Could the authors share the exact prompts used?
-
With respect to the visual characteristics of the near-misses: I think without a perceptual reference, it may be difficult to judge whether the near-miss gap is larger than the images allow us to see. For example, any scorer that grows with visual change will find near-misses harder. Could the authors clarify on whether the cm offsets are perceptible at the resolution used?
-
The authors claim that measuring failures in units of each task's boundary makes a near-miss "mean the same thing on every task". However, being equally fast past the limit does not make a failure equally visible. I think Section 6.5 seems to indicate this too: task identity explains 41% of near-miss variance vs. 6% for judge identity. Could the authors please reconcile this with their original framing?
-
The authors may also wish to consider citing / discussing relevant work on papers attempting to improve the known inaccuracies that come with using VLMs for success / progress evaluation. Some pointers: (1) Fu et al. (2024), "FuRL"; (2) Zakharov et al. (2025) "GoalLadder"; or (3) Huang et al. (2024), “The Dark Side of Rich Rewards”.
Questions for authors
See "Weaknesses".
4 comments
Response to reviewer CQ6y (Part 2/2)
W2, continued.
You also pointed out that any scorer whose output grows with visual change will find near-misses harder. We checked this and it does explain a lot of the gap. To test it, we only compared near and gross failures from the same task whose images changed by a similar amount.
| Mean near/gross gap | Spatial | Object | Goal |
|---|---|---|---|
| All failures | .194 | .198 | .112 |
| Same task, similar object pixel shift | .079 [.048, .109] | .137 [.117, .168] | .025 [-.032, .094] |
| Same task, similar image change | .087 [.052, .113] | .116 [.057, .159] | .050 [.010, .088] |
We're less sure about the Object number because it moves between .030 and .144 depending on where we put the bins. On Goal, once pixel shift is matched, the gap turns negative for 5 of 11 judges, including all five core VLMs.
We also looked at whether people have a near/gross gap themselves. They do, and how it compares with the judges' gap changes from suite to suite.
| Near/gross gap | Spatial | Object | Goal |
|---|---|---|---|
| People | .15 | .01 | .02 |
| Core judges | .16 | .21 | .08 |
| People minus judges | .00 |
| -.20 [-.27, -.14] | -.06
|
On Object people have almost no gap while the judges have a big one, so it is clearly a judge problem there. On Goal it probably is as well, but the interval just reaches zero. On Spatial people struggle about as much as the judges, so the gap by itself doesn't show that the judges are doing anything wrong. Because of this, the revision now leads with the direct comparison of people and judges on near-misses.
Finally, we tried giving the judges a better view. For the VLM judges on identical rollouts, re-rendering at 512 px actually makes the gap a bit wider (by .03, .02 and .05). A wrist camera narrows it by .01, .09 and .05, only really helping on Object. For human raters, with 512-pixel video and the tolerance stated, their near AUROC is .86, .99 and .94, so it doesn't really help people either.
W3. The paper says measuring failures in units of each task's boundary makes a near-miss "mean the same thing on every task". But equally far past the limit isn't equally visible, and Section 6.5 seems to show that. Can you reconcile these?
The multiple puts every failure the same distance past its own task's limit, measured in units of the push. We agree that this isn't the same as making failures equally easy to see, or equally close to the goal, and we revise our wording for this.
| Spatial | Object | Goal | |
|---|---|---|---|
| Median largest object shift at 1.0x, range over tasks | 5.2-13.1 px | 17.2-25.9 px | 6.9-43.6 px |
| Near failures, median distance from the target | 3.7 cm | 10.9 cm | 4.1 cm (two tasks with a target) |
So the way we'd reconcile the two is that the multiple only tells you how far past the limit a failure is. How hard that failure is to see still depends on the task, which is what Section 6.5 picks up. Over 27 tasks, the task explains 43% of the variance in near AUROC and the judge explains 4%. Although inside a single suite the two can be much closer.
We think visibility is the most likely reason the task matters so much, but we weren't able to show it. A task’s mean near AUROC correlates with how many pixels separate its near failures from its 0.75x success, but not significantly after Holm correction.
W4. Related work on FuRL, GoalLadder and "The Dark Side of Rich Rewards".
Thank you for the suggested works, and we have included all three in the related work. We found the third work on false-positive rewards to be the most relevant as a near-miss scored like a success is exactly the kind of false positive mentioned.
We thank you again for the review. The points on the latent threshold and on visibility made the paper clearer, and we’re happy to answer anything else during the discussion.
Revised manuscript uploaded
Thanks again for the time and effort you've put into reviewing our manuscript.
We'd like to let you know that the manuscript has been updated, with all revisions marked in blue.
If you have any concerns or further suggestions, we'd be glad to discuss them.
Authors of 52295
Looking forward to your reply
Dear Reviewer CQ6y,
We sincerely thank you for your efforts in reviewing our work and for your great support! Please let us know if you need any further information or if there are additional points you would like to discuss with us. We would be glad to engage further discussion with you.
Thank you very much!
Best regards,
Authors of #52295
Response to reviewer CQ6y (Part 1/2)
Thank you for the thoughtful review. We looked into your first three points and have revised the paper accordingly. Judges miss failures that people can plainly see in the same frames, with it being the most obvious on Object. On Spatial, the near-misses sit close to the limit of what these images show, for both people and judges. The general response covers two label corrections and the human-study setup (all numbers below use the corrected labels).
We agree with the argument. If a judge can't know where the threshold is, the sensible thing for it to do is give a lower score the further the object ends up from the goal. So seeing it mark down a pushed success doesn't tell us that it is ignoring completion. We've dropped the "not completion" claim and retitled Section 6 (see the general response). The exact prompts are in Appendix K of the revised PDF.
To test your point further, we did the following:
The first was whether telling the judge the threshold fixes the problem. We added one sentence to the prompt giving the tolerance in centimeters (3 to 7 cm depending on the task) and re-scored four local VLM configurations, GPT-5.5, and Claude. We found that it makes almost no difference since over the 23 judge-suite cells, near-miss AUROC moved by between -.11 and +.04. It got significantly better in 4 cells and worse in 4. Our theory is that the judges can't make use of the number because a 3 cm difference is too small to see in these images.
The second was whether a judge's score drops sharply right where the task goes from success to failure, which is what we'd expect if it did know the threshold. On Spatial only 1 of 10 functional judges shows a significant drop at 3 cm, and we can rule out drops bigger than roughly 25-40% of the judge's total fall from success to gross failure. On Object, 6 of 12 judges do show a drop at the boundary, but the object also ends up much further away at that same point, so we can't say which of the two the judge is reacting to.
We also hoped the human raters would tell us what a reasonable score looks like here, but they don't. Given the same frames, both people and core judges can barely tell untouched successes from pushed ones (general response). So we think this is still an open question.
It depends a lot on the suite. On Object the offsets are easy to see, on Goal you can see most of them, and on Spatial they are right at the edge of what the image shows. We also found that how visible a failure is explains about half or more of the gap on Spatial and Goal.
We didn't have a perceptual reference, so we ran a blinded human study to get one (the setup and the comparison with judges are in the general response).
To put the Spatial numbers in context, one Qwen3-VL vision token covers 32 px at this input size, so a typical Spatial near-miss differs from its success by about a tenth of a token. People have trouble here too since they call only 16% of the Spatial near-misses failures, although they do still rank them well below the successes.