acceptodds
Under review as a conference paper at ICLR 2027

Evaluator validity is a property of the (evaluator, task) pair

Abstract

Cheap stand-ins for expensive robot policy evaluation, whether simulators, learned world models, or vision-language judges, rest on one load-bearing argument: a high rank correlation between the cheap and the expensive ordering on a handful of policies. Careful papers check more; none of those checks replaces it. We show this evidence does not support the conclusion drawn from it, and say what to report instead. Using a deterministic ground-truth evaluator, so that every disagreement is judge error and nothing else, we score 12 (judge, task suite) pairs built from three vision-language judges and four LIBERO suites, holding the prompt, the policy family and the construction of the policy set fixed. Validity turns out to belong to the pair: within a single suite the probability of choosing the truly best policy varies by 0.835 depending only on the judge, and no judge is best on every suite. Rank correlation tracks none of it. It exceeds 0.82 in 10 of the 12 cells while the probability of picking the best policy inside that group runs from 0.212 to 1.000, and its sign changes with how the policy set was assembled and with the wording of the prompt. In the practitioner’s own units, following one of these certified evaluators gives up as much as 14.7 points of true success rate, and the worst evaluator we measure is beaten by a coin. The same failure appears between two independently released checkpoints, where one judge is significantly backwards on a true gap of 12.6 points. Our coverage is narrow, three judges and two public checkpoints, so we map this failure rather than bound it. We give the perfect-evaluator ceiling that makes any reported correlation readable; on a published leaderboard a flawless evaluator would score only ρ = 0.500 on LIBERO-Long at the budget the field uses.

Primary area:
datasets and benchmarks
Keywords:
evaluation, LLM-as-a-judge, robot policy evaluation, benchmark validity, embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q5eXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the reliability of vision-language judges for robot policy evaluation. Using LIBERO’s success predicate as the reference, it compares three judges across four suites and examines the effects of policy set construction and prompting. The paper argues that high rank correlation alone does not establish reliable policy selection or accurate success-rate estimates, and proposes a reporting protocol that distinguishes these two uses.

Critical strengths
  1. The experiments provide useful evidence that judge performance varies substantially across task suites. The comparison between OpenVLA and OpenVLA-OFT is particularly helpful: the reversed ordering on LIBERO-Spatial shows that the problem also occurs between independently released checkpoints.

  2. Separating policy selection from success-rate estimation is practically useful. The authors also clearly acknowledge the limited coverage of the experiments, report unsuccessful predictions, and distinguish the established statistical results from their empirical contributions.

Critical weaknesses
  1. The broader significance of the findings is unclear without testing a stronger multimodal judge. If a stronger model judges the same episodes reliably, the results may mainly show that the three tested models are not good enough for these tasks, rather than reveal a broader problem with robot evaluation.

  2. I am not convinced that the proposed ceiling applies to the main experiments. Section 3 assumes independent samples, but the judge and the reference here evaluate the same episodes. A perfect judge would match every reference label, so their success rates would be identical. The paper needs to explain why a ceiling based on independent evaluations is appropriate in this setting.

  3. Some statistical claims need more justification. The rule for trusting the full ordering in Section 2.2 does not explain its assumptions about the residuals or how it accounts for multiple comparisons. Also, Proposition 2 gives a sample budget that is sufficient; it does not show that smaller budgets are inadequate. The claims about budget shortfalls in Section 3.1 therefore seem too strong, as Appendix D itself cautions.

  4. The comparison with existing evaluation practice is not fully convincing. Section 1 treats a large absolute correlation as evidence that an evaluator passes validation. However, a strong negative correlation already shows that the judge largely reverses the policy ranking. The more relevant cases are those with high positive correlation but unreliable policy selection. Section 7 also acknowledges SIMPLER’s MMRV, but does not report it on the same policy sets. Including this comparison would clarify which failures existing checks already detect and what the proposed protocol adds.

Questions for authors

Please see the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer q5eX

Thank you for a clear review. Each weakness led to a change.

"The broader significance of the findings is unclear without testing a stronger multimodal judge."

We ran four stronger judges with a decision rule fixed in advance (general response, point 3). Each passes selection in three of four suites, where the original judges passed one or two, and orders most real policy pairs correctly. None passes selection everywhere, the failing suite differs between them, and together they fail the reporting test in 14 of 16 cells. On interpretation, if a judge had passed every cell, the protocol would have shown it. The finding that a high correlation does not certify the choice of a policy would still stand, because it concerns what correlation can tell a reader about any evaluator. In this setting the problem shrinks with scale for choosing a policy but not for reporting success rates, and the failing suite cannot be predicted from size. The per-setting check therefore stands, and the revision will drop any suggestion that larger judges cannot help.

"I am not convinced that the proposed ceiling applies to the main experiments."

The reviewer is right, and point 4 of the general response gives the change. The ceiling is a replication distribution for independently collected rollouts. For the same-episode design we will add a paired reference, which is what a perfect judge achieves. On our ladders both equal 1.000 in all cells, so Table 1 reads the same.

"Some statistical claims need more justification."

The ordering rule assumes that the residuals of the judge's rates around the line q = a·p + b are independent across policies, have equal variance and are roughly normal. The √2 comes from the difference of two residuals. The revision will state these assumptions and apply a Bonferroni correction over the adjacent pairs. No verdict changes in the 12 cells of Table 1 or the 9 pre-registered settings of Appendix C. Shapiro–Wilk tests give p ≥ 0.15 except for the 7B judge on Goal, where a is close to zero.

We also agree that Proposition 2 gives a sufficient budget and not a necessary one. We simulated the published OpenVLA leaderboard directly at 500 rollouts per method. This identifies the leader with probability about 0.99 on three suites but only 0.713 on Long. Reaching 95% needs about 300 rollouts on those three suites and about 2,300 on Long. The revision will replace the claim that both sample-size answers exceed 500 with these numbers and keep the shortfall claim only for Long.

"The comparison with existing evaluation practice is not fully convincing."

We agree on both points. A strong negative correlation already flags a reversal, so the revised Section 1 will state the problem as a high positive correlation with unreliable selection. All 10 cells with ρ ≥ 0.82 have positive ρ. We computed MMRV and Pearson's r on the same policy sets and will report both. MMRV catches the two worst cells, and its rank correlation with regret across the 12 cells is 0.91, so it is a good proxy for regret. What it misses is unreliable selection when the point estimates happen to be in order. In the nine cells with MMRV of 0.05 or less, P(top-1) ranges from 0.306 to 1.000. Regret in those cells reaches 9.6 points. The protocol adds how often the choice would be right if the evaluation were repeated, and a separate check for reporting success rates.

We hope these answers address your concerns, and we are happy to discuss any point further.

Thank the authors for their detailed clarifications and responses. Given my limited expertise on this topic, and after considering the other reviewers' comments and assessments, I believe it is appropriate to maintain my original score.

AuthorsOpenReview ↗

Thank you, and where your four points are addressed

Thank you for reading our responses and the other reviews. Each of your four points led to a change in the revised PDF, and we list them here in case this is useful.

  • Stronger judges. Section 4.5 and Table 2 add five larger judges, including GLM-4.6V. None of them passes the selection check in all four suites, and 17 of their 20 success-rate cells fail, so the problem is not specific to the three original judges.
  • The ceiling. Section 3 replaces it with a replication reference and a paired reference. The paired reference is what a perfect judge achieves on the same episodes, which is the setting you pointed out.
  • Statistical claims. Section 2.2 and Appendix C state the assumptions of the ordering rule and add a Bonferroni correction, which changes no verdict. The sample-size shortfall is now claimed only for Long, based on direct simulation rather than Proposition 2.
  • Existing practice. The problem is now stated as a high positive correlation with unreliable selection, and Table 1 reports MMRV on the same policy sets.

We are grateful for these comments, which made the paper more careful.

Reviewer_DfPGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the validity of vision-language judges used as inexpensive substitutes for robot-policy evaluation. Using LIBERO simulator success predicates as the reference, it evaluates three judges across four suites, primarily on controlled degradation ladders, and additionally compares OpenVLA with OpenVLA-OFT. The results show that evaluator performance depends strongly on the judge–suite combination and that high aggregate rank correlation can coexist with poor policy selection or calibration. The paper recommends reporting sensitivity, calibration, rank correlation alongside a perfect-evaluator reference, bootstrap top-1 selection agreement, and regret, and releases a reporting utility.

Critical strengths
  1. The empirical examples make an important practical distinction between ranking agreement, reliable selection of the best policy, and accurate success-rate reporting. Across the 12 judge–suite combinations, several high positive correlations coexist with low bootstrap top-1 agreement. The independently released checkpoint comparison provides useful evidence beyond artificially degraded policies, including a significantly reversed ordering for one judge on Spatial.

  2. The study includes several informative controls and candid limitations: multiple judge families and suites, alternative degradation axes and prompts, reproduction gates, and disclosure of failed preregistered predictions. The proposed reporting bundle and accompanying utility could improve evaluation practice even though the individual statistics are established quantities.

Critical weaknesses
  1. The perfect-evaluator “ceiling” does not match the paired observation design used in the main experiments and reporting utility. Section 3 and the supplementary ceiling() function draw two independent Binomial samples for each policy. In contrast, ground truth and judge predictions in this study score the same recorded episodes. A pointwise-perfect judge on those episodes would have identical per-policy rates and hence correlation one whenever the rates are nonconstant, regardless of finite-sample uncertainty about population rates. The independent-replication distribution can be a useful reference for a different sampling design, but it is not a general upper bound for paired evaluator–reference agreement. This matters because the headline leaderboard ceiling and the interpretation that a real evaluator cannot exceed it except by luck are presented as central contributions. The reference needs to be defined for the actual dependence structure and its interpretation revised.

  2. The central selection metric needs a more precise estimand and uncertainty analysis. In the released analyse() implementation, each bootstrap draw recomputes both the ground-truth rates and the judged rates, and counts success when the judged winner is also a winner under that draw’s resampled ground truth. This estimates bootstrap agreement with a resampled reference winner, rather than directly the probability of selecting a fixed population-optimal policy. The distinction is especially consequential for the close-policy leaderboard regime emphasized by the paper. Appendix O acknowledges conditionality, but does not resolve this change in target; the reported Monte-Carlo error also does not quantify uncertainty about population-level selection performance. Since P(top-1) and regret carry much of the paper’s argument, their interpretation and robustness to reference-ranking uncertainty require clearer treatment.

Questions for authors
  1. Is the perfect-evaluator reference intended for independently collected rollout sets or for the paired, same-video evaluation used here? Please provide a reference calculation that respects the paired dependence, distinguish a replication distribution from an upper bound, and explain how this changes the leaderboard interpretation.

  2. What is the precise population estimand for P(top-1) and regret? Can you distinguish bootstrap agreement with a resampled ground-truth winner from selection of a fixed population-optimal policy, and report uncertainty beyond bootstrap Monte-Carlo error, particularly for close policies?

  3. How much of the judge–suite interaction persists across a broader set of independently trained policy families and stronger or differently configured judges? The existing two-checkpoint control is valuable, but does not establish the prevalence of the failure across the simulator and world-model evaluators discussed in the motivation.

  4. Appendix J finds no significant association for three selected image statistics. Please narrow the claim that the failure is not a visibility problem: failure to reject these tests does not establish independence from visual evidence or rule out other visibility-related mechanisms.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DfPG

Thank you for two precise statistical points. Both are correct, and both led to changes.

"The perfect-evaluator 'ceiling' does not match the paired observation design used in the main experiments and reporting utility."

The reviewer is right. Our ceiling draws two independent samples, while the judge and the ground truth score the same episodes, where a pointwise-perfect judge reaches ρ = 1. It is a replication distribution and not an upper bound for paired agreement, and we will remove the sentence that a real evaluator cannot beat it except by luck. It was intended for independently collected rollouts, the situation of a simulator or world model that produces its own rollouts. For the paired design the revision will add a reference that respects the dependence. Treating the observed rates as the population, we draw ground truth for n fresh episodes per policy and compute ρ between that ordering and the population ordering, which is what a perfect judge achieves. We compute the same agreement for each judge by resampling episodes together with their verdicts.

On our ladders both references equal 1.000 in all 12 cells, so the reading of Table 1 is unchanged. The judges' agreement with the population ordering tells the same story as Table 1. Its median is −0.802 for the 32B judge on Spatial and −0.029 for the 7B judge on Goal. In the other ten cells it lies between 0.667 and 1.000. On the Long suite of the OpenVLA leaderboard both references have a median of 0.500, so even a perfect judge scoring those episodes would agree with the population ordering only at that level. The revision will state this as a property of the leaderboard, whose published ordering at 500 episodes is close to a coin flip, and not as a bound on evaluators.

"The central selection metric needs a more precise estimand and uncertainty analysis."

The reviewer describes our implementation correctly. P(top-1) as reported is the agreement with the winner of the same bootstrap draw of the ground truth. The revision will say so and compare two estimands that target the population. One fixes the winner at the observed ground-truth leader and resamples only the verdicts. The other gives the true rates a Beta posterior and asks how often the judge's pick is the best policy under it, and regret is defined the same way. Across the 12 cells the three definitions differ by at most 0.020, so Table 1 stands. They separate when policies are close. Take a ladder whose top two levels have true rates of 0.700 and 0.697. There InternVL3 gets 0.573, 0.768 and 0.450 under the three definitions. The revision will report the population estimand whenever the perfect-evaluator reference falls below 0.9.

We agree that Monte-Carlo error is not the relevant uncertainty. The revision will add intervals that resample tasks and then episodes. They are wide in most cells, for example 0.035 to 0.984 for the 7B judge on Long, and narrow at the extremes. The failing cells that carry the argument remain failures, for example the 32B judge on Spatial at 0.000 with an upper bound of 0.037.

"How much of the judge–suite interaction persists across a broader set of independently trained policy families and stronger or differently configured judges?"

We extended the real-checkpoint comparison to four suites and four policy families at matched resolution, and we added four stronger judges (general response, points 1 and 3). Of the nine policy pairs the simulator separates, the three original judges order at most four correctly, while the new judges order seven to nine. We agree that this does not establish prevalence across simulators and world models, and the revision will scope the claims to vision-language judges.

"Please narrow the claim that the failure is not a visibility problem."

We agree and will narrow the claim. We extended the analysis to all three judges, nine image statistics and both outcome classes. Of 216 tests, 19 separate a judge's accepted from its rejected episodes, all among ground-truth failures. For the 32B judge on Spatial, the case in Section 4.2, none does. Elsewhere some do, for example one statistic reaches an AUC of 0.83 for the 32B judge on Object failures. The revision will say that the statistics we tested do not explain the 32B failure on Spatial and that this does not rule out other visibility mechanisms.

We hope these answers address your concerns, and we are happy to discuss any point further.

Thanks for the detailed response. The revised interpretation of the perfect-evaluator reference and the new uncertainty analysis addressed my concerns. Given how much the top-1 estimates diverge when policies are close, I hope to see that example and implicated leaderboard interpretation in the revised version.

AuthorsOpenReview ↗

Both are now in the revised PDF

Both are now in the revised PDF. Thank you for the careful review and for this follow-up.

The example is in Section 5 and Appendix Q. On a fine-tuning ladder whose top two levels have true rates 0.700 and 0.697, InternVL3 scores 0.573 as implemented, 0.768 with the winner fixed and 0.450 under the Beta posterior, and the revision reports the population estimand in such cases.

The leaderboard interpretation is in Section 3.1, Appendix Q and the table of Appendix F. Under the population estimand, a perfect evaluator at the published budget picks the truly best Long method with probability only 0.527, against a chance level of one third. On the other three suites it does so with probability at least 0.962. The revision therefore states that the published ordering on Long does not identify a best method, whatever evaluator produced it.

Reviewer_YamoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a high rank correlation (Spearman ρ) between a cheap evaluator and ground truth is adequate evidence that the evaluator can replace expensive robot-policy evaluation. The authors use LIBERO’s deterministic success predicate as ground truth and isolate the judge by running policies in the simulator and giving the recorded video to three VLM judges (Qwen2.5-VL 7B and 32B, InternVL3-8B). The policy sets are ladders of one OpenVLA policy degraded along noise, observation and action-repeat axes, evaluated on four suites (12 judge-suite cells). Their main claims:

  1. Validity belongs to the (judge, task) pair. P(top-1) within one suite varies by up to 0.835 across judges, and no judge is best everywhere.
  2. ρ is nearly uninformative about decision quality. ρ ≥ 0.82 in 10 of 12 cells, while P(top-1) in those cells ranges from 0.212 to 1.000, and the sign of ρ flips with ladder construction and prompt wording.
  3. The same failure appears on two real checkpoints (OpenVLA vs. OpenVLA-OFT), where the 32B judge is significantly reversed on Spatial.

They also give a perfect-evaluator ceiling for ρ, borrowed from attenuation theory. They propose reporting sensitivity, calibration, ρ with its ceiling, P(top-1) and regret, with a two-cell verdict (choose vs. report), and release a tool for computing these.

Critical strengths
  1. The experimental design isolates judge error cleanly. A deterministic simulator (0 flips over 500 reruns), reproduction gates for each suite, and a judge run-to-run flip rate of 0 mean every disagreement is attributable to the judge. The 3×4 grid in Table 1 is a convincing demonstration. The Spatial cell (ρ = 0.976 vs. −0.970 for two Qwen sizes on identical footage, P(top-1) 0.643 vs. 0.000) and the Goal flat-trace result (7B accepts 3/235 true successes) make the central point concrete, and Figure 3 shows the mechanism well.

  2. The paper is unusually honest and the diagnostics are practically useful. It reports its own failed pre-registered prediction (App. C) and two refuted task-level predictors (App. N), notes the Bonferroni-corrected reading of Table 2, and states its limitations plainly. The decomposition into sensitivity, calibration, P(top-1) and regret is actionable. The finding that a prompt change can make the 7B judge good for choosing but unusable for reporting (P(top-1) 0.834, signed calibration −0.129) is a clean illustration of the two-cell verdict. The surrocheck tool lowers the cost of adoption.

  3. The perfect-evaluator ceiling is a simple, correct and underused idea. Computing it on the OpenVLA leaderboard (ρ = 0.5 on LIBERO-Long at n = 500) shows that reported ρ values in this regime are close to uninterpretable.

Critical weaknesses
  1. The scope of the evidence is much narrower than the framing. The title and abstract speak of “cheap stand-ins” in general (simulators, world models, VLM judges), and the introduction motivates the work with learned world models. The experiments cover three small open-weight VLMs (7B–32B) given nine frames, one policy lineage, one simulator family, and one binary prompt. No frontier or fine-tuned success detectors are tested, which are what recent pipelines would use. The world-model case is explicitly not measured (Section 6). The real-policy evidence is a single k = 2 pair from one lineage, so the claim that the field’s evidence “does not support the conclusion drawn from it” is shown for a specific judge setup, not for the cited literature. The paper never re-examines any cited work’s data. The authors acknowledge this, but the headline claims should be scoped to match.

  2. The “pair” claim is confounded with the prompt, and the ρ-vs-P(top-1) comparison is partly an artifact of the ladder design. Appendix I shows the 32B judge’s ρ moving from −0.970 to 0.647 across prompts, and the 7B judge’s P(top-1) from 0.643 to 0.834 under an occlusion-aware prompt. Validity is therefore at least a (judge, task, prompt) triple, and the main grid uses one prompt that was not tuned per cell. A judge declared “invalid” on Goal might be fixable with a task-appropriate prompt, which weakens the “no shortcut” conclusion. Separately, P(top-1) depends heavily on the gap between the top two ladder levels, and ρ over 6–8 points on a monotone ladder has a perfect-evaluator ceiling of 1.0. The comparison across cells is thus governed by ladder spacing as much as by the judge. The paper partly acknowledges this in Section 3 but does not control for it in Table 1.

  3. The target of the critique is partly a straw man. The “field’s own criterion of ρ ≥ 0.82” (Section 4.3, regret paragraph) appears to be a threshold chosen by the authors, not one the literature uses. The related work admits that SimplerEnv’s MMRV already weights violations by margin, and that other works report intervals and secondary scoring checks. Notably, in 2 of 12 cells ρ is negative and does flag the failure. The honest summary is that ρ is insufficient on its own, which is true but expected from Proposition 1, which the authors themselves call textbook. Much of the technical content (Propositions 1 and 2, the attenuation ceiling) is, by the authors’ own account, not new, so the contribution rests on the empirical map and the reporting protocol.

  4. The proposed protocol has an unaddressed circularity and weak validation. Estimating sensitivity, P(top-1) and regret requires ground-truth rollouts on the target tasks and policies. Appendix D implies gaps of 5.5 points need n = 500 per policy, which is roughly the cost the cheap evaluator was meant to avoid. The paper does not discuss how validation cost amortizes (for example, validating on a subset of policies or tasks and transferring), and the pre-registered attempts to predict failure were refuted, so there is no cheaper route. The thresholds (P(top-1) > 0.80, |cal| < 0.05, a ∈ [0.8, 1.25]) are conventions with no sensitivity analysis. The sensitivity a is estimated from the same ladder used to compute ρ, so its agreement with the sign of ρ is partly by construction, and the evidence that a “predicts” failure rests on one pre-registration where one of three predictions failed.

  5. The real-policy experiment (Section 4.4) has uncontrolled confounds and low statistical weight. The reversal rests on one significantly reversed cell out of six (it is the only one surviving Bonferroni), with a single seed per policy and judge verdicts treated as independent binomial samples. OFT differs from OpenVLA in action-chunk execution, so its videos may differ systematically in motion style, and the judge may be reacting to that and not to success. This is not examined (compare the visibility analysis in App. J, which covers only the 32B and 7B judges on the ladder data). The Goal rows for OpenVLA also use n = 300 against n = 500 for OFT. Reproduction gates are within error but three of four sit slightly low (Table 4). These are minor individually, but together they make the “same failure on real checkpoints” claim suggestive and not established.

Questions for authors
  1. Would the central finding hold for stronger judges (a frontier VLM, or a judge fine-tuned for success detection) and for richer inputs (more frames, final-state crops)? If validity belongs to the pair, a result showing even one higher-capacity judge that is uniformly valid across the four suites would sharpen the claim a lot. Equally, if the failure persists, that strengthens the argument considerably.

  2. Can you separate the effect of ladder spacing from judge quality in the ρ vs. P(top-1) comparison? For example, recompute P(top-1) and the ρ ceiling on ladders with matched top-two gaps, or report P(top-1) alongside a perfect-evaluator P(top-1) at the same n (the analogue of your ρ ceiling). Without this, readers cannot tell how much of the 0.212–1.000 range reflects the judge versus the ladder.

  3. How should a practitioner use the protocol when ground truth is expensive? Given your own sample-size numbers, what is the minimal validation set (policies × episodes) you would recommend, and does validity on a small calibration set transfer to new policies on the same task? The OpenVLA/OFT experiment hints at the answer being “not reliably”. Could you quantify it, for instance by checking whether sensitivity estimated on the ladder predicts the judge’s behavior on the OpenVLA/OFT pair?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Yamo (1/2)

Thank you for the detailed review. Your suspicion about the OFT videos was right. They were recorded at a different resolution, and at matched resolution the reversal disappears. We withdraw that claim (general response, point 1). Your questions on ladder spacing and transfer also changed how we present the contribution.

"The scope of the evidence is much narrower than the framing."

We agree, and we will narrow the framing to match the evidence. The abstract and introduction will refer to vision-language judges on LIBERO, and world models will appear only as motivation. We propose the same change to the title. The new evidence adds four stronger judges, including one fine-tuned for success detection. It also covers four policy families on real checkpoints (general response, points 1 and 3). Six of the eight stand-in papers we cite report Pearson's r. On our data it exceeds 0.82 in 10 of the 12 cells. In those cells P(top-1) still ranges from 0.212 to 1.000, so the critique carries over to their statistic. We have not re-examined their data, and the revision will say so.

"The 'pair' claim is confounded with the prompt."

We agree that validity belongs at least to the judge, task and prompt together, and Appendix I shows a prompt repairing one cell. To test this, we re-judged InternVL3-38B on Long with the two alternative prompts of Appendix I. Long is the one suite where this judge fails selection. Neither repairs it. P(top-1) is 0.357 and 0.660 against the threshold of 0.80. A suitable prompt may exist, but finding it needs the same per-setting check.

"Can you separate the effect of ladder spacing from judge quality in the ρ vs. P(top-1) comparison?"

As the reviewer notes, the ρ ceiling is 1.000 on our ladders. For every cell we will also report the P(top-1) of a perfect evaluator at the same n. It is 0.997 to 1.000 in all 12 cells, so the low values in Table 1 come from the judges and not from the spacing. Spacing does dominate when policies are close. On the Long suite of the OpenVLA leaderboard a perfect evaluator picks the leader with probability 0.713.

"The target of the critique is partly a straw man."

The threshold ρ ≥ 0.82 was ours, and we will drop the phrase "the field's own criterion". At ρ ≥ 0.9 the picture is the same, with P(top-1) from 0.306 to 1.000 in the nine cells above it. We will state the problem as a high positive correlation coexisting with poor selection, since a negative ρ already flags failure. MMRV catches the two worst cells but not unreliable selection when point estimates are in order, as our reply to q5eX shows. We agree the propositions are not new. The contribution is the measured map, the size of the gap and the protocol, as the introduction will say.

"The proposed protocol has an unaddressed circularity and weak validation."

The reviewer is right about the cost. We asked how many labeled episodes per policy a verdict needs to match the full verdict 90% of the time. In 6 of the 12 cells it needs more than 100. Combining judge verdicts with a few labels did not beat the labels alone, with either prediction-powered inference or the Rogan–Gladen correction. On our data, validating a judge for selection therefore costs about as much as selecting with ground truth, a limitation we will state.

On transfer, the sensitivity measured on the real pair falls inside the ladder interval in only 5 of 12 cells, below our pre-registered bar. The sign does agree in 10 cells. On a ladder of fine-tuning checkpoints, InternVL3's sensitivity also lies well above its noise-ladder interval. So the direction transfers but the size does not. Across nine threshold combinations the verdicts change in at most two cells, and every table prints the continuous values. We agree that the signs of a and ρ agree partly by construction, and will no longer present this as independent evidence.

"The real-policy experiment (Section 4.4) has uncontrolled confounds and low statistical weight."

We agree, and the general response withdraws this claim. The shift at fixed outcome that we attributed to policy style also disappears at matched resolution. General response point 2 covers determinism and the seed argument. The Table 2 z-scores are stratified by task, and the revision will add intervals that resample tasks and label the Goal rows with 300 episodes. The largest reproduction gap, on Object, is 1.2 standard errors. Appendix J will cover all three judges, and we will narrow the visibility claim, as our reply to DfPG explains.

AuthorsOpenReview ↗

Response to Reviewer Yamo (2/2)

"Would the central finding hold for stronger judges … and for richer inputs?"

See general response point 3 for stronger judges. For inputs, the reversal between the 7B and InternVL3 judges on the ladders holds at 9, 16 and 32 frames. Showing only the final frame flips 19% to 39% of verdicts, so the final state alone does not settle the judgment. Sixteen frames do not repair the failing cells of the 32B and 38B judges. We did not test final-state crops.

"How should a practitioner use the protocol when ground truth is expensive?"

To pick the best published OpenVLA-leaderboard method with 95% probability, ground truth alone needs about 300 episodes per policy on three suites and about 2,300 on Long. Validating a judge needs more than 100 labeled episodes per policy in half our cells, and the result transfers only in direction. We recommend validating on the target task with the policies to be compared, and using ground truth directly once that set is large enough to decide.

We hope these answers address your concerns, and we are happy to discuss any point further.

Reviewer_YamoOpenReview ↗

Response to Authors by Reviewer Yamo

I thank the authors for their rebuttal and additional experiments, which address some of my concerns. However, the withdrawal of the real-policy reversal claim, together with the remaining limitations in generalizability, novelty, and the practical cost of the proposed validation protocol, leaves me insufficiently convinced to change my recommendation. I will therefore maintain my original score of 2 (Weak Rejection).

AuthorsOpenReview ↗

The withdrawal does not affect the central result

The withdrawn claim does not affect the central result. Thank you for reading our response and for the clear summary.

Table 1 carries both main findings, and it uses only the noise ladders. Every level of a ladder comes from the same policy, recorded and judged the same way, so the resolution mismatch behind the withdrawn claim cannot arise there. Both findings therefore stand as submitted. Validity depends on the (judge, suite) pair, and a high positive ρ can coexist with unreliable selection.

What we withdrew is a piece of real-checkpoint support for these findings. What remains on real checkpoints is weaker but points the same way. The original judges order at most four of the nine pairs that the ground truth separates. When tasks are resampled, the judges' difference between OpenVLA and OFT excludes zero in 5 of 12 cells, while the ground-truth difference does so in all 12. No stronger judge passes selection in all four suites, and together the five fail the reporting test in 17 of 20 cells.

We agree that the scope beyond LIBERO, the novelty of the propositions and the cost of validation remain limitations. The revised PDF now states each of them, in Section 1 for scope and novelty and in Section 6 for cost, and Section 4.4 with Appendix R gives the released-checkpoint results.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response

General response with a correction and new results on shared points

We thank the reviewers. Below are a correction to Section 4.4 and new results on points raised by more than one review. Other points are answered under each review.

1. Correction to Section 4.4. The OpenVLA-OFT videos behind Table 2 were recorded at 256×256 pixels and every other policy at 224×224. We re-rendered the same OFT episodes at 224×224, and every outcome stayed the same. At matched resolution the significant reversal of the 32B judge on Spatial disappears, with z moving from −6.45 to +0.77. Shown the 256-pixel frames again, the same judge reverses again. We therefore withdraw the claim that the reversal seen on ladders also appears between real checkpoints.

What the real checkpoints do show is that the original judges rarely separate policies that the simulator separates clearly. We extended the comparison to all four suites and to two more policy families, π0.5 and SmolVLA. The table in point 3 gives the counts. That a resolution change alone produced a significant reversal is itself evidence that verdicts depend on inputs unrelated to success.

2. Reproducibility. Outcomes and verdicts repeat exactly on one GPU model, which every paired comparison used. Another GPU model flips 14% to 39% of outcomes while suite success rates move by at most 2.4 points. The seed argument does not change the episodes. Judged frames are H.264 video at about 90 kbps. Lossless frames raise true positive rates by only 0.03 to 0.14. The paper did not state these points, and the revision will.

3. Stronger judges. We ran four stronger judges with a decision rule fixed before any output existed. Qwen2.5-VL-72B, InternVL3-38B and InternVL3-78B use the prompt, frames and decoding of Table 1. Robometer-4B is fine-tuned for success detection on data that include these four LIBERO suites, and we used its success head. The selection and reporting columns count the suites out of four that a judge passes under the paper's thresholds. The last column covers the nine real policy pairs whose true rates differ.

JudgeSelectionReportingReal pairs correct / reversed
Qwen2.5-VL 7B1 (Object)1 (Object)0 / 0
InternVL3-8B2 (Object, Goal)04 / 0
Qwen2.5-VL 32B1 (Object)00 / 2
Qwen2.5-VL 72B (new)3 (all but Goal)07 / 0
InternVL3-38B (new)3 (all but Long)09 / 0
InternVL3-78B (new)3 (all but Goal)1 (Object)9 / 0
Robometer-4B, fine-tuned (new)3 (all but Spatial)1 (Object)9 / 0

Larger judges select far better, and the revision will say so. Still, no judge passes selection in all four suites, and the failing suite changes from judge to judge. Moving from InternVL3-38B to 78B repairs Long and breaks Goal. In every failing cell ρ is at least 0.986. The new judges fail the reporting test in 14 of 16 cells.

On the real checkpoints they order most pairs correctly. Three of them also separate OFT from π0.5 on Spatial with |z| between 3.1 and 5.2. Yet the true success rates of the two policies are 0.982 and 0.976. Neither 16 frames for the 32B and 38B judges nor alternative prompts for the 38B judge repair their failing cells. A colleague's rerun of InternVL3-78B on another GPU type gives the same pass and fail pattern, with 99.3% of verdicts identical. We did not test closed models, since the episodes are unreleased data that we do not send to third-party services.

4. The perfect-evaluator reference (Reviewers DfPG and q5eX). Both reviewers are right that our "ceiling" assumes two independent samples, while the judge and the ground truth score the same episodes, where a pointwise-perfect judge reaches ρ = 1. We will remove the sentence that no evaluator can beat it except by luck. We will define two references. The replication reference is the agreement between two independent evaluations. It is the submission's quantity and fits simulators or world models that generate their own rollouts. The paired reference is the agreement between the ground-truth ordering on n fresh episodes and the population ordering, which is what a perfect judge achieves. On our ladders both equal 1.000 in all 12 cells, so Table 1 reads the same. On the Long suite of the OpenVLA leaderboard both have a median of 0.500, and the revision will state this as a fact about that leaderboard rather than a bound on evaluators.

5. Changes in the revision. The revision and the released tool will apply all of the above. Table 1 will add Pearson's r, MMRV, the perfect-evaluator P(top-1) and task-resampled intervals. We will state the estimand of P(top-1) and the assumptions of the ordering rule with a Bonferroni correction. We will narrow the sample-size and visibility claims and scope the abstract and introduction to vision-language judges, and we propose the same change to the title.

We are happy to discuss any point further.

AuthorsOpenReview ↗

Update: a fifth stronger judge (GLM-4.6V)

A fifth stronger judge, GLM-4.6V, passes selection in only one of the four suites. This adds to our answers to all three reviewers on whether the central finding holds for stronger judges.

A colleague ran it on the same episodes after our first response. It is an open-weight mixture-of-experts model with 106B parameters, from a fourth model family. It used the decision rule fixed in advance and the prompt, frames and decoding of the judges in point 3 of the general response. Its thinking mode was turned off so that it answers within the same token budget.

It passes selection and reporting only on Object. There it accepts 92% of successful episodes and 6% of failed ones. On Spatial it accepts about 20% of successful and failed episodes alike. Its sensitivity on Goal and Spatial is clearly lower than that of Qwen2.5-VL-72B, with non-overlapping intervals.

JudgeSelectionReportingReal pairs correct / reversed
GLM-4.6V, 106B mixture-of-experts (new)1 (Object)1 (Object)2 / 3

On the real checkpoints it is the only stronger judge that reverses pairs. Of the nine pairs whose true rates differ, it reverses three. One reversal survives the Bonferroni correction and ranks SmolVLA above OFT on Spatial at matched resolution, with z = −3.61. It also separates OFT from π0.5 on Spatial with z = −2.87. Their true success rates are 0.982 and 0.976.

This qualifies point 3 of the general response. The other four stronger judges select far better than the original ones. A newer and larger judge from another family can still be much worse, and its failing suites could not be predicted from its size. Across the five stronger judges, reporting fails in 17 of 20 cells. We see this as further support for validating each judge on each task, and the revision will add GLM-4.6V to the table. We are happy to discuss this result.

If the authors have addressed your concerns or provided clarifications, please join the discussion to indicate whether your concerns have been resolved or if you have any further questions.

AuthorsOpenReview ↗

Revised PDF uploaded: changes by section

We have uploaded a revised PDF that makes the changes promised in our responses. The main text uses the additional page. The changes by section are as follows.

  • Abstract and Section 1. The claims are scoped to vision-language judges on LIBERO, with world models as motivation only. The problem is stated as a high positive correlation coexisting with unreliable selection, and the claim that the ladder reversal also appears between released checkpoints is removed (Yamo, DfPG, q5eX).
  • Section 2.2 and Appendix C. The assumptions of the ordering rule are stated, and a Bonferroni correction over adjacent pairs changes no verdict. The sign agreement between a and ρ is no longer presented as independent evidence (q5eX, Yamo).
  • Section 3 and Appendices E and F. The "ceiling" is replaced by a replication reference and a paired reference, and the sentence about beating it only by luck is removed. The Long result is stated as a property of that leaderboard, and the sample-size shortfall is kept only for Long, based on direct simulation. Under the population estimand, a perfect evaluator picks the truly best Long method with probability 0.527 (DfPG, q5eX).
  • Section 4.1 and Appendix G. Determinism is stated per GPU model, together with the seed argument, the video codec and the lossless-frame check (general response, point 2).
  • Table 1. It adds intervals that resample tasks and then episodes, the P(top-1) of a perfect evaluator, Pearson's r and MMRV. The phrase "the field's own criterion" is removed (all reviewers).
  • Section 4.4 and Appendix R. The section is rewritten as a correction and now covers four policy families at matched resolution. With intervals that resample tasks, the judges' difference excludes zero in 5 of the 12 OpenVLA and OFT cells, while the ground-truth difference does so in all 12 (Yamo, DfPG).
  • Section 4.5, Table 2 and Appendix S. These report the five stronger judges, including GLM-4.6V (all reviewers).
  • Section 5 and Appendix Q. The estimand of P(top-1), the population estimands with the close-policy example, and the threshold grid are added (DfPG, Yamo).
  • Section 6. A new paragraph states the cost of validating a judge and how far a validation transfers (Yamo).
  • Appendices I and J. Appendix I adds the alternative prompts for InternVL3-38B on Long. Appendix J covers all three judges, nine statistics and both outcome classes, with the narrower claim (Yamo, DfPG).

As proposed earlier, the revised PDF changes the title to "The validity of vision-language judges is a property of the (judge, task) pair", which matches the narrowed scope. The FAQ for the discussion stage permits this, and we will follow the area chair's guidance if the title should instead stay unchanged until the camera-ready version. The released tool will add the paired reference, the population estimand and the task-resampled intervals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.