acceptodds
Under review as a conference paper at ICLR 2027

Correlation Is Not Selection: Validating Automatic Evaluators for Robot Policies

Abstract

Testing robot policies on real hardware is slow, so automatic evaluators, such as reward models, vision-language judges, and world models, are increasingly used to choose which policy to deploy. What matters is the evaluator's top pick, yet evaluators are validated by how well their scores rank-correlate with success measured on real robots, over three to eight policies. We investigate whether such evaluators pick the right policy from recorded robot trials. We first compile an extensive set of 82 published evaluator–reality correlations from nine papers. Most of these rest on too few policies to support a general ranking claim: of the 36 computed over four or more policies, 25 have a 95% interval that includes zero. We then audit eight open-weight evaluator configurations on the same 5106 recorded episodes of seven robot-arm policies, against one set of double-blind human comparisons from the RoboArena benchmark. For each evaluator, we measure which policy it picks, how much worse that pick is than the humans' best, and how stable the pick is when the data is resampled. Three conclusions follow from these exploratory analyses. First, rank correlation does not tell which policy an evaluator will pick. Evaluators with the same correlation pick different policies, and they pick the humans' best policy in 16% to 80% of resamples, although these differences are not statistically established. Second, when evaluators miss the humans' best policy, they usually pick one almost as good, which the best policy would beat only slightly more than half the time. The exception, a score of how well a video's frames match the text instruction, picks the worst policy. Third, the best policy itself depends on whose preferences count and which policies compete. Counting each lab or human rater equally changes it, and adding one stronger policy drops the best evaluator's agreement to 20%. Evaluators should therefore be validated by the decision they support, against a named population and candidate set, not by correlation alone. We release the audit code, the frozen human ranking, and a report-card script that applies this audit to any new evaluator.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Policy Evaluation, Automatic Evaluators, Policy Selection, VLM-as-a-Judge, Reward Models, Human Preference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_3rqMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether automatic evaluators for robot policies reliably select the best policy. The authors study 82 previously reported correlations and evaluate 8 automatic evaluators on 5106 recorded robot episodes. The paper's findings include: (1) evaluators with similar rank correlations can select different policies and (2) the selected policy depends on the human evaluator population and set of candidate policies.

Critical strengths

The paper studies an important problem, given that many recent robot policies have relied on VLM-as-judge for various types of task-level signals, including reward models. I believe this work provides some contribution in this direction, though I do have substantial criticisms of this work in the weaknesses section.

Critical weaknesses
  • This paper is poorly presented due to (what I think) some heavy AI editing. I will list the following instances that I have noticed that make the paper impossible for me to review:
    • What does this mean: “Most of these rest on too few policies to support a general ranking claim: of the 36 computed over four or more policies, 25 have a 95% interval that includes zero.” What does including 0 mean here? This is in the abstract.
    • What does this mean: “For each evaluator, we measure which policy it picks, how much worse that pick is than the humans’ best, and how stable the pick is when the data is resampled” What is the “humans’ best”?
    • “Swapping only the top two of seven policies still keeps τ at 19/21 = 0.905.” What does this mean and where does 19/21 come from? The introduction is also poorly written.
    • What are each of these abbreviations (pg_vq, pg_fast_spec, etc) in Line 176? The presentation choice of having one sentence explanations for each in the appendix while leaving the abbreviations in the main text make this difficult to parse.
    • I don’t understand this sentence: “A claim that an evaluator “recovers the human ranking” is therefore incomplete unless it says whose ranking it means.”
    • What are “raters” and “sites”?
  • I disagree with this conclusion being surprising: “In particular, two evaluators with the same rank correlation selected different policies.” why would this be surprising? Two policies could be different from the human-preference ranking in different orderings.
  • As the paper itself notes, the set of LLM-based evaluators that were benchmarked in this study is not comprehensive and are heavily skewed by the reward model application, the LLM backbone, etc.

Given how poorly this paper is presented, I don’t believe it’s fair to request reviewers’ time to parse this.

Questions for authors

Please address the weaknesses. Unless I missed something fundamental, I believe that the presentation is so hard to parse/follow that I don't believe this paper merits consideration for acceptance.

2 comments

Thank you for listing the exact sentences. We agree that they were unclear as written, and the revised PDF defines each term where it first appears. Page numbers (p. N) refer to the submitted PDF.

W1. "What does this mean: '… of the 36 computed over four or more policies, 25 have a 95% interval that includes zero.' What does including 0 mean here?"

An interval that includes zero means that the data are too few to tell the correlation from zero. The abstract shortened the introduction's statement (p. 2) incorrectly, and Table 30 (p. 28) gives the correct split. The revised abstract says that 25 of the 36 "cannot rule out a correlation of zero". The revised introduction defines each part. "Fifteen have a 95% confidence interval that contains zero, so the data are too few to tell the correlation from zero. Ten are rank correlations over five policies, too few for an interval, and an exact permutation test gives p>0.05p>0.05 for them, where pp is the chance that a random ranking scores at least as far from zero."

W2. "What is the 'humans' best'?"

The human reference is the policy ranking fitted to the human pairwise comparisons (p. 2). The humans' best is the policy that this ranking puts first. The revised introduction defines both terms at first use.

W3. "'Swapping only the top two of seven policies still keeps τ at 19/21 = 0.905.' What does this mean and where does 19/21 come from? The introduction is also poorly written."

Kendall's τ\tau is the share of policy pairs ordered correctly minus the share ordered wrongly (p. 1). A τ\tau of 0.905 can therefore come from an evaluator that picks the second-best policy. The revised introduction writes out the arithmetic. "Seven policies form 21 pairs. Swapping the top two makes one pair wrong and keeps 20 right, so τ=(20−1)/21=0.905\tau=(20-1)/21=0.905." The revised introduction also states the paper's question in its first paragraph.

W4. "What are each of these abbreviations (pg_vq, pg_fast_spec, etc) in Line 176?"

They are short names of the seven RoboArena policies. The revised Sec. 3.1 describes the policies in words. Two start from the robot model π0\pi_0, and "five are PaliGemma variants that differ only in how they encode actions (diffusion, two FAST tokenizers, finite scalar quantization, binning)". Short names now appear only in tables and one figure, with a key in the caption of Table 2. The one-sentence definitions stay in Appendix B (p. 14).

W5. "I don't understand this sentence: 'A claim that an evaluator "recovers the human ranking" is therefore incomplete unless it says whose ranking it means.'"

The reason is in the sentences before it (Sec. 3.1, p. 4). One rater at one site supplies 53.7% of the sessions, and the effects of that rater, site and task mix cannot be separated. Sec. 4.2 then shows that equal rater or site weights change the winner (p. 6). The revision states the consequence directly. "Counting raters or sites equally changes which policy ranks first (Section 4.3). A claim of agreement with the human ranking must therefore name the raters and the weighting."

W6. "What are 'raters' and 'sites'?"

A rater is one person who judged comparisons, and a site is one laboratory that ran the robot. The terms appear in the introduction (p. 2) and Appendix B (p. 15). The revised introduction defines both at first use and adds that RoboArena has 44 raters at 12 sites.

W7. "I disagree with this conclusion being surprising: 'In particular, two evaluators with the same rank correlation selected different policies.' why would this be surprising?"

We agree. The result follows from the definition of rank correlation. The revision removes "unexpected" and this sentence from the conclusion (p. 9), and the introduction no longer lists it as a finding (p. 2).

W8. "As the paper itself notes, the set of LLM-based evaluators that were benchmarked in this study is not comprehensive …"

We agree that the set is not comprehensive. The evaluators cover the three ways an episode is scored today, with reward models, vision-language judges and a similarity score (Sec. 3.3, p. 5). The revision adds a proprietary API judge, Claude Opus 5.5, on the same inputs as InternVL3-8B (new Appendix K). It picks the human winner at a frozen regret of 0.002 [0.000, 0.036]. The released report-card script computes the audit from per-episode scores of any evaluator (abstract, p. 1, now Appendix P).

W9. "Given how poorly this paper is presented, I don't believe it's fair to request reviewers' time to parse this."

The revision rewrites the abstract and the introduction around one question, whether the policy an evaluator would deploy is the best one and what a wrong pick costs. On a new second benchmark, three of eight evaluators pick policies 12.5 to 32.6 success points below the winner, each at a positive rank correlation.

Q1. "Please address the weaknesses."

We will gladly clarify any sentence that is still unclear.

If the authors have addressed your concerns or provided clarifications, please join the discussion to indicate whether your concerns have been resolved or if you have any further questions.

Reviewer_vJESOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that automatic evaluators for robot policies are validated by rank correlation over a handful of policies, whereas what matters in practice is which policy the evaluator picks for deployment. They first reconstruct 82 published evaluator reality correlations from nine papers and shows that 25 of the 36 computed over 4 policies have 95% intervals including zero. They then audit eight open weight evaluator configurations on 5106 RoboArena/DROID episodes from seven VLA policies against a frozen Bradley-Terry human reference, measuring winner agreement, selection regret, and stability under bootstrap resampling, rater/site reweighting, and an enlarged candidate set.

Critical strengths
  • Re-deriving intervals for all 82 published correlations, and showing that most policy-level claims cannot rule out zero, is a concrete and consequential finding: it directly questions how a fast-growing class of evaluators is currently being validated, and gives the community a better protocol (selection, regret, named population).
  • Pre-registration with a clear registered/amended/exploratory split, matched inputs across evaluators, contamination and confound controls, noise floors under equally valid implementation changes, and a released report-card tool make the audit reproducible and reusable.
Critical weaknesses
  • Seven DROID policies (five sharing a PaliGemma backbone) with a top-two gap of p=0.544, and one rater supplying 54% of comparisons, mean the headline contrasts (e.g., winner agreement 0.16 vs. 0.80) have wide, overlapping intervals and are "not established." The paper's critique of prior work's small policy counts largely applies to its own evaluator comparisons, so the empirical takeaways are mostly negative/diagnostic rather than actionable rankings of evaluators.
  • World-model and real-to-sim evaluators make up most of the cited systems, yet none are audited (nor are proprietary API judges), and evaluators see only one exterior camera while human raters watch the robot. This limits how far the conclusions transfer to the evaluators practitioners actually use for policy selection.
Questions for authors
  1. Given the near-tie at the top of the reference, should winner agreement be a primary metric at all? Would you consider making tolerance-based regret the headline, and how do conclusions change?
  2. In the literature reconstruction, one study supplies 22 of the 36 policy-level correlations. How does the "25 of 36 include zero" headline look when that study is excluded or down-weighted?
2 comments

Thank you for the review. Page numbers (p. N) refer to the submitted PDF, and "now" gives the place in the revised PDF.

W1. "The paper's critique of prior work's small policy counts largely applies to its own evaluator comparisons, so the empirical takeaways are mostly negative/diagnostic …"

We agree, and the paper holds its own contrasts to that standard. The limitations call the study a case study of seven related policies with likely near-ties at the top (Sec. 5.2, p. 9). Every evaluator contrast carries the interval that seven policies support (Tables 14 and 17), and none is stated as a ranking. Regret is computed from 2,498 sessions, while τ\tau over seven policies can move only in steps of 0.095 (Sec. 5.2, p. 9). The claims the paper does make have intervals that exclude zero. These are SigLIP2's cost, 3.1 [2.8, 3.5] (now Table 16), and the gain of every positive-correlation evaluator over a random pick, 0.47 to 0.61 (Appendix C, p. 17).

For actionable findings, new Appendix L runs the audit on ArmnetBench, the replication dataset named in the limitations (p. 9). Its winner is clear, and five of eight evaluators pick it. SigLIP2 picks a policy 21.0 [15.1, 26.8] points below the winner at a positive τ\tau of +0.333. The bootstrap interval of that τ\tau excludes zero, but an exact permutation test does not. RoboReward-4B and Qwen2.5-VL-7B pick policies 12.5 and 32.6 points below the winner. Their costs are established under frozen-choice regret, the submitted cost measure (Sec. 3.2, p. 5), which we adopted for ArmnetBench after the first analysis, and not under the first rule (Appendix L). The actionable rule is to validate with the cost of the pick, because a positive τ\tau did not prevent a 21-point loss.

W2. "World-model and real-to-sim evaluators … none are audited (nor are proprietary API judges), and evaluators see only one exterior camera …"

The limitations name all three gaps (Sec. 5.2, p. 9). The revision adds three studies.

  • New Appendix J audits SIMPLER from its released success rates. On close drawer, at r=0.771r=0.771, the simulator's pick is the real third at an observed cost of 18.5 success points. The case is exploratory (sensitivity interval [−1.0, 37.2]).
  • Claude Opus 5.5 (new Appendix K) picks the reference winner at a frozen regret of 0.002 [0.000, 0.036]. Its paired contrast with LLM-as-a-Verifier, +0.139 [−0.106, +0.525], contains zero, the same near tie as for the open-weight leaders.
  • New Appendix O re-scores all eight evaluators on two more views. The other exterior camera changes no agreement, τ\tau or regret result after multiple-testing correction. On the wrist camera, SigLIP2's τ\tau goes from −0.714 to +0.619 and its regret falls by 2.94 [2.61, 3.32], so its reversal belongs to the exterior view. Three other evaluators change their pick, with cost changes whose intervals contain zero.

Q1. "Given the near-tie at the top of the reference, should winner agreement be a primary metric at all? Would you consider making tolerance-based regret the headline, and how do conclusions change?"

No to the first question, and yes to the second. The revision moves "The cost of a wrong pick" (Sec. 4.3, p. 7) to the front of the results (now Sec. 4.1). It brings the tolerance table into the main text (Table 12, p. 16, now Table 3), with intervals on the frozen picks (now Table 16). Winner agreement stays as a stability diagnostic. The paper notes that it understates a good evaluator's stability near a tie (0.59 against 0.76 in simulation, Appendix C, p. 19). The conclusions become clearer.

  • The evaluators fall into three groups by the policy they would deploy. Four pick the winner at a regret of 0.002 [0.000, 0.034], or [0.000, 0.036] for the new API judge. Four pick the runner-up (0.18 [0.00, 0.38]), and SigLIP2 picks the last policy (3.1) (Table 16 and Appendix K).
  • The evaluators also look more alike. The share of draws within 0.2 of the winner is 62% to 96% for the open-weight configurations (99.9% for the API judge), against 16% to 80% for exact winner agreement. A 0.2 gap is a win probability of 0.55 for the winner.

Q2. "In the literature reconstruction, one study supplies 22 of the 36 policy-level correlations. How does the '25 of 36 include zero' headline look when that study is excluded or down-weighted?"

The submission notes this share (p. 2). Without that study, 14 policy-level coefficients remain and 5 contain zero (now Table 31). The 25 of 36 count therefore depends on that study. The finding that holds in every view concerns reporting. With equal weight per paper (new Table 31), 5 of the 9 papers print no correlation across four or more independent policies. Few published validations quantify the uncertainty of the selection decision, and the audit supplies it. The general comment corrects our abstract's wording of the 25 of 36 count.

We hope these additions answer both weaknesses, and we welcome further questions.

If the authors have addressed your concerns or provided clarifications, please join the discussion to indicate whether your concerns have been resolved or if you have any further questions.

Reviewer_a27NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that automatic evaluators for robot policies are usually validated by rank correlation with real-robot success over only 3–8 policies, while practitioners deploy the evaluator's top-ranked policy. The authors (1) reconstruct 82 published evaluator–reality correlations and report that 25 of the 36 computed over four or more policies have intervals including zero or non-significant permutation tests; (2) propose a decision-level audit (winner agreement, selection regret, reweighted reference populations) and apply it to eight open-weight evaluators on 5,106 DROID episodes of seven policies against RoboArena human comparisons; and (3) release the frozen reference, per-episode scores and a report-card script. The headline claims are that rank correlation does not determine selection, that "the best policy depends on whose preferences count and which policies compete," and that evaluators should be validated at the decision level rather than by correlation.

Critical strengths
  • The statistical reporting is unusually candid. Registered and exploratory analyses are separated (Table 1), a double bootstrap is used for differences in winner agreement, and a power simulation shows what the audit cannot detect.
  • The controls (label leakage, human progress score, frame-shuffle arms, noise floors) are carefully done, and the released per-episode scores and report-card tool are useful.
  • The SigLIP2 result (a similarity reward that ranks policies in reverse) is a clean and informative negative finding.
Critical weaknesses

My main concern is that the title, abstract and conclusion substantially oversell the data, and that the paper's own results largely support the opposite reading.

  1. The title is not supported by the evidence. Every evaluator with positive τ picks either the reference winner or the runner-up (Sec. 4.3). Mean regret is 0.026–0.162, and the winner would beat these picks with probability 0.506–0.540, "close to a coin flip." Three evaluators pick the winner outright. The only failure is SigLIP2, and τ (−0.714) already flags it. In this audit τ did its job: positive τ gave a good pick, negative τ gave the worst. The data support "correlation works adequately here," not "correlation is not selection."
  2. The cross-evaluator inconsistencies are at noise level by the authors' own analysis. The headline example, Qwen2.5-VL-72B vs. 7B, is a winner vs. a runner-up that the winner beats with probability about 0.54, while the reference's own #1 vs. #2 gap is 0.544 [0.495, 0.591]. Every winner-agreement contrast has an interval including zero (Table 17), and the power simulation shows 0.1–0.2 true differences are undetectable. The "16% to 80%" range in the abstract thus reflects resampling noise around near-tied policies, yet it is presented as a finding.
  3. "The best policy depends on whose preferences count" is a statement about the reference, not about selection methods. The alternative winners each hold in at most 0.54 of draws, and five of the seven policies share a backbone. This shows the top of the ranking is ill-determined, not that evaluators are defective. The pi05 result is a single candidate-set change with small regret (0.13).
  4. The proposed metrics are not validated as better than τ. Winner agreement understates a good evaluator's stability (0.59 vs. 0.76 in simulation) and is sensitive to ties and aggregation (Qwen-72B: 0.591→0.889). It was also added after registration. The 25/36 literature statistic concerns generalization to new policies, 22 of 36 come from one study disclaiming ranking claims, and no world model is audited, so it does not show a selection problem either.
  5. Narrow evidence: one embodiment, one rater pool dominated by a single rater, seven policies including one far-worst, and one camera for evaluators versus full view for raters.
  6. Insufficient comparison with simulation-based policy evaluation. Prior work, such as SIMPLER (Li et al., CoRL 2024), demonstrates that simulation can provide scalable and reproducible robot-policy evaluation with correspondence to real-world performance, using paired sim-to-real evaluations across multiple embodiments and task families. Other work has explored policy evaluation using learned world models and approaches that combine large-scale simulation with a smaller number of real-world trials. Given that this study considers a single robot embodiment and seven related policies, it is unclear why simulation-based evaluation would be inadequate or impractical in this setting. The paper neither compares its automatic evaluators against a simulator-based baseline nor establishes a domain-specific reason that simulation is unsuitable. Although this omission does not invalidate the methodological argument that rank correlation alone may be insufficient, it weakens the practical motivation and leaves the incremental value of the proposed framework less clear.

An honest reading is that on this dataset standard evaluators pick the best or a near-tied policy, with differences within noise. That is a useful finding, but opposite to the framing.

References:

Questions for authors
  1. Given Sec. 4.3, are your results consistent with correlation-based validation being adequate here? If not, which result contradicts it?
  2. Besides SigLIP2, can you show a case where τ is high but regret clearly exceeds the reference's own noise?
  3. Will you revise the title and abstract to state that inter-evaluator differences are not statistically established and the findings are exploratory on one dataset?
  4. What does your audit catch that τ alone would not, given SigLIP2 is detected by τ?
  5. How would your approach compare with simulation-based policy evaluation? Given the limited set of seven related policies on a single robot embodiment, why is simulation-based evaluation insufficient for this setting? Can you compare your proposed audit with a simulator-based baseline to clarify what additional value it provides for policy selection?
3 comments

Thank you for the detailed review. We agree with your reading of the RoboArena result, and the revision changes the title and abstract to match it. The revision also adds a second benchmark, where evaluators with a positive τ\tau pick policies 12.5 and 21.0 points below the winner (Q2). Page numbers (p. N) refer to the submitted PDF, and "now" gives the place in the revised PDF.

W1. "The title is not supported by the evidence. … In this audit τ did its job: positive τ gave a good pick, negative τ gave the worst."

We agree for RoboArena. Submitted Sec. 4.3 (p. 7) reports the result you describe. Every positive-τ\tau evaluator picks the winner or the runner-up, and the winner is preferred to these picks with probability 0.506 to 0.540. The revision changes the title (Q3), and the revised abstract states this result. The revision adds an interval to the cost of these picks. The runner-up pick costs 0.18 [0.00, 0.38] in log-strength (now Table 16), and a τ\tau of +0.62 to +0.81 gives no such bound. On ArmnetBench, a positive τ\tau did not prevent costly picks (Q2).

W2. "The cross-evaluator inconsistencies are at noise level by the authors' own analysis."

Yes. The submission calls the winner-agreement differences "observed but not established" (Sec. 4.1, p. 6, now Sec. 4.2), and every interval in Table 17 (p. 19) contains zero. The revised abstract drops the 16% to 80% range and states that our data cannot tell the positive-τ\tau evaluators apart.

W3. "'The best policy depends on whose preferences count' is a statement about the reference, not about selection methods."

We agree that it concerns the reference, and the submission measures it. The revision retitles Sec. 4.3 to say this. The reference's top two policies are separated by a win probability of 0.545 [0.497, 0.594] on the sessions that all evaluators scored (Appendix B, p. 15). The 0.544 [0.495, 0.591] you quote is the same gap on all sessions, and equal rater or site weights change the winner (Sec. 4.2, Table 2, now Sec. 4.3). Every validation against that reference inherits this uncertainty, a rank correlation included, and the audit reports it. The π0.5\pi_{0.5} result is one candidate-set change with small regret, as you say. The paper concludes from it that winner agreement in one candidate set says little about the next, and that the audit has to be rerun when the set changes (Sec. 4.6, p. 8).

W4. "The proposed metrics are not validated as better than τ."

We agree for winner agreement. The revision keeps it as a stability diagnostic and leads with regret (now Sec. 4.1 and Table 3). Both measures were added after the analysis plan was fixed, as Sec. 3.1 and Table 1 state (p. 4). Q4 lists what regret reports and τ\tau does not. As you note, the literature count concerns generalization to new policies, and one study supplies 22 of the 36 (p. 2). Without that study, 5 of the remaining 14 intervals contain zero (now Table 31). Q5 covers world models.

W5. "Narrow evidence: one embodiment, one rater pool dominated by a single rater, seven policies including one far-worst, and one camera for evaluators versus full view for raters."

The submitted limitations name the single embodiment, rater pool and camera (Sec. 5.2, p. 9), and Appendix G (p. 24) removes the far-worst policy. Without that policy, three of the seven τ\tau intervals contain zero. The revision adds ArmnetBench (new Appendix L), with low-cost robots, operator-labeled rollouts and seven policies from six model families. New Appendix O re-scores all eight evaluators on the other exterior camera and on the wrist camera. SigLIP2's wrist-view τ\tau is +0.619 against −0.714, and its regret falls by 2.94 [2.61, 3.32]. The other pick changes have cost changes whose intervals contain zero.

W6. "Insufficient comparison with simulation-based policy evaluation."

We answer this point under Q5.

W7. "An honest reading is that on this dataset standard evaluators pick the best or a near-tied policy, with differences within noise. That is a useful finding, but opposite to the framing."

We agree with this reading of RoboArena. The revised abstract now states it plainly, and the new title drops the submitted claim (W1 and W2).

Q1. "Given Sec. 4.3, are your results consistent with correlation-based validation being adequate here? If not, which result contradicts it?"

Yes, on RoboArena. We can say so because the picks and their costs were measured (W1). Adequacy has to be measured on each benchmark, because on ArmnetBench a positive τ\tau did not prevent costly picks (Q2).

Q2. "Besides SigLIP2, can you show a case where τ is high but regret clearly exceeds the reference's own noise?"

At τ\tau above 0.8, no such case is in our data. At a moderate positive τ\tau, yes, where the reference is clear at the top. On ArmnetBench (new Appendix L), π0.5\pi_{0.5} leads by 12.5 points.

RoboReward-4B (τ\tau +0.429 [+0.238, +0.714]) picks the runner-up at 12.5 [6.1, 18.7] points. This cost uses frozen-choice regret (Sec. 3.2, p. 5), which fixes the pick at the policy chosen on all the data. We applied it to ArmnetBench after the first analysis. Under the first rule, which picks again in every resample, its interval contained zero (Appendix L).

SigLIP2 has a positive τ\tau on ArmnetBench (+0.333 [+0.048, +0.488]). It picks the fourth policy at 21.0 [15.1, 26.8] points, a cost that holds under session-inflated intervals and both cost rules. For both evaluators, the bootstrap interval excludes zero, but an exact permutation test does not (one-sided p=0.12p=0.12 and 0.190.19). Qwen2.5-VL-72B, at τ\tau +0.524 [+0.143, +0.683], picks the winner. The same test gives it p=0.068p=0.068.

Under the main reference, the five open-weight evaluators with the highest τ\tau pick the winner, and the costs of the other three rise as τ\tau falls. That order does not hold under the sensitivity reference fixed before scoring (Appendix L), which also counts suboptimal rollouts as successes. There RoboReward-4B has the higher τ\tau (+0.524) and still loses 11.1 [5.0, 17.3] points, while Qwen2.5-VL-72B has the lower τ\tau (+0.429) and still picks the winner. Between the two references, only the fifth and sixth policies change places, and they differ by one successful rollout of 360. Their picks differ by 12.5 and 11.1 points under the two references. On RoboArena, the positive-τ\tau costs whose intervals exclude zero all come from reweighted references (largest 0.617 [0.121, 1.159] when raters count equally, now Table 16), under which the reference's own winner holds in at most 54% of draws (Table 2, p. 6).

Q3. "Will you revise the title and abstract to state that inter-evaluator differences are not statistically established and the findings are exploratory on one dataset?"

Yes. The title is now "Selection Regret: Validating Automatic Evaluators for Robot Policies". The abstract leads with the cost of the pick and states that our data cannot tell the positive-τ\tau evaluators apart. It keeps the exploratory label of the submitted abstract (p. 1) and drops the 16% to 80% range.

Q4. "What does your audit catch that τ alone would not, given SigLIP2 is detected by τ?"

The audit reports the pick's cost and its uncertainty, which new Sec. 4.7 gathers. These are the bound on the cost on RoboArena (W1), the costly picks on ArmnetBench (Q2), the uncertainty of the reference (W3), and the fact that three of the seven τ\tau intervals exclude zero only because one policy is far worse than the rest (W5). τ\tau flags SigLIP2 when sessions are resampled, but not when sites are. Its τ\tau interval is [−0.810, +0.048] when sites are resampled (Appendix C, p. 17) and [−0.500, +0.071] with eight policies (Table 26). Its regret stays between 1.7 and 3.1 in five of the six resampling and reweighting checks, and is 3.3 with eight policies.

Q5. "How would your approach compare with simulation-based policy evaluation? … Can you compare your proposed audit with a simulator-based baseline to clarify what additional value it provides for policy selection?"

We do not argue that simulation is inadequate here. A simulator is one more evaluator, and its pick needs the same check. It generates its own episodes, so it cannot score our recorded trials, but its per-policy results go through the same report card. The submission positioned SIMPLER in Sec. 2 (p. 3), and new Appendix J applies the report card to SIMPLER's released success rates. On close drawer, at r=0.771r=0.771, the pick has an observed cost of 18.5 points. This case is exploratory, because its sensitivity interval [−1.0, 37.2] contains zero. We ran no simulator on the seven RoboArena policies, as the revised limitations state. The two DROID simulators, PolaRiS and RoboLab, share four policies with RoboArena and run joint-position versions of them, while RoboArena uses joint-velocity checkpoints. From the papers' printed and plotted values, both simulators pick π0.5\pi_{0.5} among those four, with regret 0 against our reference (now Appendix N).

We hope the revised title and abstract resolve your main concern, and we will gladly answer any further questions.

If the authors have addressed your concerns or provided clarifications, please join the discussion to indicate whether your concerns have been resolved or if you have any further questions.

Reviewer_2pSkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether automatic robot evaluators select the best policy, rather than merely produce rankings correlated with real-world performance. Through a review of published validations and a controlled evaluator comparison, the authors show that similar rank correlations can lead to different selections, while most incorrect selections are close in quality to the human reference winner. Furthermore, the study examines how these findings depend on human preferences and the candidate policies.

Critical strengths

While many existing approaches to validating robot evaluators focus on rank correlation, this work examines the policy-selection decision that such evaluations are intended to support. By comparing multiple evaluator types using the same recorded episodes and measuring selection regret alongside winner agreement (Sections 3.2–3.3), the study provides a useful examination of both the selected policy and the cost of an incorrect choice.

Critical weaknesses
  • The empirical evaluation is concentrated on a single benchmark, with the main candidate set consisting of seven policies from the Pi0 and PaliGemma model lineages (Section 3.1). Although the authors examine alternative human-reference weightings and an expanded candidate set (Sections 4.2 and 4.6), these analyses remain within the same benchmark. Consequently, it remains unclear how broadly the observed evaluator behavior generalizes across different policy families and evaluation settings.
  • The additional practical insight provided by the evaluator comparison seems somewhat limited. Section 4.1 reports substantial differences in bootstrap winner agreement, but these differences are not statistically established. Furthermore, most incorrect selections among the positive-correlation evaluators are near-ties (Section 4.3 and Figure 3). Although this usefully motivates reporting regret alongside correlation, the evidence provides limited support for practically consequential differences between these evaluators. Given that the conceptual distinction between overall rank agreement and top-policy selection is straightforward, stronger empirical findings would help establish the significance of the contribution.
Questions for authors

Among the positive-correlation evaluators, does the existing regret analysis establish any meaningful differences in selection quality (Section 4.3 and Figure 3)? In particular, which conclusions about their relative reliability remain supported when selection cost, rather than exact winner agreement, is the primary criterion?

2 comments

Thank you for the review. The main change is a second benchmark, on which evaluators with a positive rank correlation make costly picks. Page numbers (p. N) refer to the submitted PDF, and "now" gives the place in the revised PDF.

W1. "The empirical evaluation is concentrated on a single benchmark, with the main candidate set consisting of seven policies from the Pi0 and PaliGemma model lineages (Section 3.1)."

The submitted limitations name ArmnetBench as the replication dataset (Sec. 5.2, p. 9). New Appendix L runs the same eight evaluators there. ArmnetBench has low-cost robots and seven policies from six model families. Its winner, π0.5\pi_{0.5}, leads by 12.5 success points and is first in 9,998 of 10,000 resamples. The revision also audits a proprietary API judge, Claude Opus 5.5, on RoboArena (new Appendix K), which picks the reference winner at a regret of 0.002 [0.000, 0.036]. The revision re-scores all eight evaluators on two more camera views (new Appendix O) and applies the report card to SIMPLER's released simulator results (new Appendix J).

W2. "Section 4.1 reports substantial differences in bootstrap winner agreement, but these differences are not statistically established. … the evidence provides limited support for practically consequential differences between these evaluators."

On RoboArena we agree. The submission calls the winner-agreement differences "observed but not established" (Sec. 4.1, p. 6, now Sec. 4.2), and every double-bootstrap interval for a difference contains zero (Table 17, p. 19). The revision therefore makes regret the lead measure.

On ArmnetBench the differences are large. Five evaluators pick the winner. Three do not, and each has a positive τ\tau.

Evaluatorτ\tau [95% interval]PickSuccess points lost
SigLIP2+0.333 [+0.048, +0.488]fourth21.0 [15.1, 26.8]
RoboReward-4B+0.429 [+0.238, +0.714]runner-up12.5 [6.1, 18.7]
Qwen2.5-VL-7B+0.143 [−0.048, +0.429]last32.6 [26.8, 38.2]

Qwen2.5-VL-72B, at τ=+0.524\tau=+0.524 [+0.143, +0.683], picks the winner. When suboptimal rollouts also count as successes, its τ\tau and RoboReward-4B's swap, and both picks stay. An exact permutation test cannot tell the τ\tau of SigLIP2 or RoboReward-4B from a random ranking (one-sided p=0.19p=0.19 and 0.120.12). All three costs are frozen-choice regret (Sec. 3.2, p. 5), which fixes the pick at the policy chosen on all the data. We made it the ArmnetBench rule for an established cost after the first analysis. Under the first rule, which picks again in every resample, only SigLIP2's cost was established (Appendix L).

Q1. "Among the positive-correlation evaluators, does the existing regret analysis establish any meaningful differences in selection quality (Section 4.3 and Figure 3)?"

On RoboArena, no, and the audit says why. On ArmnetBench, where the winner is clear, yes (W2). The human reference's top two policies are almost tied (win probability 0.545 [0.497, 0.594], Appendix B, p. 15), so the reference cannot rank the evaluators that pick between them. Held-out regret, which scores each pick on sessions not used to make it, agrees. All six gaps between LLM-as-a-Verifier and the other positive-correlation evaluators contain zero, the largest being −0.135 [−0.276, +0.045] against RoboReward-4B (Appendix C, p. 17, interval now printed).

Q2. "In particular, which conclusions about their relative reliability remain supported when selection cost, rather than exact winner agreement, is the primary criterion?"

The revision makes cost the primary criterion. Submitted Sec. 4.3 is now Sec. 4.1, with the tolerance table as Table 3. Three conclusions hold.

  1. On RoboArena, any positive-correlation evaluator is a low-cost choice. Three open-weight configurations pick the winner and four the runner-up, and the winner is preferred to their picks with probability 0.506 to 0.540 (Sec. 4.3, p. 7). The revision adds intervals to these frozen picks, 0.002 [0.000, 0.034] and 0.18 [0.00, 0.38] in log-strength (now Table 16). The upper end is a win probability of 0.59 for the winner (now Sec. 4.7).
  2. Every positive-correlation open-weight configuration outperforms a random pick among the seven policies, with held-out regret of 0.05 to 0.19 against 0.67 (Sec. 4.3, p. 8, Table 14). Among the six close policies, LLM-as-a-Verifier (gain 0.179 [0.040, 0.340]) and Qwen2.5-VL-72B (0.160 [0.012, 0.336]) are established above a random pick when half of the sessions are used to choose (Table 13, p. 17).
  3. On ArmnetBench, cost separates the evaluators. Five pick the winner, and three lose 12.5 to 32.6 points, with frozen-choice intervals that exclude zero (W2). Among the five, winner agreement runs from 0.70 to 0.987, and these differences are not established.

We hope the second benchmark supplies the stronger empirical findings you asked for.

If the authors have addressed your concerns or provided clarifications, please join the discussion to indicate whether your concerns have been resolved or if you have any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes and shared response to all reviewers

We thank the reviewers. The paper audits the decision that an automatic robot-policy evaluator supports. For each evaluator, it reports which policy the evaluator would deploy, what that pick costs against a named human reference (its selection regret), and how certain both are. The main new result is a second benchmark, where evaluators with a positive rank correlation pick policies 12.5 and 21.0 points below the winner. The revision keeps every submitted result. Page numbers (p. N) refer to the submitted PDF, and "now" gives the place in the revised PDF, where changes are in blue.

Results already in the submitted paper

  1. On RoboArena, every evaluator with a positive rank correlation picks the human winner or the runner-up. The winner is preferred to these picks with probability 0.506 to 0.540 (Sec. 4.3, p. 7, now Sec. 4.1). We agree with Reviewer a27N that these evaluators chose well here, and the revised abstract states this result. The revision adds an interval to the cost of the runner-up pick, 0.18 [0.00, 0.38] in log-strength (now Table 16). Its upper end is a win probability of 0.59 for the winner (now Sec. 4.7). A τ\tau of +0.62 to +0.81 gives no such bound.
  2. The reference itself is uncertain. Its top two policies are separated by a win probability of 0.545 [0.497, 0.594] (Appendix B, p. 15). Counting each rater or site equally changes the winner (Sec. 4.2, Table 2, now Sec. 4.3).
  3. SigLIP2 picks the last policy at a cost of 3.1, while its τ\tau interval contains zero under site resampling (Appendix C, p. 17).

New in the revision

  1. Reviewers 2pSk, a27N and vJES asked for a second benchmark. The submitted limitations name ArmnetBench as the replication dataset (Sec. 5.2, p. 9). New Appendix L runs the same eight evaluators there, with seven policies from six model families. Its winner is clear, 12.5 success points ahead, and five evaluators pick it. SigLIP2 picks the fourth policy at 21.0 [15.1, 26.8] points, at τ=+0.333\tau=+0.333 [+0.048, +0.488]. For this τ\tau the bootstrap interval excludes zero, but an exact permutation test does not (one-sided p=0.19p=0.19). RoboReward-4B picks the runner-up at 12.5 [6.1, 18.7], and Qwen2.5-VL-7B picks the worst policy at 32.6 [26.8, 38.2]. Counting suboptimal rollouts as successes, RoboReward-4B has a higher τ\tau (+0.524) than Qwen2.5-VL-72B (+0.429), which picks the winner. Their costs are established under frozen-choice regret, the submitted cost measure (Sec. 3.2, p. 5), which we adopted for ArmnetBench after the first analysis. Under the first rule, only SigLIP2's cost was established (Appendix L).
  2. Reviewer vJES noted that no API judge was audited (Sec. 5.2, p. 9). New Appendix K audits Claude Opus 5.5. It picks the reference winner with τ=+0.905\tau=+0.905 [+0.524, +1.000] and a regret of 0.002 [0.000, 0.036].
  3. Reviewers vJES and a27N noted the single exterior camera (Sec. 5.2, p. 9). New Appendix O re-scores all eight evaluators on the other exterior camera and on the wrist camera. On the wrist view, SigLIP2's τ\tau moves from −0.714 to +0.619, and its regret falls by 2.94 [2.61, 3.32]. Three other evaluators change their pick, with cost changes whose intervals contain zero.
  4. Reviewers a27N and vJES asked about simulators. The submission positions SIMPLER in Sec. 2 (p. 3), and the revised Sec. 2 adds a paragraph on simulation-based evaluation. New Appendix J applies the report card to SIMPLER's released success rates. The simulator's pick is the observed real best on six of nine tasks. On close drawer, at r=0.771r=0.771, the pick is the real third at an observed cost of 18.5 points, which neither rr nor MMRV reports. This case is exploratory, because its sensitivity interval [−1.0, 37.2] contains zero.
  5. Reviewers vJES and a27N asked about the literature count without the study that supplies 22 of the 36 policy-level correlations (p. 2). Without it, 5 of 14 intervals contain zero. And 5 of the 9 papers report no correlation across four or more independent policies (new Table 31).

A correction

Our abstract said "25 have a 95% interval that includes zero". The exact statement is that 25 of the 36 cannot rule out a correlation of zero. Fifteen have a 95% confidence interval that contains zero. Ten have no interval, and an exact permutation test gives p>0.05p>0.05 for them (Table 30, p. 28). The revised abstract says "cannot rule out a correlation of zero", and the introduction and Figure 1 give the split.

Other changes in the revised PDF

  1. The title is now "Selection Regret: Validating Automatic Evaluators for Robot Policies". The new title answers Reviewer a27N (W1 and Q3). The "16% to 80%" range leaves the abstract.
  2. Regret leads. Submitted Sec. 4.3 is now Sec. 4.1, and its tolerance table (Table 12, p. 16) is now Table 3.
  3. New Sec. 4.7 gathers what the audit reports and τ\tau does not.
  4. Every term that Reviewer 3rqM listed is defined at first use, and the main text describes the policies in words.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.