- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether automatic evaluators for robot policies reliably select the best policy. The authors study 82 previously reported correlations and evaluate 8 automatic evaluators on 5106 recorded robot episodes. The paper's findings include: (1) evaluators with similar rank correlations can select different policies and (2) the selected policy depends on the human evaluator population and set of candidate policies.
Critical strengths
The paper studies an important problem, given that many recent robot policies have relied on VLM-as-judge for various types of task-level signals, including reward models. I believe this work provides some contribution in this direction, though I do have substantial criticisms of this work in the weaknesses section.
Critical weaknesses
- This paper is poorly presented due to (what I think) some heavy AI editing. I will list the following instances that I have noticed that make the paper impossible for me to review:
- What does this mean: “Most of these rest on too few policies to support a general ranking claim: of the 36 computed over four or more policies, 25 have a 95% interval that includes zero.” What does including 0 mean here? This is in the abstract.
- What does this mean: “For each evaluator, we measure which policy it picks, how much worse that pick is than the humans’ best, and how stable the pick is when the data is resampled” What is the “humans’ best”?
- “Swapping only the top two of seven policies still keeps τ at 19/21 = 0.905.” What does this mean and where does 19/21 come from? The introduction is also poorly written.
- What are each of these abbreviations (pg_vq, pg_fast_spec, etc) in Line 176? The presentation choice of having one sentence explanations for each in the appendix while leaving the abbreviations in the main text make this difficult to parse.
- I don’t understand this sentence: “A claim that an evaluator “recovers the human ranking” is therefore incomplete unless it says whose ranking it means.”
- What are “raters” and “sites”?
- I disagree with this conclusion being surprising: “In particular, two evaluators with the same rank correlation selected different policies.” why would this be surprising? Two policies could be different from the human-preference ranking in different orderings.
- As the paper itself notes, the set of LLM-based evaluators that were benchmarked in this study is not comprehensive and are heavily skewed by the reward model application, the LLM backbone, etc.
Given how poorly this paper is presented, I don’t believe it’s fair to request reviewers’ time to parse this.
Questions for authors
Please address the weaknesses. Unless I missed something fundamental, I believe that the presentation is so hard to parse/follow that I don't believe this paper merits consideration for acceptance.
2 comments
If the authors have addressed your concerns or provided clarifications, please join the discussion to indicate whether your concerns have been resolved or if you have any further questions.
Thank you for listing the exact sentences. We agree that they were unclear as written, and the revised PDF defines each term where it first appears. Page numbers (p. N) refer to the submitted PDF.
W1. "What does this mean: '… of the 36 computed over four or more policies, 25 have a 95% interval that includes zero.' What does including 0 mean here?"
An interval that includes zero means that the data are too few to tell the correlation from zero. The abstract shortened the introduction's statement (p. 2) incorrectly, and Table 30 (p. 28) gives the correct split. The revised abstract says that 25 of the 36 "cannot rule out a correlation of zero". The revised introduction defines each part. "Fifteen have a 95% confidence interval that contains zero, so the data are too few to tell the correlation from zero. Ten are rank correlations over five policies, too few for an interval, and an exact permutation test gives p>0.05 for them, where p is the chance that a random ranking scores at least as far from zero."
W2. "What is the 'humans' best'?"
The human reference is the policy ranking fitted to the human pairwise comparisons (p. 2). The humans' best is the policy that this ranking puts first. The revised introduction defines both terms at first use.
W3. "'Swapping only the top two of seven policies still keeps τ at 19/21 = 0.905.' What does this mean and where does 19/21 come from? The introduction is also poorly written."
Kendall's τ is the share of policy pairs ordered correctly minus the share ordered wrongly (p. 1). A τ of 0.905 can therefore come from an evaluator that picks the second-best policy. The revised introduction writes out the arithmetic. "Seven policies form 21 pairs. Swapping the top two makes one pair wrong and keeps 20 right, so τ=(20−1)/21=0.905." The revised introduction also states the paper's question in its first paragraph.
W4. "What are each of these abbreviations (pg_vq, pg_fast_spec, etc) in Line 176?"
They are short names of the seven RoboArena policies. The revised Sec. 3.1 describes the policies in words. Two start from the robot model π0, and "five are PaliGemma variants that differ only in how they encode actions (diffusion, two FAST tokenizers, finite scalar quantization, binning)". Short names now appear only in tables and one figure, with a key in the caption of Table 2. The one-sentence definitions stay in Appendix B (p. 14).
W5. "I don't understand this sentence: 'A claim that an evaluator "recovers the human ranking" is therefore incomplete unless it says whose ranking it means.'"
The reason is in the sentences before it (Sec. 3.1, p. 4). One rater at one site supplies 53.7% of the sessions, and the effects of that rater, site and task mix cannot be separated. Sec. 4.2 then shows that equal rater or site weights change the winner (p. 6). The revision states the consequence directly. "Counting raters or sites equally changes which policy ranks first (Section 4.3). A claim of agreement with the human ranking must therefore name the raters and the weighting."
W6. "What are 'raters' and 'sites'?"
A rater is one person who judged comparisons, and a site is one laboratory that ran the robot. The terms appear in the introduction (p. 2) and Appendix B (p. 15). The revised introduction defines both at first use and adds that RoboArena has 44 raters at 12 sites.
W7. "I disagree with this conclusion being surprising: 'In particular, two evaluators with the same rank correlation selected different policies.' why would this be surprising?"
We agree. The result follows from the definition of rank correlation. The revision removes "unexpected" and this sentence from the conclusion (p. 9), and the introduction no longer lists it as a finding (p. 2).
W8. "As the paper itself notes, the set of LLM-based evaluators that were benchmarked in this study is not comprehensive …"
We agree that the set is not comprehensive. The evaluators cover the three ways an episode is scored today, with reward models, vision-language judges and a similarity score (Sec. 3.3, p. 5). The revision adds a proprietary API judge, Claude Opus 5.5, on the same inputs as InternVL3-8B (new Appendix K). It picks the human winner at a frozen regret of 0.002 [0.000, 0.036]. The released report-card script computes the audit from per-episode scores of any evaluator (abstract, p. 1, now Appendix P).
W9. "Given how poorly this paper is presented, I don't believe it's fair to request reviewers' time to parse this."
The revision rewrites the abstract and the introduction around one question, whether the policy an evaluator would deploy is the best one and what a wrong pick costs. On a new second benchmark, three of eight evaluators pick policies 12.5 to 32.6 success points below the winner, each at a positive rank correlation.
Q1. "Please address the weaknesses."
We will gladly clarify any sentence that is still unclear.