- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the reliability of vision-language judges for robot policy evaluation. Using LIBERO’s success predicate as the reference, it compares three judges across four suites and examines the effects of policy set construction and prompting. The paper argues that high rank correlation alone does not establish reliable policy selection or accurate success-rate estimates, and proposes a reporting protocol that distinguishes these two uses.
Critical strengths
-
The experiments provide useful evidence that judge performance varies substantially across task suites. The comparison between OpenVLA and OpenVLA-OFT is particularly helpful: the reversed ordering on LIBERO-Spatial shows that the problem also occurs between independently released checkpoints.
-
Separating policy selection from success-rate estimation is practically useful. The authors also clearly acknowledge the limited coverage of the experiments, report unsuccessful predictions, and distinguish the established statistical results from their empirical contributions.
Critical weaknesses
-
The broader significance of the findings is unclear without testing a stronger multimodal judge. If a stronger model judges the same episodes reliably, the results may mainly show that the three tested models are not good enough for these tasks, rather than reveal a broader problem with robot evaluation.
-
I am not convinced that the proposed ceiling applies to the main experiments. Section 3 assumes independent samples, but the judge and the reference here evaluate the same episodes. A perfect judge would match every reference label, so their success rates would be identical. The paper needs to explain why a ceiling based on independent evaluations is appropriate in this setting.
-
Some statistical claims need more justification. The rule for trusting the full ordering in Section 2.2 does not explain its assumptions about the residuals or how it accounts for multiple comparisons. Also, Proposition 2 gives a sample budget that is sufficient; it does not show that smaller budgets are inadequate. The claims about budget shortfalls in Section 3.1 therefore seem too strong, as Appendix D itself cautions.
-
The comparison with existing evaluation practice is not fully convincing. Section 1 treats a large absolute correlation as evidence that an evaluator passes validation. However, a strong negative correlation already shows that the judge largely reverses the policy ranking. The more relevant cases are those with high positive correlation but unreliable policy selection. Section 7 also acknowledges SIMPLER’s MMRV, but does not report it on the same policy sets. Including this comparison would clarify which failures existing checks already detect and what the proposed protocol adds.
Questions for authors
Please see the weaknesses.
3 comments
Thank the authors for their detailed clarifications and responses. Given my limited expertise on this topic, and after considering the other reviewers' comments and assessments, I believe it is appropriate to maintain my original score.
Thank you, and where your four points are addressed
Thank you for reading our responses and the other reviews. Each of your four points led to a change in the revised PDF, and we list them here in case this is useful.
- Stronger judges. Section 4.5 and Table 2 add five larger judges, including GLM-4.6V. None of them passes the selection check in all four suites, and 17 of their 20 success-rate cells fail, so the problem is not specific to the three original judges.
- The ceiling. Section 3 replaces it with a replication reference and a paired reference. The paired reference is what a perfect judge achieves on the same episodes, which is the setting you pointed out.
- Statistical claims. Section 2.2 and Appendix C state the assumptions of the ordering rule and add a Bonferroni correction, which changes no verdict. The sample-size shortfall is now claimed only for Long, based on direct simulation rather than Proposition 2.
- Existing practice. The problem is now stated as a high positive correlation with unreliable selection, and Table 1 reports MMRV on the same policy sets.
We are grateful for these comments, which made the paper more careful.
Response to Reviewer q5eX
Thank you for a clear review. Each weakness led to a change.
"The broader significance of the findings is unclear without testing a stronger multimodal judge."
We ran four stronger judges with a decision rule fixed in advance (general response, point 3). Each passes selection in three of four suites, where the original judges passed one or two, and orders most real policy pairs correctly. None passes selection everywhere, the failing suite differs between them, and together they fail the reporting test in 14 of 16 cells. On interpretation, if a judge had passed every cell, the protocol would have shown it. The finding that a high correlation does not certify the choice of a policy would still stand, because it concerns what correlation can tell a reader about any evaluator. In this setting the problem shrinks with scale for choosing a policy but not for reporting success rates, and the failing suite cannot be predicted from size. The per-setting check therefore stands, and the revision will drop any suggestion that larger judges cannot help.
"I am not convinced that the proposed ceiling applies to the main experiments."
The reviewer is right, and point 4 of the general response gives the change. The ceiling is a replication distribution for independently collected rollouts. For the same-episode design we will add a paired reference, which is what a perfect judge achieves. On our ladders both equal 1.000 in all cells, so Table 1 reads the same.
"Some statistical claims need more justification."
The ordering rule assumes that the residuals of the judge's rates around the line q = a·p + b are independent across policies, have equal variance and are roughly normal. The √2 comes from the difference of two residuals. The revision will state these assumptions and apply a Bonferroni correction over the adjacent pairs. No verdict changes in the 12 cells of Table 1 or the 9 pre-registered settings of Appendix C. Shapiro–Wilk tests give p ≥ 0.15 except for the 7B judge on Goal, where a is close to zero.
We also agree that Proposition 2 gives a sufficient budget and not a necessary one. We simulated the published OpenVLA leaderboard directly at 500 rollouts per method. This identifies the leader with probability about 0.99 on three suites but only 0.713 on Long. Reaching 95% needs about 300 rollouts on those three suites and about 2,300 on Long. The revision will replace the claim that both sample-size answers exceed 500 with these numbers and keep the shortfall claim only for Long.
"The comparison with existing evaluation practice is not fully convincing."
We agree on both points. A strong negative correlation already flags a reversal, so the revised Section 1 will state the problem as a high positive correlation with unreliable selection. All 10 cells with ρ ≥ 0.82 have positive ρ. We computed MMRV and Pearson's r on the same policy sets and will report both. MMRV catches the two worst cells, and its rank correlation with regret across the 12 cells is 0.91, so it is a good proxy for regret. What it misses is unreliable selection when the point estimates happen to be in order. In the nine cells with MMRV of 0.05 or less, P(top-1) ranges from 0.306 to 1.000. Regret in those cells reaches 9.6 points. The protocol adds how often the choice would be right if the evaluation were repeated, and a separate check for reporting success rates.
We hope these answers address your concerns, and we are happy to discuss any point further.