- Rating
- 1
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a controlled empirical study of 3 evaluation conditions for VLA models: initial-state distribution, language instructions, and execution horizon. The authors show that changing these conditions can substantially affect success rates. They propose an evaluation procedure that combines reference-selected initial states, paired comparisons, and reporting model responses across evaluation conditions.
Critical strengths
The experiments include useful controls. In particular, the fine-tuning experiment with fixed normalization statistics and the instruction repair, filler, and instruction-switching tests help rule out several alternative explanations. These results are useful for researchers choosing evaluation settings for these models.
Critical weaknesses
-
My main concern is the limited advance over existing evaluation studies. LIBERO-PRO already shows that high LIBERO scores can coexist with severe failures under changes to object positions and other evaluation conditions. Execution-horizon sensitivity is also directly demonstrated in many VLA diagnostic papers. For the instruction study, the appended metadata and its effect on policy success were publicly reported in LIBERO-Plus (issue #65), as the submission acknowledges. The additional measurements and controls are useful, but I do not find that they add up to a substantial scientific contribution. The paper adds controlled evidence on adaptation and instruction sensitivity, but the methodological advance provided by its evaluation procedure remains unclear to me. I do not see a clear scientific advance beyond documenting these effects across more models.
-
Selecting additional states using another model’s success is a reasonable way to establish that they are solvable, but it also makes the evaluated distribution depend on the reference model. Appendix B.3 shows that reference-selected success does not exceed standard success under the tested reference choices, with some ties; it does not report how those choices affect pairwise model rankings and margins. This matters because reliable model comparison is a central motivation for the procedure. The external validation in Appendix D.2 also does not show an advantage over the standard score: both achieve Kendall’s τ of 0.60 against real-robot rankings, and the interval for the difference in rank violation includes zero. These results support using the procedure as an additional diagnostic, but leave its added value as a comparison methodology unclear.
Questions for authors
-
Which conclusion enabled by the additional controls goes beyond prior perturbation studies, and how should it change evaluation practice?
-
How do the pairwise model rankings and margins change when different reference models are used to select states in Appendix B.3?