acceptodds
Under review as a conference paper at ICLR 2027

Success Rates Are Not Model Properties: Controlled Diagnostics for VLA Evaluation

Abstract

Success rates on standard manipulation benchmarks are the primary evidence for comparing vision-language-action (VLA) models. But a success rate is not a model property: it is measured under conditions fixed by the benchmark or by the configuration distributed with the model. We show that three of these conditions, changed with the model fixed, move success by amounts that are large for some models and small for others. The three are (1) which initial states are tested, (2) what instruction the model receives, and (3) how many predicted actions it executes between observations, the execution horizon. We test multiple VLA models on LIBERO and LIBERO-Plus, and execution also in SimplerEnv. (1) For initial states, X-VLA and π0.5 both succeed on about 98% of LIBERO-Object's standard initial states at the same execution horizon, but on 22.1% and 89.5% of placements outside these states on which a separate VLA model succeeds at a high rate. After task-specific fine-tuning of π0.5, success on the same placements falls by more than the standard score changes, for every recipe we tested. (2) For the instruction, removing the parameters that LIBERO-Plus appends to most instructions raises the success of the two SmolVLA models by up to 51.7 percentage points and changes the other eight models by at most 8.2 points. For these two models, the delivered text, not just the scene, drives success. Appending two ordinary words to standard instructions also lowers their success, even though both models perform another task when given that task's instruction. (3) For execution, a horizon of ten instead of each model's default raises the standard score of five of the six models on LIBERO and also changes success on the placements from (1), while in SimplerEnv each tested configuration does worst at its shortest horizon. Because models respond differently, these conditions move the margins between models and sometimes reverse which model scores higher, so a comparison made under one setting does not necessarily hold under another. We give an evaluation procedure that reports each model's response to these conditions and states when a comparison holds, asking not only how often a model succeeds, but under which conditions it does.

Primary area:
datasets and benchmarks
Keywords:
vision-language-action models, VLA evaluation, benchmark validity, evaluation protocols, robot manipulation, robustness evaluation, action chunking, LIBERO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_bquiOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a controlled empirical study of 3 evaluation conditions for VLA models: initial-state distribution, language instructions, and execution horizon. The authors show that changing these conditions can substantially affect success rates. They propose an evaluation procedure that combines reference-selected initial states, paired comparisons, and reporting model responses across evaluation conditions.

Critical strengths

The experiments include useful controls. In particular, the fine-tuning experiment with fixed normalization statistics and the instruction repair, filler, and instruction-switching tests help rule out several alternative explanations. These results are useful for researchers choosing evaluation settings for these models.

Critical weaknesses
  1. My main concern is the limited advance over existing evaluation studies. LIBERO-PRO already shows that high LIBERO scores can coexist with severe failures under changes to object positions and other evaluation conditions. Execution-horizon sensitivity is also directly demonstrated in many VLA diagnostic papers. For the instruction study, the appended metadata and its effect on policy success were publicly reported in LIBERO-Plus (issue #65), as the submission acknowledges. The additional measurements and controls are useful, but I do not find that they add up to a substantial scientific contribution. The paper adds controlled evidence on adaptation and instruction sensitivity, but the methodological advance provided by its evaluation procedure remains unclear to me. I do not see a clear scientific advance beyond documenting these effects across more models.

  2. Selecting additional states using another model’s success is a reasonable way to establish that they are solvable, but it also makes the evaluated distribution depend on the reference model. Appendix B.3 shows that reference-selected success does not exceed standard success under the tested reference choices, with some ties; it does not report how those choices affect pairwise model rankings and margins. This matters because reliable model comparison is a central motivation for the procedure. The external validation in Appendix D.2 also does not show an advantage over the standard score: both achieve Kendall’s τ of 0.60 against real-robot rankings, and the interval for the difference in rank violation includes zero. These results support using the procedure as an additional diagnostic, but leave its added value as a comparison methodology unclear.

Questions for authors
  1. Which conclusion enabled by the additional controls goes beyond prior perturbation studies, and how should it change evaluation practice?

  2. How do the pairwise model rankings and margins change when different reference models are used to select states in Appendix B.3?

Reviewer_8BETOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper studies how initial-state coverage, instruction text, and execution horizon affect VLA success rates and model comparisons. It proposes controlled diagnostics and a reporting procedure across LIBERO, LIBERO-Plus, and SimplerEnv.
Critical strengths
  • Reliable VLA evaluation is an important problem. Showing that nearly identical benchmark scores can conceal large robustness differences provides actionable guidance for researchers and practitioners.
  • The controls are thorough: paired evaluations, alternative references and thresholds, normalization controls, instruction-following tests, and seeded replications strengthen the main observations.
  • The reporting checklist and condition-dependent comparisons could improve evaluation practice and motivate broader robustness studies.
Critical weaknesses
  • The methodological novelty is modest. Sensitivity to distribution shifts, prompts, and execution horizons is already established, and some specific issues were previously reported. The contribution is primarily systematic measurement; clearer evidence that the proposed procedure improves model selection would strengthen its significance.
  • Reference-selection robustness is only partially established. Alternative references and thresholds preserve the standard-versus-translated performance gap, but this does not establish stability of inter-model margins or rankings. Report those comparisons across reference choices and quantify uncertainty from selecting conditions using finite reference rollouts.
  • Several findings have narrow empirical scope. Large instruction effects concentrate in the two SmolVLA variants, while fine-tuning covers only π0.5 on three tasks using scripted demonstrations without rotation. The fixed-normalization control is valuable, but broader models, demonstration sources, and randomized text variations are needed to establish generality.
Questions for authors

see the weakness section.

Reviewer_NPDuOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how three evaluation choices affect reported success rates of vision-language-action models: initial-state coverage, instruction wording, and execution horizon. The authors conduct controlled studies on LIBERO, LIBERO-Plus, and SimplerEnv, varying one evaluation condition at a time while keeping the model fixed. They report substantial and model-dependent changes in success rates, including effects from translated object placements, task-irrelevant instruction text, and action execution horizons. The paper also examines how these interventions affect model comparisons and argues that success rates should be reported together with the conditions under which they are measured. The supplementary material provides implementation details, analysis scripts, and protocol checks for the reported experiments.

Critical strengths
  1. The paper studies an important evaluation problem using controlled interventions. Varying initial states, instruction text, and execution horizon while holding other factors fixed is a useful way to expose hidden protocol dependence in VLA benchmarks. The reported effects are often large, and the percentage-point changes in the main tables are internally arithmetically consistent.

  2. The work provides a potentially useful evaluation-audit perspective. The paper does not only report one model improvement, but examines how benchmark choices can change absolute success rates and model comparisons. The supplementary code and protocol documentation also make the experimental setup more inspectable than a purely results-only submission.ng.

Critical weaknesses
  1. The reference-selected state condition is vulnerable to selection bias and incomplete uncertainty accounting. The authors use VLA-JEPA to select 27 conditions from 50 candidates with a 75% threshold, then treat the selected mask as fixed when estimating uncertainty. This does not propagate uncertainty from the reference selection itself. The result may therefore reflect the particular reference model, threshold, and selected locations rather than a robust causal effect of initial-state coverage. The use of a single reference and positions outside the benchmark’s usual initialization region further weakens the interpretation of this experiment.

  2. The evidence is too limited to support the paper’s broader claims about VLA evaluation. The adaptation study covers one model and three tasks, the state study uses a small number of locations and 20 initial states per condition, and SimplerEnv uses only 24 episodes per task. The main LIBERO-Plus results also rely heavily on single runs. In addition, horizon is measured by action count even though models may have different control frequencies and action semantics, so equal horizons may correspond to different physical durations. The released artifact does not include checkpoints, raw episode-level outputs, or all generated results needed for a complete independent rerun.

Questions for authors
  1. Can the authors release the complete episode-level outcomes and the exact reference-selection mask, including the random seeds, reference model, threshold, and candidate-location list? What are the results when the mask-selection procedure is repeated across multiple references and thresholds, with uncertainty propagated through the selection step?

  2. Can the authors report horizon comparisons matched by physical execution time or control frequency, in addition to action count? Do the observed effects and model rankings remain when the comparison is made using matched wall-clock or simulated time?

  3. For every main table, how many independent runs, seeds, and episodes contribute to each number? Can the authors provide confidence intervals and explain how multiple comparisons across models, suites, horizons, and instruction variants were handled?

  4. Is there a single documented command, exact code commit, configuration, checkpoint set, and environment specification that reproduces every main-table result? If not, which results can currently be reproduced from the supplementary package alone?

Reviewer_WqZsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a VLA success rate is a property of a model together with its evaluation conditions. Holding public checkpoints fixed, it varies one condition at a time: (1) Initial states, by translating the target object in LIBERO-Object. (2) The instructions, by removing the parameter strings that LIBERO-Plus appends or by appending neutral words. (3) The execution horizon h. Models with nearly equal (98%) standard success differ by ~67 points on the translated placements (π0.5 89.5% vs. X-VLA 22.1%). Text effects are large only for the two SmolVLA models, and setting the execution horizon h = 10 instead of the default raises five of six models by 3.4-23.8 points. The paper there proposes an evaluation procedure and a reporting card.

Critical strengths
  1. Rigorous design and statistics. Each study changes one condition on fixed checkpoints, pairs episodes on shared initial states, and includes extensive robustness checks.
Critical weaknesses
  1. The general claim is broader than the evidence, and part of it was already known (L103-108).

(a) Instruction: large effects occur only for the two SmolVLA models. The other eight change by at most 8.2 points. This shows that SmolVLA is fragile to extra text, not that the instruction matters for VLAs in general.

(b) Execution: h = 10 improves exactly the five models whose default is h = 50, an unusual setting. Among h = 5, 10 and 25, these models differ by at most 8.0 points, so the large gains mainly reflect a poor default.

(c) Comparisons: ranking reversals are rare.

  1. The adaptation result and the proposed procedure are not established in general. The adaptation study uses one model, three tasks and a narrow recipe. Likewise, reference selection is costly and depends on the model pool. Yet it predicts real-robot rankings no better than the standard score.
Questions for authors
  1. Do the large effects extend beyond the SmolVLA models, the h = 50 default and LIBERO-Object, e.g., to other models or benchmarks?
  2. Does the loss after fine-tuning persist under a standard fine-tuning recipe rather than scripted, rotation-free demonstrations?
  3. Is there evidence that the proposed procedure leads to better model comparisons than the standard score?
Reviewer_RQTxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how three evaluation conditions change measured success of fixed vision-language-action models: initial object placements, the delivered instruction, and the number of actions executed before observing again. Experiments span ten models from eight families, with different subsets entering each study, primarily on LIBERO and LIBERO-Plus and with execution-horizon experiments in SimplerEnv. The authors find that similar standard scores can conceal large differences on translated placements selected by a separate successful reference model; task-specific fine-tuning can reduce success on these placements; removing benchmark-appended text substantially improves two SmolVLA models; and execution-horizon changes produce model-dependent gains. The paper proposes paired diagnostics, explicit reporting of evaluation conditions, and comparisons of model margins across conditions.

Critical strengths

1,Consequential empirical findings supported by controls that address plausible alternatives. At a common execution horizon, X-VLA and π0.5 have similar standard success but substantially different reference-selected success (Section 4.1). The adaptation analysis includes fixed-normalization controls, while the instruction study includes filler padding, below-truncation suffix tests, and task-changing instruction controls (Appendices B.5 and C.3–C.5). These experiments make the findings more informative than an uncontrolled robustness comparison.

2,Careful treatment of what model comparisons establish. The paper uses paired conditions, reports sampling units, repeats reference selection with alternative models and thresholds, and checks instruction-repair comparisons by resampling base tasks. It also reports negative or limiting results: execution changes do not reverse the ordering in the 40 comparisons among the five default-horizon-50 models, and reference-selected scores do not improve real-robot ranking correlation in Appendix D.2. This restraint strengthens the methodological contribution.

Critical weaknesses

1,Reference-selected success remains a conditional diagnostic with an incompletely established interpretation as a reusable evaluation criterion. The main analysis selects 27 of 50 task–translation conditions using one reference model. Alternative references support the qualitative standard-versus-translated gap, but this does not establish that the sizes of model margins remain stable across reference choices. Moreover, Appendix A.2 conditions uncertainty estimates on the retained set, excluding variation in reference selection. This is reasonable for describing that fixed set, but limits how confidently the procedure can be transferred to a newly selected reference or evaluation population. The paper should more explicitly distinguish evidence of a failure case from a general ranking criterion.

2,Some headline findings have narrower empirical scope than the overall framing might suggest. The large instruction-repair effects are concentrated in the two SmolVLA checkpoints; the other eight models change by at most 8.2 points. The fine-tuning result covers one model on three tasks using scripted demonstrations that never rotate, although the fixed-statistics control importantly rules out normalization as the sole explanation. These are convincing case studies, but they do not establish that similarly large effects are typical across adaptation pipelines or language perturbations. This constrains the breadth and novelty of the contribution, particularly since the paper acknowledges that the underlying evaluation choices and specific implementation issues were already known.

Questions for authors

1,Clarification affecting the evaluation proposal: When each alternative model selects the translated conditions, do the major between-model margins and their directions remain stable? Appendix B.3 establishes robustness of the within-model gap, but does not appear to establish the corresponding robustness of rankings.

2,Statistical clarification: Were the reference rollouts used to select conditions independent of the rollout samples used for the evaluated models? How sensitive are the retained set and the principal margins to a fresh reference rollout seed? Please clarify that the reported intervals are conditional on selection.

3,Clarification affecting generality: What motivated choosing tasks 0, 5, and 8 for adaptation, and were they chosen before observing translated-state performance? Were fine-tuning settings selected independently of the translated evaluation? The answer matters for interpreting the scope of the adaptation result.

These are requests to interpret the current evidence; broader real-robot validation would be future work rather than a prerequisite for accepting the simulation findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.