acceptodds
Under review as a conference paper at ICLR 2027

Measure, Don't Average: Small-Cohort Measurement of VLA Policies

Abstract

Vision-language-action (VLA) studies often compare only four to six policies, yet report benchmark means without testing whether items, ranks, and uncertainty are identified. We audit a 3,155-episode matrix over six variants and a 14,412-episode LIBERO-Plus spatial bank. Standard 2PL estimation converges while item discrimination is prior-sensitive at . A hierarchical Bernoulli model grounded in observed rollout structure predicts unseen initializations (log-loss .1865 versus .1978) and has the best unseen-task point loss (.2039 versus a predeclared .2596 baseline, but not separated from a .2084 suite mean). Under a common observed-compatible data-generating process, nominal-90% measurement-replication coverage is .912 for the hierarchy, .898 for 1PL, and .931 for 2PL; the hierarchy has the best interval score but is not uniformly better calibrated. An equal-12,800-observation control reverses our earlier policy-count explanation: discrimination recovery is .764 with repeats and .731 with . For acquisition, we implement outcome-enumerated joint weak-rank information gain and compare matched atomic task queries on independent future matrices. It beats uniform, but does not robustly beat task EIG or Fisher: 25% rank-difference intervals include zero on both pre-specified data-generating processes, no method has a median sustained- budget within 50%, and the joint method triggers no model-based stops. Architecture-DIF tests also remain negative and partly unidentified. The contribution is validated small-cohort measurement with transparent negative boundaries, not generic adaptive or certified robot evaluation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action policies, robot policy evaluation, small-cohort measurement, uncertainty quantification, hierarchical models, benchmark validity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZoboOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a hierarchical model for comparing VLA policies on LIBERO and LIBERO-Plus. It shows that confident rankings on tested tasks may not hold in a new evaluation. Adaptive task selection beats random selection but does not consistently outperform existing baselines.

Critical strengths
  1. The paper clearly shows why confidence in a ranking on tested tasks does not guarantee that it will hold in a new evaluation. This is useful for interpreting VLA comparisons.
  2. The hierarchical model achieves a statistically significant reduction in log-loss compared with the beta-binomial baseline on unseen initializations.
Critical weaknesses
  1. On unseen tasks, the model does not show a clear improvement over a simple variant–suite average. For unseen initializations, it is unclear whether the gain comes from the hierarchy or from how repeated trials are handled.
  2. The interval-score improvement is shown only on data generated by the proposed model. Without the corresponding results under the Rasch model, it is unclear whether this advantage holds more generally.
  3. Key details needed to reproduce the model are missing, including the priors, hyperparameters, and identification constraints. Without these details or access to the code, the reported results are difficult to reproduce.
Questions for authors
  1. How much of the gain on unseen initializations comes from the hierarchy versus repeat deduplication? An ablation would help separate these effects.
  2. How do the interval scores of HB, 1PL, and 2PL compare under the Rasch model?
  3. What numerical priors, hyperparameters, and identification constraints are used, and how sensitive are the results to the choice of priors?
  4. Will the code be released publicly?
Reviewer_jdAtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the unreliable issue of averaging scores from small batches of evaluations in VLA, and proposes a hierarchical Bernoulli measurement model with robot replay characteristics. This model can predict the ranking uncertainty of the evaluation targets for unseen tasks and initialization/output distinctions.

Critical strengths

This work is focused on VLA evaluation. It outputs the posterior uncertainty of strategy rankings by differentiating among various evaluation targets, and reveals the problem that relying solely on average success rate would lead to false and stable rankings.

Critical weaknesses
  1. The proposed hierarchical Bernoulli model shows no statistically significant advantage in task prediction compared to the simple baseline of the kit mean. The model gain is limited.
  2. Adaptive evaluation and architecture DIF detection both exhibit significant failures and no practical adaptive evaluation solution has been produced.
  3. The experiments rely on the data generation process generated by simulation. Some conclusions are derived from simulation modeling and are not fully equivalent to the real robot rollout noise.
Questions for authors

See the above Weaknesses.

Reviewer_86MAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Benchmark means over four to six policies are reported routinely, without asking whether the items, the ranks, or the uncertainty are identified at that sample size. This paper treats robot-policy evaluation as a measurement problem rather than an averaging problem: it assembles two audited response matrices over six policy variants, documents three concrete small-cohort pathologies, and proposes a hierarchical Bernoulli model whose trial layer reflects the repeat structure the authors actually measured. It then formalizes the point that a rank probability is incomplete without its estimand, and reports two negative results as results — an adaptive-acquisition study that beats uniform selection but not simpler targeted rules, and a differential-item-functioning analysis that cannot be identified in this design.

Critical strengths
  1. The problem is important and widespread, and the field is ignoring it. Transient visual mislocalization is a realistic deployment fault, and because a VLA executes action chunks, a brief perceptual error is baked into executed motion even after the image returns to normal (Fig. 2. The framing is new: evaluation as a measurement problem, not an averaging problem. Recasting policy evaluation as a measurement problem rather than an averaging problem, and importing the corresponding apparatus, is a genuinely different framing of a familiar complaint.
  2. Estimand awareness is formalized, not merely mentioned. Defining rank probability conditioned on a declared target, and showing the same comparison move across targets, turns "ranking is uncertain" into an operational statement.
Critical weaknesses
  1. Novelty relative to prior work is unclear. The paper cites a larger-scale IRT reliability simulation study as motivation, but it does not clearly state what remains new beyond that work. As written, it is difficult to tell whether this paper provides an independent empirical confirmation, a methodological extension, or a narrower specialization to robot-policy evaluation. The claims in the abstract overreach the evidence. The empirical scope covers six checkpoints, one benchmark family, and simulation-only experiments. Yet the abstract reads as if it is making a general statement about evaluation practice. The evidence currently supports a more limited claim about these policies, on this suite, under this ceiling proportion—and that ceiling proportion itself appears to be policy-dependent.
  2. The main contribution is presented as a tool, but the tool is not accessible. With public release deferred and the reproducibility materials kept private, readers cannot fit the model, run the screening procedure, or apply the estimand taxonomy to their own response matrices. For a paper whose strongest positive claim is a validated small-cohort measurement model, this is a serious limitation: the deliverable is a method that others cannot currently execute. Either the specification should be complete enough for reimplementation—including the pooling structure, prior scales, and the bundled parameterization—or the paper should be reframed as a case study. At present, it claims to offer a tool while withholding it.
  3. The empirical advantage is narrower than the contribution list suggests. On the harder estimand involving unseen tasks, the model does not outperform a trivial predictor; its performance is indistinguishable from simply predicting the variant–suite mean, and the reported interval includes zero. The abstract acknowledges this limitation, but the contribution list overstates the result by claiming improvement on both unseen initial states and unseen tasks. What is supported is improvement only for unseen initial states, where the interval excludes zero. The preregistered suite-level baseline is also beaten only marginally, and largely because it performs worse than directly predicting the suite mean.
Questions for authors
  1. Even if weights and code cannot be released, can the response matrices be? If not, would the paper be reframed as a case study?
  2. Please state explicitly what remains new relative to the larger-scale IRT reliability simulation study you cite as motivation.
  3. Negative results are reported as results. The limitations section is unusually complete. Could the authors please explain the underlying rationale for this framing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.