- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a hierarchical model for comparing VLA policies on LIBERO and LIBERO-Plus. It shows that confident rankings on tested tasks may not hold in a new evaluation. Adaptive task selection beats random selection but does not consistently outperform existing baselines.
Critical strengths
- The paper clearly shows why confidence in a ranking on tested tasks does not guarantee that it will hold in a new evaluation. This is useful for interpreting VLA comparisons.
- The hierarchical model achieves a statistically significant reduction in log-loss compared with the beta-binomial baseline on unseen initializations.
Critical weaknesses
- On unseen tasks, the model does not show a clear improvement over a simple variant–suite average. For unseen initializations, it is unclear whether the gain comes from the hierarchy or from how repeated trials are handled.
- The interval-score improvement is shown only on data generated by the proposed model. Without the corresponding results under the Rasch model, it is unclear whether this advantage holds more generally.
- Key details needed to reproduce the model are missing, including the priors, hyperparameters, and identification constraints. Without these details or access to the code, the reported results are difficult to reproduce.
Questions for authors
- How much of the gain on unseen initializations comes from the hierarchy versus repeat deduplication? An ablation would help separate these effects.
- How do the interval scores of HB, 1PL, and 2PL compare under the Rasch model?
- What numerical priors, hyperparameters, and identification constraints are used, and how sensitive are the results to the choice of priors?
- Will the code be released publicly?