- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the evaluation of task-conditioned adaptive compute allocation for vision-language-action (VLA) policies. The authors argue that, before developing an adaptive compute selector, it is important to determine whether observed performance differences between compute configurations reflect stable task-dependent effects or stochastic variation across repeated executions.
To address this question, the paper proposes Repeat-to-Select, a statistical framework that uses independent repeated rollouts to estimate stable differences in task-specific success probabilities. The method constructs an unbiased repeat-product estimator of squared probability differences and introduces an unbiased finite-panel centering correction to distinguish average configuration advantages from task-dependent heterogeneity. Based on this formulation, the authors derive a sharp upper bound on the potential gain of a task-conditioned oracle over the best fixed configuration, together with a local sample-complexity lower bound relating the number of tasks, repeated evaluations, and detectable effect size.
The empirical evaluation primarily examines solver depth and action execution horizon for a π0.5 policy on LIBERO-Plus. The authors additionally reanalyze six previously collected robot evaluation panels and eight public code-generation and software-agent evaluation panels to study repeat-stable effects and hindsight oracle bias. A simple ridge-based task-conditioned selector is also evaluated to examine the distinction between theoretical selection opportunity and realized predictive performance.
Critical strengths
This is a meaningful problem formulation that clarifies an important evaluation issue in adaptive compute allocation.The most compelling aspect of this paper is its observation that apparent performance differences across stochastic executions should not automatically be interpreted as predictable signals for task-conditioned routing.In particular, the distinction among hindsight oracle gains, an oracle defined using underlying task-specific success probabilities, and the performance of a learned selector is conceptually important. It addresses a common evaluation pitfall in which favorable realized outcomes are mistaken for evidence of learnable configuration preferences.
The repeat-based effect estimator, unbiased centering correction, and sharp routing bound provide a coherent theoretical formulation of this issue. Although the individual mathematical ingredients are based on established statistical techniques, their integration into a specific compute-selection evaluation problem has methodological value.This perspective is potentially relevant beyond VLA policies, including stochastic model routing and software-agent evaluation.
Critical weaknesses
1.Despite its emphasis on adaptive compute for VLA policies, the proposed methodology is largely domain-agnostic and its theoretical guarantees are restricted to task-level pre-execution selection. The connection to the richer online adaptation mechanisms commonly used in VLA policies is limited. In addition, the paper does not sufficiently establish what distinctive decision-making capabilities its statistical framework provides over existing repeated-measurement approaches.
2.Although the unbiased estimators and sharp routing bound are mathematically meaningful, the resulting quantities are only indirectly related to exploitable routing gains. The centered dispersion measure does not necessarily imply changes in the optimal configuration, and the conservative empirical routing bounds remain too wide to support decisive conclusions under the available evaluation budget.
3.The VLA experiments are not sufficiently comprehensive. The prospective evaluation is limited in model, environment, and configuration coverage, while the statistical resolution remains insufficient for some central claims. The retrospective robot analyses and cross-domain code/agent experiments provide useful supplementary evidence but do not establish a convincing end-to-end demonstration of improved adaptive compute decisions. The fragmented empirical organization and relatively dense presentation further weaken the coherence of the overall contribution.
Questions for authors
1.What distinctive advantages does Repeat-to-Select provide over existing statistical evaluation approaches?
Could the authors clarify how the proposed framework differs substantively from classical repeated-measures variance analysis, random-effects modeling, or hierarchical Bernoulli estimation? In particular, under comparable data and evaluation budgets, can the proposed method provide more accurate task-effect estimates, better-calibrated confidence bounds, greater statistical power, or more reliable compute-routing decisions than reasonable alternatives?
2.Is the current VLA evaluation sufficient to support the claimed application scope?
The main prospective results are concentrated on a single π0.5 checkpoint, a limited configuration menu, and LIBERO-Plus simulation tasks.Could the authors provide additional evidence regarding generalization across VLA architectures, task distributions, or more comprehensive compute-configuration settings? I would particularly appreciate an analysis of interactions between solver depth and execution horizon, together with stronger matched comparisons against fixed-setting and task-level selection baselines.