acceptodds
Under review as a conference paper at ICLR 2027

Repeat Before You Route: Evaluating Adaptive Compute for VLA Policies

Abstract

Adaptive robot compute depends on a simple question: which tasks benefit from a different compute setting? We introduce Repeat-to-Select, a method that measures the opportunity for this choice using repeated rollouts. A repeat-product estimator isolates stable differences in success probability; unbiased centering then separates the advantage of a better default from task-dependent selection opportunity. We derive a sharp upper bound on the gain available to a task-based selector and a local lower bound linking the number of tasks, repeats, and detectable effect size. Together, these results turn repeated outcomes into a procedure for choosing defaults, measuring selection opportunity, and allocating an evaluation budget. On 120 tasks with three repeats per setting, increasing the executed action chunk from one to ten improves success by 5.83 percentage points and reduces policy calls by 91.56%. A separate 360-episode five-action comparison supports ten-action execution with 50.17% fewer calls and a nominal success-loss bound within two points. Solver-depth selection remains statistically unresolved: its nominal ceiling is 4.17 points and its conservative ceiling is 26.72 points. Across 110,610 public code and agent outcomes, apparent oracle gains exceed cross-validated same-task selection gains in all eight panels. The method and released outcomes provide a common basis for improving fixed settings and evaluating task-based compute allocation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, adaptive computation, robot learning, repeated evaluation, compute allocation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.40 over 5 reviews, above 9% of the others.
Reviewer_sLkoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the evaluation of task-conditioned adaptive compute allocation for vision-language-action (VLA) policies. The authors argue that, before developing an adaptive compute selector, it is important to determine whether observed performance differences between compute configurations reflect stable task-dependent effects or stochastic variation across repeated executions.

To address this question, the paper proposes Repeat-to-Select, a statistical framework that uses independent repeated rollouts to estimate stable differences in task-specific success probabilities. The method constructs an unbiased repeat-product estimator of squared probability differences and introduces an unbiased finite-panel centering correction to distinguish average configuration advantages from task-dependent heterogeneity. Based on this formulation, the authors derive a sharp upper bound on the potential gain of a task-conditioned oracle over the best fixed configuration, together with a local sample-complexity lower bound relating the number of tasks, repeated evaluations, and detectable effect size.

The empirical evaluation primarily examines solver depth and action execution horizon for a π0.5 policy on LIBERO-Plus. The authors additionally reanalyze six previously collected robot evaluation panels and eight public code-generation and software-agent evaluation panels to study repeat-stable effects and hindsight oracle bias. A simple ridge-based task-conditioned selector is also evaluated to examine the distinction between theoretical selection opportunity and realized predictive performance.

Critical strengths

This is a meaningful problem formulation that clarifies an important evaluation issue in adaptive compute allocation.The most compelling aspect of this paper is its observation that apparent performance differences across stochastic executions should not automatically be interpreted as predictable signals for task-conditioned routing.In particular, the distinction among hindsight oracle gains, an oracle defined using underlying task-specific success probabilities, and the performance of a learned selector is conceptually important. It addresses a common evaluation pitfall in which favorable realized outcomes are mistaken for evidence of learnable configuration preferences.

The repeat-based effect estimator, unbiased centering correction, and sharp routing bound provide a coherent theoretical formulation of this issue. Although the individual mathematical ingredients are based on established statistical techniques, their integration into a specific compute-selection evaluation problem has methodological value.This perspective is potentially relevant beyond VLA policies, including stochastic model routing and software-agent evaluation.

Critical weaknesses

1.Despite its emphasis on adaptive compute for VLA policies, the proposed methodology is largely domain-agnostic and its theoretical guarantees are restricted to task-level pre-execution selection. The connection to the richer online adaptation mechanisms commonly used in VLA policies is limited. In addition, the paper does not sufficiently establish what distinctive decision-making capabilities its statistical framework provides over existing repeated-measurement approaches.

2.Although the unbiased estimators and sharp routing bound are mathematically meaningful, the resulting quantities are only indirectly related to exploitable routing gains. The centered dispersion measure does not necessarily imply changes in the optimal configuration, and the conservative empirical routing bounds remain too wide to support decisive conclusions under the available evaluation budget.

3.The VLA experiments are not sufficiently comprehensive. The prospective evaluation is limited in model, environment, and configuration coverage, while the statistical resolution remains insufficient for some central claims. The retrospective robot analyses and cross-domain code/agent experiments provide useful supplementary evidence but do not establish a convincing end-to-end demonstration of improved adaptive compute decisions. The fragmented empirical organization and relatively dense presentation further weaken the coherence of the overall contribution.

Questions for authors

1.What distinctive advantages does Repeat-to-Select provide over existing statistical evaluation approaches?

Could the authors clarify how the proposed framework differs substantively from classical repeated-measures variance analysis, random-effects modeling, or hierarchical Bernoulli estimation? In particular, under comparable data and evaluation budgets, can the proposed method provide more accurate task-effect estimates, better-calibrated confidence bounds, greater statistical power, or more reliable compute-routing decisions than reasonable alternatives?

2.Is the current VLA evaluation sufficient to support the claimed application scope?

The main prospective results are concentrated on a single π0.5 checkpoint, a limited configuration menu, and LIBERO-Plus simulation tasks.Could the authors provide additional evidence regarding generalization across VLA architectures, task distributions, or more comprehensive compute-configuration settings? I would particularly appreciate an analysis of interactions between solver depth and execution horizon, together with stronger matched comparisons against fixed-setting and task-level selection baselines.

Reviewer_VbeaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an estimator from repeated trials for designing an adaptive controller that balances how frequent the solver is called and how deep each solver call is allowed to run. The paper proposes an estimator and derives bounds, and verified its results on robot and agent tasks.

Critical strengths

I empathize with the goal that an adaptive controller should ideally allocate computing where it matters the most.

Critical weaknesses
  1. The clarity is lacking and I find it very difficult to understand the full picture to effectively assess the rigor of this paper. I recommend the following at the very least
  • create a visual diagram that depicts the main method end to end
  • ground the method (Section 3) in a minimal running example, such as the robot placing a bowl on a plate that you mentioned at the beginning
  1. The results section seems incomplete. I do not see how the result table (Line 341) and the plot (figure 3) relates to the usefulness of the proposed estimator. It only shows the compute-time tradeoff.
  2. Section 5 and Section 6 are only stating experiments and numbers without emphasizing on why the experiments are necessary or connected to the thesis.
Questions for authors

Please see weaknesses.

Reviewer_s4ucOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether task-dependent compute routing offers a genuine advantage over using the best fixed setting. Repeat-to-Select uses independent repeated outcomes to separate stable differences in success probability from rollout noise. It then removes the average setting advantage and derives an upper bound on the gain available to a task-identity selector.

The evaluation covers robot policies, code generation, and agent tasks. On 120 LIBERO-Plus tasks, increasing the execution horizon from one to ten actions improves success by 5.83 percentage points while reducing policy calls by 91.56%. However, the benefit of solver-depth routing remains unresolved, and the learned task-level selector underperforms the best fixed depth. Across eight public code/agent panels, apparent hindsight-oracle gains consistently exceed cross-validated selection gains.

Critical strengths

1. The paper highlights a genuine problem in evaluating adaptive compute.

A configuration that succeeds on a particular rollout is not necessarily one that could have been selected reliably before execution. The paper makes this distinction explicit, and the public-data analysis illustrates how hindsight-oracle comparisons can exaggerate the apparent benefit of routing. I find this conceptual message useful, particularly for evaluations involving stochastic outcomes.

2. The experiments are carefully documented, including results that do not support the main narrative.

The authors retain all completed episodes, report negative estimates and inconclusive findings, and provide repeat-level outcomes. This transparency makes the results easier to assess. The experiments also reveal a useful fixed-policy observation: reducing solver depth from ten to one produces similar observed success rates (76.94% vs. 76.39%) while reducing inference time by approximately one third.

Critical weaknesses

1. The proposed bound is mathematically meaningful, but its practical value for deciding whether to use a router remains unclear.

The motivation is to determine whether task-dependent routing can offer a worthwhile improvement over a fixed setting. Yet the reported bounds rarely provide a sufficiently informative answer.

Across the eight public code/agent panels, the plug-in routing ceilings are approximately 17–29 percentage points, substantially larger than the reported apparent oracle gains of 1.5–14.7 points. For example, Nano default shows only a 2.2-point apparent oracle gain and a −2.8-point cross-validated selection gain, while the reported conservative ceiling exceeds 41 points. These empirical quantities are not formal bounds on the same population parameter, but the contrast illustrates how little the ceiling narrows the practical question.

There is also a structural limitation. The dispersion (V) measures variation in relative setting effects, even when the same setting remains optimal for every task. Consider two settings where A outperforms B by 5 points on half the tasks and 25 points on the rest. The proposed inequality permits a 5-point routing gain, although the true gain is zero because A is always optimal. This is precisely the distinction between quantitative and qualitative interaction studied by Gail and Simon (1985).

The prospective depth experiment provides a nominal bootstrap ceiling of 4.17 points, but the conservative finite-sample ceiling is 26.72 points and cannot exclude a ten-point improvement. Proposition 3 further illustrates the limited detection power available with small repeat counts under its stated model.

2. The methodological contribution appears incremental, while the strongest empirical result does not require the proposed framework.

The main statistical ingredients have close classical counterparts: unbiased estimation of quadratic effects, variance-component estimation through repeated observations, and bounds on maximal deviations from the mean. The routing inequality is closely related to Samuelson's (1968) result, while the testing lower bound follows a standard Le Cam argument. Although the paper acknowledges energy statistics, its relationship to the broader treatment-effect heterogeneity literature is less clear.

Combining existing tools for a new evaluation problem can certainly be valuable. My concern is that the paper does not sufficiently establish what new decision-making capability this combination provides.

The largest empirical improvement, comparing (E=10) against (E=1), is a direct fixed-setting comparison that does not depend on Repeat-to-Select. The reduction in policy calls is also largely expected when the execution horizon is increased, and executing only one action from a ten-action chunk is a relatively extreme baseline.

The more relevant (E=5) versus (E=10) comparison shows little observed difference in success (282 versus 284 successes out of 360). It was also collected later, on different hardware, after the original results had been inspected. The authors disclose these limitations, but the comparison does not establish a clear additional benefit from the proposed analysis.

Questions for authors

Q1. Can the authors identify a reported setting where the proposed bound is sufficiently tight to rule out a practically meaningful routing gain? More specifically, what decision would Repeat-to-Select support that could not already be reached through fixed-setting comparisons or held-out selection evaluation?

Q2. Can the authors distinguish variation in setting advantages from actual changes in the optimal setting across tasks? A qualitative-interaction analysis, or a suitably validated estimate of the oracle routing gain, would help establish whether the measured heterogeneity is decision-relevant.

Q3. How were the solver depth and execution horizon chosen for the main experiments? In particular, how does the D=4D=4 prefix comparison relate to the standard OpenPI LIBERO evaluation configuration?

Reviewer_VdmkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

the paper studies adaptive compute router for robot policies. the main problem is that picking the winner by just running the task once is not the right approach due to task execution variance. so the paper proposed a method called repeat-to-select where you run multiple rollouts before select the settings. the high level idea makes total sense.

Critical strengths
  1. the idea makes a lot of sense. task execution is known to be brittle especially in robot setting. picking the winning setting from just one rollout leads to suboptimal solutions.

  2. the paper provides math and theoretical justification of the approach, which looks sound.

Critical weaknesses
  1. the novelty is limited. the variance problem is well known in many domains. the paper would benefit more on which part is specifcally challenging in this problem beyond just repeating the rollouts multiple times. and how the paper addresses those problems.

  2. the evaluation is a major weakness. there is only one experiment in the robot domain and the evaluation doesn't look convincing enough. it is also a bit surprising that the paper suddenly shift gear and discuss results in the coding agent space. i think the paper should focus more on robot? otherwise the paper should reposition itself on a general framework, which i believe the paper should be rewritten including the motivations and introductions.

  3. the problem is essentially a hyperparameter tunning problem where one can run experiments parallelly and sequentially. the models can perform chain of thoughts so we have to justify why a selection mechanism is necessary in this process since the method already runs longer and repeats. the paper didn't really put much effort in justify the need of such routing mechanism.

Questions for authors

i hope the authors could respond to my concerns in the weakness section.

Reviewer_DLRoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Repeat-to-Select, a statistical framework for evaluating whether task-dependent compute allocation can outperform the best fixed configuration in VLA policies. It introduces a repeat-based estimator for separating stable configuration effects from stochastic rollout variation, an unbiased correction for task-dependent heterogeneity, and theoretical bounds on the potential benefits and sample complexity of task-based routing.

The authors evaluate solver depth and execution horizon using π0.5 on 120 LIBERO-Plus tasks. Increasing the execution horizon from 1 to 10 improves success by 5.83 percentage points while reducing policy calls by 91.56%. The framework is further examined on six previously collected robotics panels and 110,610 public code-generation and agent outcomes. A learned task-conditioned solver-depth selector achieves 76.67% success, compared with 78.89% for the best fixed configuration.

Critical strengths

The paper provides a mathematically explicit formulation for distinguishing stochastic rollout variation from stable task-dependent differences in success probability. Its repeat-based estimator and theoretical routing bounds offer a systematic approach to evaluating apparent gains from adaptive compute selection.

Critical weaknesses

The experimental evaluation is limited in scale and diversity. The main experiments cover only 120 task variants with three repeated rollouts per configuration, despite LIBERO-Plus containing over 10,000 task variants. Only four compute configurations are included in the main prospective evaluation, and no additional prospective VLA evaluation is conducted on another robotic benchmark. Although the authors analyze previously collected robotics and public code/agent results, these do not fully address the limited coverage of the primary VLA experiments. Furthermore, the robustness of the results to alternative random seeds remains insufficiently explored, particularly for comparisons with small or statistically inconclusive performance differences.

The practical value of the proposed framework remains insufficiently demonstrated. While the framework does not necessarily require a successful learned router to be statistically valid, its usefulness for making actionable compute-allocation decisions is unclear. For solver-depth selection, the nominal routing-gain ceiling is 4.17 percentage points, whereas the conservative finite-sample ceiling reaches 26.72 points, leaving substantial uncertainty about the potential benefit of adaptation. Moreover, the only learned task-conditioned router underperforms the best fixed configuration (76.67% versus 78.89%). Consequently, the experiments demonstrate the effectiveness of choosing a better fixed execution configuration but provide limited evidence that the proposed statistical framework can reliably guide practical adaptive compute allocation.

Questions for authors
  1. Why are only 120 task variants selected from LIBERO-Plus, which contains over 10,000 variants? How representative is this subset, and can the authors provide results on a substantially larger evaluation set or an additional robotic benchmark?

  2. The experiments use only three independent seeds (211, 223, and 227). Can the authors demonstrate that the reported improvements remain consistent across additional independent seed sets? In particular, how robust are the relatively small differences between compute configurations to stochastic rollout variations?

  3. Given that the learned task-conditioned router underperforms the best fixed configuration, what empirical evidence supports the practical benefit of the proposed framework for adaptive compute allocation, beyond identifying a better fixed configuration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.