acceptodds
Withdrawn from ICLR 2027

CARE: Certifying Acceleration for Vision-Language-Action Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While vision-language-action (VLA) models have advanced rapidly, running them at every control step remains expensive. Prior work accelerates VLA inference using techniques like action chunking and visual-token pruning, typically evaluating based on latency and average task success. However, acceleration may discard information and break tasks the original policy would solve, a risk hidden by average metrics. Measuring these failures is challenging because action deviations compound over closed-loop trajectories, meaning task failure is only observable across full episodes. We therefore define an acceleration-induced failure via paired rollouts from identical initial conditions, tracking when the reference succeeds but the accelerated policy fails. To manage this, we introduce CARE, an approach for certified accelerator selection. CARE uses paired rollouts on a calibration set to provide finite-sample guarantees that acceleration-induced failure risk stays below a user-specified budget. It deploys the fastest certified candidate, falling back to the reference if none qualify. By relying only on terminal outcomes and measured compute, CARE applies unchanged across diverse acceleration mechanisms, while sequential testing and failure-triggered reference rollouts keep certification affordable. On four LIBERO suites with OpenVLA-OFT, CARE certifies – speedups while guaranteeing (at 95% confidence) that at least 85.8% of reference-solved episodes are preserved. Under tight budgets, selectors without guarantees exceed the budget in up to 75% of trials, whereas CARE stays within budget and its sequential form uses 78.9% fewer rollouts than exhaustive evaluation. CARE further generalizes to flow-step reduction for , and to Qwen3.5-9B and Llama-3.1-8B agents in Crafter.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLA inference, risk control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_s9ssOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

this paper targets an interesting deployment problem for VLAs. They select accelerated VLA policies while controlling the probability of breaking episodes that the original policy would solve. It evaluates paired, complete rollouts and uses statistical tests to certify candidates under a specified risk budget. They evaluate on Libero simulation and on the OpenVLA-OFT model they claim at least 85.8% of reference solved episodes are preserved under their speedup. Additional experiments cover flow-step reduction for pi0.5 and compute-reduced LLM agents in Crafter.

Critical strengths
  • the target metric is useful and clearly defined
  • testing easy tasks first could produce premature certification before difficult tasks appear. their balanced-round construction and proof address heterogeneous task risks
  • They do not simply choose whichever candidate happens to look safe on a small validation set. They explicitly control false certification across candidates and repeated inspections.
Critical weaknesses
  • the methodological contribution is weak. the regression metric resembles established negative-flip analysis, while certification builds directly on Learn-then-Test, multiple-testing correction, and standard anytime-valid inference. The authors should identify precisely what prior frameworks cannot accomplish and demonstrate the benefit of CARE's additions under matched evaluation conditions.
  • the headline speedups overstates the acceleration demonstrated. the reported 9 to 11 times speedups use one step replanning as the reference while candidates execute eight-action chunks!
  • The evaluation is limited to simulation, leaving the practical feasibility of CARE’s certification protocol unclear. Without this evidence, the paper demonstrates a useful statistical formulation but provides limited support for its practical value in robotic deployment.
  • the guarantees have restricted scope. The certified risk is an average over an equal-weight task mixture under the calibration distribution, so substantial degradation on individual tasks remains possible. In addition, calibration and test episodes are drawn from the same fixed pool of LIBERO initial states, which gives limited evidence about new deployment conditions. The resampling study in Section 4.2 likewise re-draws stored episodes from the same pool. The evaluation would be stronger with per-task regression results, evaluation on scenes outside the calibration pool, and full reporting of each certificate (per-candidate loss counts, e-values, and stopping points).
Questions for authors

refer to weaknesses

Reviewer_RmcMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CARE, a framework for selecting accelerated policies with finite-sample control of acceleration-induced failure (AIF): a paired, closed-loop event in which the reference policy succeeds but the accelerated policy fails. CARE combines multiple testing, task-balanced sequential certification, and failure-triggered reference execution, falling back to the reference when no candidate is certified. On OpenVLA-OFT/LIBERO, the paper reports 9.02–10.81× inference speedups over H1 and lower confidence bounds of 85.8%–95.7% on reference-success retention. An expanded-candidate experiment reduces certification rollouts by 78.9%, with additional evaluations on π0.5 and Crafter agents.

Critical strengths
  1. The risk definition is practically meaningful. AIF captures regressions that average success rates can conceal, preventing newly solved episodes from offsetting reference successes lost through acceleration. The paper also explicitly distinguishes deployment-wide AIF risk from conditional retention.
  2. The certification-cost reductions have theoretical and empirical support. Task-balanced sequential testing and failure-triggered reference queries reduce rollouts while preserving the risk guarantee. Ablations and evaluations across model families support their practical value.
Critical weaknesses
  1. The main certificates cannot yet be independently verified. The paper omits the actual Q grid, complete error allocations, and success/AIF counts, while the computation underlying Table 1’s retention bounds remains unclear. Appendix A.6 does not report the number of reference evaluations n_0 or their cost. If each of the four suites uses a reference-success lower bound with error probability 0.01, alongside a 0.04 candidate-testing budget, a direct union bound supports only 92%, rather than 95%, simultaneous confidence. The authors should clarify whether this budget is further divided across suites.

  2. The comparisons do not fully separate useful selection from conservative fallback or establish comparable evaluation costs. Zero violations under tight budgets coincide with 100% fallback. With the paper’s standard uniform correction over eight candidates, a 2% budget and 250 samples cannot certify a candidate even with zero observed AIF. In Table 3, baselines described as stopping early are nevertheless charged the full rollout cost. Table 4 lacks sufficient sequential-baseline configurations and variability across repeated streams. These issues limit the evidence for statistical and computational efficiency.

Questions for authors
  1. Which method and data produce Table 1’s retention bounds? Please provide success/AIF counts, cross-suite error allocations, n0n_0, and how additional reference evaluations are included in the cost accounting.
  2. Does Table 4 use the same candidate ordering, episode stream, and reference caching across methods? Please provide the testing configurations and clarify whether the certification-rate and cost advantages persist across repeated streams.
  3. Why are the early-stopping methods in Table 3 charged the full evaluation cost? Can violation rates, fallback rates, and speedups be reported together under tight budgets?
  4. Do the “fresh calibration streams” and experiments with n=500–2000n=500–2000 use new scenes or resampling from the fixed dataset? Do these results support conclusions about the finite empirical population or the distribution of new deployment scenes?
Reviewer_aJqkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CARE selects an accelerator for a vision-language-action policy while controlling the probability of losing episodes that the reference would solve. It defines this risk through paired, complete rollouts and combines task-balanced sequential testing with family-wise error control. Reference rollouts are needed only after candidate failures, and the procedure falls back to the reference when no candidate is certified. Experiments evaluate OpenVLA-OFT acceleration on LIBERO, flow-step reduction for π0.5, and reduced-compute language-model agents in Crafter.

Critical strengths

The paired episode-level risk targets a useful problem that average success rates can hide: an accelerated policy may lose previously solved episodes while gaining others. This makes reference preservation an explicit evaluation objective. I also value the treatment of candidate selection and sequential stopping within the same guarantee, under the stated assumptions (Theorem 1). Failure-triggered reference evaluation preserves the loss exactly while reducing evaluation cost (Lemma 1; Table 10). The black-box interface and experiments with different acceleration mechanisms add practical value, although the empirical coverage is still limited.

Critical weaknesses
  1. The empirical validation leaves important parts of the acceleration and generality claims unresolved. The headline 9–11× speedup compares H8 candidates with an H1 reference, but the paper does not provide enough timing detail to reproduce or interpret this gain: hardware, batch size, warmup, and whether timing is per call, control step, or episode. H1 is a reasonable reference for preservation, but the contribution of chunking should be separated from additional pruning or caching gains. Table 5 usefully distinguishes reduced flow computation from measured speedup; comparable clarity is needed for the main result.

The cross-domain evidence also lacks essential protocol details. Crafter reports two models, two tasks, and three seeds, without specifying the risk and error budgets, calibration size, candidate family, or learned schedules sufficiently to reproduce the certificates. Section 4.4 mentions both π0 and π0.5, while the numerical results cover π0.5. The latter may be a wording issue, but the missing Crafter protocol limits how much generality I can infer.

A further concern is the interpretation of failures as acceleration-induced. Assumption A3 permits residual simulator nondeterminism, yet reference-versus-reference rerun disagreement is not reported. I regard this as an important missing validation: without it, the contribution of ordinary execution variability relative to the target risk budget is unclear. This does not demonstrate that nondeterminism is large or invalidate risk control for the defined stochastic event; it limits the causal interpretation of that event.

  1. The contribution beyond existing risk-control methods is not convincing enough. The paper appropriately cites Learn-then-Test, Pareto Testing, and anytime-valid inference. Its distinctive elements are the paired VLA objective, task-balanced certification, and failure-triggered reference evaluation; the latter two follow relatively directly from balanced aggregation and the loss identity. I recognize the usefulness of this combination, but the paper needs a clearer account of what the adaptation enables beyond applying the existing framework, supported by stronger empirical validation. The present evidence is not sufficient for me to view it as a strong methodological contribution.

Two secondary limitations deserve clearer emphasis. At α = 2–2.5% and n = 250, zero violations coincide with 100% fallback and 1.00× speedup (Table 11). This is correct behavior, but does not demonstrate useful acceleration at those budgets. Also, the guarantee concerns an equal-weight task mixture under the stated sampling assumptions; it does not protect the worst task or establish robustness to distribution shift. I treat both as scope limitations rather than failures of the theorem.

Questions for authors
  1. Retention bound (clarification): Is the 95.7% preserved value in Table 1 obtained by applying Theorem 3 inversion to the fully paired test set? Please provide the paired counts, confidence allocation, and inversion grid/Lipschitz settings for each suite, and distinguish calibration cost from test-reporting cost. I am asking for reproducibility details, not asserting that 95.7% is inconsistent with the theorem.

  2. Resampling (clarification and additional evidence): Are trials sampled with replacement? What population produces the n > 500 rows in Table 11? Please distinguish the 10,000 resampling trials from independent environment evaluations. If fresh-scene repetitions are available, how do the selection results and uncertainty compare?

  3. Certification settings (clarification): Please provide Q, round size m, maximum n, error-budget allocation, candidate order, and the size and scene sources of Dprof for each experiment. How is Dprof separated from calibration and test data, and which settings were fixed before observing calibration outcomes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.