acceptodds
Withdrawn from ICLR 2027

VLA-ZO: Fast Zeroth-Order Adaptation for Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adapting vision-language-action (VLA) models to deployment-time distribution shifts is important for reliable robotic operation, but conventional first-order adaptation can exceed the memory budget of inference-oriented deployment platforms. Zeroth-order (ZO) optimization offers a forward-only alternative with inference-level memory, but accurate gradient estimation requires many perturbation queries, making naive ZO prohibitively slow for large VLA models. We present VLA-ZO, a framework for fast ZO adaptation that exploits the structure of VLA computation. By confining adaptation to the action side, VLA-ZO keeps the expensive vision-language prefix frozen and reuses its conditioning states across perturbation queries and optimizer steps, while schedule-aware prefetching hides state-transfer overhead. On LIBERO camera-viewpoint shifts, VLA-ZO reduces end-to-end adaptation time by 25.59 at and 32.54 at relative to baseline ZO, while improving average task success from 48.27% without adaptation to 58.17% and 63.58%, respectively. These results show that making ZO faster can make larger query budgets practical, providing a promising path toward resource-efficient VLA adaptation on deployment platforms.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, zeroth-order optimization, gradient-free optimization, efficient adaptation, on-device adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Lb6cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VLA-ZO, a framework for accelerating forward-only, zeroth-order adaptation of VLA models under visual distribution shifts. Adaptation is restricted to an action-side LoRA module, allowing the frozen vision-language conditioning states to be reused across perturbation evaluations and optimization steps. A disk-backed cache and one-step-ahead prefetching further reduce recomputation and transfer overhead.

The evaluation uses π0.5 and OpenVLA-OFT on 40 LIBERO tasks, with five target-condition demonstrations, one per scene, covering viewpoint changes and compound visual shifts. For π0.5, the full system achieves 25.59× and 32.54× speedups over uncached action-side ZO at query budgets of 16 and 64, respectively, while maintaining similar adaptation quality. Average viewpoint-shift success improves from 48.27% without adaptation to 58.17% and 63.58%.

Critical strengths
  1. A clear and practically important systems insight. Reusing the frozen prefix states within and across optimization steps is a simple, well-motivated intervention that preserves the intended optimization objective under the stated assumptions.

  2. Substantial measured acceleration across two VLA architectures. Evaluation across π0.5 and OpenVLA-OFT strengthens the case that the reuse strategy is not specific to a single action-module implementation. The improvements in adaptation success over the unadapted policies further demonstrate the practical relevance of making ZO adaptation faster.

Critical weaknesses
  1. Insufficient baseline comparisons. The proposed caching strategy could also be applied to first-order training of the action module. The experiments with FLA† and DART† show that the proposed method outperforms the evaluated zeroth-order implementations, but it remains unclear under what conditions it outperforms similarly execution-optimized first-order approaches.

  2. Insufficient evidence for practical deployment. The experiments provide convincing evidence of acceleration under specific simulation settings and on high-performance hardware. However, direct evidence of the method’s applicability to real-world robotic systems and its practical deployment benefits remains limited.

  3. Insufficient efficiency evaluation under a fixed time budget and limited experimental robustness. The current experiments fix the number of updates at 1,000 steps and adjust the learning rate according to the query budget qq, leaving the benefits of increasing perturbation directions versus optimization steps under the same time budget unclear. The authors should report success rate versus adaptation time curves.

Questions for authors
  1. On a Matched First-Order Baseline: What trainable parameters and optimizer are used for the first-order results in Figure 1? Can the authors compare the success rate, adaptation time, and peak memory usage of first-order training and the proposed method under the same action-side LoRA configuration and prefix caching strategy?

  2. On the FLA and DART Baselines: Why were FLA and DART reimplemented using zero-order gradient estimation and SGD? How does this modification affect comparability with the original methods? Can the authors disentangle the effects of system optimizations from those of optimization configurations?

  3. On Performance Under a Fixed Time Budget: The current experiments fix the number of updates at 1,000 and adjust the learning rate according to the query budget (qq). Can the authors jointly vary the query budget and number of updates, and report success rate versus end-to-end adaptation time curves to verify whether increasing the number of perturbation directions is more effective than increasing the number of optimization steps under the same time budget?

Reviewer_1v2JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VLA-ZO, a framework that accelerates adaptation of vision-language-action (VLA) models by freezing and reusing vision-language computations, while updating the action module without backpropagation. Its main contribution is improving the execution efficiency of zeroth-order optimization.

VLA policies can suffer substantial performance drops when camera viewpoints, lighting, or image quality change after deployment. Conventional fine-tuning requires backpropagation and considerable GPU memory. Zeroth-order (ZO) optimization estimates update directions by perturbing parameters and comparing losses, using only forward passes. However, evaluating qq perturbation directions requires 2q2q forward evaluations per update, making adaptation slow.

The authors show that adapting only LoRA parameters in the action module can improve performance under visual distribution shifts. This allows the expensive vision-language prefix to remain frozen and enables three optimizations:

  1. Within-step reuse: Compute the prefix outputs once per batch and reuse them across all positive and negative perturbation evaluations.
  2. Across-step reuse: Cache these outputs on disk and reload them when the same demonstration samples are revisited.
  3. Schedule-aware prefetching: Prepare and transfer the next batch’s cached states while the current batch is being processed.

The framework retains standard two-sided finite-difference estimation and ZO-SGD. When the frozen prefix is deterministic and the cached states contain all required information, reuse preserves the optimization objective and gradient estimator in exact arithmetic. Experiments evaluate π0.5 and OpenVLA-OFT on all 40 tasks in the LIBERO simulation benchmark. Adaptation uses one demonstration per scene—five demonstrations in total—and evaluation includes tasks without adaptation demonstrations. For π0.5 under camera-viewpoint shifts, the unadapted policy achieves 48.27% average success.

Within-step reuse provides the largest speedup. Larger perturbation budgets mainly help with medium and large viewpoint changes. Caching itself does not directly improve accuracy; it makes larger query budgets computationally affordable. The method also improves results under blur and lighting shifts and transfers to OpenVLA-OFT, although increasing qq does not consistently improve performance.

Critical strengths
  1. Clear problem formulation. The paper addresses post-deployment adaptation under an inference-level GPU memory budget, for which zeroth-order optimization is a natural candidate.

  2. Detailed implementation description and sufficient ablation. The paper provides thorough implementation details to support reproducibility.

  3. Cross-architecture validation. Experiments cover two distinct caching boundaries: the inputs to a flow-matching action expert and those to an L1 regression head.

Critical weaknesses

Major concerns

  1. The most critical baseline is missing: first-order optimization with action-side LoRA and the same prefix caching. The paper motivates zeroth-order optimization by arguing that backpropagation exceeds the memory budget, but the reported memory figures correspond to backpropagation through the entire model. If only the action expert is trained, as in this paper, and the prefix KV states are cached, backpropagation only needs to pass through the roughly 300-million-parameter action expert. The activation overhead should then be small and may well fit within an inference-level memory budget. Moreover, a single backward pass provides an exact gradient, which should offer substantially better speed and quality than zeroth-order optimization with q=64q=64. Figure 1b also shows that first-order optimization achieves a higher success rate than the best zeroth-order result. Without this comparison, the paper’s central premise remains unsubstantiated. This is my primary reason for leaning toward rejection.

  2. The reported speedups rely on an unnecessarily inefficient baseline. The baseline recomputes the prefix 2q2q times per sample, whereas any reasonable implementation with a frozen prefix would compute it once and reuse it. Breaking down Table 1, the two subsequent optimizations with more substantial engineering content together provide only approximately 1.6× and 1.2× speedups. The 25.59× and 32.54× figures highlighted in the abstract are therefore misleading.

  3. The novelty is limited. Freezing a backbone and caching its features are standard practices in linear probing, side-tuning, LST, and related parameter-efficient fine-tuning approaches. DeepZero already reuses features preceding the perturbed layer. Across-step caching amounts to dataset-level feature caching, while prefetching follows a standard data-loading pipeline. The authors also acknowledge that they do not introduce a new adaptation method. In my view, the contribution is insufficient for ICLR.

  4. The baseline comparisons are unfair. Both FLA and DART are modified to use zeroth-order SGD rather than their original first-order optimization methods. Poor performance from estimating gradients over 3.3 billion parameters with only q=16q=16 directions is unsurprising and should not be taken as representative of DART. At a minimum, results from the original first-order FLA and DART methods should be included for reference.

  5. The evaluation does not align with the deployment motivation. The paper repeatedly emphasizes deployment platforms and on-robot adaptation, yet all timing experiments use a server-class GPU with 96 GB of memory. Both first-order and zeroth-order methods fit on this hardware, so the memory constraint motivating the method is never actually encountered. There are no experiments on edge devices. Disk-loading costs could be substantially higher on such devices, potentially reversing the conclusions about across-step caching and prefetching. For π0.5, each cached sample occupies approximately 17 MiB, so a dataset of just over 1,000 frames requires roughly 18 GB of disk storage—also a considerable burden for an edge device.

  6. The statistical evidence is insufficient. The experiments use a single seed and a single task combination, with no error bars. Differences between execution variants are described as comparable to those between repeated runs, but the variance across repeated runs is never reported. Several results may fall within experimental noise: on OpenVLA-OFT, q=16q=16 performs worse than q=4q=4; under View+Noise, adaptation reduces performance on the Spatial suite relative to zero-shot evaluation; and on the OpenVLA-OFT Goal suite, q=64q=64 performs worse than the zero-shot baseline. These cases in which adaptation degrades performance are not discussed.

  7. Figure 2b is misleading. The claim that action-side adaptation is “28.4× faster” conflates parameter placement with caching. A fair comparison of parameter placement would compare uncached action-side adaptation against vision-side adaptation, yielding only approximately a 1.1× speedup. In addition, the success rates in Figure 2b differ from those in the corresponding rows of Table 1, without explanation.

  8. There is no comparison under equal time budgets. The conclusion argues that faster execution enables more query directions and better adaptation quality within a fixed time budget, but the paper provides no adaptation-quality curves under fixed time budgets. It also neither compares against nor combines its approach with MobiZO-style batched parallel perturbation evaluations, which offer an orthogonal acceleration strategy.

Minor issues

  • The opening paragraph of Section 3.3 appears before the material it summarizes, suggesting a misplaced paragraph.
  • “Tables 2 examines” is a grammatical error.
  • The learning-rate search grid for π0.5 is not specified.
  • Section 3.2 does not explain why adapting only the action expert can mitigate viewpoint shifts originating on the visual side. This phenomenon itself deserves further analysis.
Questions for authors

see weaknesses.

Improve Rating

When following additional experiments are provided, I will revise my recommendation to Accept.

  • Report the GPU memory usage, adaptation time, and success rate of first-order optimization with action-side LoRA and cached prefix states. The paper’s premise is justified only if this baseline exceeds the memory budget or performs worse.
  • Reproduce the timing results in Table 1 on an actual edge GPU.
  • Provide adaptation-quality curves for naive zeroth-order optimization and VLA-ZO under equal time budgets.
  • Include experimental results for the original first-order versions of FLA and DART.
Reviewer_BAmwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies one-shot adaptation of VLA policies (pi0.5 and OpenVLA-OFT) to camera-viewpoint, blur and lighting shifts on LIBERO, using zeroth-order (MeZO-style) optimization. The motivation is that backprop needs more memory than an inference-oriented deployment platform has. The authors point out that if adaptation is restricted to LoRA on the action expert (or the L1 action head), the vision-language prefix is frozen and its output conditioning state h_i depends only on the observation. VLA-ZO then (i) computes h_i once per sample and shares it across the 2q probes, (ii) caches it on disk across optimizer steps, and (iii) prefetches the next minibatch's cached states while the current probes run. A small cost model (Eqs. 4-8) predicts the gains.

On pi0.5 the paper reports 25.6x (q=16) and 32.5x (q=64) shorter adaptation time than uncached action-side ZO, while average viewpoint-shift success goes from 48.3% (zero-shot) to 58.2% and 63.6%. OpenVLA-OFT shows smaller gains (49.5% to 52.7%). FLA and DART are used as baselines, both re-implemented as ZO-SGD.

Critical strengths
  1. The execution accounting is clean and honest. The objective, perturbation directions and parameter space are unchanged by the caching, timings include cold-cache population, and the cost model matches what is measured. For example, prefetching saves about 0.76 s per update at both q=16 (2,264 -> 1,504 s) and q=64 (5,465 -> 4,709 s), so it matters most at small q, as Eq. 8 predicts. Appendix B also says plainly where exact equivalence breaks (different kernels, mixed cold microbatches). This makes the systems claims easy to check.

  2. The practical effect is real. At q=64, adaptation drops from about 42.6 hours to about 78 minutes, which makes a large query budget usable at all, and on pi0.5 the larger budget clearly helps under Medium and Large shifts (59.8 -> 66.4 and 26.2 -> 36.8). Anyone running ZO on VLAs would want this.

Critical weaknesses
  1. The decisive baseline is missing, and the motivation does not carry over to the final method. The case for ZO rests on backprop having to store activations for the whole model (20.27 vs 12.74 GiB at batch 16, 54.69 vs 27.71 GiB at batch 64). Those numbers describe first-order adaptation where the backward pass traverses the full network. Once adaptation is confined to the action expert and the prefix output is cached, which is exactly what VLA-ZO does, a backward pass only touches the action expert, and most of ZO's memory and compute advantage disappears. For OpenVLA-OFT this is even more obvious: the adapted part is a 0.79M-parameter head sitting on cached hidden states, where exact gradients should be very cheap and strictly better than q=4 or q=16 random directions. I could not find first-order LoRA on the same action-side parameters, with the same cache, in any table (time, peak memory or success). Figure 1 only gives curves with no numbers. Without this comparison the paper shows that ZO can be made faster, but not that ZO is the right tool in this setting.

  2. The speedup is largely by construction and the technical novelty is thin. The headline 25.6x / 32.5x is measured against an uncached baseline that re-executes a frozen prefix 2q times per sample, and Eq. 5 already tells us the limit is (c_pre + c_suf)/c_suf. Caching the output of a frozen backbone is standard for head or adapter tuning, and DeepZero (cited by the authors) already reuses features before the perturbed layer. Prefetching is one-step double buffering and contributes about 14% at q=64. I would be more convinced if the paper compared with other ways of making ZO cheap (low-rank or subspace perturbations such as SubZero, batched probes) or offered something beyond this one observation. Related to this, Section 3.2 / Figure 2b credits action-side placement with a 28.4x speedup, but the 1,504 s is the full VLA-ZO number with all caching. Uncached action-side ZO takes 38,487 s in Table 1 versus 42,782 s for FLA, i.e. roughly 1.1x. Placement and caching are conflated there.

  3. The baselines are not like-for-like, and the action-side choice is confounded with ZO's dimension dependence. FLA and DART are run as ZO-SGD versions of first-order methods. DART perturbs about 3.35B parameters with q=16 and 500+500 updates, and ZO estimator variance grows with dimension, so it is expected to look weak regardless of the method's merit. Numbers for the original first-order FLA/DART are not shown, so it is impossible to tell where VLA-ZO sits relative to what these methods actually achieve (Figure 1b suggests first-order is around 70%, versus 58-64% here). I also could not find the pi0.5 learning-rate sweep for the baselines (Appendix A.4 describes only the OpenVLA-OFT grid), while VLA-ZO gets a different LR per q (3e-2 at q=16, 1e-1 at q=64). In Table 2, VLA-ZO at q=64 is compared with baselines at q=16. Also, the finding that action-side matches vision-side adaptation (58.6 vs 57.1) is under ZO, where a 3.8x smaller search space helps on its own. It says little about which placement is better in general, and it confines the method to the part of the network that is not where viewpoint, blur or lighting shifts enter (success under the Large shift is still only 26-37%).

  4. The deployment claim is not tested. The abstract and conclusion motivate resource-constrained deployment platforms, but all timings are on a 96 GB RTX PRO 6000 Blackwell server GPU, everything is in LIBERO simulation, and the peak memory of VLA-ZO in the reported configuration (batch 64, two staged batches of 17.02 MiB states, cloned attention cache) is never reported. The memory figures in the introduction are at batch 16, while the experiments use batch 64. The persistent cache is also roughly 18 GiB per camera condition by my arithmetic (1,085 frames x 17.02 MiB), and by Eq. 8 the benefit of prefetching depends on disk throughput relative to c_suf, which will look very different on embedded storage.

Minor: (a) Terminology: the setting is called one-shot, but the protocol uses five demonstrations (one per scene, 1,085 frames), and the "Noise" condition is Gaussian blur according to Appendix A.2.

Questions for authors
  1. (Most important) Please report time, peak memory and success rate for first-order LoRA on the same action-side parameters, using the same cached prefix, for both pi0.5 and OpenVLA-OFT. If this is cheap and at least as accurate, what is the remaining case for ZO here?
  2. What is the peak accelerator memory and the on-disk cache size of VLA-ZO at batch 64 in the reported runs? Has anything been run under a memory cap, on a smaller device, or with the cache on slower storage?
  3. Can you show results for the original first-order FLA and DART under the same protocol, and describe the pi0.5 learning-rate sweep used for the baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.