acceptodds
Under review as a conference paper at ICLR 2027

LPA-CACHE: Latency-Plateau Aware KV Cache Reuse for Efficient Vision-Language-Action Model Inference on GPUs

Abstract

Vision-language-action (VLA) models require low-latency inference for responsive robotic control. Previous works have shown that cross-frame reuse of visual key-value (KV) representations can reduce redundant computation. However, reductions in FLOPs do not necessarily translate into proportional latency reductions. Profiling the prefill-stage of VLA language-backbone on GPUs reveals a pronounced latency-plateau effect in dominant linear operators: reducing the number of recomputed tokens within a plateau yields marginal speedup, whereas crossing a plateau boundary yields substantial latency reduction. Motivated by this observation, we propose LPA-CACHE, a training-free method for accelerating VLA inference on GPUs through latency-plateau-aware KV cache reuse. A one-time profiling procedure identifies latency-plateau boundaries on the target GPU platform. These boundaries are combined with preselected reuse layers to construct a fixed-shaped reuse schedule. At inference time, visual tokens estimated to be task-relevant from previous-step attentions are protected from reuse, while the remaining visual tokens are ranked by inter-frame relative changes and progressively reused according to layerwise reuse budgets. Guided by observed cross-layer consistency in attention-based task relevance, LPA-CACHE extracts attention weights from only a small subset of layers, allowing FlashAttention-2 to be used in the remaining layers for further speedup. Extensive experiments across several representative VLA models on two robotic simulation platforms (LIBERO and CALVIN) demonstrate that LPA-CACHE achieves up to 1.38 language-backbone prefill speedup and up to 1.15 observation-to-action inference speedup with negligible loss in task success rate.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Vision-Language-Action Models, KV Cache Reuse, Efficient Inference, Hardware-Aware Acceleration, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_J91jOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LPA-CACHE profiles GPU latency plateaus and uses their boundaries to choose visual KV reuse budgets. It protects tokens using previous-step attention, ranks the remaining tokens by inter-frame embedding changes, and extracts attention from only a few layers to retain FlashAttention compatibility.

Critical strengths
  • The paper provides clear evidence that token reduction and latency reduction are not proportional. As shown in Table 1, reducing the active length from 257 to 256 lowers reported latency from 75.23 to 69.05 ms without changing success. This is a concrete and useful systems observation.

  • Evaluation covers three VLA models and two simulation benchmarks. Reporting observation-to-action latency, including reuse overhead, is more informative than reporting FLOPs or prefill savings alone.

Critical weaknesses
  • The contribution is primarily an engineering refinement of existing KV reuse. Hardware profiling and boundary-aligned budgets are useful, but the paper offers limited new understanding beyond this combination. The resulting observation-to-action speedups are roughly 1.10–1.15× over FlashAttention baselines, and evaluation on one GPU leaves the broader applicability of the central hardware claim uncertain.

  • The experiments do not cleanly isolate plateau alignment. Compared with VLA-Cache, the method changes token ranking, relevance probing, reuse scheduling, and FlashAttention support. The boundary experiment demonstrates the latency phenomenon, but does not establish how much of the overall improvement specifically requires the proposed scheduling strategy.

Questions for authors
  • With identical token selection, probe layers, and attention implementations, how much additional benefit comes from plateau-aware scheduling?

  • Do the gains persist on another GPU, and how sensitive are they to the empirically selected reuse-entry and probe layers?

Reviewer_HgELOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LPA-CACHE, a training-free inference-acceleration method for VLA models that optimizes the language-backbone prefill stage. Latency plateaus on the target GPU are profiled offline, and the reuse budget is chosen so the active token count lands on a plateau boundary. At inference, previous- step attention protects task-relevant tokens; among the rest, positions with small relative embedding change across frames reuse cached KV. Attention is extracted only at a few probe layers so FlashAttention-2 remains usable elsewhere. On a single RTX 3090 Ti, the paper reports up to 1.38x prefill and 1.15x observation-to-action speedup with near-baseline success rates.

Critical strengths
  1. Hardware-grounded motivation with direct measurement: Table 1 shows that crossing a plateau boundary (257 -> 256 tokens) drops latency 75.23 -> 69.05 ms with unchanged success rate, giving concrete evidence for plateau-aware budgeting.
  2. Deployment-conscious design: attention extraction is limited to probe layers so most layers keep FlashAttention-2, and both prefill and full observation-to-action latency (including reuse-decision overhead) are reported rather than FLOPs only.
  3. Consistent, if modest, latency improvements across three VLA models on LIBERO and UniVLA on CALVIN.
Critical weaknesses
  1. Incremental over existing cross-frame KV-reuse work. VLA-Cache (Xu et al., NeurIPS 2025, arXiv:2502.02175) already does training-free cross-frame reuse, attention-protected task-relevant tokens, and layer-adaptive reuse ratios; GPU latency plateaus from tile/wave quantization are documented (e.g., NVIDIA's matrix-multiplication performance guide). The paper's distinct part — profiling plateaus to set the budget — is real but narrow.
  2. The key ablation isolating the new component is missing. In the main tables the budget design, token-change metric, probe-layer attention extraction, and FlashAttention-2 usage all change at once, so the gain cannot be attributed to plateau-aware budgeting specifically. A controlled comparison of plateau-aware vs. well-tuned plain fixed budgets under the same token selection and attention implementation is needed.
  3. Single-GPU evidence for a hardware-aware method. All experiments run on one RTX 3090 Ti with a single reuse entry (J = 1); entry/probe layers are chosen empirically. The claimed portability of the profiling pipeline is untested on any second GPU.
  4. Success-rate claims rest on point estimates with no confidence intervals or repeated runs, and per-subtask results vary widely (e.g., on CALVIN, lift pink block slider drops 95.4% -> 77.4% and rotate blue block left 96.9% -> 87.1% vs. the FlashAttention-2 baseline). Only VLA-Cache is compared numerically; the other cited caching baselines are discussed qualitatively but never benchmarked.
Questions for authors
  1. With token selection and attention implementation fixed, how much of the speedup comes from plateau-aware budgeting vs. a well-tuned fixed budget? Can you report matched-success or matched-latency comparisons?
  2. How does the offline latency profile translate into the chosen budgets in Section 5.3 (entry layer, target active length, probe layers, protected count), and is that selection independent of the final evaluation?
  3. How many evaluation episodes and independent runs underlie Tables 2-3, and can confidence intervals be reported? Does the pipeline reproduce its gains on a second GPU after re-profiling?
Reviewer_LCoROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LPA-Cache, a hardware-aware KV cache reuse method for Vision-Language-Action (VLA) models. Exploiting the phenomenon that GPU latency changes in a stepwise manner with active sequence length, the method uses offline profiling to determine reuse budgets and target sequence lengths. It then protects task-relevant visual tokens using text-to-vision attention scores and identifies reusable tokens based on cross-frame visual embedding changes, thereby reducing redundant computation and inference latency. The authors evaluate LPA-Cache on OpenVLA, OpenVLA-OFT, and UniVLA using the LIBERO and CALVIN simulation benchmarks. The results demonstrate reduced prefill latency while largely maintaining task success rates or task completion lengths, yielding moderate speedups. However, the experiments are primarily conducted on a single NVIDIA RTX 3090 Ti in simulation, and the improvements in end-to-end latency remain relatively limited. The method's dependence on hardware-specific profiling and empirically selected parameters, as well as the generalizability of its token reuse strategy, requires further investigation.

Critical strengths
  1. Clear optimization objective: The method targets actual inference latency by designing a KV cache reuse strategy around GPU latency plateaus, rather than focusing solely on reducing FLOPs.

  2. Practical potential: The method combines attention-based protection of important tokens with cross-frame embedding changes to identify redundant tokens, demonstrating prefill latency improvements across multiple VLA models.

Critical weaknesses
  1. Limited experimental coverage: Experiments are conducted on a single NVIDIA RTX 3090 Ti and focus primarily on the OpenVLA family and simulation tasks, leaving generalizability insufficiently validated.

  2. Limited novelty: The method primarily uses latency plateau analysis to guide token reuse. Its substantive contributions relative to existing token reuse or pruning methods remain insufficiently established.

  3. Limited acceleration gains: The optimization mainly benefits the prefill stage, with relatively modest improvements in end-to-end inference latency.

  4. Insufficient trade-off analysis: The relationship between token reuse rates, latency improvements, and task performance has not been systematically investigated.

Questions for authors
  1. Causes of latency plateaus: Could the authors provide further explanations for the underlying mechanisms of GPU latency plateaus? Which hardware characteristics or operator execution properties primarily determine their formation, and how does this analysis motivate the proposed optimization strategy?

  2. Adaptability of latency thresholds: Do the reuse budgets and target active sequence lengths require re-profiling for different GPUs? Have the authors evaluated differences in latency plateaus and method performance across different GPU architectures?

  3. Differences between offline and online latency: Do configurations optimized through offline profiling remain effective under varying GPU utilization and runtime workloads during online inference? Have the authors considered online profiling or dynamic adaptation of the reuse strategy?

  4. Trade-offs at higher token reuse rates: Could more aggressive token reuse cross additional latency plateau boundaries and yield further speedups? How would such changes affect task success rates?

  5. Effectiveness in real-world robotic control: Can the proposed method maintain its latency benefits and task performance on real robots? In particular, how reliable is the reuse strategy during continuous motion, substantial visual changes, or precision-sensitive manipulation tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.