acceptodds
Under review as a conference paper at ICLR 2027

Seeing Action-Critical Evidence: Token Pruning with Internal Action Priors for Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models translate visual observations and language instructions into continuous robotic actions. However, their dense visual token sequences introduce substantial decoding overhead, hindering efficient real-world deployment. Recent trainable token pruning methods reduce this cost, yet their pruning criteria remain largely action-agnostic: they often retain task-irrelevant background tokens while discarding fine-grained visual cues essential for precise control. In this work, we investigate whether pretrained VLA models internally encode action-aware signals for token selection. Our analysis suggests that: (i) action-critical visual tokens show an association with value vectors in deep Feed-Forward Network (FFN) layers; and (ii) these value vectors define actionable residual subspaces whose projection energy approximates local action saliency. Motivated by these insights, we propose ActPruner, a novel Action-Anchored Dynamic Token Pruning approach, which constructs an action-aware pruning metric from intrinsic action priors encoded in deep FFN layers. Technically, ActPruner constructs an Action Information Extraction Module (AIEM) with three progressive steps: Deep Prior Extraction to extract compact action anchors from deep FFN value vectors, Semantic Space Adaptation to align them to shallow pruning layers, and Instruction Modulation to generate action-conditioned pruning queries. This enables dynamic preservation of manipulation-relevant affordance regions while suppressing redundant background tokens. Extensive experiments on VLA-based robotic manipulation benchmarks verify the superiority of our ActPruner over state-of-the-art VLA pruning methods. Code is available in the supplementary material.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Visual Token Pruning, Action-Aware Token Selection, Efficient Inference, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_guVDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ActPruner, which extracts action-related directions from deep FFN value vectors and uses them to guide visual-token pruning. Semantic adaptation brings the deep features to the pruning layer, and instruction modulation makes the selection task-conditioned. The method retains only 4% of visual tokens on LIBERO while reaching nearly the dense baseline, with additional results on Meta-World and a physical robot. Personally, I like this work : )

Critical strengths
  • The attempt to use action information already encoded inside the VLA is interesting and more task-specific than attention-only pruning.
  • The three-stage design is simple enough to be added to a frozen backbone, and the overhead analysis is clear.
  • For experiments, the paper includes latest strong baselines like SpecPrune-VLA and LightVLA, and covers real-robot experiments rather than only simulator numbers. Ablations show that both semantic adaptation and instruction modulation matter, and the action-anchor analysis is useful. The comparison and analysis for other baselines are clear.
Critical weaknesses
  • Will the code be open-sourced for reproducibility?
  • The method depends on a single deep FFN layer and PCA-derived anchors. The reported layer and anchor count appear tuned on LIBERO-Long, so robustness to backbone, embodiment, and task changes is still uncertain. Other backbones like StarVLA can be also considered.
  • Meta-World performance remains below the dense baseline (52.5% vs. 55.0%), which is important given the paper's broad claim of preserving control quality. This is not a big weakness but authors can revise related statements to make this paper more solid and rigorous.
  • The real-world evaluation has four tasks and appears to use a small number of trials; stronger statistical reporting would help distinguish a real gain from task-specific variation.
  • The analysis argues that FFN value vectors encode action priors, but the evidence is mostly correlational. A comparison with learned probes or multi-layer features would better support this interpretation. Related intepretion can be more extended in the revised version.
Questions for authors

See above.

Reviewer_eZShOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ActPruner, a visual token pruning method for VLA models guided by internal action-related representations. The method extracts PCA anchors from deep FFN weights and combines cross-layer representation adaptation with instruction-conditioned queries to enable differentiable dynamic token selection.

Experiments cover LIBERO, Meta-World, and real-world robotic manipulation tasks. On LIBERO, ActPruner retains only approximately 4% of visual tokens while achieving a 97.0% average success rate, comparable to the dense baseline (97.1%).

Critical strengths

S1. Competitive efficiency–accuracy trade-off.

ActPruner substantially reduces visual tokens while largely preserving the task success rates of dense VLA models. On LIBERO, it outperforms the trainable pruning baseline LightVLA in both success rate and token retention (97.0% / 4% vs. 96.5% / 15%). Additional evaluations on Meta-World and real-world tasks, together with measured inference latency, further support its empirical performance and computational efficiency.

S2. Clear methodological motivation supported by systematic empirical analysis.

The paper motivates its approach through an analysis of the association between FFN representations and action saliency. It further provides extensive ablations involving PCA anchors, learnable queries, the adapter, instruction modulation, and anchor configurations. Although the underlying techniques are not individually novel, the components serve well-motivated purposes, and the diagnostic experiments provide useful empirical support for the design choices.

Critical weaknesses

W1. The independent technical contribution of FFN-PCA requires further clarification.

FFN-PCA is the most distinctive component of the method, but its success-rate gains over Learnable Queries and Random Selected Value Vectors in Table 4 are relatively modest (94.6% vs. 93.4% / 93.6%), with corresponding token retention rates of 6%, 6%, and 7%.

These results and the appendix diagnostics support the utility of PCA anchors. However, further clarification of the alternative approaches' parameterization, anchor update strategies, and training configurations would help assess the independent contribution of FFN-PCA. Moreover, the current analysis of FFN representations primarily provides a motivation for action-related priors, while their precise underlying mechanism remains only partially explained.

W2. Cross-architecture efficiency evaluation and real-world deployment comparisons remain incomplete.

Detailed inference latency and computational cost analyses are primarily conducted on OpenVLA-OFT, without comparable system-level speedup measurements for π0.5. Given the differences in their action-generation architectures, the current results do not fully establish ActPruner's practical computational benefits across different VLA backbones.

Furthermore, the real-world experiments mainly compare ActPruner against dense π0.5 and the training-free ADP, without a direct comparison to trainable pruning methods such as LightVLA.

Questions for authors

Q1. Could the authors clarify whether the different anchor designs in Table 4 are comparable in terms of parameterization, training budget, and key implementation settings, and explain how differences in token retention affect the interpretation of the results? Based on the existing ablations, what is the main independent advantage of FFN-PCA over directly learned queries?

Q2. Could the authors report the inference latency, computational cost, and speedup of ActPruner on π0.5 in Meta-World or real-world deployment settings to further validate its cross-architecture efficiency benefits?

Q3. In the real-world experiments, given that ActPruner requires training whereas ADP is training-free, could the authors include a trainable pruning baseline if feasible, such as LightVLA? This would make the comparison more convincing.

2 comments
Reviewer_eZShOpenReview ↗

Response to Authors of Submission 36652

Thank you for the detailed responses and additional experiments.

The clarification of the anchor configurations and the new action-Jacobian analysis strengthen the evidence supporting the FFN-PCA design. The additional efficiency measurements on π0.5 and the real-world comparison with LightVLA also directly address my previous concerns.

These additions make the paper's methodological justification and empirical evaluation more convincing. I appreciate the paper's clear motivation, coherent design, and systematic validation. Overall, I consider this a solid and well-executed study. Great work!

Thank you again for the thoughtful rebuttal and additional experiments.

After further consideration, I believe the paper's overall contributions warrant a stronger recommendation. While the independent improvement from FFN-PCA over simpler query designs is relatively modest, the overall method is well-motivated and coherently designed. The systematic ablations and strong efficiency–accuracy results provide convincing evidence of its practical value. The additional experiments also substantially address my earlier concerns and further strengthen the paper.

I have therefore decided to raise my rating to 4.

Reviewer_PrkyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ActPruner, a trainable visual-token pruning framework that derives selection queries from the principal directions of a pretrained VLA model’s deep FFN down-projection matrix. These directions are transformed by a learned adapter and conditioned on language instructions before selecting visual tokens. The paper motivates this design through action-saliency diagnostics and a conditional theoretical connection between token-removal sensitivity and projection energy onto a task-conditioned subspace.

Critical strengths
  1. Reducing visual-token processing while preserving manipulation performance is practically valuable. The reported LIBERO results 97.0% success with approximately 4% token retention and a 1.65× per-step inference speedup suggest a useful operating point.
  2. Appendix E compares anchor constructions under matched retention budgets, using a fixed dense policy and identical observations. Aggregating uncertainty at the episode level also avoids treating temporally correlated policy queries as independent samples. These experiments provide meaningful evidence that leading PCA directions and learned adaptation improve action-output preservation relative to the evaluated controls.
Critical weaknesses

The theoretical action subspace is not identified with the PCA subspace used by ActPruner. Lemma 1 establishes that local FFN responses lie in the column span of WdownW_{\mathrm{down}}. This structural property does not identify action-sensitive directions within that span. If the matrix has full row rank, the span is the entire residual space, making the inclusion alone insufficiently selective.

Assumption 2 subsequently posits a low-dimensional subspace that approximately captures the action Jacobian. The method instead extracts leading principal directions of the centered weight matrix. Weight variance need not correspond to downstream action sensitivity: a high-variance direction may have little influence on the action output, while a lower-variance direction may be important.

Appendix E supports the empirical usefulness of PCA anchors, but it does not establish that they approximate the subspace required by the proposition. Accordingly, the evidence supports describing these anchors as useful FFN-derived priors more directly than it supports identifying them with intrinsic manipulation intents.

Questions for authors
  1. How closely do the leading PCA directions align with action-Jacobian-sensitive directions? Could you measure the action sensitivity captured by the PCA subspace, for example through (|J_i(I-\Pi_P)|), relative to random and low-variance subspaces?
  2. Does the saliency-based interpretation remain predictive when large token subsets are removed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.