acceptodds
Under review as a conference paper at ICLR 2027

Redundancy or Relevance: What Language-Guided Token Pruning Selects in Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) model token pruning methods rank tokens typically by computing an importance score based on attention weights, spatial redundancy via inter-token similarity, or task relevance via language conditioning. In this work, we analyze pre-decoder language-vision representations of VLA models. We start this paper by proposing a simple, yet effective, language-vision cosine-similarity based pruning method. Surprisingly, the simple pruning method outperformed more complex state-of-the-art pruning methods. So to better understand how pruning works, we study in more details how cross-modal similarities drive pruning. We find that in many cases pruning is effective in VLAs, not because of semantic grounding, but rather due to the criterion's projection geometry acting as a redundancy suppressor along the visual typicality axis. Since each backbone produces a uniquely structured embedding space, we characterize its geometry to determine what the score actually ranks. We show that when the geometry allows, this signal could be sufficient to determine which tokens to drop, despite the lack of strong semantic alignment. We test over four different VLA models with three different robotic datasets, showing how our language pruning method and the simpler method achieve high accuracy with the lowest latency. For example, on OpenVLA-OFT, we achieve up to latency reduction compared to vanilla OpenVLA-OFT, with only 1% drop in success rate.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action models, Token Pruning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CYvTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a simple training-free VLA pruning rule that keeps the visual patches most similar to the language instruction, and analyzes what this score actually measures. It finds that on most backbones the instruction contributes little: the score mainly reflects how atypical a patch is within the current frame, so the rule acts as a redundancy filter rather than instruction-guided selection. A language-free rule selects mostly the same patches, and swapping the instruction changes success by only about 3 points. At a 25% keep ratio, the method reaches 96.2% on OpenVLA-OFT (LIBERO) versus 97.2% unpruned with a 1.68× speedup (VLA-Pruner: 96.5%), reaches 79.1% on CogACT (vs. 79.7%) and 92.2% on π0.5 (vs. 96.8%) with about 2.5× speedup, but fails on LLaVA-VLA (CALVIN chain length 1.85 vs. 3.61).

Critical strengths

A clear mechanistic explanation of what a widely used pruning signal actually measures. "Language-guided" pruning in VLAs is mainly redundancy suppression. Several recent VLA pruning methods (VLA-IAP, TEAM-VLA, SemanticVLA) build on instruction-similarity signals, and this paper tells them what that signal is really doing.

A useful diagnostic insight. It gives practitioners a cheap diagnostic (measuring α and anisotropy) to predict whether similarity-based pruning will work on a new backbone.

Critical weaknesses

The core claim that the simple method surpasses SOTA is not supported. The abstract and Section 3.1 state that cosine-similarity pruning "outperforms more complex state-of-the-art pruning methods." However, in Table 3 (OpenVLA-OFT, 25% kept), VLA-Pruner has a higher average success rate (96.5) than the language criterion (96.2) and the hub rule (95.9), and VLA-Cache matches it on Object. The advantage lies only in latency (97.1 vs. 107.5 ms).

The experimental scope is too narrow for the generality claimed. All method comparisons (Tables 3–6) use a single 25% keep ratio; the substitution study (Table 2) uses 12.5% for OpenVLA-OFT and π0.5, but includes no baselines.

Questions for authors

Pruning-ratio sweep. Report success vs. keep ratio (e.g., 50%, 25%, 12.5%) for language, hub, DivPrune, VLA-Pruner and random selection on OpenVLA-OFT and π0.5.

2 comments

We thank the reviewer for their insightful comments and suggestions. We have tried to cover all raised points, including in the weaknesses, and we are happy to modify based on the discussion.

Response to the weakness on the SOTA claim

We agree with the reviewer. The contribution of the paper is the analysis, and that was our intention while writing, but our wording did not present it that way. We thank the reviewer for pointing this out. The claim that the criterion outperforms state-of-the-art methods was not supported, and we have removed it. The abstract, the introduction and Section 3.1 now say that the criterion performs competitively with existing pruning methods on OpenVLA-OFT.

We have also clarified the paper's contribution. It explains what language–vision cosine similarity ranks in VLAs, and why such a simple score can be competitive at all. The comparison with existing methods places that behaviour in context, rather than ranking the methods.


For the second weakness, we agree that the submitted version overstated the generality of the results. The revision no longer claims that the criterion outperforms existing methods or that it applies to all VLAs. It describes what the criterion ranks on the four policies we study, and the conditions under which that holds. We also extended the evaluation. Appendix C now reports success at 50%, 25% and 12.5% of the visual tokens kept on LIBERO-Long. These results place the 12.5% substitution study of Table 2 against existing methods at the same budget.


Response to the question on success against keep ratio

We added these results to Appendix C (Table 9) in the revised version. Success rate (%) on LIBERO-Long, 500 episodes per cell:

ModelMethod50%25%12.5%
OpenVLA-OFTRandom93.682.841.8
DivPrune94.092.275.8
VLA-Pruner93.693.073.7
Language93.692.488.0
Hub −xi-x_i93.491.270.0
π0.5\pi_{0.5}Random91.889.864.0
DivPrune92.089.086.8
Language (raw)90.252.446.4
Language (ABTT)90.285.272.6
Hub −xi-x_i91.485.271.4

Random selection and DivPrune on π0.5\pi_{0.5} score above both of our rules at 50% and 25%. This suggests the policy benefits more from spreading the budget across the frame than from concentrating it on the objects. VLA-Pruner's released code supports only OpenVLA and OpenVLA-OFT, so we report it on OpenVLA-OFT.

To test whether coverage is what the criterion lacks on π0.5\pi_{0.5}, Appendix D combines the two. It keeps half of the budget by the language score and fills the other half with DivPrune's selection. With 25% of the tokens kept (LIBERO average, Table 10):

PolicyLanguageLanguage + DivPruneDivPrune
π0.5\pi_{0.5}92.295.795.6
OpenVLA-OFT96.296.694.1

Dear Reviewer,

Thank you again for your feedback. As described in our response above, we have revised our wording regarding the unsupported claims, and we have added the pruning-ratio results at 50%, 25%, and 12.5%, as you suggested (Appendix C). As the discussion period closes soon, we would be grateful if you could let us know whether our revisions address your concerns or if there is anything further we can clarify.

Thank you.

Reviewer_QjMtOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines what language-guided visual-token pruning actually selects in vision-language-action (VLA) models. The authors introduce a simple mean cosine similarity score and decompose it into a component associated with visual typicality and an orthogonal residual component, analyzing how representation geometry affects token selection across backbones. Instruction substitution experiments, comparisons with hub-based selection, and evaluations across models suggest that, in some models, the benefits of pruning primarily reflect redundancy suppression rather than semantic alignment with the task instruction. The experiments cover four VLA models and three robotics benchmarks and also identify settings in which the mechanism fails.

Critical strengths
  1. Insightful observations supported by a coherent analysis. Starting from the effectiveness of a simple pruning rule, the paper examines its dependence on the specific instruction content. The geometric decomposition, instruction substitution experiments, and comparisons across backbones together provide a clear framework for understanding why pruning works and when it fails.

  2. A simple method with broad evaluation and meaningful efficiency gains. The method requires no additional training and uses a simple scoring and selection procedure. The evaluation spans multiple models and benchmarks, with substantial latency reductions for OpenVLA-OFT and CogACT at relatively small costs in success rate. Reporting performance degradation in settings such as LLaVA-VLA also helps delineate the method's applicability.

Critical weaknesses
  1. Potential language dependence in the π0.5 control. Section 3.3 estimates the ABTT directions from a joint matrix of visual and language embeddings and uses them to transform the visual features. Under the current description, the subsequent hub score may therefore retain an indirect dependence on language through preprocessing. Removing the explicit language query alone does not establish that the entire control is independent of language information. This issue directly affects the credibility of the proposed interpretation for π0.5.

  2. Limited validation of budget sensitivity and the scope of the conclusions. The main performance comparisons focus on selected retention ratios and lack a systematic analysis of budget sensitivity. It is therefore difficult to determine whether the performance advantages and mechanistic observations hold across compression levels. Performance costs also vary considerably across models, and some settings require ABTT or different scoring rules to obtain favorable results. The paper should more precisely specify the scoring rules and geometric conditions under which its conclusions hold, and avoid generalizing results from particular settings into broad claims of superiority over existing pruning methods.

Questions for authors
  1. ABTT and language dependence. Does the hub-based control for π0.5 use ABTT directions computed from the joint visual–language matrix? How are these directions handled when the instruction used for scoring is replaced? Could a control using visual-only preprocessing, or other direct evidence, establish whether implicit language dependence affects the main conclusions?

  2. Budget selection and sensitivity. How were the retention ratios selected for each model? Could you report success rate and measured latency at several budget settings for representative models, and check whether the main mechanistic observations remain consistent across budgets? Tables 3–5 use 25% retention, whereas the LLaVA-VLA discussion states a "75% retention budget." Please clarify whether these refer to different experimental settings or reflect a confusion between retention and compression ratios.

  3. Complexity estimates and timing scope. What assumptions underlie the approximations in Eqs. (12)–(14) of Appendix A.1? Please clarify how the quadratic attention term and the costs of scoring and ABTT are accounted for, and whether the reported end-to-end latency includes these steps. This would help reconcile the theoretical cost estimates with the measured speedups.

2 comments

We thank the reviewer for their insightful comments and suggestions. We have tried to cover all raised points, including in the weaknesses, and we are happy to modify based on the discussion.

Response to weakness 1 (language dependence in the π0.5\pi_{0.5} control)

The reviewer is right. In the submitted version, the hub rule on π0.5\pi_{0.5} ranked patches after the joint ABTT transform. That transform is fit on both the patch and the prompt embeddings, so the control was not strictly language-free.

In the revision version we just submitted, we have updated results for π0.5\pi_{0.5} and have now removed this dependence. The hub rule now ranks the raw π0.5\pi_{0.5} embeddings, with no transform, so no language information enters it at any stage. ABTT is now used only by the language criterion. Now the hub rule remains the language-free control and language and hub rule both reach almost the same success rates.

π0.5\pi_{0.5}, 25% kept, LIBERO averageSuccess (%)
Hub rule −xi-x_i, raw embeddings (revised)91.6
Language criterion + ABTT92.2
Language criterion, raw embeddings75.7

Response to weakness 2 (budget sensitivity and scope of the claims)

Budget sensitivity. We have now added an evaluation on LIBERO-Long at three budgets, 50%, 25% and 12.5% of the visual tokens kept, with random selection, DivPrune, and VLA-Pruner. The following Table shows these results and they are also included in Appendix C in the revised version. VLA-Pruner's released code supports only OpenVLA and OpenVLA-OFT, so we report it on OpenVLA-OFT.

ModelMethod50%25%12.5%
OpenVLA-OFTRandom93.682.841.8
DivPrune94.092.275.8
VLA-Pruner93.693.073.7
Language93.692.488.0
Hub −xi-x_i93.491.270.0
π0.5\pi_{0.5}Random91.889.864.0
DivPrune92.089.086.8
Language (raw)90.252.446.4
Language (ABTT)90.285.272.6
Hub −xi-x_i91.485.271.4

These results show that OpenVLA-OFT benefits more from the criterion even at lower drop-rates as can be seen in Table 9 where the Language has the highest success rate at the highest drop rate. However, for π0.5\pi_{0.5} spread is more important, with DivPrune performing the best. Appendix D also shows the results of combining both Language and DivPrune. In that setting, the combination shows the best overall average success rate, suggesting that both approaches can be complementary and could be combined to get the best results on suitable backbones. We test with OpenVLA-OFT and π0.5\pi_{0.5}.

Scope of the claims. We removed the claim of superiority over existing methods. The paper now states the criterion is competitive on OpenVLA-OFT. We state the conditions under which the criterion ranks patches by typicality:

  • The query points away from the visual hub (α<0\alpha < 0).
  • Typicality varies within the frame (anisotropy AA well below 1).

OpenVLA-OFT and CogACT satisfy both. π0.5\pi_{0.5} satisfies them only after ABTT. LLaVA-VLA does not, because there α≈0\alpha \approx 0. The paper now presents these conditions as findings of the analysis, not as general claims about pruning.


Response to the question on ABTT and language dependence

  1. Does the hub control use the joint ABTT direction? In the submitted version, yes. In the revision it ranks the raw embeddings with no transform, so no language information enters it at any stage.

  2. What happens when the scoring text is replaced? In the substitution experiments (Table 2), the ABTT direction is refit on the patch rows together with the rows of the substitute text. The true instruction therefore has no effect on which patches are kept.

  3. Visual-only preprocessing.

We examined vision-only preprocessing in the revised version in Appendix E, Table 11. Fitting the direction on the patches alone removes the hub itself. Only the joint fit keeps a usable hub and gives a negative α\alpha. The joint fit is needed only for the language criterion.

Response to the question on budget selection and sensitivity

How the budgets were chosen. All main tables keep 25% of the visual tokens per camera, for every model. At 50%, all methods are within about two points of one another, so the selectors cannot be told apart. 25% is the largest budget at which they separate. The substitution study (Table 2) uses 12.5% for OpenVLA-OFT and π0.5\pi_{0.5}, because at 25% the criterion and the hub rule differ by less than a point on OpenVLA-OFT, which leaves no gap for the scoring text to explain.

Several budgets. Appendix C now reports success at 50%, 25% and 12.5% on LIBERO-Long for OpenVLA-OFT and π0.5\pi_{0.5}. We measured latency at the 25% budget used in the main tables. Prefill compute scales with the fraction of tokens kept (Appendix G, Eq. 14), so the cost at 50% and 12.5% follows.

"75% retention". This was a typo, a confusion between retention and compression. Every experiment, including LLaVA-VLA, keeps 25% of the visual tokens and drops 75%. We corrected the text, which now says "25% of the visual tokens kept" throughout.

Response to the question on complexity and timing scope

Complexity estimates. Eq. (12) counts the cost of one decoder layer on nn tokens. The first term covers the projections and the MLP, and the second term, 2n2D2n^2D, is the attention. We added the attention term in the revision. Eq. (14) then approximates the saving by the fraction of tokens removed. The attention term shrinks faster than the token count, so it only adds to the saving.

Cost of scoring and ABTT. We added these costs to Appendix G (Eq. 15). The language score and the hub rule cost about 3(N+M)D3(N+M)D multiply-adds per camera. The cost of ABTT is also linear in (N+M)D(N+M)D.

Timing scope. The reported latency includes the time of token selection, which includes scoring and ABTT where used.

Reviewer_jFJ9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a simple, yet effective language-vision similarity-based VLA pruning method. Moreover, it provides an in-depth analysis of the role of cross-modal similarity in pruning, revealing that, in several VLA models, the score primarily favors visually atypical patches rather than content semantically relevant to the task instruction.

Critical strengths
  1. The paper provides new insights into what cross-modal similarity actually measures in VLA token pruning. Through extensive qualitative and quantitative analyses, it reveals several important findings:

    1. The score primarily favors visually atypical patches.
    2. The task-specific instruction has little effect on pruning performance, but the text embeddings themselves can still provide an important signal for token selection.
    3. The behavior of the scoring criterion depends on the geometry of the backbone’s embedding space.
  2. The proposed pruning method is simple and training-free, yet achieves competitive performance–latency trade-offs in several evaluated settings.

Critical weaknesses
  1. The evaluation lacks real-world robot experiments. In relatively clean simulation environments such as LIBERO, visually atypical patches may naturally correspond to foreground objects, making background tokens easier to discard. However, visual typicality may be a less reliable indicator of redundancy in more complex real-world scenes. Hence real-robot experiments would help better assess the robustness of the proposed method.

  2. Despite its in-depth analysis of cross-modal similarity, the practical pruning strategy remains backbone-dependent. Language-based scoring outperforms the hub rule on OpenVLA-OFT, CogACT, and π0.5\pi_{0.5}, but performs substantially worse on LLaVA-VLA. Moreover, the results on π0.5\pi_{0.5} require an additional common-direction removal step (ABTT) before scoring. These backbone-specific choices limit the generality of the proposed approach.

Questions for authors
  1. Regarding findings (1) and (2) in Strength 1, I wonder whether these phenomena also occur in VLMs. This would help clarify whether the observed behavior is inherited from the pretrained VLM backbone or emerges during VLA fine-tuning.

  2. I wonder how the proposed criterion compares with [CLS] attention-based token selection, as used in LLaVA-PruMerge and VisPruner. Since these approaches also obtain selection signals before the decoder, they should be included as baselines in the performance evaluation.

1 comment

We thank the reviewer for their insightful comments and suggestions. We have tried to cover all raised points, including the weaknesses, and we are happy to modify based on the discussion.

Response to weakness 1 (real-world experiments)

We thank the reviewer for this point and agree that clean simulated scenes favor typicality. We could not run closed-loop experiments on a real robot arm during the discussion period. However, to test whether the geometry carries over to real images, we added an offline analysis on real robot frames to Appendix A of the revised version. We passed frames from CogACT's real-robot training data (the RT-1 dataset and BridgeData V2) through the policy and found the same behaviour as in simulation:

  • α\alpha is negative in every frame.
  • The score follows typicality.
  • The criterion and the hub rule keep largely the same patches.

Appendix A shows kept and dropped patches on real-world frames. Appendix B also shows the patches each rule keeps on one frame from each of several tasks from simulation, for OpenVLA-OFT, CogACT and LLaVA-VLA.


Response to weakness 2 (backbone dependence)

We agree that no single rule works best on every backbone, and we remove any wording that suggests that the criterion by itself is a general pruning method. The revised abstract and introduction reflect this.

We see the backbone dependence as one of the paper's findings. The decomposition explains why the criterion behaves differently on each backbone, using two quantities, α\alpha and anisotropy AA. The criterion suppresses typical patches when α<0\alpha < 0, needs ABTT when AA is close to 1, and fails when α≈0\alpha \approx 0 or α>0\alpha > 0.


Response to the question on VLMs

Thank you for this question. To understand if this is inherited from the base VLM, we added an analysis looking at the base VLM of each of the models in Appendix F. We passed the same frames and prompts through each policy's base VLM and compared the geometry with the fine-tuned VLA:

Modelα\alpha (VLA / base VLM)AA (VLA / base VLM)
OpenVLA-OFT / Prismatic−0.29-0.29 / +0.03+0.030.360.36 / 0.230.23
CogACT / Prismatic−0.27-0.27 / +0.00+0.000.280.28 / 0.230.23
π0.5\pi_{0.5} / PaliGemma+0.53+0.53 / +0.44+0.440.950.95 / 0.790.79
π0.5\pi_{0.5} / PaliGemma, after ABTT−0.87-0.87 / −0.80-0.800.010.01 / 0.630.63

Findings

  • Prismatic: in the base VLM, α\alpha is close to zero, so the score does not rank by typicality there. The negative α\alpha is acquired during VLA fine-tuning.
  • PaliGemma: the base VLM already has a positive α\alpha and strong anisotropy, and π0.5\pi_{0.5} inherits both.

We are also happy to include other general VLMs to the appendix, however, this will not show if the properties are inherited to VLAs, and can likely be an interesting next study.


Response to the question on [CLS]-attention baselines

Thank you for this suggestion. We have now added VisPruner results with OpenVLA-OFT and updated Table 4. On OpenVLA-OFT, with 25% of the visual tokens kept, the LIBERO averages are:

MethodAvg. success (%)Latency (ms)
VisPruner95.6104.6
Hub rule −xi-x_i95.996.9
Language criterion96.297.1
VLA-Pruner96.5113.0

OpenVLA-OFT's DINOv2 encoder has a [CLS] token so we used it. The SigLIP encoder of OpenVLA-OFT and π0.5\pi_{0.5} does not have [CLS] token.

Comments on the paper

2 comments
AuthorsOpenReview ↗

A new version based on the first set of reviews

We thank the reviewers for their constructive comments and feedback. We have uploaded an updated version of the paper that aims to take all the suggestions and weaknesses into account, clarifying parts of the text, scoping our claims, and adding new experiments. We are happy to consider any further comments from the reviewers. Again, we thank you for your reviews and comments!

Dear reviewers, we have now updated the submission to add some of the latencies that were missing in our previous edited version. We look forward to receiving any further feedback you have based on our edited version, and for any discussion or points you would like to raise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.