acceptodds
Under review as a conference paper at ICLR 2027

Policy-Functional Token Pruning for Efficient Vision-Language-Action Models

13 views

Abstract

Visual token pruning can accelerate vision-language-action (VLA) inference, but accurate importance rankings do not necessarily yield an effective retained set. Our token-deletion diagnostics show that attention closely tracks how much removing a token changes an intermediate action-related representation, yet different influential tokens can change it in similar directions. In light of these observations, we introduce Policy-Functional Token Pruning (PFT-PRUNE), a training-free method that considers both individual influence and pairwise directional alignment. Specifically, at an early attention layer, we characterize each visual token by the local sensitivity of an action-related representation to its key, and use shared random projections to construct a compact policy-functional fingerprint (PFF). PFF energy provides the scalar priority, while fingerprint alignment identifies potential functional redundancy; Functional NMS uses both to select tokens under an exact budget before physical pruning. Across four LIBERO suites, PFT-PRUNE achieves 94.1% and 93.2% average success on and OpenVLA-OFT at 20% nominal visual-token retention, with inference speedups of and , respectively. On four physical-robot tasks, it achieves a speedup at 50% nominal retention while preserving 95.9% of 's average success rate.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, visual token pruning, efficient inference, policy-functional representation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hvTAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PFT-PRUNE, a training-free method for pruning visual tokens in vision-language-action models based on their functional impact on action predictions. It introduces Policy-Functional Fingerprints (PFFs) to estimate token importance and redundancy, and Functional NMS to select a diverse set of tokens under a fixed budget. Experiments on LIBERO and real-world robotic tasks show that PFT-PRUNE reduces inference latency while preserving task success rates.

Critical strengths

S1 - well-motivated problem: existing visual pruning methods follow the pruning methodology learned from vision language models and focus on attention scores or visual similarity, which may not reflect a token's actual impact on action generation.

S2 - technically interesting approach with strong experimental coverage: PFFs provide a principled, action-oriented measure of token importance and functional redundancy and the method is evaluated on two VLA architectures, multiple LIBERO benchmarks, and real-world robot tasks, with comparisons against several pruning baselines.

S3 - practical potential: At 20% token retention, the method achieves 1.23× speedup for π0.5 and 1.59× for OpenVLA-OFT while maintaining competitive success rates.

Critical weaknesses

W1 - the local Jacobian as a proxy for token deletion: the theoretical analysis relies on simplified attention settings. It remains unclear how accurately the proposed local Jacobian-based fingerprints capture the actual effects of token removal in full VLA models.

W2 - moderate end-to-end gains: despite substantial token reduction, observed speedups are relatively modest, for example 1.11x speedup for pi0.5 at 50% retention illustrating that the relationship between token reduction and wall-clock latency is not straightforward. A more detailed runtime breakdown would clarify the computational overhead of fingerprint estimation and token selection.

W3 - limited robustness evidence: the experimental evidence is not yet sufficient to establish robust superiority across task regimes. Statistical uncertainty and generalization across environments, viewpoints, and task distributions are insufficiently explored.

W4 - heuristic design choices: the sensitivity to random projection count and the Functional NMS threshold is not fully established. For example, functional NMS suppresses a candidate whencosine similarity exceeds a fixed threshold, with 0.9 used as the default. This is intuitive, but it introduces a hyperparameter whose appropriate value may depend on the model, layer, camera view, fingerprint dimension, and task.

Questions for authors

Q1 - how well do PFF estimates correlate with the actual change in action predictions after token removal in the full model?

Q2 - what are the runtime costs of fingerprint computation, NMS, and token gathering, and how do they affect end-to-end speedup?

Q3 - how sensitive are performance and latency to the random projection count and NMS threshold?

Q4 - can the method generalize to more diverse real-world tasks and distribution shifts, for example long horizon task and precision-sensitive manipulations?

7 comments
AuthorsOpenReview ↗

Response To Reviewer hvTA (Part 1/4)

We thank you for your thorough review and thoughtful comments. Your questions have helped us clarify the scope of our theoretical analysis and strengthen the empirical evaluation. Below, we address each concern with additional experiments and clarifications. Changes to the manuscript are highlighted in blue for easy reference. We would be happy to clarify any remaining questions or concerns.

W1. the local Jacobian as a proxy for token deletion: the theoretical analysis relies on simplified attention settings. It remains unclear how accurately the proposed local Jacobian-based fingerprints capture the actual effects of token removal in full VLA models.

Q1. how well do PFF estimates correlate with the actual change in action predictions after token removal in the full model?

A1. Thank you for the question. We directly evaluate how token selection based on PFF estimates affects the complete policy's final action predictions. We remove all discarded tokens simultaneously, run the remaining policy layers and measure action RMSE relative to the unpruned output. On OpenVLA-OFT LIBERO-Long, PFT-Prune reduces full-action RMSE by 13.0% relative to PFF-Energy Top-K and 12.8% relative to PFF energy + native-key cosine at the same token budget.

We evaluate 250 observations from 50 episodes across all ten LIBERO-Long tasks, using the Long subset of our four-suite diagnostic. All variants use the same checkpoint, observations, pruning boundary and per-camera budgets, retaining 102 visual tokens. We measure RMSE over seven normalized action coordinates for the full eight-step action chunk and its first five steps. Entries are mean ± sample standard deviation across ten task means, computed by first averaging observations within episodes and then episodes within tasks.

VariantFull 8-step RMSE ↓First-5 RMSE ↓
Full PFT-Prune0.05229 ± 0.012030.05024 ± 0.01334
PFF-Energy Top-K0.06008 ± 0.012630.05662 ± 0.01379
PFF energy + native-key cosine0.05994 ± 0.013710.05656 ± 0.01444
Attention Top-K0.06064 ± 0.011500.05812 ± 0.01327

PFT-Prune achieves lower mean RMSE than all three controls for both the full action chunk and its first five steps, supporting improved final-action preservation under simultaneous token removal on LIBERO-Long.

Proposition 3.1 establishes the deletion-cosine identity for single-query, single-head attention, while Appendix A.3 derives the multi-query, multi-head Jacobian used by our implementation. The experiment above complements this local analysis by directly evaluating the final actions produced after pruning.

AuthorsOpenReview ↗

Response To Reviewer hvTA (Part 2/4)

W2. moderate end-to-end gains: despite substantial token reduction, observed speedups are relatively modest, for example 1.11x speedup for pi0.5 at 50% retention illustrating that the relationship between token reduction and wall-clock latency is not straightforward. A more detailed runtime breakdown would clarify the computational overhead of fingerprint estimation and token selection.

Q2. what are the runtime costs of fingerprint computation, NMS, and token gathering, and how do they affect end-to-end speedup?

A2. Thank you for the question. We profile the unpruned policy and PFT-Prune under the same conditions at both 50% and 20% nominal retention. Including all pruning overhead, complete-model latency decreases from 60.76 ms to 55.14 and 48.81 ms, respectively. The breakdown below shows that the savings mainly come from VLM computation, while image encoding, the initial dense layer and action generation account for substantial remaining latency.

We evaluate all three settings on the same 250 LIBERO-Long observations across ten tasks, using the same checkpoint and action noise on a single RTX 4090 at batch size one. PFT-Prune retains 384 and 154 visual tokens at 50% and 20% nominal retention, respectively. Each setting is measured over three passes, and the table reports mean ±\pm sample standard deviation across 750 calls. The stages cover complete model inference, including all pruning overhead.

Profiled stage (ms)Dense policyPFT 384 (50%)PFT 154 (20%)Saved (ms)
VLM: image encoding + transformer37.47 ± 0.1527.39 ± 0.1921.75 ± 0.075.64
Fingerprints + probe setup0.002.82 ± 0.442.66 ± 0.630.17
Functional NMS0.000.76 ± 0.140.75 ± 0.140.01
Energy/cosine, fill + bookkeeping0.000.39 ± 0.070.37 ± 0.070.01
Token gathering/compaction0.000.44 ± 0.040.43 ± 0.050.02
Action: 10 denoising steps22.99 ± 0.1822.58 ± 0.0222.11 ± 0.010.46
Other preprocessing/finalization0.30 ± 0.030.76 ± 0.070.74 ± 0.100.02
Complete model total60.76 ± 0.2855.14 ± 0.5048.81 ± 0.716.33

At 50% and 20% nominal retention, total pruning overhead is 4.41 and 4.21ms, respectively. Including this overhead, complete-model latency decreases by 9.2% and 19.7%. Reducing the retained budget from 384 to 154 tokens saves a further 6.33~ms, primarily through reduced VLM computation, while pruning overhead remains similar. The overall speedup is limited by computation that token pruning does not remove, including image encoding, the initial dense layer and action generation.

We also evaluate implementation optimizations in a separate deployment replay. These reduce fingerprint computation plus selection from 3.12 to 0.86~ms and complete-model latency from 47.02 to 44.64ms. These measurements use a separate timing setup and are not directly comparable to the instrumented profile above.

AuthorsOpenReview ↗

Response To Reviewer hvTA (Part 3/4)

W4. heuristic design choices: the sensitivity to random projection count and the Functional NMS threshold is not fully established. For example, functional NMS suppresses a candidate whencosine similarity exceeds a fixed threshold, with 0.9 used as the default. This is intuitive, but it introduces a hyperparameter whose appropriate value may depend on the model, layer, camera view, fingerprint dimension, and task.

Q3. how sensitive are performance and latency to the random projection count and NMS threshold?

A3. Thank you for the question. We evaluate how probe count and the Functional NMS threshold affect both task success and inference latency. The results support R=2R=2 as a practical success--latency tradeoff and show that the default threshold γ=0.90\gamma=0.90 achieves mean success close to the best observed value in the tested range.

Probe-Count Sensitivity. We evaluate π0.5\pi_{0.5} with R=1,2,4,8,16R=1,2,4,8,16, fixing γ=0.90\gamma=0.90 and retaining 154 visual tokens. Each setting uses the same four probe seeds, with 200 episodes per suite and seed across all four LIBERO suites, totaling 16,000 episodes. All other evaluation settings are held fixed. Success is reported as mean ±\pm sample standard deviation across seeds, after averaging the four suites within each seed.

#RRSuccess (%)Overall Latency (ms)Fingerprints+Selection (ms)
194.53 ± 0.4845.00 ± 0.010.81
294.91 ± 0.4445.05 ± 0.030.86
493.88 ± 0.4745.17 ± 0.030.96
893.72 ± 1.0345.42 ± 0.071.17
1693.47 ± 0.9545.78 ± 0.051.59

At R=2R=2, mean success reaches 94.91%, while complete-model latency is only 0.05 ms higher than at R=1R=1. Increasing RR further raises fingerprint and selection costs without improving mean success in this sweep, supporting R=2R=2 as a practical success–latency tradeoff. Latency is measured on the same 250 observations over three passes per seed and probe count, and is reported as mean ±\pm sample standard deviation across four seed-level means. These measurements use a four-thread replay, separate from the 16-thread deployment replay in A2.

NMS Threshold Sensitivity. We extend the π0.5\pi_{0.5} threshold study in Appendix to OpenVLA-OFT, evaluating γ=0.80,0.85,0.90,0.95\gamma=0.80,0.85,0.90,0.95 with three probe seeds. We fix R=2R=2 and retain 102 visual tokens, with all other evaluation settings held fixed across thresholds. Each threshold is evaluated over 500 episodes per suite and seed across all four LIBERO suites, totaling 24,000 episodes. All success rates are reported as mean ±\pm sample standard deviation across three seeds. For overall success, we first average the four suites within each seed.

γ\gammaSpatialObjectGoalLongOverallLatency (ms)
0.8094.20 ± 0.6990.73 ± 3.3096.20 ± 0.5390.80 ± 0.8792.98 ± 1.2345.46 ± 1.26
0.8594.47 ± 0.6491.60 ± 3.1096.80 ± 0.5390.47 ± 0.1293.33 ± 0.6345.23 ± 1.10
0.90 (default)94.53 ± 0.5891.67 ± 3.2396.47 ± 0.6489.87 ± 0.9993.13 ± 0.6345.11 ± 0.93
0.9593.93 ± 0.7090.20 ± 3.6095.93 ± 0.4288.13 ± 1.0192.05 ± 0.6645.09 ± 1.10

Across the tested thresholds, mean overall success ranges from 92.05% to 93.33%. The default γ=0.90\gamma=0.90 achieves 93.13%, only 0.20 percentage points below the highest observed mean at γ=0.85\gamma=0.85. Mean inference latency ranges from 45.09 to 45.46 ms. These results support γ=0.90\gamma=0.90 as a reasonable default in the evaluated setting.

Reviewer_pCK1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free visual token pruning method for VLA models. It uses randomized Jacobian fingerprints to estimate token influence and directional redundancy, followed by Functional NMS for budgeted selection. Experiments on two VLA architectures across LIBERO and real-world robot trials demonstrate good success rates with reduced inference latency.

Critical strengths
  1. Clear motivation. The paper identifies a meaningful limitation of scalar importance-based token pruning and motivates functional redundancy through deletion diagnostics.
  2. It's an interesting approach. Jacobian-based fingerprints and Functional NMS provide a principled way to incorporate directional information into token selection without additional training.
  3. Extensive evaluation. Experiments cover two VLA architectures, four LIBERO suites, and 2,000 physical-robot trials. The ablations and diagnostic analyses are well organized.
Critical weaknesses
  1. Limited theoretical justification. Proposition 3.1 establishes exact equivalence only for single query and single head attention, which doesn't generally extend to multi-head attention, pre-RoPE fingerprints, or final action predictions. And similarity between individual token-deletion effects don't establish redundancy under joint token removal.
  2. Weak finite-probe guarantees. Proposition 3.2 establishes unbiased estimation of Jacobian inner products in expectation, but provides no strong guarantee that (R=2) preserves the pairwise similarities or rankings used by Functional NMS.
  3. Token budget comparisons are not entirely straightforward. For pi0.5, nominal retention includes 256 masked-camera tokens assigned zero budget. Thus, 20% nominal retention corresponds to approximately 30% retention of active-view tokens. Different pruning schedules across baselines further complicate comparisons at matched nominal budgets.
  4. Efficiency gains and architectural generality are kinda limited. Despite aggressive nominal pruning, the wall-clock speedups remain modest, especially on pi0.5 and the physical robot. The method also requires architecture-specific proxy construction and access to internal gradients.
Questions for authors
  1. How well does PFF similarity predict joint token-deletion effects or changes in final action predictions, beyond the intermediate representation (F)?
  2. How stable are functional NMS decisions with only two random probes? Do different probe realizations substantially change the selected tokens?
  3. Could the authors provide a breakdown of fingerprint computation, functional NMS, and remaining forward-pass latency? How much of the policy-inference speedup translates into end-to-end robotic execution?
5 comments
AuthorsOpenReview ↗

Response To Reviewer pCK1 (Part 1/4)

We thank you for your thorough review. We appreciate the positive comments, as well as the thoughtful points raised. Please kindly find our response to your comments below. Additionally, all modifications to the manuscript have been highlighted in blue for easy reference. We would be happy to clarify any remaining questions or concerns.

W1. Limited theoretical justification. Proposition 3.1 establishes exact equivalence only for single query and single head attention, which doesn't generally extend to multi-head attention, pre-RoPE fingerprints, or final action predictions. And similarity between individual token-deletion effects don't establish redundancy under joint token removal.

Q1. How well does PFF similarity predict joint token-deletion effects or changes in final action predictions, beyond the intermediate representation (F)?

A1. Thank you for highlighting these distinctions.

Multi-Head Attention. Appendix A.3 already derives the local Jacobian for multiple action queries and attention heads, including output projection and applicable local transformations through the chain rule. Our implementation computes PFFs from this Jacobian using shared vector–Jacobian products. Proposition 3.1 provides the exact deletion-cosine identity for single-query, single-head attention, while Appendix A.3 derives the multi-query, multi-head Jacobian used to construct our fingerprints.

Pre-RoPE Coordinates. Appendix B.4 distinguishes the native-key coordinates used by our implementation from the attention coordinates used in Proposition 3.1. With K~i=RiKi\widetilde K_i=R_iK_i, where RiR_i is the orthogonal RoPE transformation, the chain rule gives Ji=J~iRiJ_i=\widetilde J_iR_i and hence ∥Ji∥F=∥J~i∥F\|J_i\|_F=\|\widetilde J_i\|_F. PFF energy is also preserved when using the same probes. However, cross-token inner products involve the relative rotation RiRj⊤R_iR_j^\top, so PFF cosine can change between these coordinate systems. Our implementation measures sensitivity in pre-RoPE coordinates; the exact deletion-cosine identity remains restricted to the attention coordinates specified in Proposition 3.1.

Joint Token Removal And Final Actions. To evaluate the effects beyond the local representation, we remove all discarded tokens simultaneously, run the remaining policy layers and compare the final action predictions with the unpruned output. We use 250 observations from 50 episodes across all ten OpenVLA-OFT LIBERO-Long tasks. All variants share the checkpoint, observations, pruning boundary and per-view budgets, retaining 102 visual tokens. We report RMSE for the full eight-step action chunk and its first five steps over seven normalized action coordinates.

VariantFull 8-step RMSE ↓First-5 RMSE ↓
Full PFT-Prune0.0523 ± 0.01200.0502 ± 0.0133
PFF-Energy Top-K0.0601 ± 0.01260.0566 ± 0.0138
PFF energy + native-key cosine0.0599 ± 0.01370.0566 ± 0.0144
Native-key norm + native-key cosine0.0977 ± 0.01270.0913 ± 0.0163

PFT-Prune reduces full-action RMSE by 13.0% relative to PFF-Energy Top-K and 12.8% relative to PFF energy + native-key cosine. These results show that PFF-based selection better preserves final action predictions under simultaneous token removal on OpenVLA-OFT LIBERO-Long, complementing the local Jacobian analysis with evidence from the complete policy.

AuthorsOpenReview ↗

Response To Reviewer pCK1 (Part 2/4)

W2. Weak finite-probe guarantees. Proposition 3.2 establishes unbiased estimation of Jacobian inner products in expectation, but provides no strong guarantee that (R=2) preserves the pairwise similarities or rankings used by Functional NMS.

Q2. How stable are functional NMS decisions with only two random probes? Do different probe realizations substantially change the selected tokens?

A2. Thank you for raising this question. Proposition 3.2 establishes unbiased inner-product estimation. We complement this result with empirical measurements of pairwise NMS threshold decisions, final token selections and task success at R=2R=2.

Functional NMS Stability With Two Random Probes. We evaluate OpenVLA-OFT with five probe seeds (10028–10032) on the same 1,000 observations, with 250 observations per LIBERO suite. We fix R=2R=2, γ=0.90\gamma=0.90 and the per-camera token budgets. For each camera view, we compare whether PFF cosine exceeds the suppression threshold for each of the 32,640 unordered token pairs. The table reports agreement percentages averaged over all ten seed pairs and observations. Agreement includes pairs that remain above the threshold and pairs that remain below it. Final token selection also depends on energy ordering, retained anchors and budget filling, which we assess through selected-set overlap below.

ViewSpatialObjectGoalLongOverall
Main98.5099.5899.1298.6498.96
Wrist99.3499.6599.3699.4799.45

Selected-Token Overlap. Using the same observations and probe seeds, we measure pairwise Jaccard similarity (intersection over union) between the final retained token sets, averaging over all ten seed pairs and observations. We compare against independent random selections with the same per-camera budgets: 41 of 256 main-view tokens and 61 of 256 wrist-view tokens. At approximately 20% token retention, mean pairwise Jaccard similarity is 0.4933, compared with an expected value of 0.1158 for random selection. The selected sets therefore share substantial overlap across probe seeds, although the exact retained tokens vary.

SuiteObservationsRandomPFTRatio
Spatial2500.11580.50484.36×4.36\times
Object2500.11580.49714.29×4.29\times
Goal2500.11580.48554.19×4.19\times
Long2500.11580.48584.19×4.19\times
Overall1,0000.11580.49334.26×4.26\times

Task Success And Probe Count. Separately, we evaluate π0.5\pi_{0.5} with R=1,2,4,8,16R=1,2,4,8,16, using the same four probe seeds across all four LIBERO suites. Each probe count covers 3,200 episodes, totaling 16,000 episodes. We keep the checkpoint, token budget, NMS threshold, initial states and action-noise schedule fixed. Success and latency are reported as mean ± sample standard deviation across four probe-seed means, with each seed mean computed over the four suites for success and three replay passes on the same 250 observations for latency. At R=2R=2, overall success is 94.91±0.44%94.91\pm0.44\%, with a range of 1.00 percentage point across seeds. Increasing the probe count raises computation without improving mean success in this sweep, supporting two probes as a practical default in the evaluated setting. Success and latency are reported as mean ± sample standard deviation across four probe-seed means, with each seed mean computed over the four suites for success and three replay passes on the same 250 observations for latency.

#RRSuccess (%)Overall Latency (ms)Fingerprints+Selection (ms)
194.53 ± 0.4845.00 ± 0.010.81
294.91 ± 0.4445.05 ± 0.030.86
493.88 ± 0.4745.17 ± 0.030.96
893.72 ± 1.0345.42 ± 0.071.17
1693.47 ± 0.9545.78 ± 0.051.59
AuthorsOpenReview ↗

Response To Reviewer pCK1 (Part 3/4)

W3. Token budget comparisons are not entirely straightforward. For pi0.5, nominal retention includes 256 masked-camera tokens assigned zero budget. Thus, 20% nominal retention corresponds to approximately 30% retention of active-view tokens. Different pruning schedules across baselines further complicate comparisons at matched nominal budgets.

A3. Thank you for highlighting this distinction. All methods evaluated on the same policy use the same nominal retention-rate definition. For π0.5\pi_{0.5}, this rate is calculated over 768 visual slots, including 256 masked slots that receive zero budget. Thus, nominal 20% retention corresponds to retaining 154 of 512 active visual tokens, or 30.08%. This denominator is used consistently for PFT-Prune and the baselines.

PolicyVisual slotsActive tokensMain / WristActive-Token Retention (%)
π0.5\pi_{0.5}76851262 / 92 = 15430.08%
OpenVLA-OFT51251241 / 61 = 10219.92%

For the component comparisons in A1, we use the same pruning boundary and per-camera token budgets across all variants, isolating the effect of token selection. Across different pruning methods, the same nominal retention rate can still lead to different computational costs because tokens are removed at different layers. We therefore compare task success alongside complete-policy inference latency, including the overhead of token selection and gathering.

Reviewer_EwMJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PFT-Prune, a training-free visual token pruning method for efficient Inference in VLA models. Unlike conventional approaches based on individual token importance or feature similarity, it considers both the magnitude and direction of a token's impact on action-related representations.To achieve this, it constructs a Policy-Functional Fingerprint (PFF) by compressing Jacobian using a shared random projection, and selects tokens via Functional NMS leveraging PFF energy and alignment. The experimental results demonstrate that the method effectively reduces inference time while maintaining a high task success rate compared to existing pruning methods.

Critical strengths
  1. Comparing the direction of action-related impacts rather than relying on conventional feature similarity is a core contribution. The paper experimentally demonstrates that PFF Alignment reflects the actual deletion direction better than Attention-Profile Similarity.

  2. Inference acceleration was confirmed across different VLA models and real-world robots.

Critical weaknesses
  1. Even if individual tokens exhibit similar deletion effects, it is not guaranteed that their simultaneous removal will yield a negligible joint impact. Prior work has also noted that local representation effects may differ from downstream pruning effects [1]. This paper lacks direct validation showing how functional alignment contributes to joint deletion effects and final action preservation.

  2. Token pruning leveraging feature diversity has been explored in prior literature [2]. However, while the performance gain of PFF Alignment over native-key cosine similarity is prominent in OpenVLA-OFT, it remains marginal in π0.5\pi_{0.5}. Given the extra computational cost of gradient calculations, a deeper analysis is needed to identify under what conditions PFF Alignment outperforms conventional feature similarity and why.

[1] Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization, NeurIPS 2025

[2] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models, CVPR 2025.

Questions for authors
  1. Are there experimental results demonstrating whether PFT-Prune reduces joint deletion effects and final action deviation more effectively than Energy Top-K or native-key cosine-based selection?

  2. What accounts for the marginal performance difference between PFF Alignment and native-key cosine similarity in π0.5\pi_{0.5}?

  3. To what extent do variations in the random probe seed affect token selection and overall task success rates?

6 comments
AuthorsOpenReview ↗

Response To Reviewer EwMJ (Part 1/3)

We thank you for your thorough review. We appreciate the positive comments, as well as the thoughtful points raised. Please kindly find our response to your comments below. Additionally, all modifications to the manuscript have been highlighted in blue for easy reference. We would be happy to clarify any remaining questions or concerns.

W1: Even if individual tokens exhibit similar deletion effects, it is not guaranteed that their simultaneous removal will yield a negligible joint impact. Prior work has also noted that local representation effects may differ from downstream pruning effects [1]. This paper lacks direct validation showing how functional alignment contributes to joint deletion effects and final action preservation.

Q1. Are there experimental results demonstrating whether PFT-Prune reduces joint deletion effects and final action deviation more effectively than Energy Top-K or native-key cosine-based selection?

A1. Thank you for highlighting this concern and pointing us to Balanced Token Pruning [1]. We have added this reference and discussed its relevance in the revised manuscript. We agree that preserving local representations alone does not establish preservation of downstream policy outputs. We therefore evaluate how removing multiple tokens simultaneously affects the policy's final action predictions on OpenVLA-OFT LIBERO-Long. At the same token budget, PFT-Prune reduces full-action RMSE by 13.0% relative to PFF-Energy Top-K and 12.8% relative to PFF energy + native-key cosine.

We evaluate 250 observations from 50 episodes across all ten LIBERO-Long tasks. For each observation, we remove all discarded tokens simultaneously, run the remaining policy layers and compare the final action chunk with the unpruned output. All variants use the same setting with 102 visual tokens. We measure RMSE over seven normalized action coordinates for the full eight-step action chunk and its first five steps. These results support improved final-action preservation when multiple tokens are removed simultaneously at the same token budget. Entries are mean ±\pm SD across ten task means.

VariantFull 8-step RMSE ↓First-5 RMSE ↓
Full PFT-Prune0.0523 ± 0.01200.0502 ± 0.0133
Native-key norm + native-key cosine0.0977 ± 0.01270.0913 ± 0.0163
Attention priority + PFF cosine0.0549 ± 0.00990.0531 ± 0.0116
PFF-Energy Top-K0.0601 ± 0.01260.0566 ± 0.0138
PFF energy + native-key cosine0.0599 ± 0.01370.0566 ± 0.0144
Value-side PFF0.0554 ± 0.01200.0523 ± 0.0148
Native-key norm + PFF cosine0.0886 ± 0.01840.0834 ± 0.0185
Attention Top-K0.0606 ± 0.01150.0581 ± 0.0133

[1] Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization, NeurIPS 2025

AuthorsOpenReview ↗

Response To Reviewer EwMJ (Part 2/3)

W2. Token pruning leveraging feature diversity has been explored in prior literature [2]. However, while the performance gain of PFF Alignment over native-key cosine similarity is prominent in OpenVLA-OFT, it remains marginal in π0.5\pi_{0.5}. Given the extra computational cost of gradient calculations, a deeper analysis is needed to identify under what conditions PFF Alignment outperforms conventional feature similarity and why.

Q2. What accounts for the marginal performance difference between PFF Alignment and native-key cosine similarity in π0.5\pi_{0.5}?

A2. We appreciate the reference to DivPrune [2] and agree that feature diversity has been explored for visual-token pruning. Our distinction lies in the similarity measure: PFF alignment compares tokens through their local effects on the policy representation, whereas native-key cosine compares the key vectors themselves. Our alignment-only comparison keeps PFF-energy ranking, the NMS procedure and per-view token budgets fixed, changing only the pairwise similarity measure. Functional alignment can therefore be useful when similarity between key vectors does not accurately reflect similarity in their effects on the policy representation.

On π0.5\pi_{0.5}, PFF alignment improves LIBERO-Long success from 88.0% to 91.0%, while Spatial is unchanged and Object and Goal decrease by 0.5 and 1.5 percentage points, respectively. The improvement on Long is therefore partly offset by changes in the other suites, yielding a small overall gain from 93.9% to 94.1%. These results show that the benefit varies across suites, with the largest observed improvement on LIBERO-Long.

On OpenVLA-OFT LIBERO-Long, the experiment in A1 further shows that replacing native-key cosine with PFF cosine reduces full-action RMSE by 12.8% under the same ranking procedure and token budget. This provides complementary evidence that functional alignment can improve final-action preservation.

Computational Cost. Both methods in the alignment-only comparison compute PFF energy, so PFF cosine reuses the gradients already computed for token ranking. Compared with methods that do not compute PFF energy, gradient computation remains an additional cost. We compute fingerprints by differentiating a local attention proxy with frozen model weights and two shared random probes. In our π0.5\pi_{0.5} profile, fingerprint computation and probe setup take 2.66 ms, and total pruning overhead is 4.21 ms. Including all pruning costs, complete-model latency decreases from 60.76 to 48.81 ms relative to the unpruned policy under the same conditions.

Profiled stage (ms)Dense policyPFT-Prune
VLM forward: image encoding + transformer37.47 ± 0.1521.75 ± 0.07
Fingerprints, including gradients and probe setup0.002.66 ± 0.63
Functional NMS0.000.75 ± 0.14
Energy/cosine, fill and selection bookkeeping0.000.37 ± 0.07
Token gathering/compaction0.000.43 ± 0.05
Action generation: 10 denoising steps22.99 ± 0.1822.11 ± 0.01
Other preprocessing, finalization and runtime0.30 ± 0.030.74 ± 0.10
Complete model total60.76 ± 0.2848.81 ± 0.71

[2] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models, CVPR 2025.

AuthorsOpenReview ↗

Response To Reviewer EwMJ (Part 3/3)

Q3. To what extent do variations in the random probe seed affect token selection and overall task success rates?

A3. Thank you for highlighting the importance of robustness to random probe seeds. We conducted experiments to evaluate both selected-token overlap and task-success variation. On OpenVLA-OFT, five probe seeds yield a mean pairwise Jaccard similarity of 0.4933 across 1,000 fixed observations. Separately, on π0.5\pi_{0.5}, four probe seeds yield overall success of 94.91±0.44%94.91\pm0.44\% across 3,200 episodes, with a range of 1.00 percentage point.

Task Success Across Probe Seeds. We evaluate π0.5\pi_{0.5} with four probe seeds, keeping the model, pruning settings, initial states and action-noise schedule unchanged. Each seed covers 800 episodes across all four LIBERO suites, totaling 3,200 episodes. We report mean ± sample standard deviation across seeds. Overall success ranges from 94.375% to 95.375%, a spread of 1.00 percentage point, indicating limited variation across the four tested probe seeds.

Probe seedSpatialObjectGoalLongOverall
Paper (Table 1)96.094.595.091.094.1
1798.595.596.089.094.750
2998.092.597.589.594.375
4398.097.097.088.595.125
7198.096.095.592.095.375
Mean (%)98.12595.25096.50089.75094.906
STD0.251.940.911.550.44

Selected-Token Overlap Across Probe Seeds. We evaluate OpenVLA-OFT with five probe seeds on the same 1,000 observations, keeping R=2R=2, γ=0.90\gamma=0.90 and the 102-token budget fixed. We report pairwise Jaccard similarity (intersection over union), averaged over all ten seed pairs and observations. For comparison, we include the expected Jaccard similarity of independent random selections under the same per-camera budgets.

At approximately 20% token retention, the mean pairwise Jaccard similarity is 0.4933, compared with 0.1158 for random selection under the same per-camera budgets. Thus, the selected sets overlap substantially across probe seeds, although the exact retained tokens vary. For deployment, we fix the probes and use deterministic sorting and fill rules to support repeatability for the same input and numerical backend.

SuiteObservationsRandomPFTRatio
Spatial2500.11580.50484.36×4.36\times
Object2500.11580.49714.29×4.29\times
Goal2500.11580.48554.19×4.19\times
Long2500.11580.48584.19×4.19\times
Overall1,0000.11580.49334.26×4.26\times

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their thorough reviews and constructive feedback. We appreciate their recognition of the clear motivation for functional redundancy in token pruning (Reviewers pCK1 and hvTA), the value of incorporating action-related directional information into token selection (Reviewers EwMJ, pCK1 and hvTA), and the broad evaluation across VLA architectures, LIBERO suites and physical-robot tasks (Reviewers pCK1 and hvTA). We also appreciate the recognition of inference acceleration across different VLA models and real-world robots (Reviewer EwMJ) and the method's practical potential (Reviewer hvTA).

We have added experiments to evaluate final-action preservation, quantify inference overhead, assess sensitivity to probe count, probe seed and NMS threshold, and test sequential physical manipulation under changes in object appearance and geometry.

1. Final-Action Preservation Under Joint Pruning. We directly evaluate final-action deviations from the unpruned policy after removing all discarded tokens simultaneously. On OpenVLA-OFT LIBERO-Long, PFT-Prune reduces full-action RMSE by 13.0% relative to PFF-Energy Top-K, 12.8% relative to PFF energy + native-key cosine, and 13.8% relative to Attention Top-K at the same 102-token budget.

2. Pruning Overhead And Inference Latency. We profile the unpruned policy and PFT-Prune under the same conditions, reporting the time spent on fingerprint computation, Functional NMS, token gathering and the remaining forward computation. These measurements explain how visual-token reduction affects complete-policy inference latency. We also evaluate implementation optimizations that reduce pruning overhead while maintaining task success.

3. Sensitivity To Probe Count, Probe Seed And NMS Threshold. We evaluate how probe count, probe seed and the Functional NMS threshold affect task success and inference latency. On π0.5\pi_{0.5}, two probes achieve the highest observed mean success among the tested probe counts. On OpenVLA-OFT, the default NMS threshold achieves mean success within 0.20 percentage points of the highest observed value in the threshold sweep. We also report variation in selected tokens and task success across probe seeds.

4. Additional Real-World Evaluation. We add a sequential puzzle-assembly task requiring four successive grasp–align–fit cycles, with success defined by correctly fitting all four pieces. The evaluation pieces differ in color and shape from those in the training data. This extends our physical-robot evaluation to longer manipulation sequences with repeated geometric alignment under changes in object appearance and geometry.

We reply to each reviewer's questions in detail below their reviews. Please kindly check them out. Thank you and please feel free to ask any further questions.

Area_Chair_6SQuOpenReview ↗

reviewer discussion

Dear Reviewers,

The authors have submitted their rebuttal. Please read their response carefully and engage in the discussion as soon as possible. Let us know whether the rebuttal addresses your concerns and update your final rating accordingly.

Thank you for your time and service to ICLR.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.