- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PFT-PRUNE, a training-free method for pruning visual tokens in vision-language-action models based on their functional impact on action predictions. It introduces Policy-Functional Fingerprints (PFFs) to estimate token importance and redundancy, and Functional NMS to select a diverse set of tokens under a fixed budget. Experiments on LIBERO and real-world robotic tasks show that PFT-PRUNE reduces inference latency while preserving task success rates.
Critical strengths
S1 - well-motivated problem: existing visual pruning methods follow the pruning methodology learned from vision language models and focus on attention scores or visual similarity, which may not reflect a token's actual impact on action generation.
S2 - technically interesting approach with strong experimental coverage: PFFs provide a principled, action-oriented measure of token importance and functional redundancy and the method is evaluated on two VLA architectures, multiple LIBERO benchmarks, and real-world robot tasks, with comparisons against several pruning baselines.
S3 - practical potential: At 20% token retention, the method achieves 1.23× speedup for π0.5 and 1.59× for OpenVLA-OFT while maintaining competitive success rates.
Critical weaknesses
W1 - the local Jacobian as a proxy for token deletion: the theoretical analysis relies on simplified attention settings. It remains unclear how accurately the proposed local Jacobian-based fingerprints capture the actual effects of token removal in full VLA models.
W2 - moderate end-to-end gains: despite substantial token reduction, observed speedups are relatively modest, for example 1.11x speedup for pi0.5 at 50% retention illustrating that the relationship between token reduction and wall-clock latency is not straightforward. A more detailed runtime breakdown would clarify the computational overhead of fingerprint estimation and token selection.
W3 - limited robustness evidence: the experimental evidence is not yet sufficient to establish robust superiority across task regimes. Statistical uncertainty and generalization across environments, viewpoints, and task distributions are insufficiently explored.
W4 - heuristic design choices: the sensitivity to random projection count and the Functional NMS threshold is not fully established. For example, functional NMS suppresses a candidate whencosine similarity exceeds a fixed threshold, with 0.9 used as the default. This is intuitive, but it introduces a hyperparameter whose appropriate value may depend on the model, layer, camera view, fingerprint dimension, and task.
Questions for authors
Q1 - how well do PFF estimates correlate with the actual change in action predictions after token removal in the full model?
Q2 - what are the runtime costs of fingerprint computation, NMS, and token gathering, and how do they affect end-to-end speedup?
Q3 - how sensitive are performance and latency to the random projection count and NMS threshold?
Q4 - can the method generalize to more diverse real-world tasks and distribution shifts, for example long horizon task and precision-sensitive manipulations?
7 comments
Response To Reviewer hvTA (Part 2/4)
W2. moderate end-to-end gains: despite substantial token reduction, observed speedups are relatively modest, for example 1.11x speedup for pi0.5 at 50% retention illustrating that the relationship between token reduction and wall-clock latency is not straightforward. A more detailed runtime breakdown would clarify the computational overhead of fingerprint estimation and token selection.
Q2. what are the runtime costs of fingerprint computation, NMS, and token gathering, and how do they affect end-to-end speedup?
A2. Thank you for the question. We profile the unpruned policy and PFT-Prune under the same conditions at both 50% and 20% nominal retention. Including all pruning overhead, complete-model latency decreases from 60.76 ms to 55.14 and 48.81 ms, respectively. The breakdown below shows that the savings mainly come from VLM computation, while image encoding, the initial dense layer and action generation account for substantial remaining latency.
We evaluate all three settings on the same 250 LIBERO-Long observations across ten tasks, using the same checkpoint and action noise on a single RTX 4090 at batch size one. PFT-Prune retains 384 and 154 visual tokens at 50% and 20% nominal retention, respectively. Each setting is measured over three passes, and the table reports mean sample standard deviation across 750 calls. The stages cover complete model inference, including all pruning overhead.
| Profiled stage (ms) | Dense policy | PFT 384 (50%) | PFT 154 (20%) | Saved (ms) |
|---|---|---|---|---|
| VLM: image encoding + transformer | 37.47 ± 0.15 | 27.39 ± 0.19 | 21.75 ± 0.07 | 5.64 |
| Fingerprints + probe setup | 0.00 | 2.82 ± 0.44 | 2.66 ± 0.63 | 0.17 |
| Functional NMS | 0.00 | 0.76 ± 0.14 | 0.75 ± 0.14 | 0.01 |
| Energy/cosine, fill + bookkeeping | 0.00 | 0.39 ± 0.07 | 0.37 ± 0.07 | 0.01 |
| Token gathering/compaction | 0.00 | 0.44 ± 0.04 | 0.43 ± 0.05 | 0.02 |
| Action: 10 denoising steps | 22.99 ± 0.18 | 22.58 ± 0.02 | 22.11 ± 0.01 | 0.46 |
| Other preprocessing/finalization | 0.30 ± 0.03 | 0.76 ± 0.07 | 0.74 ± 0.10 | 0.02 |
| Complete model total | 60.76 ± 0.28 | 55.14 ± 0.50 | 48.81 ± 0.71 | 6.33 |
At 50% and 20% nominal retention, total pruning overhead is 4.41 and 4.21ms, respectively. Including this overhead, complete-model latency decreases by 9.2% and 19.7%. Reducing the retained budget from 384 to 154 tokens saves a further 6.33~ms, primarily through reduced VLM computation, while pruning overhead remains similar. The overall speedup is limited by computation that token pruning does not remove, including image encoding, the initial dense layer and action generation.
We also evaluate implementation optimizations in a separate deployment replay. These reduce fingerprint computation plus selection from 3.12 to 0.86~ms and complete-model latency from 47.02 to 44.64ms. These measurements use a separate timing setup and are not directly comparable to the instrumented profile above.
Response To Reviewer hvTA (Part 3/4)
W4. heuristic design choices: the sensitivity to random projection count and the Functional NMS threshold is not fully established. For example, functional NMS suppresses a candidate whencosine similarity exceeds a fixed threshold, with 0.9 used as the default. This is intuitive, but it introduces a hyperparameter whose appropriate value may depend on the model, layer, camera view, fingerprint dimension, and task.
Q3. how sensitive are performance and latency to the random projection count and NMS threshold?
A3. Thank you for the question. We evaluate how probe count and the Functional NMS threshold affect both task success and inference latency. The results support as a practical success--latency tradeoff and show that the default threshold achieves mean success close to the best observed value in the tested range.
Probe-Count Sensitivity. We evaluate with , fixing and retaining 154 visual tokens. Each setting uses the same four probe seeds, with 200 episodes per suite and seed across all four LIBERO suites, totaling 16,000 episodes. All other evaluation settings are held fixed. Success is reported as mean sample standard deviation across seeds, after averaging the four suites within each seed.
| # | Success (%) | Overall Latency (ms) | Fingerprints+Selection (ms) |
|---|---|---|---|
| 1 | 94.53 ± 0.48 | 45.00 ± 0.01 | 0.81 |
| 2 | 94.91 ± 0.44 | 45.05 ± 0.03 | 0.86 |
| 4 | 93.88 ± 0.47 | 45.17 ± 0.03 | 0.96 |
| 8 | 93.72 ± 1.03 | 45.42 ± 0.07 | 1.17 |
| 16 | 93.47 ± 0.95 | 45.78 ± 0.05 | 1.59 |
At , mean success reaches 94.91%, while complete-model latency is only 0.05 ms higher than at . Increasing further raises fingerprint and selection costs without improving mean success in this sweep, supporting as a practical success–latency tradeoff. Latency is measured on the same 250 observations over three passes per seed and probe count, and is reported as mean sample standard deviation across four seed-level means. These measurements use a four-thread replay, separate from the 16-thread deployment replay in A2.
NMS Threshold Sensitivity. We extend the threshold study in Appendix to OpenVLA-OFT, evaluating with three probe seeds. We fix and retain 102 visual tokens, with all other evaluation settings held fixed across thresholds. Each threshold is evaluated over 500 episodes per suite and seed across all four LIBERO suites, totaling 24,000 episodes. All success rates are reported as mean sample standard deviation across three seeds. For overall success, we first average the four suites within each seed.
| Spatial | Object | Goal | Long | Overall | Latency (ms) | |
|---|---|---|---|---|---|---|
| 0.80 | 94.20 ± 0.69 | 90.73 ± 3.30 | 96.20 ± 0.53 | 90.80 ± 0.87 | 92.98 ± 1.23 | 45.46 ± 1.26 |
| 0.85 | 94.47 ± 0.64 | 91.60 ± 3.10 | 96.80 ± 0.53 | 90.47 ± 0.12 | 93.33 ± 0.63 | 45.23 ± 1.10 |
| 0.90 (default) | 94.53 ± 0.58 | 91.67 ± 3.23 | 96.47 ± 0.64 | 89.87 ± 0.99 | 93.13 ± 0.63 | 45.11 ± 0.93 |
| 0.95 | 93.93 ± 0.70 | 90.20 ± 3.60 | 95.93 ± 0.42 | 88.13 ± 1.01 | 92.05 ± 0.66 | 45.09 ± 1.10 |
Across the tested thresholds, mean overall success ranges from 92.05% to 93.33%. The default achieves 93.13%, only 0.20 percentage points below the highest observed mean at . Mean inference latency ranges from 45.09 to 45.46 ms. These results support as a reasonable default in the evaluated setting.
Response To Reviewer hvTA (Part 1/4)
We thank you for your thorough review and thoughtful comments. Your questions have helped us clarify the scope of our theoretical analysis and strengthen the empirical evaluation. Below, we address each concern with additional experiments and clarifications. Changes to the manuscript are highlighted in blue for easy reference. We would be happy to clarify any remaining questions or concerns.
A1. Thank you for the question. We directly evaluate how token selection based on PFF estimates affects the complete policy's final action predictions. We remove all discarded tokens simultaneously, run the remaining policy layers and measure action RMSE relative to the unpruned output. On OpenVLA-OFT LIBERO-Long, PFT-Prune reduces full-action RMSE by 13.0% relative to PFF-Energy Top-K and 12.8% relative to PFF energy + native-key cosine at the same token budget.
We evaluate 250 observations from 50 episodes across all ten LIBERO-Long tasks, using the Long subset of our four-suite diagnostic. All variants use the same checkpoint, observations, pruning boundary and per-camera budgets, retaining 102 visual tokens. We measure RMSE over seven normalized action coordinates for the full eight-step action chunk and its first five steps. Entries are mean ± sample standard deviation across ten task means, computed by first averaging observations within episodes and then episodes within tasks.
PFT-Prune achieves lower mean RMSE than all three controls for both the full action chunk and its first five steps, supporting improved final-action preservation under simultaneous token removal on LIBERO-Long.
Proposition 3.1 establishes the deletion-cosine identity for single-query, single-head attention, while Appendix A.3 derives the multi-query, multi-head Jacobian used by our implementation. The experiment above complements this local analysis by directly evaluating the final actions produced after pruning.