acceptodds
Under review as a conference paper at ICLR 2027

ProtoP: Region-Anchored Visual Token Pruning for Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models translate the reasoning capabilities of large vision-language backbones into continuous robotic control, but their deployment is bottlenecked by the cost of processing dense visual token sequences through Transformer backbones. Token pruning offers a natural remedy, yet we identify a critical failure mode of existing attention-based methods: spatial collapse. At high compression ratios, global Top- election concentrates the retained tokens onto one or two high-attention hotspots - typically the gripper or the currently manipulated object - even though the budget would suffice to cover every region of the scene, leaving the model blind to the rest of it and causing catastrophic task failure. The failure is one of coverage, not of token importance. We introduce ProtoP (Prototype-Preserving Token Pruning), a training-free pruning strategy that structurally prevents spatial collapse. ProtoP partitions the visual attention map into spatially coherent regions around local attention peaks ("prototypes"), then allocates the token budget across regions proportionally to prototype importance, with a one-token-per-region floor that guarantees every semantically distinct region survives. At moderate pruning ratios, where sufficient tokens remain to cover the scene, ProtoP matches the strongest baselines; as the ratio increases and spatial collapse becomes the dominant failure mode, the gap widens sharply, reaching 93.3% (OpenVLA-OFT) and 86.0% () task success at 90% pruning versus 82.2% and 72.8% for the best competing method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Token Pruning, Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_gw1tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free visual token pruning method for VLA models. The motivation is that, as the pruning ratio increases, attention-based global Top-k pruning tends to concentrate the retained tokens on a few dominant attention targets. The result is spatial collapse, so a small number of tokens must also be kept for the other task-relevant attention regions. ProtoP scores each visual token by the mean attention it receives from text and action query tokens at an early LLM layer. It then finds local attention maxima and uses watershed segmentation to partition the remaining foreground into regions. The token budget is allocated across regions in proportion to each region's peak score, and every region keeps at least one token. At 90% pruning on LIBERO, ProtoP reaches 93.3% success with OpenVLA-OFT and 86.0% with π0.5, outperforming VLA-Pruner and other methods. At 60–75% pruning, ProtoP performs on par with the baselines.

Critical strengths

Large gains at extreme pruning. At 90% pruning on LIBERO, ProtoP improves over the latest baseline (VLA-Pruner) by about 11.1 points on OpenVLA-OFT and about 13.2 points on π0.5. The high-compression regime is where pruning is most valuable for deployment, and it is also where existing methods break down.

Critical weaknesses

Effect of the scoring layer. FastV, SparseVLM and VLA-Pruner score tokens at layer 2, while ProtoP scores at layer 3. The ablation study shows how much this matters: scored at layer 2, ProtoP reaches 81.7%, below VLA-Pruner's 82.2%; at layer 1 it reaches 84.0%, and at layer 3 it jumps to 92.8%. Yet the paper's central claim is that coverage (the one-token-per-region floor), not the scoring, drives the improvement.

Limited novelty. The paper's contribution is a diagnosis plus a relatively simple fix, and the method is somewhat sensitive to hyperparameter choices.

Questions for authors

Layer indexing and compute. Is "layer 3" 0-indexed? Is FastV/VLA-Pruner's "K = 2" defined the same way? Please compare at matched TFLOPs or latency. More diversity-aware baselines (DivPrune, CDPruner, DART)

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors #1

Thank you for the review. We appreciate your recognition of the large gains at extreme pruning (+11.1 and +13.2 points over VLA-Pruner), the regime where pruning matters most for deployment and where existing methods break down. Your point about the scoring-layer confound made our central claim more precise. Changes in the revised PDF are in blue.

Notation: Table R# denotes a table in this rebuttal (numbered across all responses). "Main-paper Table #" and "Appendix Table S#" refer to the revised PDF. New experiments use OpenVLA-OFT on LIBERO at 90% pruning, with 200 episodes per suite. In tables, Δ is the difference in success rate (percentage points) from the reference row; ∗ p < 0.05, ∗∗ p < 0.01, ∗∗∗ p < 0.001 (exact McNemar test on paired episodes); unmarked differences are not significant.

[W1] "FastV, SparseVLM and VLA-Pruner score tokens at layer 2, while ProtoP scores at layer 3. ... Yet the paper's central claim is that coverage (the one-token-per-region floor), not the scoring, drives the improvement."

With all baselines moved to layer 3, ProtoP still leads by 14–27 points, so the gain does not come from the layer alone. The layer and coverage are both needed, and only coverage lets a method benefit from the deeper layer.

To separate the scoring layer from the coverage floor, we evaluated every baseline at both its default layer and ProtoP's layer 3 (with exactly 52 tokens), and we also evaluated ProtoP at layer 2 (Table R4).

Table R4. Effect of the scoring layer (Appendix Table S11). Each row shows the average at the default layer and at layer 3; Δ compares the layer-3 average with ProtoP.

MethodAvg, default layer → layer 3Δ at layer 3Object at layer 3
FastV67.0 → 75.5−19.3∗∗∗36.0
SparseVLM (single-step)71.4 → 71.4−23.4∗∗∗23.0
VLA-Pruner83.0 → 80.8−14.0∗∗∗47.0
Global Top-k (ProtoP's score)– → 67.4−27.4∗∗∗19.5
ProtoP (floor rule)†86.5 → 93.6−1.289.0
ProtoP (exact budget)94.8(ref.)93.0

† For ProtoP, the left value is at layer 2 (100 episodes per suite).

  • The layer is a shared hyperparameter. Every attention-based pruner has one, so it is not a ProtoP-specific advantage. VLA-Pruner's own ablation reports 58.5% at pruning layer K = 2 and 74.5% at K = 3 (their Table 6). In main-paper Table 2, FastV and VLA-Pruner use the layers their authors recommend.
  • Only coverage can exploit a deeper layer. Without coverage, the sharper score concentrates tokens further: VLA-Pruner drops from 83.0% to 80.8%, and on Object from 75.5% to 47.0%. With coverage, the same move raises ProtoP from 86.5% to 93.6%.
  • At layer 2, ProtoP is not significantly above VLA-Pruner (86.5 vs 84.0%, p = 0.22). So the layer and coverage are both needed, and coverage is what lets the better score pay off. The paper now presents them as separate factors (Sec. 5.3, 5.4, Appendix G.3).
AuthorsOpenReview ↗

Rebuttal by Authors #2

[W2] "Limited novelty. The paper's contribution is a diagnosis plus a relatively simple fix, and the method is somewhat sensitive to hyperparameter choices."

As mentioned, there are two contributions: the diagnosis reveals the issue, leading to our simple, but effective method on every suite. The method is also insensitive to most hyperparameters.

Diagnosis. We identify spatial collapse as the main failure mode of attention-based pruning at high compression. Every evaluated attention-based pruner, including the VLA-specific VLA-Pruner, concentrates its tokens on one or two hotspots and loses the rest of the scene. The revision confirms this diagnosis with ground-truth object masks that are independent of our method: removing the tokens on task objects lowers success by 33 points, whereas removing as many random tokens lowers it by 2 (Appendix G.6–G.7).

Simple, but effective. Not every simple fix works. At a matched score, layer, and token count (main-paper Table 5), each of the other simple rules fails on some suite:

  • The fixed grid fails on Long (83.5 vs 91.5%, p = 0.007).
  • Per-camera quotas and peaks + Top-k fail on Object (79.5 / 81.5 vs 93.0%, p < 0.001).
  • Feature diversity without attention (DivPrune) collapses to 35.6% (Appendix G.10).

ProtoP is the only rule that holds on every suite. It needs no training and improves over VLA-Pruner by 11–13 points on two backbones.

Sensitivity. Most choices have little effect:

  • The foreground threshold is flat: 93.0–93.4% over 0.25–1.0 (Appendix Table S4).
  • Peak detectors are interchangeable (Appendix Table S6).
  • d_min ∈ {1, 2} stays within 2 points (main-paper Table 4(b)).

The most sensitive choice is the scoring layer, which affects all attention-based methods alike.

[Q] "Is "layer 3" 0-indexed? Is FastV/VLA-Pruner's "K = 2" defined the same way?" Yes, all indices are 0-indexed over decoder layers:

  • ProtoP scores at index 3.
  • FastV with K = 2 scores at index 1.
  • VLA-Pruner and single-step SparseVLM score at index 2.

Sec. 5.1 and Appendix A.3 now state this.

[Q] "Please compare at matched TFLOPs or latency."

At matched FLOPs, ProtoP is the most accurate method we evaluated. In latency it is not faster, so the paper now claims accuracy at a given budget rather than efficiency.

We added operating points with similar analytic LLM FLOPs: progressive SparseVLM, ProtoP's selection inside the same progressive schedule, and VLA-Pruner with a larger budget (Table R5).

Table R5. Accuracy at similar compute (Appendix Table S12). Δ is relative to ProtoP.

MethodTFLOPsSuccess (%)Δ
ProtoP2.5894.8(ref.)
Progressive SparseVLM2.6589.9 (Object 75.5)−4.9∗∗∗
Progressive ProtoP2.6592.3 (Object 86.0)−2.5∗
VLA-Pruner (85% pruning)†2.6892.8−1.7
Progressive SparseVLM2.9095.0+0.2

† 100 episodes per suite, compared with ProtoP on the same episodes (94.5%). The other rows use 200 paired episodes per suite.

FLOPs. At matched FLOPs, ProtoP is significantly more accurate than progressive SparseVLM, which needs 12% more FLOPs to match ProtoP's accuracy. No evaluated operating point is both cheaper and more accurate than ProtoP.

Latency. ProtoP's benefit is accuracy at a given budget, not wall-clock time. VLA-Pruner at 85% pruning runs at 95 ms vs 109 ms for ProtoP, because our region construction runs on the CPU. The paper says this explicitly.

[Q] "More diversity-aware baselines (DivPrune, CDPruner, DART)"

We evaluated DivPrune, which collapses at 90% pruning. CDPruner and DART are now cited.

  • DivPrune (feature diversity without attention) reaches 35.6% on average (Appendix G.10).
  • Fixed grid (spatial diversity without attention) fails on Long.

Diversity helps only when it is aligned with task-conditioned attention, which is what ProtoP's regions provide. CDPruner and DART were not evaluated, and the paper says so.

Dear Reviewer gw1t,

Thank you again for your review. As the author-reviewer discussion period is coming to a close, we wanted to check whether our response addressed your concerns.

To separate the scoring layer from coverage, we evaluated every baseline at layer 3 as well as at its default layer. ProtoP still leads by 14 to 27 points, and at layer 2 the gap to VLA-Pruner is not significant, so we now present the layer and coverage as two separate factors rather than one. We also clarified that all layer indices are 0-indexed, added matched-FLOPs operating points, and evaluated DivPrune, which reaches 35.6%.

If any part of your concern remains open, we would be glad to answer it while there is still time.

Best regards, The Authors

Reviewer_Tu1DOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ProtoP, a training-free visual token pruning method for vision-language-action models. It identifies spatial collapse under aggressive global attention-based pruning, where retained tokens concentrate on a few hotspots and omit other task-relevant regions. ProtoP detects local attention peaks, partitions the attention map using watershed segmentation, and allocates tokens across regions while retaining at least one token per region. Evaluation on OpenVLA-OFT and π₀.₅, primarily on LIBERO with additional benchmark and small-scale real-robot experiments, shows improved task success under aggressive pruning compared with the evaluated baselines.

Critical strengths
  1. The paper addresses a practically relevant failure mode in VLA inference. Preserving information about secondary objects and future interaction targets is important for multi-stage manipulation, and the spatial-collapse analysis provides an intuitive motivation for considering coverage alongside attention scores.

  2. The method is lightweight and requires no additional training. The reported improvements at 90% pruning are substantial on LIBERO across two VLA architectures, suggesting that region-aware selection could be a useful practical component for efficient inference.

Critical weaknesses
  1. Limited methodological originality. The proposed method combines established components—local peak detection, watershed segmentation, and region-wise budget allocation—with a straightforward coverage constraint. The paper does not sufficiently establish what this combination contributes beyond simpler spatially diverse selection strategies or existing structural-preservation approaches. In particular, a baseline that retains local peaks and fills the remaining budget using global Top-k is missing. Without such comparisons, it remains unclear whether watershed-based partitioning and region-wise allocation constitute a meaningful methodological advance.

  2. Insufficiently controlled experimental comparisons. ProtoP uses attention from layer 3, whereas the main attention-based baselines use layer 2. This difference is consequential: Table 4(a) shows that ProtoP itself improves from 81.7% to 92.8% when changing the scoring layer from 2 to 3. Consequently, the main comparisons do not adequately isolate the contribution of the proposed selection mechanism. SparseVLM is also adapted from progressive pruning to a single pruning step, and VLA-Pruner is modified for OpenVLA-OFT. These adaptations warrant stronger validation, together with comparisons using matched scoring layers, attention sources, and actual retained token counts. The current evidence therefore leaves uncertainty about both the source and the extent of the claimed advantage.

Questions for authors
  1. Clarification and controlled evidence: How much of ProtoP’s advantage remains when it and the attention-based baselines use the same scoring layer, attention source, and actual retained token budget? In particular, can the authors provide a comparison at layer 3, given the large layer-dependent difference in Table 4(a)?

  2. Evidence needed to assess the methodological contribution: How does ProtoP compare with retaining all detected local peaks and filling the remaining budget through global Top-k, without watershed segmentation or region-wise allocation? This comparison would help determine whether the full proposed mechanism is necessary.

  3. Clarification of comparative coverage: Why was VLA-IAP, which is discussed in the related work and also preserves structural anchors, excluded from the experimental comparisons? Is there directly comparable evidence that establishes ProtoP’s additional benefit over this closely related approach?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors #1

Thank you for the review. We are glad that you found spatial collapse a practically relevant failure mode for multi-stage manipulation, and the method lightweight, training-free, and effective at 90% pruning on two architectures. Your suggested peaks + Top-k baseline is now a main-text control, and it helped us state precisely what region-wise allocation adds. Changes in the revised PDF are in blue.

Notation: Table R# denotes a table in this rebuttal (numbered across all responses). "Main-paper Table #" and "Appendix Table S#" refer to the revised PDF. New experiments use OpenVLA-OFT on LIBERO at 90% pruning, with 200 episodes per suite. In tables, Δ is the difference in success rate (percentage points) from the reference row; ∗ p < 0.05, ∗∗ p < 0.01, ∗∗∗ p < 0.001 (exact McNemar test on paired episodes); unmarked differences are not significant.

[W1] "...a baseline that retains local peaks and fills the remaining budget using global Top-k is missing." [Q2] "How does ProtoP compare with retaining all detected local peaks and filling the remaining budget through global Top-k, without watershed segmentation or region-wise allocation?"

We added this baseline to the main paper. ProtoP outperforms it (94.8 vs 91.9%), mainly on Object.

We implemented it together with other simple spreading rules. All use ProtoP's score (layer 3) and keep exactly 52 tokens, so only token placement differs (Table R1).

Table R1. Token placement at a matched score, layer, and budget (= main-paper Table 5).

SelectionObjectLongAvg (4 suites)Δ Avg
Global Top-k19.570.067.4−27.4∗∗∗
Peaks + Top-k (your proposal)81.592.091.9−2.9∗∗
Per-camera Top-k79.592.589.4−5.4∗∗∗
Fixed grid93.083.592.9−1.9
ProtoP (exact budget)93.091.594.8(ref.)

Peaks + Top-k falls short on Object (81.5 vs 93.0%, p < 0.001). Once the peaks are kept, global filling sends the rest of the budget back to the dominant hotspot. Allocating the remainder region by region is what prevents this.

[W1] "...it remains unclear whether watershed-based partitioning and region-wise allocation constitute a meaningful methodological advance."

The advance lies in identifying the constraint that attention-based pruners are missing, and in showing that region-wise allocation is the rule that satisfies it on every suite. The components themselves are standard by design. Table R1 shows that the constraint is decisive:

  • Coverage recovers the loss. Any spatial coverage rule recovers most of the 27-point loss of global Top-k. Even the score-agnostic fixed grid beats peaks + Top-k on Object (93.0 vs 81.5%, p < 0.001). Existing attention-based pruners lack coverage, and they collapse on Object even at our layer: FastV, SparseVLM, and VLA-Pruner reach 23–47% (Table R4).
  • Only ProtoP holds on every suite. The fixed grid fails on Long (83.5 vs 91.5%, p = 0.007), and per-camera quotas and peaks + Top-k fail on Object. ProtoP's regions come from the attention map, so it is the only rule without a failing suite.

The paper now presents the contribution in these terms: diagnosis, principle, and an instantiation that holds across suites (abstract, Sec. 5.4, conclusion).

AuthorsOpenReview ↗

Rebuttal by Authors #2

[W2] "ProtoP uses attention from layer 3, whereas the main attention-based baselines use layer 2. This difference is consequential..." [Q1] "...can the authors provide a comparison at layer 3, given the large layer-dependent difference in Table 4(a)?" (Q1)

Yes. At layer 3, every baseline remains 14–27 points below ProtoP.

We evaluated FastV, SparseVLM, and VLA-Pruner at both their default layers and ProtoP's layer 3 (with exactly 52 tokens), and we also evaluated ProtoP at layer 2 (Table R4).

Table R4. Effect of the scoring layer (Appendix Table S11). Each row shows the average at the default layer and at layer 3; Δ compares the layer-3 average with ProtoP.

MethodAvg, default layer → layer 3Δ at layer 3Object at layer 3
FastV67.0 → 75.5−19.3∗∗∗36.0
SparseVLM (single-step)71.4 → 71.4−23.4∗∗∗23.0
VLA-Pruner83.0 → 80.8−14.0∗∗∗47.0
Global Top-k (ProtoP's score)– → 67.4−27.4∗∗∗19.5
ProtoP (floor rule)†86.5 → 93.6−1.289.0
ProtoP (exact budget)94.8(ref.)93.0

† For ProtoP, the left value is at layer 2 (100 episodes per suite).

  • The layer is a shared hyperparameter. Every attention-based pruner has one. VLA-Pruner's own ablation reports 58.5% at pruning layer K = 2 and 74.5% at K = 3 (their Table 6).
  • Moving the baselines to our layer does not close the gap. Without coverage, the sharper deep-layer score concentrates more tokens on the hotspot: VLA-Pruner's Object score falls from 75.5% to 47.0%. With coverage, the same move raises ProtoP from 86.5% to 93.6%.
  • At layer 2, the gap disappears. ProtoP and VLA-Pruner are not significantly different there (86.5 vs 84.0%, p = 0.22; 100 episodes per suite). This is consistent with our account: both the layer and coverage are needed, and only coverage lets a method benefit from the better layer (Sec. 5.4, Appendix G.3).

[W2] "SparseVLM is also adapted from progressive pruning to a single pruning step, and VLA-Pruner is modified for OpenVLA-OFT. These adaptations warrant stronger validation..."

We validated the SparseVLM adaptation against the original progressive version. At matched compute, ProtoP remains significantly more accurate.

Beyond the single-step adaptation, we implemented the original progressive SparseVLM, and we also ran ProtoP's selection inside the same progressive schedule (Appendix Table S12).

  • Why single-step. The single-step SparseVLM follows the protocol and released code of VLA-Pruner, which compares SparseVLM as single-layer pruning under identical token budgets.
  • Progressive SparseVLM. Our three-stage version (layers 2/6/15) omits the token-recycling step. It keeps more tokens in the early layers, and it matches ProtoP (95.0 vs 94.8%, p = 0.90) at 12% more LLM FLOPs. At matched FLOPs, it drops to 89.9% (p < 0.001), with Object falling to 75.5%, the same collapse signature.
  • Progressive ProtoP. The schedule and the selection rule are orthogonal. Using ProtoP's selection inside the same progressive schedule recovers 10.5 points on Object (p < 0.001). Region-wise allocation therefore helps regardless of the pruning schedule.
  • VLA-Pruner. It keeps its dual-level scoring and diversity step. OpenVLA-OFT decodes all actions in one parallel pass, so its action attention is taken from the action tokens of that pass (Appendix A.3).

[Q3] "Why was VLA-IAP, which is discussed in the related work and also preserves structural anchors, excluded from the experimental comparisons?"

No official implementation was available. Sec. 2 now contrasts it with ProtoP (geometric priors vs. regions from the attention map) and states that we do not compare against it empirically.

Dear Reviewer Tu1D,

Thank you again for your review. As the author-reviewer discussion period is coming to a close, we wanted to check whether our response addressed your concerns.

We added the peaks + Top-k baseline you proposed, and it is now a control in the main paper. It reaches 91.9% against 94.8% for ProtoP, with the gap concentrated on the Object suite (81.5 vs 93.0%). We also moved every attention-based baseline to our scoring layer 3, where they remain 14 to 27 points below ProtoP, which addresses the layer confound you raised, and we validated the SparseVLM adaptation against the original progressive version.

If any part of your concern remains open, we would be glad to answer it while there is still time.

Best regards, The Authors

Reviewer_9oEqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ProtoP is a visual token pruning method for vision-language-action models that aims to preserve spatial coverage. It builds per-camera attention maps from an early layer, detects local peaks, partitions the foreground using watershed, and allocates tokens to regions according to peak importance with a one-token-per-region floor. The pruning rule requires no additional training. Experiments on OpenVLA-OFT and pi0.5 evaluate success rates and latency across pruning ratios, with further ablations, a small real-robot study, and visual perturbation analyses.

Critical strengths
  1. The 90% nominal-pruning results are strong on two VLA backbones: 93.3% versus 82.2% for VLA-Pruner on OpenVLA-OFT and 86.0% versus 72.8% on pi0.5, gains of 11.1 and 13.2 percentage points. Relative to each unpruned backbone, Table 2 reports approximately 1.63x and 1.60x latency speedups. This is a meaningful empirical result, even though it does not by itself identify which part of the selection rule causes the gain.

  2. The paper includes more than a headline table: Appendix B compares allocation weights, foreground thresholds, peak detectors, and lower pruning ratios; Appendix C adds a one-task physical-robot check and a prototype-stability diagnostic under visual perturbations. Table S9 discloses that OpenVLA-OFT actually retains fewer than the nominal tokens, so the reported gains cannot simply be attributed to keeping extra tokens. These analyses are useful, though they answer narrower questions than the central mechanism claim.

Critical weaknesses
  1. The distinctive contribution is not isolated from existing coverage based pruning. Prior work already combines saliency and coverage [1] or uses diverse visual tokens and structural anchors for efficient VLA inference [2], [3]. ProtoP's potentially new element is its attention-peak/watershed construction, yet it is not compared with fixed-grid or per-camera quotas, spatial farthest-point selection, or another simple diversity rule under matched scores and actual token counts. Appendix B.1 varies allocation weights inside the same watershed regions; it does not test whether those regions are needed. Appendix A.3 changes progressive SparseVLM into single-layer pruning and adapts VLA-Pruner to joint-pass attention/set unions, while ProtoP scores at layer 3 rather than the baselines' layer 2. The adaptations are disclosed, but conclusions about the original methods are correspondingly limited. Table S9's under-allocation rules out an extra-token explanation, yet makes actual-count-matched quality/latency comparisons more important. At 90% on OpenVLA-OFT, ProtoP is more accurate but slower than VLA-Pruner (107.5 versus 102.4 ms), so a blanket efficiency-superiority claim is not supported.

  2. The claimed semantic-coverage guarantee and the causal account exceed the evidence. A token floor can cover detected nonempty foreground basins when K <= B, but a semantic object can be filtered out, lack a peak, or share a basin; Algorithm 1 cannot guarantee survival of every semantically distinct region. Its max(1,floor(B*w_k)) rule also need not respect the global budget: B=10 with weights 0.9/0.05/0.05 allocates 9/1/1, or 11 tokens, if all regions have capacity. This is a formal counterexample to the unrestricted rule, not a claim that the reported runs overspent; Table S9 reports under-allocation in the evaluated OpenVLA-OFT settings. There is no independent quantitative object/region-coverage analysis establishing the paper's stronger claim that the performance difference is only coverage rather than token importance. Evaluation uncertainty further limits the cross-benchmark claims: DROID has only two derived tasks and a 3-point advantage over SparseVLM at 90%; the VLABench trial denominator differs between main text and appendix; and the one-task real-robot result has ten trials per configuration (50% versus 30% at 90% pruning). The perturbation study measures prototype stability, not closed-loop success under perturbation. These are substantive limits on the claimed mechanism and generality.

References

[1] SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs, DOI: 10.52202/085713-5394

[2] EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models, DOI: 10.52202/085713-1365

[3] Training-free interaction-aligned visual token pruning for efficient embodied manipulation, Available: https://arxiv.org/abs/2603.22991

Questions for authors
  1. At matched scoring source, layer, actual retained-token count, and end-to-end cost, does ProtoP outperform fixed-grid/per-camera quotas or a simple spatial-diversity selector? Please distinguish this from Appendix B.1's within-watershed allocation ablation, and state how the SparseVLM and VLA-Pruner adaptations affect the scope of comparison.

  2. Will the semantic-coverage claim be narrowed to detected nonempty foreground basins with K <= B? Please provide the full budget algorithm for K>B, region-size caps, floor-induced overshoot or unused remainder, and actual-count distributions for both backbones. Can detected-basin and annotated-object coverage be measured separately rather than treating them as equivalent?

  3. For pi0.5, which module and denoising step provide action-query attention, where is pruning applied, and how are positions and KV states handled? Does the implementation use Equation (1)'s query mean or the appendix's union of modality-selected tokens? Please report the scoring and selection cost within end-to-end timing and confirm identical compiled-denoising settings across methods.

  4. Please reconcile the VLABench trial denominator (100 per task in the main text versus 50 in Appendix A.2), specify construction of the two DROID-derived tasks, and explain the default Goal 92.4 in Table 4(a) versus 94.4 in Table 2. What paired-episode or repeated-run uncertainty supports the smaller gaps, and what are the trial-level outcomes for the one real-robot task?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors #1

Thank you for the detailed review. We are glad that you found the 90%-pruning results on two backbones meaningful (+11.1 and +13.2 points over VLA-Pruner), and the supplementary analyses useful (ablations, the real-robot check, and the disclosure of actual token counts). The budget counterexample and the request for matched-selector controls led directly to the exact-budget rule and the new Sec. 5.4. Changes in the revised PDF are in blue.

Notation: Table R# denotes a table in this rebuttal (numbered across all responses). "Main-paper Table #" and "Appendix Table S#" refer to the revised PDF. Unless noted, new experiments use OpenVLA-OFT on LIBERO at 90% pruning, with 200 episodes per suite. In tables, Δ is the difference in success rate (percentage points) from the reference row; ∗ p < 0.05, ∗∗ p < 0.01, ∗∗∗ p < 0.001 (paired exact McNemar test); unmarked differences are not significant.

[W1] "...it is not compared with fixed-grid or per-camera quotas ... or another simple diversity rule under matched scores and actual token counts." [Q1] "At matched scoring source, layer, actual retained-token count, and end-to-end cost, does ProtoP outperform fixed-grid/per-camera quotas or a simple spatial-diversity selector?"

Yes. At a matched score, layer, and token count, ProtoP outperforms all of these rules, and the comparison is now in the main paper (new Sec. 5.4).

All rules use ProtoP's score (layer 3) and keep exactly 52 tokens, so only token placement differs (Table R1).

Table R1. Token placement at a matched score, layer, and budget (= main-paper Table 5).

SelectionObjectLongAvg (4 suites)Δ AvgLatency (ms)
Global Top-k19.570.067.4−27.4∗∗∗101.6
Per-camera Top-k79.592.589.4−5.4∗∗∗102.0
Peaks + Top-k81.592.091.9−2.9∗∗106.0
Fixed grid93.083.592.9−1.9105.0
ProtoP (exact budget)93.091.594.8(ref.)109.3

Every rule that enforces spatial coverage recovers most of the 27-point loss of global Top-k: what attention-based pruners lack is coverage, not a better score.

  • Fixed grid. It ignores the score and fails on Long (83.5 vs 91.5%, p = 0.007), where the relevant objects change from stage to stage. Yet on Object, this score-agnostic grid outperforms peaks + Top-k, which does use the score (93.0 vs 81.5%, p < 0.001). Spreading tokens widely matters more than following the score precisely.
  • Per-camera quotas and peaks + Top-k. Both let the remaining budget flow back to the hotspot, and both fail on Object (79.5 / 81.5 vs 93.0%, p < 0.001).
  • ProtoP. It is the only rule within one point of the best on every suite. Its regions come from the attention map, so coverage also follows what the policy attends to.

DivPrune (feature diversity without attention) collapses to 35.6% (Appendix G.10).

[W1] "Appendix A.3 changes progressive SparseVLM into single-layer pruning ... while ProtoP scores at layer 3 rather than the baselines' layer 2. The adaptations are disclosed, but conclusions about the original methods are correspondingly limited."

We added layer-matched baselines and the original progressive SparseVLM. ProtoP's advantage holds at a matched layer and at matched compute.

  • Protocol. We followed the protocol and released code of VLA-Pruner, which compares all methods, including SparseVLM, as single-layer pruning under identical token budgets, so a budget also fixes the compute. Appendix A.3 now states this.
  • Layer-matched baselines. At layer 3 with 52 tokens, FastV, single-step SparseVLM, and VLA-Pruner reach 75.5 / 71.4 / 80.8%, all below ProtoP's 94.8% (p < 0.001; Appendix Table S11).
  • Progressive pruning. Our three-stage SparseVLM prunes at layers 2/6/15, without the token-recycling step. It keeps 237 and then 110 tokens in the early layers, so it uses 12% more LLM FLOPs at the same final budget. We also ran "progressive ProtoP", which replaces Top-k with ProtoP's selection at every stage of the same schedule (Table R2).

Table R2. Progressive pruning and compute-matched operating points (200 paired episodes per suite; Appendix Table S12).

MethodFinal tokensTFLOPsAvgΔ AvgObject
ProtoP (single-stage)522.5894.8(ref.)93.0
Progressive SparseVLM512.9095.0+0.295.0
Progressive ProtoP512.8995.3+0.594.5
Progressive SparseVLM, matched FLOPs352.6589.9−4.9∗∗∗75.5
Progressive ProtoP, matched FLOPs352.6592.3−2.5∗86.0

Progressive SparseVLM matches ProtoP only with more compute. At matched compute it collapses on Object, the same suite where global selection fails. Swapping in ProtoP's selection under the same schedule, score, and FLOPs recovers 10.5 points on Object (p < 0.001) and 2.4 points overall (p = 0.029). Single-stage ProtoP remains the most accurate at this compute.

AuthorsOpenReview ↗

Rebuttal by Authors #2

[W1] "...ProtoP is more accurate but slower than VLA-Pruner (107.5 versus 102.4 ms), so a blanket efficiency-superiority claim is not supported."

We removed this claim; the paper now claims accuracy at a given token budget. Region construction runs on the CPU and adds 7.7 ms over GPU Top-k (109.3 vs 101.6 ms). Appendix A.5 now reports end-to-end time.

[W2] "...a semantic object can be filtered out, lack a peak, or share a basin; Algorithm 1 cannot guarantee survival of every semantically distinct region." [Q2] "Will the semantic-coverage claim be narrowed to detected nonempty foreground basins with K <= B?"

Yes. The guarantee now covers detected attention regions only, and unprotected cases are listed as a limitation. With ground-truth masks, 14.1% of task objects contain no peak, and 8.3% are removed by the threshold (Appendix G.6).

[W2] "Its max(1,floor(B*w_k)) rule also need not respect the global budget..." [Q2] "Please provide the full budget algorithm for K>B, region-size caps, floor-induced overshoot or unused remainder, and actual-count distributions..."

We replaced the rule with one that keeps exactly B tokens, and we report the realized token counts of the submitted rule.

Overshoot. Your counterexample is correct; overshoot did occur. On OpenVLA-OFT, the submitted rule (the "floor rule" in the revision) kept more than 52 tokens in 8.7% of 16,836 queries, up to 75, with a mean of 46.7 (Appendix Table S13). K ranged from 12 to 50 (mean 26.7), so K > B never occurred.

Exact rule. The revised Algorithm 1:

  • If K ≥ B, keep the B highest-scoring prototypes.
  • Otherwise, keep one token per region and distribute the rest in proportion to prototype score (largest-remainder rounding), capped by region size, with the surplus re-distributed.

It always keeps 52 tokens and scores 94.8% vs 93.6% for the floor rule (p = 0.27). Main-paper Tables 2–4 keep the floor rule, as Sec. 4.2 states.

[W2] "There is no independent quantitative object/region-coverage analysis establishing the paper's stronger claim that the performance difference is only coverage rather than token importance." [Q2] "Can detected-basin and annotated-object coverage be measured separately rather than treating them as equivalent?"

Yes. We now measure annotated-object coverage separately and test its effect causally. We also softened the claim from "only coverage" to "primarily coverage".

We use the simulator's instance masks, independent of the watershed partition (Appendix Tables S14–S15).

  • Causal test. At a fixed budget, removing ProtoP's tokens on task objects lowers success from 93.3% to 60.2% (p < 0.001). Removing as many random tokens gives 91.3% (p = 0.14).
  • Coverage. ProtoP covers all of its own regions and 88% of the task objects; global selectors cover 70–81%.
  • Necessary but not sufficient. Forcing a token onto every task object does not rescue global Top-k (57.0 → 58.0%). Object coverage is necessary, and the rest of the attended layout also matters.

[W2] "...DROID has only two derived tasks and a 3-point advantage over SparseVLM at 90%; the VLABench trial denominator differs between main text and appendix..." [Q4] "...specify construction of the two DROID-derived tasks, and explain the default Goal 92.4 in Table 4(a) versus 94.4 in Table 2."

We clarified both setups and fixed the two errors. We agree that the 3-point margin over SparseVLM on DROID is not significant on its own (Fisher p = 0.61), and Sec. 5.2 now says so. However, the strongest baseline changes across benchmarks, while ProtoP stays at or near the top on every one (Table R7).

Table R7. Success rate (%) of single-stage pruners across benchmarks (main-paper Tables 2 and 3).

SettingFastVSparseVLMVLA-PrunerProtoP
LIBERO 90% (OpenVLA-OFT)66.071.382.293.3
LIBERO 90% (π0.5)35.161.072.886.0
VLABench 75%33.027.039.039.0
VLABench 90%2.019.019.028.0
DROID 80%15.564.052.062.5
DROID 90%0.038.521.541.5
Largest gap to the best−50.9−25.0−20.0−1.5
  • SparseVLM is the strongest baseline on DROID, but it trails ProtoP by 22–25 points on LIBERO.

  • VLA-Pruner is the strongest baseline on LIBERO and VLABench, but at 90% on DROID it reaches about half of ProtoP's success (21.5 vs 41.5%, p < 0.001).

  • ProtoP is the best method in five of the six settings and within 1.5 points of the best in the sixth. Each baseline fails on some benchmark; ProtoP does not.

  • VLABench: 50 trials per task (add_condiment, select_drink). "100" in the main text was an error and is fixed.

  • DROID: "put the cube in the bowl" and "put the can in the mug", 100 trials each.

  • Main-paper Table 4(a): the Goal entry at layer 3 was entered incorrectly. The correct value is 94.4, as in main-paper Table 2 (average 93.3); both are fixed.

AuthorsOpenReview ↗

Rebuttal by Authors #3

[Q4] "What paired-episode or repeated-run uncertainty supports the smaller gaps, and what are the trial-level outcomes for the one real-robot task?"

We do not claim the smaller gaps. The 92.4 vs 94.4 discrepancy was an entry error, not run-to-run variance (see above), and our claims rest only on gaps that are significant on paired episodes.

  • Repeatability. Re-running the ProtoP configuration of main-paper Table 2 reproduces all 800 episodes exactly. In the segmentation-enabled environment of Appendix G.7–G.8, reruns are identical on Object, differ by at most 1/200 on Goal, and differ by up to 11/200 on Long. We therefore do not interpret gaps of a few episodes on Long.
  • Paired tests. All methods run from the same initial states, and we report exact McNemar tests with discordant counts (Appendix G.1). Non-significant gaps are labeled as such, e.g., floor vs. exact budget (p = 0.27) and fixed grid vs. ProtoP (p = 0.096).
  • Small gaps we do not claim. The 3-point DROID margin over SparseVLM and the VLABench margins are not significant on their own; we argue only for consistency across benchmarks (Table R7). At 60–75% pruning, where the gaps in main-paper Table 2 are small, we describe ProtoP as matching the baselines.
  • Gaps we do claim. At 90% pruning the gaps are 11–13 points, and they are significant on paired episodes (e.g., VLA-Pruner 83.0 vs. ProtoP 94.8%, p < 0.001; Appendix Table S11).

Table R8. Real-robot successes out of 10 trials ("Pick up the doll"; Appendix C.1). The unpruned policy succeeds in 7/10.

Method60%75%90%
FastV5/104/102/10
SparseVLM2/102/102/10
VLA-Pruner5/107/103/10
ProtoP7/107/105/10

With 10 trials per configuration, these differences are not significant (e.g., 5/10 vs. 3/10 at 90%, Fisher p = 0.65). The paper therefore presents this study as a feasibility check: ProtoP runs on physical hardware under clutter, with a trend consistent with simulation.

[W2] "The perturbation study measures prototype stability, not closed-loop success under perturbation."

We added closed-loop success under perturbation. ProtoP is significantly more robust than VLA-Pruner, and as robust as the other coverage rule (fixed grid).

The experiments cover Object and Long. Gaussian noise or a 14%-area occluder was applied to both cameras at every step, with 400 episodes per method (Table R3; Appendix G.9).

Table R3. Closed-loop success under visual perturbations at 90% pruning (Appendix G.9). Δ is relative to ProtoP.

MethodSuccess (%)Δ
Unpruned81.5+17.0∗∗∗
Fixed grid65.5+1.0
ProtoP64.5(ref.)
VLA-Pruner51.0−13.5∗∗∗

Robustness follows coverage: both coverage rules beat the global selector by 13–15 points. A large occluder still hurts every method, including the unpruned policy. Appendix C.2 is also corrected: the occluder of Appendix Table S8 had a side of 14% of the image side (about 2% of the area).

[Q3] "For pi0.5, which module and denoising step provide action-query attention, where is pruning applied, and how are positions and KV states handled? Does the implementation use Equation (1)'s query mean or the appendix's union of modality-selected tokens? ... confirm identical compiled-denoising settings across methods."

For π0.5, scoring and pruning happen once per control step inside the prefix (Gemma backbone), before any denoising step. The action expert then denoises on the pruned prefix. Appendices A.1 and A.4 now describe this.

  • Queries. The π0.5 prefix contains no action tokens, so no denoising step is involved. Eq. 1 uses the language tokens as queries, at layer index 3 of the Gemma backbone.
  • Where pruning happens. After the scoring layer, pruned visual tokens are removed from the hidden states, and the remaining prefix layers run on the retained tokens only. All 10 denoising steps reuse this pruned prefix.
  • Positions and KV states. Retained tokens keep their original RoPE position IDs. The KV cache of the layers already computed is sliced to the retained positions, so the action expert attends only to retained tokens. Action tokens use position IDs offset by the original (unpruned) prefix length, as in training.
  • Score. It is the query mean of Eq. 1. "Union" in Appendix A.4 meant the union of query positions and is reworded.
  • Denoising settings. All pruning methods run through the same inference path with the same sampler and 10 denoising steps; only the token-selection rule differs.

Dear Reviewer 9oEq,

Thank you again for your review. As the author-reviewer discussion period is coming to a close, we wanted to check whether our response addressed your concerns.

Your budget counterexample was correct. The submitted rule kept more than 52 tokens in 8.7% of queries, and we replaced it with a rule that keeps exactly B tokens. We also added the matched-selector comparison you asked for to the main paper (new Sec. 5.4), removed the efficiency claim, added closed-loop success under perturbation, and corrected the VLABench denominator and the Table 4(a) entry.

If any part of your concern remains open, we would be glad to answer it while there is still time.

Best regards, The Authors

Reviewer_FPQxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProtoP, a training-free visual token pruning method for vision-language-action models. The method partitions attention maps into regions using local attention peaks and watershed segmentation, then allocates tokens across regions while retaining at least one token per detected region. The central motivation is that global attention-based pruning can concentrate tokens on a few hotspots and discard other task-relevant regions. Experiments on two VLA backbones show substantial improvements over the evaluated pruning baselines at aggressive compression ratios, with additional ablations and small-scale real-robot validation.

Critical strengths

Relevant practical problem. Reducing visual-token processing costs while preserving manipulation performance is an important problem for VLA deployment.

Simple and interpretable method. Region-based allocation provides a clear alternative to global token ranking and can be applied without additional training.

Promising results under aggressive pruning. At 90% pruning on LIBERO, ProtoP reports 93.3% success on OpenVLA-OFT and 86.0% on π₀.₅, compared with 82.2% and 72.8% for VLA-Pruner.

Critical weaknesses

W1. The causal evidence for spatial collapse as the dominant failure mechanism remains incomplete. Section 3 provides plausible qualitative examples and a performance comparison supporting the spatial-collapse hypothesis. However, region coverage is defined using the same watershed partition that motivates the proposed method. Consequently, guaranteeing coverage under this partition does not independently establish that the retained regions correspond to task-relevant entities, or that losing such coverage is the dominant cause of task failure.

W2. Coverage of attention-defined regions does not guarantee preservation of semantic or reasoning-relevant information. The algorithm guarantees representation of detected watershed regions, but the paper sometimes extends this to claims about preserving every semantically distinct region. This requires further justification: an object may contain multiple attention peaks, different objects may occupy the same basin, and weakly attended but relevant objects may be excluded by foreground filtering.

Questions for authors

Can you quantify task-relevant region or object coverage independently of the watershed partition and relate it to task success?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the review. We appreciate that you found the problem practically relevant, the region-based allocation simple, interpretable, and training-free, and the results under aggressive pruning promising. Your request to measure coverage independently of our own partition led to a new ground-truth analysis, and we narrowed the semantic-coverage claim accordingly. Changes in the revised PDF are in blue.

Notation: Table R# denotes a table in this rebuttal (numbered across all responses). "Main-paper Table #" and "Appendix Table S#" refer to the revised PDF. All runs below use OpenVLA-OFT on LIBERO at 90% pruning (52 tokens) from identical initial states. In tables, Δ is the difference in success rate (percentage points) from the reference row; ∗ p < 0.05, ∗∗ p < 0.01, ∗∗∗ p < 0.001 (paired exact McNemar test); unmarked differences are not significant.

[W1] "...region coverage is defined using the same watershed partition that motivates the proposed method. Consequently, guaranteeing coverage under this partition does not independently establish that the retained regions correspond to task-relevant entities, or that losing such coverage is the dominant cause of task failure." [Q] "Can you quantify task-relevant region or object coverage independently of the watershed partition and relate it to task success?"

Yes. Using ground-truth object masks, which are independent of the watershed partition, we show that losing object coverage causes task failure, and that coverage tracks success across selectors.

We used LIBERO's ground-truth instance masks of the task objects. An object counts as covered if at least one retained token lies on its mask, a definition that depends neither on the attention map nor on ProtoP. With it, we ran three analyses.

1. Causal intervention. We add or remove tokens on task objects while keeping the budget fixed at 52 (Object/Goal/Long, 200 paired episodes per suite). Each edit is paired with a control that edits tokens at random positions instead.

Table R6. Editing the selected token set using ground-truth object masks (Appendix Table S15). Δ is relative to the unmodified selector.

SelectionAvgΔ
ProtoP (unmodified)93.3(ref.)
− tokens on task objects (13.4 on average)60.2−33.1∗∗∗
− as many random other tokens (control)91.3−2.0
Global Top-k (unmodified)57.0(ref.)
+ ≥1 token per task object58.0+1.0
+ ≥1 random token (control)57.2+0.2
+ ≥4 tokens per task object52.7−4.3∗∗
+ random tokens (control)57.3+0.3
  • Removing coverage causes failure. Removing the tokens on task objects costs 33 points; removing as many other tokens costs 2. The difference from the control is also significant (p < 0.001). The simulator, not ProtoP, defines which tokens are removed.
  • Necessary but not sufficient. Forcing tokens onto the objects does not rescue global Top-k: the rest of the attended layout also matters.

2. Measurement (Appendix Table S14). Global Top-k, FastV, and VLA-Pruner put the most tokens on objects, yet they cover only 70–81% of task objects, and cover all of them in only 45–60% of observations. ProtoP covers 88%, and covers all of them in 77% of observations (p ≤ 0.013 vs the global selectors).

3. Association (Appendix Table S16; 7 selectors, coverage logged at every query). Episodes in which all task objects are covered in fewer than half of the queries succeed in 51.5% of cases, against 88–94% otherwise. Coverage is measured along the policy's own trajectory, so part of this association may be reverse causation. Over the first 10 queries only, the association holds across selectors but not within one selector, so we use it only as supporting evidence.

With partition-independent ground truth, these results confirm the diagnosis: spatial collapse is primarily a coverage failure. Robust selectors keep the wider attended layout while still following attention (Sec. 3, 5.4).

[W2] "...an object may contain multiple attention peaks, different objects may occupy the same basin, and weakly attended but relevant objects may be excluded by foreground filtering."

We agree that the guarantee needed to be stated precisely. It now covers detected attention regions only, and every claim about preserving "every semantically distinct region" is removed. We quantified each case on 156 task-object instances (Appendix G.6):

  • Multiple peaks: 65.4% of objects contain two or more peaks, and 80.1% span two or more regions. This adds tokens to the object rather than losing it.
  • Shared basin: only 5.0% of regions overlap more than one task object.
  • Excluded objects: 8.3% are removed by foreground filtering, and 14.1% contain no peak. The floor does not protect these, which is why ProtoP covers 88% rather than 100% of task objects. This is now listed as a limitation.

Dear Reviewer FPQx,

Thank you again for your review. As the author-reviewer discussion period is coming to a close, we wanted to check whether our response addressed your concerns.

Following your request, we measured coverage using LIBERO ground-truth instance masks, which are independent of our watershed partition. Removing the retained tokens that fall on task objects lowers success from 93.3% to 60.2%, while removing the same number of random tokens lowers it only to 91.3%. We also narrowed the coverage guarantee to detected attention regions and listed the unprotected cases as a limitation.

If any part of your concern remains open, we would be glad to answer it while there is still time.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.