- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SOAR, a training-free visual token pruning method guided by cross-modal subspace geometry. It reconstructs visual representations from text representations and combines normalized reconstruction residuals with attention relevance and residual-space diversity to select tokens. Experiments across three VLMs and eight benchmarks demonstrate the effectiveness of the method.
Critical strengths
- The paper provides a useful geometric perspective on visual token redundancy by examining how visual representations become increasingly reconstructable from text representations across layers.
- The proposed method is clearly described and easy to follow.
- The evaluation covers multiple models, token budgets, and benchmarks, with component ablations and measured efficiency results supporting the effectiveness of the method.
Critical weaknesses
- The incremental contribution of the proposed CMR scoring is modest in the reported ablation. Replacing the combined score with attention-only scoring reduces average retained performance from 96.59% to 96.07%, whereas removing diversity selection reduces it to 90.13%. Raw-space diversity also recovers most of the performance, reaching 95.70%. These results suggest that diversity selection accounts for a substantial part of the benefit, making the independent contribution of the proposed geometric guidance less compelling.
- The connection between reconstructability and safe pruning is not directly established. Evaluation on referring-expression grounding or segmentation tasks, such as RefCOCO, would help assess whether pruning preserves fine-grained spatial information.
- CMR is not strictly a measure of information outside the text subspace. Tikhonov regularization shrinks components within the text subspace, so the reconstruction residual generally contains both out-of-subspace components and unreconstructed in-subspace components. This limits the interpretation of CMR as purely text-independent visual information and leaves its connection to visual redundancy insufficiently justified.
- The evaluated models are relatively dated. Experiments on Qwen3-VL and Qwen3.5 would help assess the method’s effectiveness on more recent VLMs.
- Lack of discussion and experimental comparison with closely related methods such as PyramidDrop[1] and FlowCut[2].
[1] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction (CVPR 2025)
[2] FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models (NeurIPS 2025)
Questions for authors
See weakness
10 comments
Response to Reviewer 2GfA:W1(2/2)
3. Key conclusions
- Performance with uniform scoring (Uniform) — confirming the independent advantage of the CMR residual space (+0.78 percentage points): When the importance prior is removed entirely and only uniform scoring is used, CMR residual-space diversity reaches 95.09%, significantly outperforming raw-space diversity at 94.31% (a lead of 0.78 percentage points). This directly answers where the contribution of the "diversity step" comes from: diversity-based deduplication in the raw space alone yields limited gains; it is precisely the residual geometry, obtained after CMR removes the text-shared components, that enables the diversity step to accurately capture complementary visual features.
- Performance with attention only — demonstrating CMR's dual advantage in both space and scoring (+0.81 and +0.52 percentage points): When residual-space diversity is combined with attention alone, it outperforms "attention only + raw-space diversity" by a significant 0.81 percentage points, thanks to the deduplication advantage of the CMR residual space; further introducing CMR scalar scoring on top of this (Attention × CMR) improves overall performance by another 0.52 percentage points (with the gain mainly on MME). This shows that CMR not only strengthens diversity selection through the residual space, but its scalar score itself also provides a cross-modal importance prior that the attention mechanism cannot replace.
- Fully separating the contribution of CMR from that of the basic diversity step — correcting the underestimation bias (total contribution +1.33 percentage points): Since "residual-space diversity + attention only" in fact still retains CMR's role in reconstructing the space, fully separating the complete contribution of CMR from the basic diversity step requires using "attention only + raw-space diversity", which contains no CMR at all, as the baseline for comparison. Under this comparison, the true total improvement brought by CMR reaches 1.33 percentage points, made up exactly of two additive parts: CMR residual space empowering diversity (+0.81 percentage points) and CMR participating in the initial scoring (+0.52 percentage points) — far larger than the apparent gap (0.52 percentage points) shown in Table 4 of the main text when only the scoring factor is removed, fully confirming the dominant role of CMR in SOAR's performance through both space reconstruction and importance scoring.
Response to Reviewer 2GfA:W2(1/2)
W2 The connection between reconstructability and safe pruning is not directly established. Evaluation on referring-expression grounding or segmentation tasks, such as RefCOCO, would help assess whether pruning preserves fine-grained spatial information.
Mechanistically, under the causal attention mask, visual tokens serve only as key-value representations, read unidirectionally by post-image text queries through linear mappings:
- Low CMR (high text reconstructability): indicates that the visual token is highly aligned with the text subspace; its semantic information has been absorbed by the post-image text stream and can be linearly represented by the text subspace, making it cross-modally aligned redundancy that can be safely pruned.
- High CMR (low text reconstructability): the token retains independent geometric residual components that the text subspace cannot linearly explain, corresponding to local appearance details and object boundary features not yet encoded into the text.
Meanwhile, unlike token-merging strategies that mix local features, SOAR directly selects and retains original visual tokens, fully preserving the original spatial position encoding of each selected token (e.g., the 2D/mRoPE of Qwen2.5-VL). As long as attention relevance is used to lock onto the target region, and high CMR together with residual-space diversity is used to select and retain target edges and non-redundant anchors, fine-grained spatial coordinates are not lost.
To directly verify this link, as requested by the reviewer, we add a full evaluation on the three major referring expression comprehension (REC) benchmarks on Qwen2.5-VL-7B.
Experimental setup and evaluation protocol
- Model and benchmark setup: Based on Qwen2.5-VL-7B, we conduct full evaluation on 8 standard subsets: RefCOCO (val / testA / testB), RefCOCO+ (val / testA / testB), and RefCOCOg (val / test, long sentences with complex spatial relations). The metric is the standard bounding-box localization accuracy [email protected] (), and the Average column reports the average performance retention over the 8 subsets relative to the full-budget baseline (100% Upper Bound).
- Baselines and compression budgets: With 25% and 50% of visual tokens retained on average, we compare under the same protocol with the mainstream compression methods FastV, PyramidDrop, and VisionZip; SOAR directly uses the default hyperparameter configuration, without additional tuning for the grounding task.
Table: Performance comparison of Qwen2.5-VL-7B on the three major referring expression comprehension (REC) benchmarks ([email protected])
| Method | RefCOCO val | RefCOCO testA | RefCOCO testB | RefCOCO+ val | RefCOCO+ testA | RefCOCO+ testB | RefCOCOg val | RefCOCOg test | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| All visual tokens (100%) | |||||||||
| Qwen-2.5-VL-7B | 89.45 | 92.56 | 85.16 | 83.50 | 89.02 | 79.15 | 86.76 | 87.24 | 100.0% |
| Retain 50% visual tokens on average | |||||||||
| FastV | 73.85 | 73.38 | 74.21 | 66.75 | 68.88 | 62.65 | 71.06 | 71.86 | 81.2% |
| PyramidDrop | 77.52 | 80.82 | 72.07 | 70.27 | 75.48 | 63.33 | 74.86 | 75.65 | 85.1% |
| VisionZip | 80.92 | 84.55 | 76.91 | 74.02 | 79.52 | 67.89 | 78.12 | 79.34 | 89.7% |
| SOAR (Ours) | 87.04 | 90.92 | 82.18 | 80.31 | 85.83 | 72.05 | 84.57 | 84.86 | 96.3% |
| Retain 25% visual tokens on average | |||||||||
| FastV | 43.57 | 46.81 | 40.86 | 39.47 | 43.78 | 36.02 | 43.04 | 42.69 | 48.5% |
| PyramidDrop | 46.46 | 53.83 | 37.23 | 42.29 | 47.76 | 32.81 | 45.32 | 44.91 | 50.4% |
| VisionZip | 60.12 | 66.95 | 55.41 | 58.33 | 64.88 | 48.52 | 57.09 | 56.44 | 67.8% |
| SOAR (Ours) | 73.68 | 78.41 | 67.54 | 66.45 | 73.06 | 58.12 | 69.68 | 69.59 | 80.2% |
Response to Reviewer 2GfA:W1(1/2)
TO Reviewer 2GfA
We sincerely thank Reviewer 2GfA for the careful review, the clear summary, and the positive assessment of our work's geometric perspective on visual redundancy, its clear method presentation, and its comprehensive experiments and efficiency validation.
W1 The incremental contribution of the proposed CMR scoring is modest in the reported ablation. Replacing the combined score with attention-only scoring reduces average retained performance from 96.59% to 96.07%, whereas removing diversity selection reduces it to 90.13%. Raw-space diversity also recovers most of the performance, reaching 95.70%. These results suggest that diversity selection accounts for a substantial part of the benefit, making the independent contribution of the proposed geometric guidance less compelling
1. The dual role of CMR First, we would like to emphasize that the novelty of CMR is not limited to providing a new scalar importance score; it also constructs, via the cross-modal residual, the feature space for diversity selection. Within the SOAR framework, CMR simultaneously serves two parallel core functions:
The "attention-only" variant in Table 4 of the main text only removes the CMR factor from the initial score, while diversity-based deduplication is still performed in the CMR residual space. This variant therefore isolates only the scalar scoring role of CMR and underestimates the core contribution of the residual space.
2. Decoupling experiment: setup and comparison results
We also add the complete set of control-combination experiments to address the reviewer's concern about the contribution of cmr:
Main decoupling comparison table
Note: The three rows marked with * are newly added experiments; the other three rows are taken from Table 4 of the main text. ΔAvg. is the change relative to the full SOAR configuration (first row), in percentage points.