acceptodds
Under review as a conference paper at ICLR 2027

SOAR: Visual Token Pruning via Cross-Modal Subspace Overlap Guidance in VLMs

Abstract

Abundant visual information strengthens vision-language model (VLM) perception, yet massive visual tokens raise inference costs. Existing visual token pruning methods have achieved significant progress by leveraging inter-token similarity or attention scores. However, such pruning guidance is largely restricted to local token-to-token similarity, ignoring the global geometric and structural relationships between the visual and textual subspaces. In this paper, we revisit VLM inference from a geometric perspective and present a novel guidance scheme for token compression. In particular, we identify a key observation: as LLM layers deepen, the subspaces spanned by the visual tokens and the textual tokens gradually align, and their overlap expands. To quantify this phenomenon, we propose Cross-Modal Subspace Overlap (CSO) to measure the extent to which visual tokens are spanned by text representations, revealing that more visual tokens in deeper layers can be linearly predicted by the text subspace. We accordingly propose Cross-Modal Residuals (CMR), which projects visual tokens onto the text subspace via Tikhonov-regularized least squares and exploits reconstruction residuals as an efficient geometric proxy to score tokens based on their unaligned features. Finally, based on CMR, we present SOAR (Subspace Overlap and Alignment Residuals), a training-free visual token pruning framework. Experiments on diverse VLM architectures verify the effectiveness of SOAR. For instance, on LLaVA-NeXT-7B, SOAR keeps only of visual tokens while preserving of the original average performance, achieving a prefill speedup, end-to-end speedup, and a KV-cache reduction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Token Compression, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2GfAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SOAR, a training-free visual token pruning method guided by cross-modal subspace geometry. It reconstructs visual representations from text representations and combines normalized reconstruction residuals with attention relevance and residual-space diversity to select tokens. Experiments across three VLMs and eight benchmarks demonstrate the effectiveness of the method.

Critical strengths
  1. The paper provides a useful geometric perspective on visual token redundancy by examining how visual representations become increasingly reconstructable from text representations across layers.
  2. The proposed method is clearly described and easy to follow.
  3. The evaluation covers multiple models, token budgets, and benchmarks, with component ablations and measured efficiency results supporting the effectiveness of the method.
Critical weaknesses
  1. The incremental contribution of the proposed CMR scoring is modest in the reported ablation. Replacing the combined score with attention-only scoring reduces average retained performance from 96.59% to 96.07%, whereas removing diversity selection reduces it to 90.13%. Raw-space diversity also recovers most of the performance, reaching 95.70%. These results suggest that diversity selection accounts for a substantial part of the benefit, making the independent contribution of the proposed geometric guidance less compelling.
  2. The connection between reconstructability and safe pruning is not directly established. Evaluation on referring-expression grounding or segmentation tasks, such as RefCOCO, would help assess whether pruning preserves fine-grained spatial information.
  3. CMR is not strictly a measure of information outside the text subspace. Tikhonov regularization shrinks components within the text subspace, so the reconstruction residual generally contains both out-of-subspace components and unreconstructed in-subspace components. This limits the interpretation of CMR as purely text-independent visual information and leaves its connection to visual redundancy insufficiently justified.
  4. The evaluated models are relatively dated. Experiments on Qwen3-VL and Qwen3.5 would help assess the method’s effectiveness on more recent VLMs.
  5. Lack of discussion and experimental comparison with closely related methods such as PyramidDrop[1] and FlowCut[2].

[1] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction (CVPR 2025)

[2] FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models (NeurIPS 2025)

Questions for authors

See weakness

10 comments
AuthorsOpenReview ↗

Response to Reviewer 2GfA:W1(1/2)

TO Reviewer 2GfA

We sincerely thank Reviewer 2GfA for the careful review, the clear summary, and the positive assessment of our work's geometric perspective on visual redundancy, its clear method presentation, and its comprehensive experiments and efficiency validation.

W1 The incremental contribution of the proposed CMR scoring is modest in the reported ablation. Replacing the combined score with attention-only scoring reduces average retained performance from 96.59% to 96.07%, whereas removing diversity selection reduces it to 90.13%. Raw-space diversity also recovers most of the performance, reaching 95.70%. These results suggest that diversity selection accounts for a substantial part of the benefit, making the independent contribution of the proposed geometric guidance less compelling

1. The dual role of CMR First, we would like to emphasize that the novelty of CMR is not limited to providing a new scalar importance score; it also constructs, via the cross-modal residual, the feature space for diversity selection. Within the SOAR framework, CMR simultaneously serves two parallel core functions:

  • Scoring: the initial guidance score is obtained by multiplicatively fusing the text-to-visual attention score with the CMR score.
  • Providing the space: the subsequent greedy diversity selection is performed in the CMR residual space, after the text-shared components are removed.

The "attention-only" variant in Table 4 of the main text only removes the CMR factor from the initial score, while diversity-based deduplication is still performed in the CMR residual space. This variant therefore isolates only the scalar scoring role of CMR and underestimates the core contribution of the residual space.

2. Decoupling experiment: setup and comparison results

We also add the complete set of control-combination experiments to address the reviewer's concern about the contribution of cmr:

  • Combinations: CMR residual-space diversity and raw-space diversity are each paired with uniform scoring (all visual tokens receive equal initial scores, so the retained set is determined entirely by diversity) and with attention-only scoring. The combinations of raw space with uniform or attention-only scoring contain no CMR at all.
  • Evaluation setting: Strictly aligned with the protocol of Table 4 in the main text, we evaluate SQA, TextVQA, POPE, and MME on LLaVA-1.5-7B retaining 64 visual tokens (11.1%) with the default hyperparameters (η,Htop)=(0.75,12)(\eta, H_{\text{top}}) = (0.75, 12). All combinations are re-run with the same code in the same round of experiments.

Main decoupling comparison table

Initial scoreDiversity spaceAvg. %FullΔAvg.
Attention × CMRCMR residual96.590.00
Attention onlyCMR residual96.07−0.52
Uniform*CMR residual95.09−1.50
Attention × CMRRaw space95.70−0.89
Attention only*Raw space95.26−1.33
Uniform*Raw space94.31−2.28

Note: The three rows marked with * are newly added experiments; the other three rows are taken from Table 4 of the main text. ΔAvg. is the change relative to the full SOAR configuration (first row), in percentage points.

AuthorsOpenReview ↗

Response to Reviewer 2GfA:W1(2/2)

3. Key conclusions

  • Performance with uniform scoring (Uniform) — confirming the independent advantage of the CMR residual space (+0.78 percentage points): When the importance prior is removed entirely and only uniform scoring is used, CMR residual-space diversity reaches 95.09%, significantly outperforming raw-space diversity at 94.31% (a lead of 0.78 percentage points). This directly answers where the contribution of the "diversity step" comes from: diversity-based deduplication in the raw space alone yields limited gains; it is precisely the residual geometry, obtained after CMR removes the text-shared components, that enables the diversity step to accurately capture complementary visual features.
  • Performance with attention only — demonstrating CMR's dual advantage in both space and scoring (+0.81 and +0.52 percentage points): When residual-space diversity is combined with attention alone, it outperforms "attention only + raw-space diversity" by a significant 0.81 percentage points, thanks to the deduplication advantage of the CMR residual space; further introducing CMR scalar scoring on top of this (Attention × CMR) improves overall performance by another 0.52 percentage points (with the gain mainly on MME). This shows that CMR not only strengthens diversity selection through the residual space, but its scalar score itself also provides a cross-modal importance prior that the attention mechanism cannot replace.
  • Fully separating the contribution of CMR from that of the basic diversity step — correcting the underestimation bias (total contribution +1.33 percentage points): Since "residual-space diversity + attention only" in fact still retains CMR's role in reconstructing the space, fully separating the complete contribution of CMR from the basic diversity step requires using "attention only + raw-space diversity", which contains no CMR at all, as the baseline for comparison. Under this comparison, the true total improvement brought by CMR reaches 1.33 percentage points, made up exactly of two additive parts: CMR residual space empowering diversity (+0.81 percentage points) and CMR participating in the initial scoring (+0.52 percentage points) — far larger than the apparent gap (0.52 percentage points) shown in Table 4 of the main text when only the scoring factor is removed, fully confirming the dominant role of CMR in SOAR's performance through both space reconstruction and importance scoring.
AuthorsOpenReview ↗

Response to Reviewer 2GfA:W2(1/2)

W2 The connection between reconstructability and safe pruning is not directly established. Evaluation on referring-expression grounding or segmentation tasks, such as RefCOCO, would help assess whether pruning preserves fine-grained spatial information.

Mechanistically, under the causal attention mask, visual tokens serve only as key-value representations, read unidirectionally by post-image text queries through linear mappings:

  • Low CMR (high text reconstructability): indicates that the visual token is highly aligned with the text subspace; its semantic information has been absorbed by the post-image text stream and can be linearly represented by the text subspace, making it cross-modally aligned redundancy that can be safely pruned.
  • High CMR (low text reconstructability): the token retains independent geometric residual components that the text subspace cannot linearly explain, corresponding to local appearance details and object boundary features not yet encoded into the text.

Meanwhile, unlike token-merging strategies that mix local features, SOAR directly selects and retains original visual tokens, fully preserving the original spatial position encoding of each selected token (e.g., the 2D/mRoPE of Qwen2.5-VL). As long as attention relevance is used to lock onto the target region, and high CMR together with residual-space diversity is used to select and retain target edges and non-redundant anchors, fine-grained spatial coordinates are not lost.

To directly verify this link, as requested by the reviewer, we add a full evaluation on the three major referring expression comprehension (REC) benchmarks on Qwen2.5-VL-7B.

Experimental setup and evaluation protocol

  • Model and benchmark setup: Based on Qwen2.5-VL-7B, we conduct full evaluation on 8 standard subsets: RefCOCO (val / testA / testB), RefCOCO+ (val / testA / testB), and RefCOCOg (val / test, long sentences with complex spatial relations). The metric is the standard bounding-box localization accuracy [email protected] (IoU≥0.5\text{IoU} \ge 0.5), and the Average column reports the average performance retention over the 8 subsets relative to the full-budget baseline (100% Upper Bound).
  • Baselines and compression budgets: With 25% and 50% of visual tokens retained on average, we compare under the same protocol with the mainstream compression methods FastV, PyramidDrop, and VisionZip; SOAR directly uses the default hyperparameter configuration, without additional tuning for the grounding task.

Table: Performance comparison of Qwen2.5-VL-7B on the three major referring expression comprehension (REC) benchmarks ([email protected])

MethodRefCOCO valRefCOCO testARefCOCO testBRefCOCO+ valRefCOCO+ testARefCOCO+ testBRefCOCOg valRefCOCOg testAvg.
All visual tokens (100%)
Qwen-2.5-VL-7B89.4592.5685.1683.5089.0279.1586.7687.24100.0%
Retain 50% visual tokens on average
FastV73.8573.3874.2166.7568.8862.6571.0671.8681.2%
PyramidDrop77.5280.8272.0770.2775.4863.3374.8675.6585.1%
VisionZip80.9284.5576.9174.0279.5267.8978.1279.3489.7%
SOAR (Ours)87.0490.9282.1880.3185.8372.0584.5784.8696.3%
Retain 25% visual tokens on average
FastV43.5746.8140.8639.4743.7836.0243.0442.6948.5%
PyramidDrop46.4653.8337.2342.2947.7632.8145.3244.9150.4%
VisionZip60.1266.9555.4158.3364.8848.5257.0956.4467.8%
SOAR (Ours)73.6878.4167.5466.4573.0658.1269.6869.5980.2%
Reviewer_Bgb8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SOAR is a training-free visual-token pruning method built on a geometric observation: a layer-level Cross-Modal Subspace Overlap (CSO, Eq 1) between visual tokens and the text subspace rises with depth in LLaVA-1.5-7B (Fig 2). Each visual token, centered by the mean text vector, is reconstructed from the text tokens by Tikhonov-regularized least squares with a spectrum-set λ (Eqs 2-7); the normalized residual CMR_i (Eq 8) marks content the text cannot explain. CMR is fused with a FlashAttention-compatible text-to-visual attention score as Score_i = a_i² CMR_i² (Eqs 9-10, Appendix C), and K tokens are kept by greedy diversity selection on residual directions (Eqs 11-14). Experiments cover LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B on eight image benchmarks (seven for Qwen), plus wall-clock and KV-cache measurements on POPE (Table 5).

Critical strengths
  1. The efficiency claims are measured end to end, and they reproduce. Table 5 reports wall-clock total and prefill time and per-sample KV cache on one RTX 4090 at batch 1, with all scoring and selection inside the timed region (L471-472). The abstract's end-to-end figure recomputes exactly: on LLaVA-NeXT, total time drops from 46:11 to 18:32, and 2771/1112 = 2.49x. Reporting real latency with the method's own overhead, not FLOPs alone, makes the central efficiency claim checkable, and I credit it.

  2. The geometric analysis is honest about its own scale. A final-layer POPE CSO of 0.209 looks small on a [0, 1] axis, and the paper confronts this: Appendix B derives a random-subspace floor and an attainable ceiling, and Table 6 places 0.209 at 29x the floor. Calibrating a new metric against explicit floor and ceiling values, instead of asserting that the effect is large, makes the size of the effect checkable.

  3. The extra machinery is priced, and the price is modest. CMR is closed form with only an N_t × N_t inverse (Eqs 4 and 7), and the attention signal comes from an auxiliary FlashAttention pass rather than a materialized attention matrix (Appendix C). Against Random pruning, which the paper uses as an ideal fast baseline, SOAR adds 19% total time on LLaVA-NeXT (Table 5). At the tested resolutions the scorer does not eat its own speedup, which makes SOAR usable as a drop-in.

Critical weaknesses
  1. The advertised phenomenon is documented prior work, and CMR is never isolated by the ablation. Contribution 1 (L87) claims a "novel representation phenomenon", yet IMA, uncited here, reports that "after each block the cross-modal alignment increases" (Shukor and Cord, NeurIPS 2024, arXiv 2405.16700). In Table 4 (11.1%), CMR-only scores 96.00, below attention-only at 96.07; by the table's one-factor design both rows keep residual-space diversity, which needs the CMR reconstruction (Eq 11), so no CMR-free run is reported. Reporting attention scoring with raw-space diversity, and reframing Contribution 1 as a new measurement of a known effect, would fix this.

  2. The headline accuracy lead is thin, and its exact size cannot be checked. The text says SOAR "surpasses SCOPE ... by 0.6 percentage points" at 64 tokens (L372-373), yet averaging per-benchmark ratios to Vanilla over the displayed cells of Table 1 gives 95.94 vs 95.49, a 0.45 gap. The same rule misses other printed Averages in both directions, and Tables 1-3 never state their averaging rule or how MME's raw score enters it. Stating the rule and its precision would settle the margin and costs no experiments.

  3. The flagship LLaVA-NeXT table drops SCOPE, the baseline the paper calls strongest (L373), and one baseline is uncited. SCOPE is absent from Table 2, although its own paper reports 97.1% at the same 320-token budget on its six-benchmark set (Deng et al., NeurIPS 2025, arXiv 2510.24214). SpecFlow appears in Tables 1-2 but not in the References. Adding SCOPE to Table 2, citing SpecFlow, stating whether baselines were re-run or copied, and comparing with CDPruner (NeurIPS 2025, arXiv 2506.10967), which the paper cites but never runs, needs no new method work.

Questions for authors
  1. Please report attention scoring with raw-space diversity and no CMR anywhere, at 11.1% on the four Table 4 benchmarks with the same hyperparameters. Does full SOAR still lead, and by how much?
  2. How is the Average computed, and how does MME's raw score enter it? Displayed-cell ratios to Vanilla average 95.94 (SOAR) and 95.49 (SCOPE) at 64 tokens; please reconcile with the printed 96.0 / 95.4 and the 0.6 claim.
  3. Were the Tables 1-3 baselines re-run or copied? SpecFlow self-reports 95.6% on LLaVA-1.5 at 88.9% pruning (arXiv 2607.10640) against 94.4 in Table 1. Which pruning layer(s), η, and H_top produced the main tables?
  4. Table 4's full SOAR (96.59) equals the best setting of the Appendix D sweep (Tables 7-8), run on the same four benchmarks, whose mean over settings is 96.21. Were the ablation rows tuned the same way?
  5. Since the top 30 visual PCs hold about 40-70% of the energy (L777), a text span sharing a few beats the random floor (L786). With another sample's text states, what are final-layer POPE CSO (0.209) and Table 4's 96.59?
12 comments
AuthorsOpenReview ↗

Response to Reviewer Bgb8:Q1(1/2)

TO Reviewer Bgb8

We sincerely thank Reviewer Bgb8 for the careful review, the clear summary, and the positive assessment of our work's measured end-to-end efficiency, its geometric calibration against lower and upper bounds, and its low-overhead plug-and-play nature.

Q1.Please report attention scoring with raw-space diversity and no CMR anywhere, at 11.1% on the four Table 4 benchmarks with the same hyperparameters. Does full SOAR still lead, and by how much?

1. The dual role of CMR First, we would like to emphasize that the novelty of CMR is not limited to providing a new scalar importance score; it also constructs, via the cross-modal residual, the feature space for diversity selection. Within the SOAR framework, CMR simultaneously serves two parallel core functions:

  • Scoring: the initial guidance score is obtained by multiplicatively fusing the text-to-visual attention score with the CMR score.
  • Providing the space: the subsequent greedy diversity selection is performed in the CMR residual space, after the text-shared components are removed.

The "attention-only" variant in Table 4 of the main text only removes the CMR factor from the initial score, while diversity-based deduplication is still performed in the CMR residual space. This variant therefore isolates only the scalar scoring role of CMR and underestimates the core contribution of the residual space.

2. Decoupling experiment: setup and comparison results

Following the reviewer's suggestion, we fully decouple the diversity metric space from the initial score and add the complete set of control combinations:

  • Combinations: CMR residual-space diversity and raw-space diversity are each paired with uniform scoring (all visual tokens receive equal initial scores, so the retained set is determined entirely by diversity) and with attention-only scoring. The combinations of raw space with uniform or attention-only scoring contain no CMR at all.
  • Evaluation setting: Strictly aligned with the protocol of Table 4 in the main text, we evaluate SQA, TextVQA, POPE, and MME on LLaVA-1.5-7B retaining 64 visual tokens (11.1%) with the default hyperparameters (η,Htop)=(0.75,12)(\eta, H_{\text{top}}) = (0.75, 12). All combinations are re-run with the same code in the same round of experiments.

Main decoupling comparison table

Initial scoreDiversity spaceAvg. %FullΔAvg.
Attention × CMRCMR residual96.590.00
Attention onlyCMR residual96.07−0.52
Uniform*CMR residual95.09−1.50
Attention × CMRRaw space95.70−0.89
Attention only*Raw space95.26−1.33
Uniform*Raw space94.31−2.28

Note: The three rows marked with * are newly added experiments; the other three rows are taken from Table 4 of the main text. ΔAvg. is the change relative to the full SOAR configuration (first row), in percentage points.

AuthorsOpenReview ↗

Response to Reviewer Bgb8:Q1(2/2)

3. Key conclusions

  • Performance with uniform scoring (Uniform) — confirming the independent advantage of the CMR residual space (+0.78 percentage points): When the importance prior is removed entirely and only uniform scoring is used, CMR residual-space diversity reaches 95.09%, significantly outperforming raw-space diversity at 94.31% (a lead of 0.78 percentage points). This directly answers where the contribution of the "diversity step" comes from: diversity-based deduplication in the raw space alone yields limited gains; it is precisely the residual geometry, obtained after CMR removes the text-shared components, that enables the diversity step to accurately capture complementary visual features.
  • Performance with attention only — demonstrating CMR's dual advantage in both space and scoring (+0.81 and +0.52 percentage points): When residual-space diversity is combined with attention alone, it outperforms "attention only + raw-space diversity" by a significant 0.81 percentage points, thanks to the deduplication advantage of the CMR residual space; further introducing CMR scalar scoring on top of this (Attention × CMR) improves overall performance by another 0.52 percentage points (with the gain mainly on MME). This shows that CMR not only strengthens diversity selection through the residual space, but its scalar score itself also provides a cross-modal importance prior that the attention mechanism cannot replace.
  • Fully separating the contribution of CMR from that of the basic diversity step — correcting the underestimation bias (total contribution +1.33 percentage points): Since "residual-space diversity + attention only" in fact still retains CMR's role in reconstructing the space, fully separating the complete contribution of CMR from the basic diversity step requires using "attention only + raw-space diversity", which contains no CMR at all, as the baseline for comparison. Under this comparison, the true total improvement brought by CMR reaches 1.33 percentage points, made up exactly of two additive parts: CMR residual space empowering diversity (+0.81 percentage points) and CMR participating in the initial scoring (+0.52 percentage points) — far larger than the apparent gap (0.52 percentage points) shown in Table 4 of the main text when only the scoring factor is removed, fully confirming the dominant role of CMR in SOAR's performance through both space reconstruction and importance scoring.
AuthorsOpenReview ↗

Response to Reviewer Bgb8:Q2

Q2 How is the Average computed, and how does MME's raw score enter it? Displayed-cell ratios to Vanilla average 95.94 (SOAR) and 95.49 (SCOPE) at 64 tokens; please reconcile with the printed 96.0 / 95.4 and the 0.6 claim.

The Average column in Tables 1–3 of the main text is defined as the unweighted arithmetic mean of the ratios of each benchmark score to the score of the full model (Vanilla) in the same table, i.e.,

Average=1B∑b=1BsbsbVanilla×100%\text{Average} = \frac{1}{B} \sum_{b=1}^{B} \frac{s_b}{s_b^{\text{Vanilla}}} \times 100\%

where Tables 1 and 2 include 8 benchmarks and Table 3 includes 7 benchmarks. For MME, its raw total score is directly divided by the raw MME total score of Vanilla in the same table; after conversion to a relative ratio, it enters the average with equal weight alongside the other benchmarks.

We thank the reviewer for the careful verification; the two values pointed out in your question do indeed contain calculation and rounding errors. Computed directly from the cells displayed in Table 1, when retaining 64 visual tokens the averages of SOAR and SCOPE should be 95.94% and 95.49%, respectively, a lead of 0.45 percentage points, rather than the 96.0% and 95.4% printed in the original table and the 0.6 percentage points stated in lines 372–373 of the main text. In the revised version, we will add the above formula to Section 5.1, correct the text in lines 372–373, and uniformly recompute the Average column of Tables 1–3 to two decimal places based on the values displayed in the tables. The revised results are as follows:

Revised Average column of Table 1 (LLaVA-1.5-7B)

MethodRetain 192Retain 128Retain 64
FastV89.05%83.20%73.55%
SparseVLM97.17%95.61%90.64%
DART98.27%97.07%92.81%
VisionZip97.78%96.37%93.07%
HoloV98.80%97.25%94.44%
SCOPE98.39%97.59%95.49%
SpecFlow98.44%97.24%94.43%
SOAR99.14%98.40%95.94%

Revised Avg column of Table 2 (LLaVA-NeXT-7B)

MethodRetain 320
FastV88.12%
SparseVLM90.27%
VisionZip93.71%
DART96.06%
HoloV96.23%
SpecFlow96.65%
SOAR97.49%

Revised Avg column of Table 3 (Qwen2.5-VL-7B)

MethodRetain 33.3%Retain 22.2%Retain 11.1%
FastV97.50%95.02%87.54%
HoloV98.47%97.26%94.37%
SOAR99.74%98.88%96.73%

The raw per-benchmark scores underlying the three tables above are identical to those in Tables 1–3 of the submission. After the revision, SOAR remains the best across all models and budgets, and the ranking of the methods is unchanged.

Reviewer_dYeGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies training-free visual token pruning in VLMs from a geometric angle. It first introduces a layer-level metric, Cross-Modal Subspace Overlap (CSO): the fraction of the energy of centered visual tokens that the text-token subspace can explain. On LLaVA-1.5-7B (TextVQA, POPE, MME), CSO increases monotonically with depth (Spearman ρ=0.99\rho=0.99), reaching about 29× a random-subspace baseline at the last layer. Building on this, SOAR scores each visual token by its Cross-Modal Residual (CMR): the normalized residual after a Tikhonov-regularized least-squares reconstruction from the text tokens, with λ\lambda set adaptively from the text Gram spectrum. CMR is multiplied with a head-filtered text-to-visual attention score, and tokens are then selected greedily with a residual-space diversity penalty (squared cosine between residuals).

On LLaVA-1.5-7B, SOAR retains 99.1%/98.4%/96.0% of full performance at 192/128/64 tokens, above SCOPE, HoloV, and VisionZip. It also leads at 320 tokens on LLaVA-NeXT-7B and at all three ratios on Qwen2.5-VL-7B, and achieves 2.49× end-to-end and 3.62× prefill speedup on LLaVA-NeXT. Ablations cover the scoring signals, fusion type, and diversity space; the appendix adds hyperparameter sweeps and an attention–CMR overlap analysis.

Critical strengths

S1. A new, cheap, and well-motivated pruning signal grounded in a calibrated observation. CSO is not just a plot: the paper compares it against the Nt/DN_t/D random-subspace baseline and shows near-random overlap at layer 0 rising to about 29× at the last layer, with ρ=0.99\rho=0.99 on three benchmarks. CMR turns this into a per-token score with a closed-form ridge solution whose λ\lambda is set from the text spectrum, so it adds little overhead and needs no training. Attention and CMR select nearly disjoint token sets (e.g., 2/115 overlap at layer 16), which explains why combining them helps.

S2. Consistent results across backbones and budgets, with real efficiency measurements. SOAR has the best average on LLaVA-1.5-7B at all three budgets (99.1/98.4/96.0%), on LLaVA-NeXT-7B at 320 tokens, and on Qwen2.5-VL-7B at all three ratios. The advantage grows under aggressive compression (+2.3 over HoloV at 11.1% on Qwen2.5-VL). Efficiency is reported as wall-clock time with scoring included: 2.49× end-to-end, 3.62× prefill, and KV cache down from 1156MB to 192MB on LLaVA-NeXT, close to random pruning in runtime while 6.6 F1 points higher.

Critical weaknesses

W1. The core signal contributes little beyond attention, and the full-model number is tuned on the evaluation benchmarks. In Table 4, CMR alone (96.00) and attention alone (96.07) are nearly identical, and their product adds about 0.5 points (96.59). Removing diversity selection costs 6.5 points, so most of the performance comes from greedy diversity selection, a family of methods already used by DivPrune and DPP-style pruning. Moreover, 96.59 is the best cell of the η×Htop\eta\times H_{top} grid in Appendix D, evaluated on the same four benchmarks; the grid mean is about 96.2, the same as additive fusion. The claim that CMR is a key driver of the gains therefore needs evidence on held-out data.

W2. Missing implementation details, narrow baselines, and an incomplete mechanistic account. The paper never states at which layer(s) pruning is applied, which is essential for both reproducibility and the speed–accuracy trade-off. On Qwen2.5-VL only FastV and HoloV are compared, LLaVA-NeXT has a single budget, and there are no larger models, video, or multiple seeds. It is not stated whether baseline numbers were re-run or copied. Efficiency is compared only to Random and SparseVLM. Finally, in LLaVA-style models visual tokens precede the text and cannot attend to it under causal masking, so "visual tokens spanned by the text subspace" likely reflects text tokens absorbing visual content. This alternative explanation also predicts that highly attended tokens have low residuals, matching the observed low overlap, and the paper should discuss it.

Questions for authors
  1. Pruning layer and schedule. At which layer(s) is SOAR applied on each backbone, and is pruning done once or progressively? How do accuracy and speed change with the pruning layer?
  2. Held-out ablation. Can you fix η\eta and HtopH_{top} using a split disjoint from the reported benchmarks and re-run Table 4 (attention-only, CMR-only, additive, multiplicative), ideally on all eight benchmarks and at 128/192 tokens as well?
  3. Diversity selection with other scores. How does residual-space diversity selection perform when paired with attention only, or with a uniform score? This separates the contribution of CMR from that of the diversity step.
  4. Baseline provenance and causal direction. Were the baseline numbers re-run under your setup or copied from papers? And how do you reconcile the convergence interpretation with causal masking, where visual tokens cannot attend to text?
10 comments
AuthorsOpenReview ↗

Response to Reviewer dYeG: Q1

TO Reviewer dYeG

We sincerely thank Reviewer dYeG for the careful review, the clear summary, the positive assessment of our work, and several highly constructive comments.

Q1. Pruning layers and pruning schedule.

SOAR adopts three-stage progressive pruning. Taking LLaVA-1.5-7B (32 Transformer layers in total, with layer indices 0–31) as an example, pruning is performed at layers 1, 10, and 15. The initial number of visual tokens is 576. In the 64-token setting, the layer-averaged token retention satisfies:

576+9×92+5×54+17×2232=64\frac{576 + 9 \times 92 + 5 \times 54 + 17 \times 22}{32} = 64

The other backbones (LLaVA-NeXT, Qwen2.5-VL) likewise follow the mainstream settings in this area (e.g., SparseVLM and PDrop) and complete three-stage progressive pruning within the first half of the network (see the appendix of the revised manuscript for details). On the one hand, image information flows into text tokens mainly in the shallow-to-middle layers, so completing pruning in the first half of the network reduces computational cost as early as possible while preserving the key cross-modal interactions; on the other hand, since visual redundancy gradually increases with network depth, we adopt progressive pruning so that the number of retained tokens decreases stage by stage with depth, avoiding the premature loss of useful visual information in shallow layers.

Regarding the impact on speed and accuracy, we conducted new, comprehensive experiments to address this question. Performance is taken as the mean percentage relative to the Vanilla baseline over four benchmarks (MME, TextVQA, POPE, and SQA), and the Pope runtime of the unpruned full-token baseline and of every ablation configuration is re-measured uniformly on the same RTX 4090. On the 32-layer LLaVA-1.5-7B, we fix the layer-averaged visual token budget at 64.

GroupPruning layer indicesTokens retained per stagePrefill timeAvg. performance retention
Default1, 10, 1592, 54, 225:0296.59%
Group A: different layers1, 12, 1882, 48, 204:5996.14%
Group A: different layers1, 10, 2083, 49, 205:0295.95%
Group B: single stage1474:0893.36%
Group B: two stages1, 1582, 194:3896.07%
Group C: steeper allocation1, 10, 15120, 44, 104:5995.88%
Group C: flatter allocation1, 10, 1572, 64, 305:0695.64%
Reference: full tokens—5769:55—

Overall conclusion: Except for the two extreme control settings of delaying the first pruning layer and one-shot single-stage pruning, all other combinations are within 1.11 percentage points of the default configuration in accuracy, indicating that SOAR's performance does not rely on carefully chosen layer indices or allocation slopes. The default configuration achieves a 1.97× prefill speedup over the re-measured full baseline, and is simply one configuration point that strikes a relatively good balance between speed and accuracy.

AuthorsOpenReview ↗

Response to Reviewer dYeG: Q2

Q2. Held-out ablation.

First, we would like to clarify that the hyperparameters in the paper were determined only once, on LLaVA-1.5-7B (i.e., Htop=12,η=0.75H_{\text{top}} = 12, \eta = 0.75), and were then reused unchanged on LLaVA-NeXT-7B and Qwen2.5-VL-7B. These two models differ from LLaVA-1.5-7B in base architecture, number of visual tokens, and per-benchmark score distributions, so this in itself amounts to a generalization check at the model level.

As suggested by the reviewer, we independently selected the hyperparameters (η,Htop)(\eta, H_{\text{top}}) on a held-out set strictly isolated from the reported evaluation benchmarks, and, under multiple visual token budgets, re-ran all scoring variants of Table 4 in the main text, pairing the original paper parameters with the new parameters item by item.

  • Held-out set construction: Based on LLaVA-1.5-7B, we sample 1,000 examples each from GQA, the TextVQA training set, and the ScienceQA training set. Since the GQA training set is included in the fine-tuning data, we instead use the val-balanced split, which is disjoint from the evaluation set.
  • Paired parameter control: Using the hyperparameter grid of Appendix D, we select the new parameters without touching the test sets, and fix them uniformly across the four scoring variants.
  • Multi-budget evaluation scope: Paired evaluation is completed on the 7 benchmarks excluding VQAv2 (consistent with the evaluation set of Table 3 in the main text) under three budgets.

2. Experimental results

ComponentsOriginal setting (η,Htop)=(0.75,12)(\eta, H_{\text{top}}) = (0.75, 12)Selected on held-out set (η,Htop)=(0.98,28)(\eta, H_{\text{top}}) = (0.98, 28)
Retain 192 Tokens (33.3%)
SOAR full99.14%99.06% (−0.08%)
Attention score only99.08%99.04% (−0.04%)
CMR score only99.03%98.99% (−0.04%)
(Attention + CMR) score99.05%99.01% (−0.04%)
Retain 128 Tokens (22.2%)
SOAR full98.42%98.40% (−0.02%)
Attention score only98.39%98.36% (−0.03%)
CMR score only98.33%98.29% (−0.04%)
(Attention + CMR) score98.27%98.23% (−0.04%)
Retain 64 Tokens (11.1%)
SOAR full95.96%95.82% (−0.14%)
Attention score only95.81%95.71% (−0.10%)
CMR score only95.69%95.58% (−0.11%)
(Attention + CMR) score95.60%95.49% (−0.11%)

*Note: The Average column is the average performance retention over GQA, MMB, MMB-CN, MME, POPE, SQA, and TextVQA relative to the Vanilla full-token baseline;

From the table above we conclude: the model is highly insensitive to the hyperparameters, and changing the configuration barely affects overall performance. All paired performance differences are within 0.15 percentage points, and the fluctuations under medium and high budgets are negligible. This is consistent with the flat sensitivity trend in Appendix D, demonstrating that the Tikhonov-regularized projection and attention-head selection are numerically very stable and require no fine-grained tuning on the test sets.

AuthorsOpenReview ↗

Response to Reviewer dYeG: Q3

Q3. Diversity selection with other scores.

1. The dual role of CMR First, we would like to emphasize that the novelty of CMR is not limited to providing a new scalar importance score; it also constructs, via the cross-modal residual, the feature space for diversity selection. Within the SOAR framework, CMR simultaneously serves two parallel core functions:

  • Scoring: the initial guidance score is obtained by multiplicatively fusing the text-to-visual attention score with the CMR score.
  • Providing the space: the subsequent greedy diversity selection is performed in the CMR residual space, after the text-shared components are removed.

The "attention-only" variant in Table 4 of the main text only removes the CMR factor from the initial score, while diversity-based deduplication is still performed in the CMR residual space. This variant therefore isolates only the scalar scoring role of CMR and underestimates the core contribution of the residual space.

2. Decoupling experiment: setup and comparison results

Following the reviewer's suggestion, we fully decouple the diversity metric space from the initial score and add the complete set of control combinations:

  • Combinations: CMR residual-space diversity and raw-space diversity are each paired with uniform scoring (all visual tokens receive equal initial scores, so the retained set is determined entirely by diversity) and with attention-only scoring. The combinations of raw space with uniform or attention-only scoring contain no CMR at all.
  • Evaluation setting: Strictly aligned with the protocol of Table 4 in the main text, we evaluate SQA, TextVQA, POPE, and MME on LLaVA-1.5-7B retaining 64 visual tokens (11.1%) with the default hyperparameters (η,Htop)=(0.75,12)(\eta, H_{\text{top}}) = (0.75, 12). All combinations are re-run with the same code in the same round of experiments.

Main decoupling comparison table

Initial scoreDiversity spaceAvg. %FullΔAvg.
Attention × CMRCMR residual96.590.00
Attention onlyCMR residual96.07−0.52
Uniform*CMR residual95.09−1.50
Attention × CMRRaw space95.70−0.89
Attention only*Raw space95.26−1.33
Uniform*Raw space94.31−2.28

Note: The three rows marked with * are newly added experiments; the other three rows are taken from Table 4 of the main text. ΔAvg. is the change relative to the full SOAR configuration (first row), in percentage points.

3. Key conclusions

  • Uniform scoring: the independent advantage of the CMR residual space (+0.78 pp): With the importance prior removed entirely (uniform scoring only), CMR residual-space diversity reaches 95.09%, significantly outperforming raw-space diversity at 94.31% (+0.78 pp). This directly answers where the diversity step's contribution comes from. Diversity-based deduplication in the raw space alone yields limited gains. The diversity step captures complementary visual features accurately because it operates in the residual geometry left after CMR removes the text-shared components.
  • Attention only: CMR's dual advantage in space and scoring (+0.81 and +0.52 pp): Combined with attention-only scoring, residual-space diversity outperforms "attention only + raw-space diversity" by 0.81 pp, owing to the deduplication advantage of the CMR residual space. Adding the CMR scalar score on top (Attention × CMR) improves overall performance by another 0.52 pp, mainly on MME. CMR therefore strengthens diversity selection through the residual space, and its scalar score also provides a cross-modal importance prior that attention cannot replace.
  • Fully separating CMR from the basic diversity step: correcting the underestimation (total +1.33 pp): "Residual-space diversity + attention only" still retains CMR's role in reconstructing the space. Fully isolating CMR's contribution therefore requires the CMR-free "attention only + raw-space diversity" as the baseline. Against this baseline, CMR's true total gain reaches 1.33 pp and splits exactly into two additive parts: the CMR residual space empowering diversity (+0.81 pp) and CMR in the initial scoring (+0.52 pp). This is far larger than the apparent 0.52 pp gap in Table 4 of the main text, where only the scoring factor is removed. It confirms that CMR plays the dominant role in SOAR's performance through both space reconstruction and importance scoring.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.