- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses visual KV compression in hybrid-attention vision-language models (e.g., Qwen3.5, Kimi Linear, which interleave recurrent layers with full-attention layers). Its central observation is that existing methods assess entry importance only by a visual token's contribution to full attention, without subtracting information already carried by the recurrent state — so a token with large attention may add little new information.
Based on this, the authors propose FA-ESI (Full-Attention Extra Semantic Information), a training-free method for active visual KV compression. For a target full-attention layer, it replays the preceding GDN layer on that layer's normalized inputs to construct an auxiliary recurrent state; it then estimates "extra information" as the difference between full-attention values and state readouts, conditions this on the query via current attention weights, projects it onto the subspace spanned by the top-ranked prediction candidates, and computes alignment scores to obtain token-level scores that guide dynamic retention of visual KV entries during decoding.
Across 14 benchmarks (multi-image understanding, long-video understanding, multi-step reasoning, GUI-agent tasks), FA-ESI compresses ~95% of active visual KV entries while retaining 98.98%–99.24% of Vanilla performance, achieving best or tied-best results on 12 of 14 benchmarks. Decoding throughput improves 1.19–1.22× and end-to-end speedup is 1.11–1.18× at matched output lengths.
Critical strengths
-
It turns "what full attention adds" into a quantity that is decomposable and transferable. Eq. (7) holds exactly, turning "extra information" from a metaphor into a verifiable component of the layer output. Any definition of "state readout" can be plugged into this framework and tested, and it transfers to Nemotron (Mamba-2). This is what determines my judgment: my concerns target attribution, not the framework — and because the framework is correct, revision is feasible. Without this decomposition, a paper that merely adds a GDN residual to SIVS would have no salvageable core.
-
On GUI-agent tasks, ~95% compression costs almost nothing. On the 2B model, MMBench-GUI 74.627 vs Vanilla 74.572 and ScreenSpot-Pro 43.896 vs 44.529, while SIVS, LOOK-M, and VisPruner collapse on these same benchmarks (3.605, 0.190, etc.); decoding throughput is 1.19–1.22× on a consumer GPU. Even if I discount the budget-mismatched baselines and keep only the Vanilla column, this result alone supports acceptance — it does not depend on the comparisons I questioned.
Critical weaknesses
-
The mechanism is not supported by the paper's own ablation. In Table 4, FA-EI — the residual ρ used alone — is indistinguishable from random selection (53.81 vs 53.38), and Q-FA-EI is worse than plain attention (73.02 vs 76.59); gains appear only after the SIVS-style prediction-conditioned projection is added. The decisive control is missing: SIVS's scoring applied directly to under the same shared-across-layers selection and budget. On 2B, SIVS scores 96.32% @ 92.1% compression vs 99.24% @ 95.0% here — different operating points; on 4B the gap is 1.1 points with no variance. The advantage may therefore come from shared selection (78.07 vs 74.27), not from modeling complementarity.
-
The efficiency claim is not established in the regime it targets. Building the state requires replaying the full multimodal prompt during prefill, per target FA layer (15 for 2B, 19 for 4B), yet neither replay cost nor TTFT is reported; speedups are only 1.19–1.22× / 1.11–1.18× and measured only on 2B/4B with a single RTX 3090, with no efficiency data above 9B.
Questions for authors
None
11 comments
Response to Reviewer Yua4 (Part 2/6)
W1.2
On 2B, SIVS scores 96.32% @ 92.1% compression vs 99.24% @ 95.0% here — different operating points;
Thank you for pointing out the difference in operating points. We agree that the original Table 1 alone could not establish relative accuracy at a matched visual KV budget. On Qwen3.5-2B, the original SIVS configuration achieved 92.081% average compression, compared with 94.973% for FA-ESI.
We therefore evaluated SIVS with the same 5% target visual KV retention as FA-ESI on all 14 benchmarks for both model sizes. We use the definition of Rel. from Table 1: for each benchmark, we divide the method's score by the corresponding Vanilla score, then average these ratios across the 14 benchmarks.
- Qwen3.5-2B. Measured average compression was 94.970% for SIVS (5%) and 94.973% for FA-ESI. FA-ESI scored higher on all 14 benchmarks and retained 99.243% of Vanilla performance on average, compared with 95.991% for SIVS (5%), a difference of 3.252 percentage points. On MMBench the scores were 71.827 and 67.260, respectively, and on ScreenSpot-Pro they were 43.896 and 40.544.
- Qwen3.5-4B. Measured average compression was 94.966% for SIVS (5%) and 94.972% for FA-ESI. FA-ESI scored higher on 13 of 14 benchmarks and tied on LongVideoBench at 51.085. Mean relative performance was 98.984% for FA-ESI and 96.906% for SIVS (5%), a difference of 2.078 percentage points.
The revised Table 1 reports the original adaptive SIVS configuration alongside SIVS (5%), allowing readers to compare both the original algorithm at its reported operating point and the two methods at a matched target budget.
Our original comparison evaluated each method as a complete algorithm using the best-performing configuration reported in its paper, as described in Section 4.1. Adaptive selection is part of the SIVS algorithm. At each refresh, SIVS retains visual KV entries whose importance scores exceed , a threshold determined by the mean and standard deviation of the current scores. The number of retained entries thus depends on the score distribution. We used the reported configuration and retained this selection rule to reproduce the original SIVS method as faithfully as possible.
Figure 2 of the original submission also examined accuracy across compression levels on TextVQA. For both 2B and 4B, the FA-ESI accuracy curve lies above the SIVS curve across the compression levels compared in the figure, including the 90%–98% range shown in the insets. Figure 2 covered TextVQA. The new SIVS (5%) column extends the matched-budget comparison to all 14 benchmarks.
Table 1 (selected columns). Accuracy at matched 5% visual KV retention, with adaptive SIVS for reference. Each model lists adaptive SIVS, SIVS (5%), and FA-ESI in that order. Rel. and Comp. are percentages.
Benchmark 2B SIVS 2B SIVS (5%) 2B FA-ESI 4B SIVS 4B SIVS (5%) 4B FA-ESI
-------------- ------- ------------ --------- ------- ------------ ---------
MMBench 71.517 67.260 71.827 80.805 78.640 80.805
TextVQA 76.230 77.590 78.070 81.868 81.580 82.418
MileBench 51.788 51.525 52.936 47.517 47.419 47.907
MuirBench 40.808 40.769 41.769 53.654 54.654 54.885
MMRB 56.277 55.460 58.131 63.585 63.142 64.580
MIRBench 40.893 40.032 40.832 46.882 46.201 47.255
Video-MME 50.112 49.850 50.150 56.209 53.209 56.025
LongVideoBench 46.672 46.073 46.372 50.935 51.085 51.085
MMBench-Video 1.02 1.00 1.10 1.60 1.49 1.60
MVBench 54.947 55.263 55.895 61.158 61.260 61.474
MMBench-GUI 74.239 74.463 74.627 69.999 70.000 70.111
ScreenSpot-Pro 39.405 40.544 43.896 56.610 58.191 58.634
UI-Vision 24.001 24.585 26.410 31.046 30.101 31.283
AndroidCtrl. 40.750 41.437 42.008 45.797 46.886 48.352
Rel. (%) 96.318 95.991 99.243 97.853 96.906 98.984
Comp. (%) 92.081 94.970 94.973 95.073 94.966 94.972
Response to Reviewer Yua4 (Part 3/6)
W1.3
on 4B the gap is 1.1 points with no variance.
Thank you for pointing out that we originally reported only point estimates. The 1.1-point gap refers to mean relative performance across 14 benchmarks on Qwen3.5-4B: 98.984% for FA-ESI and 97.853% for the original adaptive SIVS, a difference of 1.131 percentage points. To examine the task-level gains underlying this average, we report paired 95% confidence intervals for each benchmark's score difference (FA-ESI minus original SIVS) on both model sizes. We matched predictions by question ID and resampled records sharing an image, video, or question group together. Both methods used the same sampled groups and weights in each replicate. We recomputed each benchmark's score using its original scoring and aggregation rules and took the 2.5th and 97.5th percentiles of 10,000 paired cluster bootstrap differences.
On 4B, the paired interval lies entirely above zero on 10 of 14 benchmarks. These gains span text understanding, multi-image reasoning, and GUI tasks, and vary in size:
- TextVQA: +0.550 points (95% CI [0.132, 0.969]).
- MMRB: +0.995 points (95% CI [0.214, 1.803]).
- ScreenSpot-Pro: +2.024 points (95% CI [0.633, 3.479]).
- AndroidControl: +2.556 points (95% CI [1.725, 3.395]), computed from the unrounded scores. Subtracting the separately rounded values in Table 1 gives 2.555 points.
On 2B, the paired interval lies entirely above zero on 12 of 14 benchmarks. Examples include TextVQA (+1.840 points, 95% CI [1.233, 2.437]), ScreenSpot-Pro (+4.491 points, 95% CI [2.530, 6.452]), and UI-Vision (+2.409 points, 95% CI [1.564, 3.290]). For the few intervals that include zero, we do not claim a clear advantage.
We also corrected a rounding and reporting error in the original Table 1. On 2B MMRB, the FA-ESI and SIVS scores were corrected from 58.130 and 56.280 to 58.131 and 56.277, respectively. The analysis uses these corrected scores, giving a difference of 1.854 points (95% CI [0.423, 3.305]). The revised Table 1 includes this correction, and Table 9 in Appendix B.2 reports all paired intervals for both models.
The average advantage on 4B is modest, about 1.1 percentage points, but the gains extend across most tasks. Ten of the 14 benchmarks individually support a positive score difference, and the observed gains on ScreenSpot-Pro and AndroidControl both exceed two points. On 2B, 12 benchmarks show the same direction of evidence. Thus, the task-level advantage over the original SIVS configuration is evident at both model sizes.
Table 9. Paired 95% confidence intervals for FA-ESI minus adaptive SIVS, in each benchmark’s score units.
Benchmark 2B 4B
-------------- ---------------------- ----------------------
MMBench [+0.047, +0.586] [0.000, 0.000]
TextVQA [+1.233, +2.437] [+0.132, +0.969]
MileBench [+0.128, +2.790] [+0.063, +0.742]
MuirBench [+0.079, +1.859] [-0.078, +2.435]
MMRB [+0.423, +3.305] [+0.214, +1.803]
MIRBench [-0.072, +0.098] [+0.052, +0.711]
Video-MME [+0.004, +0.073] [-0.216, +0.253]
LongVideoBench [-0.328, +0.127] [+0.018, +0.291]
MMBench-Video [+0.042586, +0.115524] [-0.033433, +0.032146]
MVBench [+0.163, +1.762] [+0.074, +0.573]
MMBench-GUI [+0.068, +0.727] [+0.009, +0.221]
ScreenSpot-Pro [+2.530, +6.452] [+0.633, +3.479]
UI-Vision [+1.564, +3.290] [+0.036, +0.452]
AndroidCtrl. [+0.382, +2.139] [+1.725, +3.395]
Response to Reviewer Yua4 (Part 1/6)
W1.1
Thank you for identifying this gap in the original ablations. We agree that the FA-EI and Q-FA-EI results alone do not sufficiently establish the role of recurrent-state subtraction within the complete scoring pipeline. The original Table 4 lacks the key control you requested: applying the same prediction-conditioned semantic scorer directly to FA values under the same budget and shared-across-layers selection strategy.
Table 4b of the revised manuscript reports this FA-only ablation. It removes only the subtraction of the GDN readouts, while keeping the target layer, shared-across-layers selection, retention budget, refresh interval, and number of prediction candidates identical to full FA-ESI.
Full FA-ESI outperforms FA-only in all 12 settings across six benchmarks and both model sizes, with improvements ranging from 0.811 to 5.718 score points. In particular:
These results supply the evidence missing from the original ablations: with prediction-conditioned semantic scoring retained, subtracting the auxiliary GDN readouts performs better than using FA values directly. This is consistent with the weaker performance of FA-EI and Q-FA-EI: the contribution of subtraction should be evaluated within the complete scoring pipeline, rather than inferred solely from variants without semantic projection.
Table 4(b). GDN readout ablation on six benchmarks: full FA-ESI, FA-only, and norm-matched readouts (Norm.).