acceptodds
Under review as a conference paper at ICLR 2027

What Does Full Attention Add? State-Complementary Visual KV Compression for Hybrid Vision-Language Models

Abstract

Vision-language models are increasingly adopting hybrid architectures that interleave full-attention layers with recurrent modules. Recurrent modules summarize historical information in fixed-capacity states, while full attention preserves direct access to historical visual tokens through explicit key–value (KV) caches. Existing visual KV compression methods typically assess entry importance through visual contributions to full attention, without explicitly accounting for information already represented in recurrent states. This motivates us to consider the additional information that full attention provides beyond recurrent-state representations when compressing visual KV entries in hybrid architectures. We propose Full-Attention Extra Semantic Information (FA-ESI), a training-free method for active visual KV compression that explicitly models this complementarity. For a target full-attention layer, FA-ESI replays the preceding Gated DeltaNet on the target layer's normalized inputs to construct an auxiliary recurrent state. It estimates full-attention extra information by comparing FA values with the corresponding state readouts. Current attention weights condition this information on the query. Projection onto the subspace of top-ranked prediction candidates followed by alignment scoring yields token-level FA-ESI scores for dynamic visual KV selection during decoding. Across benchmarks spanning multi-image understanding, long-video understanding, and GUI-agent tasks, FA-ESI achieves approximately 95% compression of active visual KV entries while retaining 98.984%–99.243% of Vanilla's performance on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Hybrid Architectures, Visual KV Compression, Recurrent State Models, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Yua4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses visual KV compression in hybrid-attention vision-language models (e.g., Qwen3.5, Kimi Linear, which interleave recurrent layers with full-attention layers). Its central observation is that existing methods assess entry importance only by a visual token's contribution to full attention, without subtracting information already carried by the recurrent state — so a token with large attention may add little new information.

Based on this, the authors propose FA-ESI (Full-Attention Extra Semantic Information), a training-free method for active visual KV compression. For a target full-attention layer, it replays the preceding GDN layer on that layer's normalized inputs to construct an auxiliary recurrent state; it then estimates "extra information" as the difference between full-attention values and state readouts, conditions this on the query via current attention weights, projects it onto the subspace spanned by the top-ranked prediction candidates, and computes alignment scores to obtain token-level scores that guide dynamic retention of visual KV entries during decoding.

Across 14 benchmarks (multi-image understanding, long-video understanding, multi-step reasoning, GUI-agent tasks), FA-ESI compresses ~95% of active visual KV entries while retaining 98.98%–99.24% of Vanilla performance, achieving best or tied-best results on 12 of 14 benchmarks. Decoding throughput improves 1.19–1.22× and end-to-end speedup is 1.11–1.18× at matched output lengths.

Critical strengths
  1. It turns "what full attention adds" into a quantity that is decomposable and transferable. Eq. (7) holds exactly, turning "extra information" from a metaphor into a verifiable component of the layer output. Any definition of "state readout" can be plugged into this framework and tested, and it transfers to Nemotron (Mamba-2). This is what determines my judgment: my concerns target attribution, not the framework — and because the framework is correct, revision is feasible. Without this decomposition, a paper that merely adds a GDN residual to SIVS would have no salvageable core.

  2. On GUI-agent tasks, ~95% compression costs almost nothing. On the 2B model, MMBench-GUI 74.627 vs Vanilla 74.572 and ScreenSpot-Pro 43.896 vs 44.529, while SIVS, LOOK-M, and VisPruner collapse on these same benchmarks (3.605, 0.190, etc.); decoding throughput is 1.19–1.22× on a consumer GPU. Even if I discount the budget-mismatched baselines and keep only the Vanilla column, this result alone supports acceptance — it does not depend on the comparisons I questioned.

Critical weaknesses
  1. The mechanism is not supported by the paper's own ablation. In Table 4, FA-EI — the residual ρ used alone — is indistinguishable from random selection (53.81 vs 53.38), and Q-FA-EI is worse than plain attention (73.02 vs 76.59); gains appear only after the SIVS-style prediction-conditioned projection is added. The decisive control is missing: SIVS's scoring applied directly to vFv^F under the same shared-across-layers selection and budget. On 2B, SIVS scores 96.32% @ 92.1% compression vs 99.24% @ 95.0% here — different operating points; on 4B the gap is 1.1 points with no variance. The advantage may therefore come from shared selection (78.07 vs 74.27), not from modeling complementarity.

  2. The efficiency claim is not established in the regime it targets. Building the state requires replaying the full multimodal prompt during prefill, per target FA layer (15 for 2B, 19 for 4B), yet neither replay cost nor TTFT is reported; speedups are only 1.19–1.22× / 1.11–1.18× and measured only on 2B/4B with a single RTX 3090, with no efficiency data above 9B.

Questions for authors

None

11 comments
AuthorsOpenReview ↗

Response to Reviewer Yua4 (Part 1/6)

W1.1

The mechanism is not supported by the paper's own ablation. In Table 4, FA-EI — the residual ρ used alone — is indistinguishable from random selection (53.81 vs 53.38), and Q-FA-EI is worse than plain attention (73.02 vs 76.59); gains appear only after the SIVS-style prediction-conditioned projection is added. The decisive control is missing: SIVS's scoring applied directly to vFv^{F} under the same shared-across-layers selection and budget.

Thank you for identifying this gap in the original ablations. We agree that the FA-EI and Q-FA-EI results alone do not sufficiently establish the role of recurrent-state subtraction within the complete scoring pipeline. The original Table 4 lacks the key control you requested: applying the same prediction-conditioned semantic scorer directly to FA values under the same budget and shared-across-layers selection strategy.

Table 4b of the revised manuscript reports this FA-only ablation. It removes only the subtraction of the GDN readouts, while keeping the target layer, shared-across-layers selection, retention budget, refresh interval, and number of prediction candidates identical to full FA-ESI.

Full FA-ESI outperforms FA-only in all 12 settings across six benchmarks and both model sizes, with improvements ranging from 0.811 to 5.718 score points. In particular:

  • TextVQA: The score increases from 74.226 to 78.070 for 2B and from 80.086 to 82.418 for 4B.
  • MMBench-GUI: The score increases from 68.909 to 74.627 for 2B and from 67.133 to 70.111 for 4B.
  • Improvements also hold on MMBench, MuirBench, ChartQA, and InfoVQA for both model sizes.

These results supply the evidence missing from the original ablations: with prediction-conditioned semantic scoring retained, subtracting the auxiliary GDN readouts performs better than using FA values directly. This is consistent with the weaker performance of FA-EI and Q-FA-EI: the contribution of subtraction should be evaluated within the complete scoring pipeline, rather than inferred solely from variants without semantic projection.

Table 4(b). GDN readout ablation on six benchmarks: full FA-ESI, FA-only, and norm-matched readouts (Norm.).

Benchmark    Model  FA-ESI  FA-only   Norm.
-----------  -----  ------  -------  ------
TextVQA         2B  78.070   74.226  77.738
TextVQA         4B  82.418   80.086  82.285
MMBench         2B  71.827   67.105  67.105
MMBench         4B  80.805   78.638  78.638
MMBench-GUI     2B  74.627   68.909  74.627
MMBench-GUI     4B  70.111   67.133  69.244
MuirBench       2B  41.769   40.064  41.646
MuirBench       4B  54.885   53.593  53.633
ChartQA         2B  77.560   76.749  77.519
ChartQA         4B  83.960   81.913  83.462
InfoVQA         2B  74.902   72.530  74.821
InfoVQA         4B  81.697   80.435  80.879
AuthorsOpenReview ↗

Response to Reviewer Yua4 (Part 2/6)

W1.2

On 2B, SIVS scores 96.32% @ 92.1% compression vs 99.24% @ 95.0% here — different operating points;

Thank you for pointing out the difference in operating points. We agree that the original Table 1 alone could not establish relative accuracy at a matched visual KV budget. On Qwen3.5-2B, the original SIVS configuration achieved 92.081% average compression, compared with 94.973% for FA-ESI.

We therefore evaluated SIVS with the same 5% target visual KV retention as FA-ESI on all 14 benchmarks for both model sizes. We use the definition of Rel. from Table 1: for each benchmark, we divide the method's score by the corresponding Vanilla score, then average these ratios across the 14 benchmarks.

  • Qwen3.5-2B. Measured average compression was 94.970% for SIVS (5%) and 94.973% for FA-ESI. FA-ESI scored higher on all 14 benchmarks and retained 99.243% of Vanilla performance on average, compared with 95.991% for SIVS (5%), a difference of 3.252 percentage points. On MMBench the scores were 71.827 and 67.260, respectively, and on ScreenSpot-Pro they were 43.896 and 40.544.
  • Qwen3.5-4B. Measured average compression was 94.966% for SIVS (5%) and 94.972% for FA-ESI. FA-ESI scored higher on 13 of 14 benchmarks and tied on LongVideoBench at 51.085. Mean relative performance was 98.984% for FA-ESI and 96.906% for SIVS (5%), a difference of 2.078 percentage points.

The revised Table 1 reports the original adaptive SIVS configuration alongside SIVS (5%), allowing readers to compare both the original algorithm at its reported operating point and the two methods at a matched target budget.

Our original comparison evaluated each method as a complete algorithm using the best-performing configuration reported in its paper, as described in Section 4.1. Adaptive selection is part of the SIVS algorithm. At each refresh, SIVS retains visual KV entries whose importance scores exceed τ=μs+ασs\tau=\mu_s+\alpha\sigma_s, a threshold determined by the mean and standard deviation of the current scores. The number of retained entries thus depends on the score distribution. We used the reported configuration and retained this selection rule to reproduce the original SIVS method as faithfully as possible.

Figure 2 of the original submission also examined accuracy across compression levels on TextVQA. For both 2B and 4B, the FA-ESI accuracy curve lies above the SIVS curve across the compression levels compared in the figure, including the 90%–98% range shown in the insets. Figure 2 covered TextVQA. The new SIVS (5%) column extends the matched-budget comparison to all 14 benchmarks.

Table 1 (selected columns). Accuracy at matched 5% visual KV retention, with adaptive SIVS for reference. Each model lists adaptive SIVS, SIVS (5%), and FA-ESI in that order. Rel. and Comp. are percentages.

Benchmark       2B SIVS  2B SIVS (5%)  2B FA-ESI    4B SIVS  4B SIVS (5%)  4B FA-ESI
--------------  -------  ------------  ---------    -------  ------------  ---------
MMBench          71.517        67.260     71.827     80.805        78.640     80.805
TextVQA          76.230        77.590     78.070     81.868        81.580     82.418
MileBench        51.788        51.525     52.936     47.517        47.419     47.907
MuirBench        40.808        40.769     41.769     53.654        54.654     54.885
MMRB             56.277        55.460     58.131     63.585        63.142     64.580
MIRBench         40.893        40.032     40.832     46.882        46.201     47.255
Video-MME        50.112        49.850     50.150     56.209        53.209     56.025
LongVideoBench   46.672        46.073     46.372     50.935        51.085     51.085
MMBench-Video      1.02          1.00       1.10       1.60          1.49       1.60
MVBench          54.947        55.263     55.895     61.158        61.260     61.474
MMBench-GUI      74.239        74.463     74.627     69.999        70.000     70.111
ScreenSpot-Pro   39.405        40.544     43.896     56.610        58.191     58.634
UI-Vision        24.001        24.585     26.410     31.046        30.101     31.283
AndroidCtrl.     40.750        41.437     42.008     45.797        46.886     48.352
Rel. (%)         96.318        95.991     99.243     97.853        96.906     98.984
Comp. (%)        92.081        94.970     94.973     95.073        94.966     94.972
AuthorsOpenReview ↗

Response to Reviewer Yua4 (Part 3/6)

W1.3

on 4B the gap is 1.1 points with no variance.

Thank you for pointing out that we originally reported only point estimates. The 1.1-point gap refers to mean relative performance across 14 benchmarks on Qwen3.5-4B: 98.984% for FA-ESI and 97.853% for the original adaptive SIVS, a difference of 1.131 percentage points. To examine the task-level gains underlying this average, we report paired 95% confidence intervals for each benchmark's score difference (FA-ESI minus original SIVS) on both model sizes. We matched predictions by question ID and resampled records sharing an image, video, or question group together. Both methods used the same sampled groups and weights in each replicate. We recomputed each benchmark's score using its original scoring and aggregation rules and took the 2.5th and 97.5th percentiles of 10,000 paired cluster bootstrap differences.

On 4B, the paired interval lies entirely above zero on 10 of 14 benchmarks. These gains span text understanding, multi-image reasoning, and GUI tasks, and vary in size:

  • TextVQA: +0.550 points (95% CI [0.132, 0.969]).
  • MMRB: +0.995 points (95% CI [0.214, 1.803]).
  • ScreenSpot-Pro: +2.024 points (95% CI [0.633, 3.479]).
  • AndroidControl: +2.556 points (95% CI [1.725, 3.395]), computed from the unrounded scores. Subtracting the separately rounded values in Table 1 gives 2.555 points.

On 2B, the paired interval lies entirely above zero on 12 of 14 benchmarks. Examples include TextVQA (+1.840 points, 95% CI [1.233, 2.437]), ScreenSpot-Pro (+4.491 points, 95% CI [2.530, 6.452]), and UI-Vision (+2.409 points, 95% CI [1.564, 3.290]). For the few intervals that include zero, we do not claim a clear advantage.

We also corrected a rounding and reporting error in the original Table 1. On 2B MMRB, the FA-ESI and SIVS scores were corrected from 58.130 and 56.280 to 58.131 and 56.277, respectively. The analysis uses these corrected scores, giving a difference of 1.854 points (95% CI [0.423, 3.305]). The revised Table 1 includes this correction, and Table 9 in Appendix B.2 reports all paired intervals for both models.

The average advantage on 4B is modest, about 1.1 percentage points, but the gains extend across most tasks. Ten of the 14 benchmarks individually support a positive score difference, and the observed gains on ScreenSpot-Pro and AndroidControl both exceed two points. On 2B, 12 benchmarks show the same direction of evidence. Thus, the task-level advantage over the original SIVS configuration is evident at both model sizes.

Table 9. Paired 95% confidence intervals for FA-ESI minus adaptive SIVS, in each benchmark’s score units.

Benchmark                           2B                      4B
--------------  ----------------------  ----------------------
MMBench               [+0.047, +0.586]          [0.000, 0.000]
TextVQA               [+1.233, +2.437]        [+0.132, +0.969]
MileBench             [+0.128, +2.790]        [+0.063, +0.742]
MuirBench             [+0.079, +1.859]        [-0.078, +2.435]
MMRB                  [+0.423, +3.305]        [+0.214, +1.803]
MIRBench              [-0.072, +0.098]        [+0.052, +0.711]
Video-MME             [+0.004, +0.073]        [-0.216, +0.253]
LongVideoBench        [-0.328, +0.127]        [+0.018, +0.291]
MMBench-Video   [+0.042586, +0.115524]  [-0.033433, +0.032146]
MVBench               [+0.163, +1.762]        [+0.074, +0.573]
MMBench-GUI           [+0.068, +0.727]        [+0.009, +0.221]
ScreenSpot-Pro        [+2.530, +6.452]        [+0.633, +3.479]
UI-Vision             [+1.564, +3.290]        [+0.036, +0.452]
AndroidCtrl.          [+0.382, +2.139]        [+1.725, +3.395]
Reviewer_xJ8cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FA-ESI, a training-free visual KV selection method for hybrid VLMs. It replays a GDN state to estimate what full attention adds beyond recurrent memory, then ranks visual tokens using a prediction-conditioned score. On 14 benchmarks with Qwen3.5-2B/4B, it retains 98.98%-99.24% of full-cache performance with about 5% of visual KV entries active, and reports 1.11-1.18x end-to-end speedups.

Critical strengths
  1. The hybrid-specific idea is sensible: keep visual KV entries for information the recurrent state may miss.
  2. The evaluation covers 14 benchmarks and multiple model sizes. Accuracy at a 5% active-KV budget is strong.
Critical weaknesses
  1. Eq. (5) directly subtracts FA values and auxiliary GDN readouts without establishing that their representations are aligned. This weakens the interpretation as "extra information."
  2. The cross-scale results are uneven: on Qwen3.5-35B-A3B, MMBench-GUI drops from 81.2 to 66.0 (Table 2). Full visual KV also remains stored for rescoring, so the 95% figure is not a memory saving.
Questions for authors
  1. How are the FA and GDN value spaces aligned in Eq. (5)? Please test normalized readouts or FA-only scoring on a GUI task.
  2. What causes the large 35B-A3B MMBench-GUI drop?
  3. Please report peak KV memory and clarify whether end-to-end timing includes the prefill replay.
10 comments
AuthorsOpenReview ↗

Response to Reviewer xJ8c (Part 1/5)

W1

Eq. (5) directly subtracts FA values and auxiliary GDN readouts without establishing that their representations are aligned. This weakens the interpretation as "extra information."

Thank you for raising the question of representation alignment in Eq. (5). We use the mapped GDN readout as an auxiliary recurrent reference and estimate the contribution of each FA value relative to that reference by taking their difference. Equation (4) specifies the correspondence between heads. For a given reference, Eq. (7) holds exactly with the attention weights, output gate, and output projection held fixed. Whether the GDN readout provides a useful reference for visual token selection is an empirical question.

We therefore added two controls in Table 4b. FA-only removes only the readout subtraction and applies the same prediction-conditioned semantic scorer directly to the FA values, with all other scoring and selection settings unchanged. The normalized-readout variant retains the subtraction and matches each mapped readout’s norm to that of its corresponding FA value before subtraction.

Across six benchmarks and two model sizes, full FA-ESI scored higher than FA-only in all 12 comparisons. It also scored higher than the normalized-readout variant in 11 comparisons and tied in one. We interpret “extra information” as the estimated contribution relative to this auxiliary recurrent reference. We present these findings as empirical support for subtracting the GDN readouts at their original magnitudes within the scoring pipeline rather than a theoretical proof of exact semantic alignment between the two representations. We give the head mapping and the requested GUI results in our response to Question 1 below.

Table 4(b). GDN readout ablation on six benchmarks: full FA-ESI, FA-only, and norm-matched readouts (Norm.).

Benchmark    Model  FA-ESI  FA-only   Norm.
-----------  -----  ------  -------  ------
TextVQA         2B  78.070   74.226  77.738
TextVQA         4B  82.418   80.086  82.285
MMBench         2B  71.827   67.105  67.105
MMBench         4B  80.805   78.638  78.638
MMBench-GUI     2B  74.627   68.909  74.627
MMBench-GUI     4B  70.111   67.133  69.244
MuirBench       2B  41.769   40.064  41.646
MuirBench       4B  54.885   53.593  53.633
ChartQA         2B  77.560   76.749  77.519
ChartQA         4B  83.960   81.913  83.462
InfoVQA         2B  74.902   72.530  74.821
InfoVQA         4B  81.697   80.435  80.879

W2.1

The cross-scale results are uneven: on Qwen3.5-35B-A3B, MMBench-GUI drops from 81.2 to 66.0 (Table 2).

Thank you for highlighting this result. We agree that averaging across tasks obscures the substantial drop on MMBench-GUI for Qwen3.5-35B-A3B. We first tested the effect of the target FA layer. Moving from layer 19, the configuration used in Table 2, to layer 23 raised the score from 66.042 to 73.655. This reduced the gap to the full-cache Vanilla baseline from 15.163 to 7.550 points, recovering about half of the original gap.

We also evaluated the same model on two additional GUI tasks using the original layer-19 configuration from Table 2. FA-ESI scored 1.330 points below Vanilla on UI-Vision Layout Grounding and 1.075 points below Vanilla on ScreenSpot-Pro. With the model and target layer held fixed, MMBench-GUI therefore showed the largest drop of the three GUI tasks we tested. Our response to Q2 gives the absolute scores and discusses the remaining gap.

Table 2 baselines and supplementary target-layer ablation on Qwen3.5-35B-A3B. Table 2 gives the Vanilla and layer-19 MMBench-GUI scores; layers 23, 27, and 31 are additional rebuttal results.

Configuration                Score
--------------------------  ------
Vanilla (full cache)        81.205
FA-ESI, layer 19 (Table 2)  66.042
FA-ESI, layer 23            73.655
FA-ESI, layer 27            71.252
FA-ESI, layer 31            68.805
AuthorsOpenReview ↗

Response to Reviewer xJ8c (Part 2/5)

W2.2

Full visual KV also remains stored for rescoring, so the 95% figure is not a memory saving.

Thank you for highlighting this distinction. FA-ESI retains the complete visual KV cache so that all visual tokens can be rescored and previously inactive tokens can be selected again at later refreshes. The approximately 95% compression ratio in Table 1 measures the reduction in the number of visual KV entries used by the main full-attention computation during decoding, relative to using all visual KV entries. This design preserves dynamic reselection while reducing the number of visual KV entries accessed at each decoding step.

The efficiency measurements show that reducing active visual KV yields end-to-end speedups even after accounting for reselection. On the same inputs and at matched output lengths, FA-ESI achieved 1.405–1.446× the decoding throughput of Vanilla and 1.206–1.237× end-to-end speedups on 2B. On 4B, the corresponding figures were 1.527–1.548× and 1.277–1.391×. End-to-end timing includes the full prefill, initial replay, and all subsequent state updates, scoring, selection, and KV updates (Figure 5 and Tables 10–11).

With approximately 5% of visual KV entries active, FA-ESI retained 99.243% and 98.984% of Vanilla performance on average across the 14 main benchmarks on 2B and 4B, respectively (Table 1).

We also measured the storage cost separately. Across three input lengths and two model sizes, peak attention KV memory was 8–32 MiB (0.78%–4.17%) higher with FA-ESI than with Vanilla (Table 6). We give the absolute peaks in our response to Question 3.

Table 6. Peak attention KV memory after generating 512 tokens (MiB). Increase is relative to Vanilla, with the percentage in parentheses.

Input tokens  Model  Vanilla  FA-ESI     Increase
------------  -----  -------  ------  -----------
15,736           2B      192     200   +8 (4.17%)
15,736           4B      512     528  +16 (3.13%)
62,854           2B      768     776   +8 (1.04%)
62,854           4B    2,048   2,064  +16 (0.78%)
125,678          2B    1,488   1,504  +16 (1.08%)
125,678          4B    3,968   4,000  +32 (0.81%)

Table 10. Decoding throughput on 2B and 4B (tokens/s). Speedup is FA-ESI relative to Vanilla.

Model  Tokens  Vanilla   SIVS   FA-ESI  Speedup
-----  ------  -------  -----  -------  -------
2B        512   84.676  5.391  122.467   1.446×
2B      1,024   90.840  5.448  127.669   1.405×
2B      2,048   93.925  5.225  132.867   1.415×
4B        512   45.094  7.447   69.798   1.548×
4B      1,024   47.092  7.505   71.906   1.527×
4B      2,048   47.372  7.586   72.894   1.539×

Table 11. End-to-end latency and refresh costs on 2B and 4B. Latencies are in seconds. E2E and Attn. are FA-ESI speedups over Vanilla; Score and Select are ms per refresh.

Model Tokens Vanilla     SIVS  FA-ESI    E2E Attn. Score Select
----- ------ ------- -------- ------- ------ ----- ----- ------
2B     8,192 115.616  912.082  95.858 1.206× 5.46× 1.641  0.128
2B    12,288 150.303 1279.970 121.508 1.237× 5.01× 1.642  0.127
2B    16,384 185.231 1698.629 151.076 1.226× 5.16× 1.641  0.128
4B     8,192 236.588 1183.797 178.166 1.328× 8.02× 3.174  0.127
4B    12,288 326.191 1810.265 234.553 1.391× 6.37× 3.198  0.127
4B    16,384 373.768 2259.644 292.771 1.277× 5.82× 3.187  0.128
AuthorsOpenReview ↗

Response to Reviewer xJ8c (Part 3/5)

Q1

How are the FA and GDN value spaces aligned in Eq. (5)? Please test normalized readouts or FA-only scoring on a GUI task.

Thank you for the suggestion. Equation (4) maps the GDN readouts to the FA head layout. In the Qwen3.5 models evaluated here, we concatenate each adjacent pair of 128-dimensional GDN head readouts into one 256-dimensional vector for the corresponding FA head. For each visual token and head, Eq. (5) then subtracts the mapped readout rr from the FA value vv, with no further projection or normalization.

We tested both controls you requested. FA-only scores vv directly, keeping the prediction-conditioned scorer, target layer, shared selection, retention budget, and refresh interval fixed. Normalized readout preserves the direction of rr and matches its norm to that of vv before subtraction. It scores v−r~v-\tilde r, where r~=r∥v∥2/max⁡(∥r∥2,10−8)\tilde r=r\lVert v\rVert_2/\max(\lVert r\rVert_2,10^{-8}).

On MMBench-GUI, the scores for full FA-ESI / FA-only / normalized readout were 74.627 / 68.909 / 74.627 on 2B and 70.111 / 67.133 / 69.244 on 4B. Subtracting the original readout improved on FA-only by 5.718 and 2.978 points, respectively. Norm matching tied the full method on 2B and scored lower on 4B.

Table 4(b) gives all six benchmarks at both model sizes. The other five results, in the same method order, are:

  • TextVQA. 2B: 78.070 / 74.226 / 77.738. 4B: 82.418 / 80.086 / 82.285.
  • MMBench. 2B: 71.827 / 67.105 / 67.105. 4B: 80.805 / 78.638 / 78.638.
  • MuirBench. 2B: 41.769 / 40.064 / 41.646. 4B: 54.885 / 53.593 / 53.633.
  • ChartQA. 2B: 77.560 / 76.749 / 77.519. 4B: 83.960 / 81.913 / 83.462.
  • InfoVQA. 2B: 74.902 / 72.530 / 74.821. 4B: 81.697 / 80.435 / 80.879.

Full FA-ESI outperformed FA-only in all 12 settings and the normalized-readout variant in 11, with one tie. Equation (4) establishes token-wise and head-wise correspondence. The ablations provide empirical support for the subtraction in Eq. (5): with the scorer and budget fixed, using the mapped GDN readout at its original magnitude as a recurrent reference improves visual KV selection.

Table 4(b). GDN readout ablation on six benchmarks: full FA-ESI, FA-only, and norm-matched readouts (Norm.).

Benchmark    Model  FA-ESI  FA-only   Norm.
-----------  -----  ------  -------  ------
TextVQA         2B  78.070   74.226  77.738
TextVQA         4B  82.418   80.086  82.285
MMBench         2B  71.827   67.105  67.105
MMBench         4B  80.805   78.638  78.638
MMBench-GUI     2B  74.627   68.909  74.627
MMBench-GUI     4B  70.111   67.133  69.244
MuirBench       2B  41.769   40.064  41.646
MuirBench       4B  54.885   53.593  53.633
ChartQA         2B  77.560   76.749  77.519
ChartQA         4B  83.960   81.913  83.462
InfoVQA         2B  74.902   72.530  74.821
InfoVQA         4B  81.697   80.435  80.879
Reviewer_6d7LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FA-ESI, a training-free method for active visual KV compression in hybrid vision-language models. It compares recurrent-state readouts with full-attention values and applies prediction-conditioned semantic scoring to select visual KV entries during decoding. Experiments across various tasks demonstrate its effectiveness on reducing visual KV entries while preserving performance.

Critical strengths
  1. Considering the complementarity between recurrent states and full-attention KV entries is interesting, and the motivation is clear.
  2. Experiments cover multiple model scales and diverse tasks. The results on GUI grounding are particularly informative because these tasks require preserving fine-grained visual evidence.
  3. Efficiency comparisons include measured decoding throughput and end-to-end latency with matched output lengths, which is promising for practical use.
Critical weaknesses
  1. It would be better to verify the contribution of recurrent-state subtraction. Table 4 compares several scoring variants, but does not report the same prediction-conditioned semantic scorer applied directly to full-attention values without subtracting GDN readouts.
  2. The practical accuracy–efficiency advantage over competing methods is insufficiently established. Figure 5 compares runtime only against Vanilla, whereas SIVS is missed. Besides, the performance improvements over SIVS are relatively minor.
  3. Preserving performance on several video benchmarks provides limited evidence. Table 5 shows that removing all visual KV entries leaves LongVideoBench and MVBench performance unchanged on both main models, while Video-MME also changes very little.
Questions for authors

Please refer to the Weaknesses

10 comments
AuthorsOpenReview ↗

Response to Reviewer 6d7L (Part 1/5)

W1

It would be better to verify the contribution of recurrent-state subtraction. Table 4 compares several scoring variants, but does not report the same prediction-conditioned semantic scorer applied directly to full-attention values without subtracting GDN readouts.

Thank you for pointing out this missing control. We have added the requested FA-only ablation, which removes only the subtraction of the GDN readouts and applies the same prediction-conditioned semantic scorer directly to the FA values. All other settings, including the target layer, shared-across-layers selection, retention budget, refresh interval, and number of prediction candidates, are identical to those of full FA-ESI. This comparison therefore isolates the contribution of recurrent-state subtraction within the complete scoring pipeline.

Full FA-ESI outperforms FA-only in all 12 settings across six benchmarks and both model sizes, with improvements ranging from 0.811 to 5.718 score points. Specifically:

  • TextVQA: The score increases from 74.226 to 78.070 for 2B and from 80.086 to 82.418 for 4B.
  • MMBench-GUI: The score increases from 68.909 to 74.627 for 2B and from 67.133 to 70.111 for 4B.
  • Improvements also hold on MMBench, MuirBench, ChartQA, and InfoVQA for both model sizes.

With the other factors held fixed, these results show that the gains cannot be attributed solely to prediction-conditioned semantic scoring: subtracting the auxiliary recurrent readouts provides an additional benefit. Table 4(b) reports this control across all six benchmarks and both model sizes.

Table 4(b). GDN readout ablation on six benchmarks: full FA-ESI, FA-only, and norm-matched readouts (Norm.).

Benchmark    Model  FA-ESI  FA-only   Norm.
-----------  -----  ------  -------  ------
TextVQA         2B  78.070   74.226  77.738
TextVQA         4B  82.418   80.086  82.285
MMBench         2B  71.827   67.105  67.105
MMBench         4B  80.805   78.638  78.638
MMBench-GUI     2B  74.627   68.909  74.627
MMBench-GUI     4B  70.111   67.133  69.244
MuirBench       2B  41.769   40.064  41.646
MuirBench       4B  54.885   53.593  53.633
ChartQA         2B  77.560   76.749  77.519
ChartQA         4B  83.960   81.913  83.462
InfoVQA         2B  74.902   72.530  74.821
InfoVQA         4B  81.697   80.435  80.879
AuthorsOpenReview ↗

Response to Reviewer 6d7L (Part 2/5)

W2.1

The practical accuracy–efficiency advantage over competing methods is insufficiently established. Figure 5 compares runtime only against Vanilla, whereas SIVS is missed.

We extended the original Figure 5 efficiency evaluation to include SIVS and three larger Qwen3.5 models (9B, 27B, and 35B-A3B) alongside 2B and 4B. Both SIVS and FA-ESI use 5% target visual KV retention and a refresh interval of 10 decoding steps. Within each model, methods use the same hardware and output length.

Across the evaluated settings, FA-ESI achieves 1.32–1.55× the decoding throughput of Vanilla and 1.21–1.39× end-to-end speedups over Vanilla. Its end-to-end speedups over SIVS range from 2.514× to 11.244× under the evaluated implementations and settings.

Figure 5 compares decoding throughput and end-to-end speedup across the five models. Table 11 gives the corresponding absolute latencies and retains the component measurements reported in the original Figure 5. On 9B at 8,192 output tokens, for example, attention is 11.938× faster than Vanilla attention. Each FA-ESI refresh takes 10.920 ms for scoring and 0.155 ms for visual KV selection. Including prefill, replay, and all decoding and refresh costs, the end-to-end speedup for this run is 1.214× over Vanilla.

SIVS's lack of an overall speedup is related to the additional computation required for periodic importance scoring. In the evaluated implementation, each scoring refresh in SIVS requires additional forward computations through both full and blind reference branches. Although sparse attention reduces the visual KV entries used by the main decoding path, repeatedly computing these reference branches introduces overhead that can offset or exceed the time saved by sparse attention.

FA-ESI reduces periodic scoring overhead by caching an auxiliary recurrent state and updating it incrementally. It constructs and caches the auxiliary GDN state needed for scoring during initialization. At each subsequent scoring refresh, it updates this state using only the tokens added since the previous state update, then scores visual tokens using the updated state, without rerunning the full/blind reference forwards. Consequently, even with the same retention ratio and refresh interval, the two methods incur different computational costs for dynamic selection.

All end-to-end measurements include prefill, initial replay, and the state-update, scoring, selection, and KV-update overhead during decoding. The measured speedups therefore reflect the overall benefit with all of these costs included.

Table 5 reports TTFT and replay cost.

Table 10. Decoding throughput on all five models (tokens/s). Speedup is FA-ESI relative to Vanilla.

Model    Tokens  Vanilla    SIVS   FA-ESI  Speedup
-------  ------  -------  ------  -------  -------
2B          512   84.676   5.391  122.467   1.446×
2B        1,024   90.840   5.448  127.669   1.405×
2B        2,048   93.925   5.225  132.867   1.415×
4B          512   45.094   7.447   69.798   1.548×
4B        1,024   47.092   7.505   71.906   1.527×
4B        2,048   47.372   7.586   72.894   1.539×
9B          512   21.933  11.342   29.007   1.323×
9B        1,024   22.116  11.265   29.195   1.320×
9B        2,048   22.153  11.263   29.236   1.320×
27B         512   19.154   9.894   25.259   1.319×
27B       1,024   19.517   9.910   25.744   1.319×
27B       2,048   19.659  10.055   26.024   1.324×
35B-A3B     512   55.869  29.140   83.934   1.502×
35B-A3B   1,024   58.102  29.370   88.375   1.521×
35B-A3B   2,048   59.262  29.840   90.714   1.531×
AuthorsOpenReview ↗

Response to Reviewer 6d7L (Part 3/5)

W2.1 (continued)

Table 11. End-to-end latency and refresh costs on all five models. Latencies are in seconds. E2E and Attn. are FA-ESI speedups over Vanilla; Score and Select are ms per refresh.

Model   Tokens Vanilla     SIVS  FA-ESI    E2E   Attn.  Score Select
------- ------ ------- -------- ------- ------ ------- ------ ------
2B       8,192 115.616  912.082  95.858 1.206×   5.46×  1.641  0.128
2B      12,288 150.303 1279.970 121.508 1.237×   5.01×  1.642  0.127
2B      16,384 185.231 1698.629 151.076 1.226×   5.16×  1.641  0.128
4B       8,192 236.588 1183.797 178.166 1.328×   8.02×  3.174  0.127
4B      12,288 326.191 1810.265 234.553 1.391×   6.37×  3.198  0.127
4B      16,384 373.768 2259.644 292.771 1.277×   5.82×  3.187  0.128
9B       8,192 495.695 1026.639 408.385 1.214× 11.938× 10.920  0.155
9B      12,288 681.028 1396.415 549.133 1.240× 11.154× 10.921  0.154
9B      16,384 869.453 1848.068 693.673 1.253× 10.376× 10.923  0.153
27B      8,192 548.403 1140.652 453.795 1.208×  7.818×  5.611  0.182
27B     12,288 757.666 1562.631 611.679 1.239×  7.145×  5.597  0.181
27B     16,384 966.281 2063.668 772.257 1.251×  6.512×  5.583  0.182
35B-A3B  8,192 229.454  493.440 184.018 1.247×  4.986×  2.774  0.173
35B-A3B 12,288 297.467  623.670 228.991 1.299×  4.585×  2.773  0.173
35B-A3B 16,384 365.623  791.670 274.013 1.334×  4.445×  2.774  0.172

Table 5. TTFT and replay at 1,024 output tokens. TTFT is in seconds; initial and periodic replay are in ms (periodic per refresh). Share is initial replay / FA-ESI TTFT.

Model    Vanilla TTFT  FA-ESI TTFT  Initial  Share (%)  Periodic
-------  ------------  -----------  -------  ---------  --------
2B            124.930      127.558  711.986      0.558     0.837
4B             86.271       80.657  250.838      0.311     0.850
9B            125.266      126.786  690.358      0.545     0.288
27B           132.819      139.919  532.793      0.381     0.258
35B-A3B        93.111       94.549  331.022      0.350     0.186

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of additional experiments and highlighted revision

We thank the reviewers for their feedback. We have uploaded a highlighted revision and posted point-by-point responses under each review. For the Area Chair and reviewers, we summarize the main additions below.

  • Scoring and selection. We added FA-only and norm-matched readout controls, along with a crossed ablation of shared versus per-layer selection and GDN readout subtraction. Holding shared selection fixed, subtraction improved scores in all six model–task settings tested (Table 4b–c).
  • Accuracy at a matched budget. We evaluated SIVS at the same 5% target visual KV retention as FA-ESI across all 14 benchmarks. FA-ESI scored higher on all 14 at 2B and on 13 at 4B, with one tie (Table 1). We also report per-benchmark paired bootstrap confidence intervals against the original adaptive SIVS configuration (Table 9).
  • Efficiency and storage. We compared Vanilla, SIVS, and FA-ESI across five models from 2B to 35B-A3B, including throughput, end-to-end latency, TTFT, replay, and refresh costs (Figure 5; Tables 5 and 10–12). We also measured peak attention KV memory on 2B and 4B: the shared active buffer added 8–32 MiB over Vanilla (Table 6).
  • Diagnostic experiments. We reran post-prefill visual KV removal on three video benchmarks (Tables 7–8). For the 35B-A3B drop on MMBench-GUI, changing the target layer from 19 to 23 raised the score from 66.042 to 73.655, with a remaining 7.550-point gap to Vanilla. With the original layer-19 configuration held fixed, FA-ESI scored 15.163 points below Vanilla on MMBench-GUI, compared with 1.330 points below Vanilla on UI-Vision Layout Grounding and 1.075 points below Vanilla on ScreenSpot-Pro. The full layer comparison is in our response to Reviewer xJ8c, Q2.
AuthorsOpenReview ↗

Final rebuttal summary: evidence and contributions — Submission 23192

Dear Area Chair and Reviewers,

Thank you for reviewing our paper. We have submitted point-by-point responses and a highlighted revision. Before the final discussion, we summarize how the new evidence addresses the main concerns and strengthens the case for acceptance.

FA-ESI addresses visual KV selection in hybrid architectures, where recurrent states and full-attention KV both retain visual information. It scores the contribution of visual KV relative to an auxiliary recurrent reference, together with prediction-conditioned semantic scoring, to guide dynamic selection. The reviews recognized this motivation and the GUI results, while requesting stronger evidence for the subtraction, matched-budget accuracy, and practical efficiency.

1. Controlled ablations establish the contribution of GDN readout subtraction within the complete scorer.

The new FA-only control retains the same prediction-conditioned scorer, target layer, shared selection, budget, and refresh interval, removing only GDN readout subtraction. Full FA-ESI outperforms FA-only in all 12 comparisons across six benchmarks and two model sizes, by 0.811–5.718 points. It also outperforms norm-matched readouts in 11 settings and ties in one. The crossed selection/subtraction ablation shows that, with shared selection fixed, subtraction improves scores in all six settings. Shared selection also contributes and avoids repeated scoring and selection across layers. These controls separately test the roles of subtraction and shared selection (Table 4b–c).

2. Complete matched-budget comparisons strengthen the accuracy evidence.

We added SIVS at the same 5% target visual KV retention across all 14 benchmarks on both models. FA-ESI scores higher on 14/14 benchmarks at 2B and 13/14 at 4B, with one tie. Mean relative performance is 99.243% versus 95.991% on 2B and 98.984% versus 96.906% on 4B (Table 1). Separately, we address uncertainty against the original adaptive SIVS configuration using 10,000 paired cluster bootstrap replicates. The per-benchmark 95% intervals for FA-ESI minus original SIVS lie entirely above zero on 12/14 benchmarks at 2B and 10/14 at 4B, supporting gains on most tasks (Table 9).

3. Measured efficiency includes dynamic-selection costs and extends to larger models.

We expanded the comparison to 2B, 4B, 9B, 27B, and 35B-A3B, including SIVS at the same target retention and refresh interval. Each within-model comparison uses matched hardware and output lengths. FA-ESI achieves 1.32–1.55× Vanilla's decoding throughput and 1.21–1.39× end-to-end speedups, and outperforms SIVS under the evaluated implementations and settings. End-to-end timing has always included prefill, initial replay, state updates, scoring, selection, and KV updates. We also report TTFT and component costs separately (Figure 5; Tables 5 and 10–12). Peak attention KV measurements quantify the storage cost of retaining the full cache for reselection: the shared active buffer adds 8–32 MiB on 2B/4B (Table 6).

4. Additional diagnostics clarify strengths and current limitations.

Reruns and sample-level inspection explain why three multiple-choice video benchmarks are relatively insensitive to visual KV removal after prefill. GUI grounding tasks requiring complete coordinates provide more discriminating evidence: removing all visual KV causes large drops, while FA-ESI retains near-full-cache scores with about 5% active visual KV (Appendix B.1).

For the 35B-A3B drop on MMBench-GUI, changing the target layer from 19 to 23 recovers 7.613 points, leaving a 7.550-point gap. Two additional GUI evaluations with the original layer-19 configuration fixed show drops of only 1.330 points on UI-Vision Layout Grounding and 1.075 on ScreenSpot-Pro. These results locate the task-dependent limitation more precisely. We retain the possible contribution of the MoE architecture as a hypothesis requiring further investigation (response to Reviewer xJ8c, Q2).

We believe the paper merits acceptance for its approach to visual KV scoring in hybrid memory architectures, supported by controlled ablations, matched-budget comparisons, and measured runtime benefits. The expanded evidence shows that approximately 95% reduction in active visual KV can preserve strong task performance while providing end-to-end acceleration with selection costs included. We respectfully ask that the final assessment consider this evidence and the contribution to efficient hybrid vision-language model inference.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.