acceptodds
Under review as a conference paper at ICLR 2027

Image Keys and Values are Asymmetric in MLLM Text-to-Image Attention

Abstract

Multimodal Large Language Models (MLLMs) integrate visual information via text-to-image (T2I) attention in the decoder. While recent studies identify various T2I attention phenomena, the underlying signals driving this attention remain underexplored. We investigate this through a structural analysis of image Key and Value representations, decomposing each into a visual prior and an instance-aware residual. This uncovers two Key–Value asymmetries: within each layer, image Keys are prior-dominated while Values are residual-dominated; across layers, the residual grows in Values but fades in deep-layer Keys. A cross-distribution analysis further shows the Key prior is input-agnostic, retaining high similarity under extreme distributional shifts. This reveals a routing–content gap: deep layers carry rich instance-aware content in Values, yet route attention through Keys dominated by an input-agnostic prior. Notably, within the same model family, the larger model exhibits a greater instance-aware component in its image Keys, providing complementary support for residual amplification. We therefore propose Image Key Steering, a training-free approach that amplifies the instance-aware component of deep-layer image Keys. Across four MLLMs and ten benchmarks, Image Key Steering consistently improves base-model performance, with the largest gains on tasks requiring precise visual grounding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Text-to-Image Attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Mcj1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the roles of image Keys and Values in MLLM text-to-image attention. It identifies a Key–Value asymmetry, where Keys are largely dominated by a shared visual prior while Values contain stronger instance-aware information, especially in deeper layers. Based on this observation, the authors propose Image Key Steering, a training-free method that amplifies the instance-aware Key residual. Experiments across multiple MLLMs and benchmarks show consistent improvements, particularly for visual grounding tasks.

Critical strengths

The paper connects an interesting representation-level observation to a simple and effective intervention. The extensive ablations, including calibration-distribution and cross-image residual controls, make the empirical story reasonably convincing.

Critical weaknesses

The interpretation of the shared mean as an input-agnostic “routing prior” is somewhat strong given the relatively simple decomposition. Also, the model coverage is still limited to a small set of open MLLMs, so it is unclear how universal the observed asymmetry is for newer architectures.

Questions for authors

How consistently does the Key–Value asymmetry hold for newer MLLM architectures beyond the models evaluated here? Also, how well would a completely fixed configuration, including a single calibration set and amplification factor per model, transfer across all task categories?

3 comments

We thank the reviewer for the positive assessment of our representation-level analysis, simple and effective intervention, and extensive ablations. We appreciate the constructive comments on the interpretation of the shared mean, generalization to newer architectures, and transfer under a fully fixed configuration, which we address below.

Revisions addressing your comments are highlighted in purple\color{#BF0040}{\text{purple}} in the revised manuscript.

W1. Interpretation of the Shared Mean as a Routing Prior

A1. We thank the reviewer for highlighting the distinction between statistical decomposition and functional interpretation. Our decomposition is inspired by prior mean-based analyses of Transformer representations (Song and Zhong; Doshi et al.), separating a shared mean from per-image deviations. The interpretation as a “routing prior” is supported by additional empirical evidence beyond this decomposition.

First, our cross-distribution analysis on LLaVA-1.5-7B shows that Key-prior directions remain highly similar when estimated from natural images, Gaussian noise, and solid-color images, with cosine similarity above 0.90 at every layer. This supports directional stability across the tested distributions. Second, our actual-Query analysis shows that the shared component remains dominant after projection onto real text Queries. The attention-distribution controls further distinguish its role: prior-only Keys better preserve the full-context attention distribution, whereas residual-only Keys better preserve the relative allocation among image tokens.

Accordingly, we interpret the visual prior operationally as an empirically stable shared Key component that serves as a routing scaffold, rather than as a claim of complete semantic disentanglement. The resulting attention remains input-dependent through both the text Queries and the instance-aware residuals. Likewise, “input-agnostic” refers to the stability of the Key-prior direction across the tested input distributions (Fig. 2), rather than to universal invariance.

W2 & Q1. Model Coverage and Key–Value Asymmetry on Newer Architectures

A2. Following your suggestion, we have extended our layer-wise analysis to InternVL3-8B using a token-pooled prior.

Keys have larger shared-component fractions than Values at all 28 decoder layers. Averaged across layers, the shared component accounts for approximately 65.3% of Key energy, while the residual accounts for 86.0% of Value energy. The following table summarizes four equal-depth intervals; entries are layer averages, and residual energies are reported per image token.

LayersKey prior (%)Value residual (%)Key residual energyValue residual energy
0–680.575.7628.237.6
7–1359.986.5656.6181.5
14–2058.189.0605.1238.9
21–2762.692.6538.61741.6

These results support both the overall composition asymmetry and contrasting middle-to-late residual trends in InternVL3-8B. From layers 7–13 to layers 21–27, mean Key residual energy decreases by 18.0%, whereas Value residual energy increases by approximately 9.6×. These interval-level trends coexist with layer-wise fluctuations.

This additional evidence extends our observations to a newer model family. We have added these results to Appendix S4.2 and clarified their empirical scope, distinguishing shared overall patterns from architecture-specific layer profiles.

Q2. Transfer of a Fully Fixed Configuration Across Task Categories

A3. We directly evaluated this setting on LLaVA-OneVision-7B using one fixed set of 200 unlabeled COCO images, disjoint from all evaluation images, together with the same α=1.5\alpha = 1.5 and steering layers 20–27 across MME, MMVP, and POPE. These benchmarks represent general vision-language understanding, vision-centric reasoning, and visual hallucination, respectively. The calibration set and steering configuration were fixed before evaluation, without using performance on these benchmarks for selection.

MethodMME ↑MMVP ↑POPE accuracy ↑POPE F1 ↑
LLaVA-OneVision-7B1580.032.084.083.5
+ Image Key Steering, fully fixed configuration1595.334.084.783.8

Under this single configuration, improvements remain positive across all three task categories: +15.3 points on MME, +2.0 percentage points on MMVP, and +0.7 percentage points in POPE accuracy. This provides direct evidence that the method can transfer across task categories with the calibration set, amplification factor, and steering layers jointly fixed. We include this unified control and its configuration in Appendix S7.

Thank you for your new results. I will maintain my positive score.

We sincerely thank the reviewer for the positive assessment and for the constructive suggestions, which helped us strengthen the paper.

Reviewer_DB9UOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the structure of image key value representations in the text to image attention.Using ANOVA style decomposition, the authors separate image representations into a shared visual prior and image depedent residual. This paper identifies two asymmetries, image keys contain a relatively large shared component compared with values, and the image depedent compoment of keysdescreases in deepter decoder layers while tar of values increases. Based on these observations, the authors propose image key steering, a training free intervention that amplifes the residual compoment of image keys in selected deep layers. Experiment across 4 MLLMs and 10 vision language benchmarks report improvements. fine grained perception and hallucination reduction. The paper also incudes ablations on intervention location, key vs value steering, calibration distributions.

Critical strengths

this representation level analysis has a direct practical application. the paper goes beyond analyzing attention weights and investigates structural differences between image keys and values. The observation that their image dependent components evolve differently across decoder layers provides a useful hypothesis about multimodal attention. The prompsed image key steering offers a training free way to test this hypothesis.

It has broad experimental evaluation. Authors evaluate multiple model families and scales across a diverse set of benchmakrs. Improvments on fine grianed visual tasks and hallucination metrics are good. The key vs value , layer selection, prior scaling, cross image residual controls provide meaningful support for the design choices and strengthen empirical contribution

Critical weaknesses

The main interpretation is that amplifying image specific key residuals improves routing to relevant visual information. However the transformation also changes key logit magnitures and potentially the balance between image and non image attention. The author talk about generic key scaling and cross image residual controls which are useful, but they do not fully isolate changes in attention sharpness, image attention mass and positional effects from improvements in semantic routing.

Although improments are reported across the evaluated models, several gains on the stronger models are relatively small. For example Qwen2VL-7B improves from 1684.6 to 1690.0 on MME and from 63.5 to 64.4 on CV-Bench2D. The paper provides limited paired uncertainty analysis for these deterministic benchmarks. Also the key prior dominance observed for LLAVA is less uniform in Qwen2VL, where many intermediate layers have larger residual fractions. This makes generality of the propsed interpretation less clear, even though the inervention is empirically useful.

Questions for authors

For Qwen2VL, the visual prior is estimated as pooled vector shared across token positions, whereas LLAVA uses position specific priors. How does this difference interact with positional encoding, particularly RoPE/mRoPE, during steering? Could authors clarify how much of the residual represents semantic image variation rather than positional or project related variation?

Could the authors provide paired confidence intervals or significance analyses for the smaller benchmark improvements, especially on Quen2VL-7B? Also does the held out amplication factor selection protocol reported in Table S1 extend to other evaluated models ?

4 comments

We sincerely thank the reviewer for the thoughtful and constructive assessment of our work. We appreciate the recognition of our representation-level analysis, its practical connection to Image Key Steering, and the breadth of our evaluation. Below, we address each concern in detail.

Revisions addressing your comments are highlighted in orange\color{#FF8000}{\text{orange}} in the revised manuscript.

W1. Disentangling Semantic Routing from Attention Sharpness, Mass, and Positional Effects

A1. This is a great question. Image Key Steering modifies image Keys before attention normalization, so changes in attention mass, sharpness, and token allocation are coupled consequences of the resulting logit changes. Our controls test whether the image-matched residual matters, without separately attributing performance gains to each downstream attention property.

A norm-matched cross-image residual leaves performance near baseline (MME: 1504.9; CV-Bench$^{2D}:56.8),whereasthecurrent−imageresidualimprovesbothto1520.6and61.4.GenericKeyscalingyieldsasmallerCV−Bench: 56.8), whereas the current-image residual improves both to 1520.6 and 61.4. Generic Key scaling yields a smaller CV-Bench^{2D}$ gain (58.3 vs. 61.4), although it performs better on MME. These results support the importance of residual–image correspondence beyond perturbation magnitude alone.

Our image-conditional analysis further factors out total image-attention mass when comparing within-image allocation: the residual-only distribution better preserves the original allocation than the prior-only distribution (JS: 0.071 vs. 0.259). This supports the residual’s role in token allocation, while allowing sharpness to contribute. For positional effects, LLaVA’s position-wise prior is preserved, and the cross-image control uses corresponding token positions. The fixed-resolution Qwen2-VL analysis additionally shows that the structural trends and steering gains persist after separating shared positional structure (A4).

W2. Small Gains and Paired Uncertainty Analysis

A2. Thank you for raising the question of effect size and uncertainty on stronger models. The Qwen2VL-7B gains should be viewed in the context of its strong baseline and comparable inference-time interventions. Under the same evaluation, Vissink improves MME by 3.9 points and CV-Bench$^{2D}$ by 0.1, whereas Image Key Steering improves them by 6.3 and 0.9 points, respectively. More broadly, our method improves 12 of the 13 reported Qwen2VL metrics, with gains spanning all ten benchmarks.

To summarize their magnitude across benchmarks with different scales, we macro-average the relative improvements within each task category, giving each benchmark equal weight. For hallucination, we use POPE accuracy and CHAIR_I, expressing the latter as a relative reduction. Image Key Steering yields average relative improvements of 0.69% on six general vision-language tasks, 3.30% on two vision-centric tasks, and 2.47% on two hallucination benchmarks, compared with 0.32%, 3.25%, and 1.59% for Vissink.

Our evaluations use deterministic decoding, which removes generation-sampling variability. To quantify uncertainty over evaluation examples, we have additionally reported paired bootstrap confidence intervals for the score differences, as detailed in A5.

W3. Generality of Key Prior Dominance on Qwen2-VL

A3. Thank you for prompting us to clarify the scope of the cross-model interpretation. Qwen2-VL exhibits a more mixed intermediate-layer profile than LLaVA, while retaining the two Key–Value contrasts that motivate steering. First, its Key prior fraction remains consistently higher than its Value prior fraction across layers. The Value residual fraction exceeds 74% at every layer and 93% in deeper layers, whereas Keys retain substantially larger shared components. Second, the Key residual energy peaks at layer 16 and declines from 1.99 to approximately 1.1 by layer 27. In contrast, the Value residual energy grows from 0.019 to 7.70 across the decoder, including an approximately nine-fold increase over the final seven layers.

The shared finding is therefore a relative Key–Value compositional asymmetry and divergent late-layer residual trends. Its generality does not require the Key prior to exceed the same fraction at every layer or in every architecture. For Qwen2-VL, steering targets layers 21–27, within the late-layer regime where Key residual energy declines while Value residual energy grows. Improvements across all ten Qwen2-VL benchmarks further demonstrate that the intervention remains effective under this different layer-wise profile. Accordingly, the revised abstract, introduction, and conclusion now describe image Keys as generally prior-dominated (highlighted in blue, in response to Reviewer J98o).

Q1. Interaction of the Pooled Prior with RoPE/mRoPE

A4. Thank you for the question. We have explicitly clarified the Key spaces used for decomposition and steering in the revised decomposition and implementation sections (section 3.1; section4; section S2; section S4). For LLaVA, both energy analysis and steering operate on pre-RoPE Key projections, using a position-specific prior; the modified Keys subsequently undergo native RoPE. For Qwen2-VL, both operate on post-mRoPE Keys. Its pooled prior is therefore estimated and applied in the position-encoded Key space, and is broadcast without a subsequent mRoPE rotation. Qwen2-VL’s energy analysis retains the KV-head dimension, while its steering implementation additionally averages the prior across KV heads.

Consequently, Qwen2-VL’s pooled residual can contain shared positional structure and should not be interpreted as purely semantic variation. We quantify this using 300 MME images resized to 448×448448 \times 448, yielding identical 256-token grids. Separating the post-mRoPE representations into global, shared positional, and image-dependent components shows that shared positional variation accounts for 14.3% of pooled Key residual energy across layers 21–27. After absorbing this variation into a position-wise prior, the Key instance-component energy still declines by 41% from its layer-14 peak to layer 27, whereas the Value instance-component energy grows by over two orders of magnitude across the decoder.

Both steering controls operate after mRoPE and apply the same KV-head averaging rule, differing only in whether the prior is pooled over token positions or estimated position-wise. Position-wise steering retains a comparable MME-P improvement: 1659.6 → 1670.3, versus 1659.6 → 1669.0 with the pooled prior. Thus, the structural trends and steering benefits persist after explicitly separating shared positional variation in this fixed-resolution setting. We use “instance-aware” operationally to denote image-dependent deviations from the shared prior; these can include encoder/projector effects and content–position interactions, without being exclusively semantic.

Q2. Paired Confidence Intervals and Significance Tests

A5. We have supplemented the Qwen2-VL-7B results with paired bootstrap confidence intervals in Appendix S2, focusing on the smaller improvements highlighted by the reviewer.

We used 10,000 bootstrap resamples, sampling images jointly for the base model and Image Key Steering while retaining all questions associated with each image. For every resample, we recomputed the official benchmark scores and their difference. This preserves the pairing between the two methods and the grouped-question structure of MME.

BenchmarkBaseImage Key SteeringDifference95% CI of difference
MME1684.61690.9+6.3[+2.8, +9.7]
CV-Bench$^{2D}$63.564.4+0.9[+0.2, +1.6]

Differences are computed as Image Key Steering minus Base. Both 95% confidence intervals exclude zero, supporting positive improvements on the two benchmarks despite their modest magnitude. These intervals quantify uncertainty over evaluation examples. Deterministic decoding removes generation-sampling variability, while the paired analysis assesses the stability of the observed improvements under resampling of the evaluation set.

Q3. Held-out α Selection on Other Models

A6. The category-level α-selection protocol applies to all evaluated models. Table S1 originally reported held-out results for LLaVA-1.5-7B. We have additionally evaluated Qwen2-VL-7B on the held-out 90% of MME and POPE, extending this check to a second model family and covering general understanding and hallucination.

We excluded the original 10% α-selection examples and retained the previously selected α, steering layers, and calibration protocol without further tuning.

ModelBenchmark / MetricSelected αBase (held-out 90%)Steering (held-out 90%)Difference
Qwen2-VL-7BMME ↑1.11682.51689.8+7.3
Qwen2-VL-7BPOPE Acc ↑1.187.888.7+0.9

Differences are computed as Steering minus Base. Improvements persist on both benchmarks after excluding the α-selection examples: +7.3 points on MME and +0.9 percentage points on POPE accuracy. These results complement Table S1 by showing that the gains also extend to examples excluded from α selection in Qwen2-VL-7B.

Thank you for the detailed reply and the experiments. The new confidence intervals, positional analysis, and held out evaluations address most of my concerns and provide stronger support for the proposed method. I also appreciate the clarification regarding the Key Value asymmetry in Qwen2-VL. While the role of attention sharpness and mass is still not fully separated, I think the additional analysis provides reasonable evidence for the approach. Overall, the rebuttal has strengthened my confidence in the paper, and I am comfortable maintaining my Weak Accept.

We sincerely thank the reviewer for carefully reading our response and for the encouraging assessment. We greatly appreciate your time and constructive feedback throughout the discussion.

Reviewer_J98oOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors decompose image keys into a calibration mean and a residual, then amplify the residual in selected deep layers at inference time. Different energy profiles for keys and values motivate this choice. Experiments on four MLLMs and ten benchmarks test whether the intervention transfers beyond the examples used for calibration.

Critical strengths

The comparisons in Tables 3 to 5 give a concrete reason to target the key residual in deep layers. They test keys against values, residual amplification against prior modification, and different layer ranges. I find this connection between the decomposition and the intervention convincing. The evaluation across four MLLMs and ten benchmarks also helps establish that the gains extend beyond a single model or task.

The calibration checks in Appendix S2 and Tables S1 and S5 provide supporting evidence: steering choices transfer beyond the selection examples, and the prior can be reused across evaluation image pools.

Critical weaknesses

My main concern is whether the decomposition supports the same explanation across architectures. Equation 6 subtracts a separate mean at each position for LLaVA; Equation 7 subtracts a single pooled mean for Qwen2-VL. With pooling, a positional pattern shared by every image can remain in the residual. Calling that residual instance-dependent therefore requires more care than in the position-wise case.

The gains support the intervention, but they do not resolve this difference in what the residual contains. Separating stable positional structure from variation across images would make the cross-model explanation more convincing. The Qwen key-prior fractions reported in Appendix S4 also qualify the claim of uniform prior dominance. My concern is with the interpretation of the gains, rather than their existence.

Questions for authors

How much of Qwen2-VL's pooled residual varies across images, and how much reflects stable differences between token positions? Comparing both decompositions in a fixed-resolution setting would help answer this without another benchmark suite. Please also explain how variable token counts enter the prior and residual energy calculations in Equations 8 and 9.

4 comments

Thank you for recognizing the link between our representation analysis and intervention, the broad evaluation, and the calibration-transfer evidence. We address residual composition, prior dominance, and energy calculations below.

Revisions are highlighted in blue\color{#0000FF}{\text{blue}} in the revised manuscript.

W1–W2 and Q1. Pooled vs. Position-wise Decomposition

A1. We thank the reviewer for highlighting the importance of a consistent explanation across architectures. The decomposition supports this consistency despite different prior estimators: Keys have larger prior fractions than Values, with an overall decline in Key residual energy and growth in Value residual energy toward deeper layers. Our additional InternVL3-8B analysis using a token-pooled prior (Appendix S4.2) supports both findings: mean Key prior and Value residual fractions are 65.3% and 86.0%; from layers 7–13 to 21–27, mean Key residual energy falls by 18.0%, while Value residual energy grows 9.6-fold.

LLaVA’s position-wise prior absorbs shared positional structure, whereas Qwen2-VL’s pooled prior leaves it in the residual. Pooling provides one prior across dynamic resolutions without requiring separate estimates for every token grid, as clarified in Section 3.1.

Following your suggestion, we analyze Qwen2-VL-7B on 300 MME images at 448×448448 \times 448 resolution (256 visual tokens) to examine whether positional structure affects these conclusions. We separate the pooled residual into shared positional structure and image-dependent variation as follows:

hi,t−μ=μt−μ⏟pt+hi,t−μt⏟ui,t,h_{i,t}-\mu=\underbrace{\mu_t-\mu}_{p_t}+\underbrace{h_{i,t}-\mu_t}_{u_{i,t}},

μt=1M∑i=1Mhi,t,μ=1N∑t=1Nμt.\mu_t=\frac{1}{M}\sum_{i=1}^{M}h_{i,t},\qquad \mu=\frac{1}{N}\sum_{t=1}^{N}\mu_t.

Here, p_t captures stable position-dependent structure, and u_i,t captures image-dependent variation at each position. Averaging the layer-wise energy fractions over steering layers 21–27, these components account for 14.3% and 85.7% of the pooled Key residual energy, respectively (6.8% and 93.2% for Values). Thus, the pooled residual predominantly captures image-dependent variation in this setting, although it is not exclusively image-dependent. Both asymmetries persist with the position-wise decomposition. Averaged over steering layers 21–27, the shared (global + positional) component accounts for 65.6% of Key energy and 12.0% of Value energy. The image-dependent Key energy peaks at layer 14 and declines by 41% by layer 27, while the corresponding Value energy grows by more than two orders of magnitude across the decoder.

We further compare both steering formulations at the same fixed resolution, using α=1.10\alpha=1.10 and L=21,…,27\mathcal{L}=\\{21,\ldots,27\\}:

MethodMME perception
Base1659.62
Pooled-prior steering1669.02
Position-wise-prior steering1670.34

As shown in the table above, the position-wise prior yields a similar numerical improvement. Together, these results quantify the residual’s composition beyond performance gains alone and show that the improvement persists with a position-wise prior in this setting. We have added this analysis to Appendix S4 and clarified the distinction between the estimators in Section 3.1.

We agree that the Qwen2-VL profile in Appendix S4 does not show uniform Key-prior dominance across all layers. The cross-model finding is that Key prior fractions are generally higher than Value prior fractions, while their magnitude varies across layers and architectures. We have accordingly revised the abstract, introduction, and conclusion to describe image Keys as generally prior-dominated, and the cross-model summary in §3 to emphasize this Key–Value contrast.

Q2. Variable Token Counts in Energy Calculations

A2. For Qwen2-VL, prior estimation first averages visual tokens within each image and then averages across images with equal weight. Energy calculation follows the same image-weighting convention. Specifically, let hi∈RNi×dh_i\in\mathbb{R}^{N_i\times d} denote the representation of image i, with the KV-head and channel dimensions flattened into d. The component energies are:

Eprior=∥μ∥22d,E_{\mathrm{prior}}=\frac{\lVert\mu\rVert_2^2}{d},

Eres=1∣Dcal∣∑i=1∣Dcal∣∥hi−1Niμ⊤∥F2Nid.E_{\mathrm{res}}=\frac{1}{|\mathcal{D}_{\text{cal}}|}\sum_{i=1}^{|\mathcal{D}_{\text{cal}}|}\frac{\lVert h_i-\mathbf{1}_{N_i}\mu^\top\rVert_F^2}{N_i d}.

Broadcasting the prior over N_i positions and applying the same normalization gives ∥μ∥22/d\lVert\mu\rVert_2^2/d. Thus, images with more visual tokens do not receive greater weight solely because of their resolution.

The reported Qwen2-VL fractions are computed after averaging the component energies:

ηprior=EpriorEprior+Eres,ηres=1−ηprior.\eta_{\mathrm{prior}} = \frac{E_{\mathrm{prior}}}{E_{\mathrm{prior}}+E_{\mathrm{res}}}, \qquad \eta_{\mathrm{res}}=1-\eta_{\mathrm{prior}}.

We have clarified the normalization convention next to Eqs. 8–9 and added the explicit Qwen2-VL calculation in Appendix S4.

AuthorsOpenReview ↗

Happy to Clarify Any Remaining Questions

Dear Reviewer,

Thank you again for your thoughtful review. In our fixed-resolution Qwen2-VL analysis, shared positional structure accounts for 14.3% of pooled Key residual energy across the steering layers, and image-dependent variation accounts for 85.7%. The Key–Value asymmetries persist under the position-wise decomposition, and both priors yield similar gains on MME perception.

We also clarified that Key prior fractions are generally higher than Value prior fractions, with variation across layers and architectures, and explained how the energy calculations weight images equally when visual token counts differ.

As the discussion period draws to a close, we would be grateful to hear whether these results and clarifications address your concerns. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Best regards,
The Authors

I appreciate the effort you put into the rebuttal and the new experiments. The fixed-resolution Qwen2-VL numbers settle my question on the pooled residual (85.7% image-dependent energy in the steering-layer key residuals at 448x448, with MME gains holding under a position-wise prior), and clarifying the equal per-image weighting resolves my normalization concern too. That said, I still have doubts about the interpretation at native resolution, where grid sizes vary and steering boosts shared positional structure together with image-specific variation. The fixed-grid setup makes your story plausible, but it still leaves open how much each factor actually drives the gains when grids vary. I will leave my rating at 2 (weak reject).

Dear Reviewer,

Thank you for your follow-up and for confirming that the fixed-resolution analysis clarifies the composition of the pooled residual and that equal per-image weighting addresses the normalization concern.

Three of the four models in our main evaluation (LLaVA-1.5-7B, LLaVA-1.5-13B, and VILA-13B) use position-wise priors that absorb shared positional structure. Image Key Steering improves their MME scores by 15.8, 26.7, and 26.0 points, respectively. For Qwen2-VL at fixed resolution, steering with the position-wise and pooled priors improves MME Perception by 10.7 and 9.4 points, respectively.

The remaining question is how much shared positional structure contributes to Qwen2-VL’s gains at native resolution. Qwen2-VL is the only dynamic-resolution model in our main evaluation. The results from the three fixed-resolution models and the controlled Qwen2-VL comparison support our central finding that amplifying image-dependent Key residuals can improve performance.

We would appreciate your taking these results into account in your final assessment.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion

Dear Area Chair and Reviewers,

Thank you for your time and constructive feedback on our paper. We especially thank Reviewer DB9U and Reviewer Mcj1 for their positive responses and for maintaining their positive ratings. Their feedback helped us strengthen this paper. As the discussion draws to a close, we would like to summarize the new evidence suggested by reviewers, the corresponding manuscript revisions, and the main contributions of our work.

  • We added a fixed-resolution Qwen2-VL-7B analysis following Reviewer J98o’s suggestion to separate shared positional structure from image-dependent variation. Both Key–Value asymmetries persisted, and position-wise steering yielded a gain comparable to pooled-prior steering on MME Perception. We also clarified how we normalize energy for images with different token counts and revised the manuscript to distinguish the shared Key–Value contrast from architecture-specific layer profiles.

  • We clarified what the attention controls establish about the image-matched residual in response to Reviewer DB9U’s questions and documented where steering operates relative to RoPE and mRoPE. We added paired bootstrap confidence intervals for Qwen2-VL-7B gains on MME and CV-Bench 2D, both of which exclude zero. The gains on MME and POPE also persisted after excluding the examples used to select the steering strength.

  • We extended the representation analysis to InternVL3-8B following Reviewer Mcj1’s suggestion and observed both Key–Value asymmetries. We clarified the evidence for interpreting the shared Key component as a routing prior. We also evaluated LLaVA-OneVision-7B with a single calibration set, steering strength, and layer range, and found improvements on MME, MMVP, and POPE.

We would like to highlight three contributions of our work.

  1. A representation-level account of the routing–content gap. Across the models studied, image Keys have larger shared fractions than image Values. In deeper layers, the image-dependent residual tends to weaken in Keys while growing in Values, revealing a gap between the image-specific content held in Values and the image-specific signals in Keys that help route attention to it.
  2. A training-free intervention derived from this finding. Image Key Steering amplifies the image-dependent residual in deep-layer Keys while preserving the shared prior. It requires a small unlabeled calibration set and adds no trainable parameters.
  3. Evaluation across models and tasks. We evaluated Image Key Steering on four models and ten benchmarks spanning general vision–language understanding, vision-centric reasoning, and hallucination. The new experiments show that gains can persist on examples held out from steering-strength selection and across task categories under a single configuration.

We appreciate the reviewers’ feedback, which helped us sharpen our claims and strengthen the connection between our representation analysis and the intervention. ​​We also welcome any last-minute questions from Reviewer J98o and would be glad to address them before the discussion period closes.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.