- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the roles of image Keys and Values in MLLM text-to-image attention. It identifies a Key–Value asymmetry, where Keys are largely dominated by a shared visual prior while Values contain stronger instance-aware information, especially in deeper layers. Based on this observation, the authors propose Image Key Steering, a training-free method that amplifies the instance-aware Key residual. Experiments across multiple MLLMs and benchmarks show consistent improvements, particularly for visual grounding tasks.
Critical strengths
The paper connects an interesting representation-level observation to a simple and effective intervention. The extensive ablations, including calibration-distribution and cross-image residual controls, make the empirical story reasonably convincing.
Critical weaknesses
The interpretation of the shared mean as an input-agnostic “routing prior” is somewhat strong given the relatively simple decomposition. Also, the model coverage is still limited to a small set of open MLLMs, so it is unclear how universal the observed asymmetry is for newer architectures.
Questions for authors
How consistently does the Key–Value asymmetry hold for newer MLLM architectures beyond the models evaluated here? Also, how well would a completely fixed configuration, including a single calibration set and amplification factor per model, transfer across all task categories?
3 comments
Thank you for your new results. I will maintain my positive score.
We sincerely thank the reviewer for the positive assessment and for the constructive suggestions, which helped us strengthen the paper.
We thank the reviewer for the positive assessment of our representation-level analysis, simple and effective intervention, and extensive ablations. We appreciate the constructive comments on the interpretation of the shared mean, generalization to newer architectures, and transfer under a fully fixed configuration, which we address below.
Revisions addressing your comments are highlighted in purple in the revised manuscript.
W1. Interpretation of the Shared Mean as a Routing Prior
A1. We thank the reviewer for highlighting the distinction between statistical decomposition and functional interpretation. Our decomposition is inspired by prior mean-based analyses of Transformer representations (Song and Zhong; Doshi et al.), separating a shared mean from per-image deviations. The interpretation as a “routing prior” is supported by additional empirical evidence beyond this decomposition.
First, our cross-distribution analysis on LLaVA-1.5-7B shows that Key-prior directions remain highly similar when estimated from natural images, Gaussian noise, and solid-color images, with cosine similarity above 0.90 at every layer. This supports directional stability across the tested distributions. Second, our actual-Query analysis shows that the shared component remains dominant after projection onto real text Queries. The attention-distribution controls further distinguish its role: prior-only Keys better preserve the full-context attention distribution, whereas residual-only Keys better preserve the relative allocation among image tokens.
Accordingly, we interpret the visual prior operationally as an empirically stable shared Key component that serves as a routing scaffold, rather than as a claim of complete semantic disentanglement. The resulting attention remains input-dependent through both the text Queries and the instance-aware residuals. Likewise, “input-agnostic” refers to the stability of the Key-prior direction across the tested input distributions (Fig. 2), rather than to universal invariance.
W2 & Q1. Model Coverage and Key–Value Asymmetry on Newer Architectures
A2. Following your suggestion, we have extended our layer-wise analysis to InternVL3-8B using a token-pooled prior.
Keys have larger shared-component fractions than Values at all 28 decoder layers. Averaged across layers, the shared component accounts for approximately 65.3% of Key energy, while the residual accounts for 86.0% of Value energy. The following table summarizes four equal-depth intervals; entries are layer averages, and residual energies are reported per image token.
These results support both the overall composition asymmetry and contrasting middle-to-late residual trends in InternVL3-8B. From layers 7–13 to layers 21–27, mean Key residual energy decreases by 18.0%, whereas Value residual energy increases by approximately 9.6×. These interval-level trends coexist with layer-wise fluctuations.
This additional evidence extends our observations to a newer model family. We have added these results to Appendix S4.2 and clarified their empirical scope, distinguishing shared overall patterns from architecture-specific layer profiles.
Q2. Transfer of a Fully Fixed Configuration Across Task Categories
A3. We directly evaluated this setting on LLaVA-OneVision-7B using one fixed set of 200 unlabeled COCO images, disjoint from all evaluation images, together with the same α=1.5 and steering layers 20–27 across MME, MMVP, and POPE. These benchmarks represent general vision-language understanding, vision-centric reasoning, and visual hallucination, respectively. The calibration set and steering configuration were fixed before evaluation, without using performance on these benchmarks for selection.
Under this single configuration, improvements remain positive across all three task categories: +15.3 points on MME, +2.0 percentage points on MMVP, and +0.7 percentage points in POPE accuracy. This provides direct evidence that the method can transfer across task categories with the calibration set, amplification factor, and steering layers jointly fixed. We include this unified control and its configuration in Appendix S7.