acceptodds
Under review as a conference paper at ICLR 2027

Perturbed Visual Keys and Values as a Contrastive Reference for Multi-Image Reasoning

Abstract

Multi-image reasoning requires multimodal large language models (MLLMs) to integrate visual evidence across images. However, how cross-image interaction shapes the visual representations used for generation remains unclear. We examine this interaction through visual Keys and Values, which govern how text queries attend to and aggregate visual information. We observe that cross-image interaction changes both components: disabling its induced change in either one lowers accuracy. Describing later-image Keys and Values as coefficients along directions defined by earlier images, we find that cross-image interaction makes more of these coefficients negative. Removing or reversing the negative ones progressively weakens support for the correct answer. This degradation serves as an effective reference branch for Key-Perturbed Contrastive Decoding (K-PCD) and Value-Perturbed Contrastive Decoding (V-PCD), two training-free methods that contrast original predictions against internally perturbed visual branches at inference time. Across seven models from three families and four multi-image benchmarks, K-PCD and V-PCD achieve the two highest average accuracies and outperform state-of-the-art baselines in cross-model average accuracy on three of four benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-image Reasoning, Multimodal Large Language Models, Contrastive Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_byVKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate how images interact in an MLLM and how it changes the KV representations of visual tokens. Based on this, they propose two training-free contrastive decoding methods to improve multi-image reasoning.

First, the authors discover that cross-image interactions introduce negative shifts in later image representations, along directions defined by earlier images. Removing such a shift weakens the model's performance. Thus, they propose K-PCD and V-PCD, which selectively reverse these negative components to introduce degradation into the prediction for contrastive decoding.

Results show K-PCD and V-PCD achieve overall accuracies of 51.89% and 51.69%, respectively, and outperform the strongest baseline by 1% consistently on multiple benchmarks.

Critical strengths
  1. Interesting analysis of geometric phenomena in multi-image interactions. The negative shifts in later images are an interesting discovery and raise a hypothesis about how MLLMs process visual information.

  2. A training-free contrastive decoding method with decent performance, as demonstrated by experiments.

  3. Extensive evaluations across seven models from three model families and four benchmarks support the effectiveness of the proposed decoding methods across different architectures.

Critical weaknesses
  1. Insufficient baseline comparison. The authors compare against baseline methods, including VCD, FOCUS, and DelimScaling. However, the authors should add a comparison to a very closely related method: [1] directly perturbs the visual attention by masking the visual tokens to produce contrastive logits, which is potentially very similar to perturbing the KV representations.

  2. The improvement over the strongest baseline method is rather limited, with only a 1% advantage.

Questions for authors
  1. How does KV perturbation differ from DEGAP [1]? In particular, what are the advantages of perturbing K and V over directly modifying the visual attention? An empirical comparison would help establish the benefit of the proposed perturbation mechanism.

  2. The joint KV perturbation in Section 4.3.2 performs worse than individual KV perturbation. The authors attribute the worse performance of joint KV-PCD to interference between the two methods. Could they clarify whether the Key and Value perturbation strengths are tuned independently? Would allowing separate, potentially smaller strengths for Keys and Values mitigate the degradation?

  3. How do the hyperparameters generalize to new tasks? The authors selected their hyperparameters using a 10% subset from each of the benchmarks, meaning they are adapted to these specific tasks. How do the methods perform on some held-out tasks? Could the authors evaluate cross-benchmark hyperparameter transfer, where parameters selected on one benchmark are applied directly to another without retuning?

[1] Seo, H., Jeong, Y., Kang, M., Park, J., & Choi, Y. S. (2026). DEGAP: Dynamic Entropy-Guided Attention Perturbation for Contrastive Decoding in Large Vision-Language Models. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 109025–109044. https://proceedings.mlr.press/v306/seo26b.html

5 comments

Thank you for the constructive review and for recognizing our geometric analysis and evaluation across model families.

Revisions addressing your comments are highlighted in purple\color{#7B2CBF}{\text{purple}} in the revised manuscript.

W1 / Q1(a). How does K/V perturbation differ from DEGAP?

Thank you for pointing us to DEGAP. The distinction lies in how the reference branch is constructed. DEGAP directly perturbs visual attention and dynamically selects the affected layers according to model confidence [1]. Our methods select negative reconstruction terms along earlier-image directions and edit these terms within visual Keys or Values, while retaining the original attention mask. K-PCD changes the Keys used to compute attention scores, whereas V-PCD changes the content vectors aggregated by attention. This provides a geometrically defined choice of what to perturb and allows the two intervention sites to be examined separately. The contrastive decoding rule is shared; our contribution is the representation-level reference construction and the analysis motivating it.

W1 / Q1(b). Empirical comparison with DEGAP.

We evaluate DEGAP alongside K-PCD and V-PCD across three models and four benchmarks, excluding the fixed 10% selection subset. All three methods are compared on the same held-out examples within each model–benchmark setting.

ModelBenchmarkDEGAP (%)K-PCD (%)V-PCD (%)
Qwen2.5-VL-7BMantis71.4371.4370.92
Qwen2.5-VL-7BMIRB60.0262.4361.74
Qwen2.5-VL-7BMuirBench45.9046.3746.20
Qwen2.5-VL-7BQBench275.3375.0075.56
InternVL3-8BMantis66.8468.3769.39
InternVL3-8BMIRB51.6652.0052.35
InternVL3-8BMuirBench37.2237.2238.93
InternVL3-8BQBench276.2276.3376.44
LLaVA-OV-7BMantis63.2762.2464.29
LLaVA-OV-7BMIRB46.8548.0049.26
LLaVA-OV-7BMuirBench35.3835.3035.30
LLaVA-OV-7BQBench274.5674.1174.00
Average12 settings58.7259.0759.53

Averaged equally over the 12 settings, K-PCD and V-PCD achieve 59.07% and 59.53%, compared with 58.72% for DEGAP. Using unrounded scores, the mean gains are +0.34 percentage points for K-PCD (95% CI [−0.20, +0.88], Holm-adjusted p=0.216) and +0.81 percentage points for V-PCD (95% CI [+0.35, +1.26], Holm-adjusted p=0.00098). The paired bootstrap and randomization test keep each question together across models and group MuirBench counterparts. Holm correction covers both comparisons against DEGAP. Section S10.2 of the revised manuscript reports these results.

W2. The improvement over the strongest baseline is approximately one percentage point.

Thank you for raising this point. With DEGAP added, the strongest baseline is no longer the same in every setting, so we report held-out comparisons against both DEGAP and FOCUS. All numbers exclude the fixed 10% selection subset.

Against DEGAP (3 models ×\times 4 benchmarks; table in W1 / Q1(b)). On the nine held-out target settings in Table S17, DEGAP averages 60.68%, compared with 60.01% for FOCUS. V-PCD improves over DEGAP by +0.81 pp (95% CI [+0.35, +1.26], Holm-adjusted p=0.00098). K-PCD is on par with DEGAP: +0.34 pp (95% CI [−0.20, +0.88], Holm-adjusted p=0.216).

Against FOCUS (7 models ×\times 4 benchmarks; revised Table S13, where the held-out K-PCD accuracy is corrected from 51.00% to 51.03%).

ComparisonSettingsFOCUS (%)Ours (%)Δ\Delta (pp)95% CI (pp)p (Holm)
K-PCD vs. FOCUS2850.2151.03+0.818[+0.268, +1.360]0.00297
V-PCD vs. FOCUS2850.2151.38+1.174[+0.590, +1.763]0.00016

The revised analysis in Section S10 uses held-out per-question correctness, shares question draws across models within each benchmark, and groups MuirBench items with their counterparts. Both gains over FOCUS remain statistically significant.

We agree that the gains are moderate in size. Beyond accuracy, the paper contributes the analysis behind the method: controlled interventions show that cross-image-induced changes in both visual Keys and Values contribute to answer accuracy, the geometric analysis characterizes how cross-image interaction changes their coefficients along earlier-image directions, and the editing diagnostic shows that removing or reversing negative reconstruction terms weakens the correct answer's margin.

Q2(a). Were Key and Value strengths tuned independently in the original joint variant?

The original joint variant selects one shared perturbation strength for Keys and Values. For LLaVA-OV-7B on Mantis, its selected configuration is αK=αV=1.0\alpha_K=\alpha_V=1.0 and β=0.1\beta=0.1.

Q2(b). Can separate, potentially smaller strengths mitigate the degradation?

We search αK,αV∈0.25,0.5,1.0\alpha_K, \alpha_V \in \\{0.25, 0.5, 1.0\\} and β∈0.02,0.05,0.1,0.3\beta \in \\{0.02, 0.05, 0.1, 0.3\\} on the same 21-example selection subset. The selected configuration is αK=1.0\alpha_K=1.0, αV=0.25\alpha_V=0.25, and β=0.3\beta=0.3. Evaluation on the same 196 held-out examples gives:

MethodHeld-out accuracy (%)
Original joint63.78
Independently tuned joint63.78
K-PCD62.24
V-PCD64.29

The independently tuned joint variant remains at 63.78%, below V-PCD's 64.29% in this setting. Changing Keys also changes the attention weights used to aggregate the edited Values, which could explain the result. We have added this control in Section S6.1 of the revised manuscript.

Q3(a). How is held-out evaluation separated from parameter selection?

Our held-out comparisons exclude the fixed 10% subset used for parameter selection and evaluate the selected configurations on the remaining examples. Cross-benchmark transfer, which tests strengths on benchmarks other than the one used for their selection, is addressed separately below.

Q3(b). Can parameters selected on one benchmark transfer without target-benchmark retuning?

For each model and intervention site, we select α\alpha and β\beta on MIRB and apply the same two strengths to Mantis, MuirBench, and QBench2 without using target-benchmark examples to select them. The transferred variants and target-tuned variants are evaluated on the same held-out target examples:

ModelTarget benchmarkK-PCD target-tuned (%)K-PCD MIRB-transfer (%)V-PCD target-tuned (%)V-PCD MIRB-transfer (%)FOCUS (%)
Qwen2.5-VL-7BMantis71.4371.4370.9269.9070.41
Qwen2.5-VL-7BMuirBench46.3746.1546.2046.4545.64
Qwen2.5-VL-7BQBench275.0075.1175.5676.0075.11
InternVL3-8BMantis68.3767.3569.3968.8866.33
InternVL3-8BMuirBench37.2237.3138.9338.9336.75
InternVL3-8BQBench276.3376.3376.4475.5675.33
LLaVA-OV-7BMantis62.2463.7864.2965.8262.24
LLaVA-OV-7BMuirBench35.3035.3435.3035.3034.36
LLaVA-OV-7BQBench274.1174.2274.0074.2273.89
Average9 settings60.7160.7861.2261.2360.01

MIRB-transfer accuracy averages 60.78% versus 60.71% with target-benchmark selection for K-PCD, and 61.23% versus 61.22% for V-PCD. Both transferred variants also exceed FOCUS's 60.01% average on these targets. We have added this transfer experiment in Section S10.3 of the revised manuscript.

[1] Seo et al. DEGAP: Dynamic Entropy-Guided Attention Perturbation for Contrastive Decoding in Large Vision-Language Models. ICML 2026.

AuthorsOpenReview ↗

Brief Follow-Up on Our Response

Dear Reviewer,

Thank you again for your specific suggestions. We added the requested held-out comparison with DEGAP across three models and four benchmarks. V-PCD improved on DEGAP by 0.81 percentage points on average, and K-PCD performed comparably. On LLaVA-OneVision-7B with Mantis, tuning the Key and Value strengths independently yielded the same accuracy as the original joint variant. We also found that strengths selected on MIRB gave nearly the same average accuracy as target-tuned strengths across three other benchmarks.

As the discussion period draws to a close, we would be grateful to hear whether these experiments address your questions about the DEGAP comparison, joint tuning, and hyperparameter transfer. If any questions remain, we would be happy to follow up.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_FxrkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates the cross-image interaction for multi-image reasoning of VLMs. Detailed analysis on visual keys and values highlights the critical importance of cross-image-induced shifts and negative component along earlier image directions. Based on these analytical findings, the authors propose Key-Perturbed Contrastive Decoding (K-PCD) and Value-Perturbed Contrastive Decoding (V-PCD). The experiments on three categories of VLMs and four multi-image benchmarks demonstrate the effectiveness of K-PCD and V-PCD among existing training-free inference-time methods.

Critical strengths
  1. This paper deeply investigates the mechanism of visual keys and values in multi-image reasoning. The analytical results reveals how cross-image-induced shifts and negative components along earlier-image directions could affect the final answer accuracy.
  2. The authors further propose K-PCD and V-PCD, two new training-free inference-time method to improve the performance of multi-image reasoning. These methods utilize perturbed visual representations to form degraded reference branch. Experimental results on Qwen2.5-VL, InternVL3, and LLaVA-OneVision showcase the effectiveness of the proposed methods on four multi-image benchmarks.
Critical weaknesses
  1. The VLMs used in the experiments are outdated. Moreover, the performance of K-PCD and V-PCD on larger VLMs (e.g., ~30B and ~100B) is not verified.
  2. This paper is somewhat difficult to follow.
Questions for authors

None

2 comments

Thank you for recognizing our analysis of visual Keys and Values and the effectiveness of the training-free decoding approach.

Revisions addressing your comments are highlighted in orange\color{#FF8000}{\text{orange}} in the revised manuscript.

W1(a). Evaluation on more recent models.

Thank you for the suggestion. We extend the evaluation to InternVL3.5-14B and Qwen3-VL-32B-Instruct, newer generations of two model families used in our original evaluation. Vanilla decoding, K-PCD, and V-PCD use the same evaluation protocol:

ModelBenchmarkVanilla (%)K-PCD (%)V-PCD (%)
InternVL3.5-14BMantis73.775.176.0
InternVL3.5-14BMIRB59.060.260.5
Qwen3-VL-32B-InstructMantis74.776.075.6
Qwen3-VL-32B-InstructMIRB64.365.865.6

Both K-PCD and V-PCD improve over vanilla decoding in all eight comparisons on these newer models, with gains from +0.9 to +2.3 percentage points. We have added these evaluations in Section S10.1 of the revised manuscript.

W1(b). Evaluation at larger model scales.

Table S14 of the original submission reported three settings for InternVL3-14B. We add Qwen3-VL-32B-Instruct, a dense model at approximately 30B scale, and GLM-4.5V. GLM-4.5V is a mixture-of-experts model whose language backbone has 106B total parameters and 12B activated parameters per token.

ModelBenchmarkVanilla (%)K-PCD (%)V-PCD (%)
InternVL3-14BMantis72.875.677.0
InternVL3-14BMIRB53.456.156.4
InternVL3-14BMuirBench38.941.539.6
Qwen3-VL-32B-InstructMantis74.776.075.6
Qwen3-VL-32B-InstructMIRB64.365.865.6
GLM-4.5VMantis77.478.378.3
GLM-4.5VMIRB65.866.866.9

Both K-PCD and V-PCD improve over vanilla decoding in all seven model–benchmark settings shown in the table. Across W1(a) and W1(b), this holds for all nine distinct settings across four models from three families, including six new settings. Section S10.1 reports all nine settings together.

W2. Clarity of presentation.

Thank you for the feedback. The paper starts by showing that changes in visual Keys and Values induced by cross-image interaction contribute to answer accuracy (Section 2.1). We then characterize these changes geometrically and find that cross-image interaction generally makes negative reconstruction coefficients along earlier-image directions more prevalent (Section 2.2). Removing or reversing negative reconstruction terms in the current representation lowers the correct answer's mean margin (Section 2.3). This finding motivates K-PCD and V-PCD, which use predictions from the perturbed branch as a contrastive reference (Section 3).

In the revised manuscript, Section 2 opens with a short overview of these steps. The transitions into Sections 2.2, 2.3, and 3 explain how the accuracy results motivate the geometric analysis, why we then test the negative reconstruction terms directly, and how the margin results inform the design of the reference branch. We also use “negative coefficients” for the scalar reconstruction weights and “negative reconstruction terms” for the corresponding vectors edited by Eq. 9.

We thank the reviewer again for the suggestions, which helped us broaden the evaluation and improve the clarity of the paper.

AuthorsOpenReview ↗

Happy to Clarify Any Remaining Questions

Dear Reviewer,

Thank you again for your suggestions on model coverage and presentation. We added evaluations on InternVL3.5-14B, Qwen3-VL-32B-Instruct, and GLM-4.5V. Both K-PCD and V-PCD improve over vanilla decoding in all six new model–benchmark settings. We also revised the overview and transitions to make it easier to follow how the geometric analysis and margin experiments motivate the design of the contrastive reference branch.

As the discussion period draws to a close, we would appreciate hearing whether the new evaluations and revisions address your concerns. We would be glad to clarify any part of the paper that still needs explanation.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_UJTWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how cross-image interaction changes visual keys and values, then builds a contrastive reference by editing negative reconstruction terms in those representations. K-PCD and V-PCD contrast the edited reference with the original branch. The contribution is this reference construction and its activation/geometric analysis; the logit-contrast rule is inherited and credited. Evaluation covers seven models and four benchmarks, with additional controls and held-out comparisons in the appendix.

Critical strengths
  • K/V activation replacement, fixed-query readout decomposition, and coefficient comparisons in a fixed coordinate frame give a more specific account than attention maps alone (Fig. 1; Tables 1, S1–S3). The paper distinguishes joint reconstruction coefficients from orthogonal projections and acknowledges the limits of that interpretation.
  • Controls include norm-matched random perturbations, positive/all-coefficient edits, A-PCD, hidden-state interventions, and parameter sensitivity (Tables 6, S7, S12). S10 tunes VCD, DelimScaling, and missing FOCUS configurations on the same subset; S13 reports positive paired held-out differences against FOCUS after excluding those examples. Two branches also offer a structural cost advantage over FOCUS's image-count-dependent branch count. The runtime check covers only 32 short-output Mantis examples.
Critical weaknesses
  • Negative reconstruction terms remain common when direct cross-image attention is disabled (Table S3). Eq. 9 edits selected terms in the current representation, rather than isolating the increment caused by cross-image interaction. The analyses support an association and decoding shows a useful reference, but the gain's specific cause is less clear. The existing random and edit controls provide relevant counterevidence. Also, the first-token margin diagnostic uses only 64 samples per model (Table 2/S4); its connection to final decoding performance needs clarification.
  • The extra gain over A-PCD is concentrated on Mantis. A/K/V-PCD average 51.13/51.89/51.69 overall, versus 49.10/49.13/49.17 across the other three benchmarks (Table S7). This is the relevant comparison for the extra directional machinery. The held-out FOCUS comparison in S13 leaves this A-PCD comparison unanswered. Per-benchmark held-out differences and the A-PCD tuning protocol would clarify where directional editing helps. Resampling also needs to account for question identities shared across model settings and paired benchmark items; the description leaves this uncertain.
  • The experiments need more models to enhance the findings.
Questions for authors
  • Which margin/edit controls use the same examples, layers, and perturbation magnitudes? Could the mechanism statement distinguish an effective negative-term reference from effects isolated as cross-image-induced?
  • Could existing logs provide held-out K/V-PCD minus A-PCD differences per benchmark and the A-PCD selection protocol? Does S13 resampling preserve shared-question and paired-item dependencies?
6 comments

Thank you for the detailed and constructive review, and for recognizing our activation analysis and experimental controls. We have conducted additional experiments and clarified the mechanism and evaluation protocols. Revisions addressing your comments are highlighted in blue\color{#1F5FBF}{\text{blue}} in the revised manuscript.

W1(a–b) / Q1. What does Eq. 9 edit, and how is it related to cross-image interaction?

We agree with the reviewer's distinction. Eq. 9 edits the current negative reconstruction terms, rather than exclusively the increment induced by cross-image interaction. Negative terms also exist when direct cross-image attention is disabled, partly reflecting image distinctness in the centered coordinate frame. Isolating the interaction-induced increment requires a paired disabled run, whereas our inference-time method uses only the current representation.

Our analysis has partially distinguished these quantities. Specifically, in Section 2.2, we compare the original and disabled representations in a fixed coordinate frame defined by the disabled run. On 71 three- and four-image Mantis examples, 65.4–81.9% of Key coefficients decrease across the six model–depth settings; Value coefficients decrease in a majority of cases in five of six settings (Table 1).

Separately, Figure 1 establishes the functional importance of cross-image-induced K/V changes. Replacing only Keys or Values with their disabled-run counterparts, while retaining the original causal mask, reduces accuracy from 62.5% to 54.5% and 50.5%, respectively, on 200 Mantis examples with LLaVA-OV-7B.

To further clarify the relationship between these changes and the terms edited by Eq. 9, we conducted two additional diagnostics on 199 Mantis examples (Section S5.2).

(i) Negative-term reversal with cross-image interaction enabled versus disabled. We apply reversal (α=1\alpha=1) under both conditions. The table reports the mean unedited correct-answer margin (m₀) and the mean margin loss (gαg_{\alpha}) after reversal.

Modelm₀, on / offK: gαg_{\alpha}, on / offV: gαg_{\alpha}, on / off
LLaVA-OV-7B1.141 / 0.1160.754 / -0.0260.781 / -0.027
Qwen2.5-VL-7B2.362 / 1.2551.949 / 0.8331.971 / 1.010

Reversal produces larger absolute mean margin losses with cross-image interaction enabled in all four settings. On LLaVA-OV-7B, the loss is approximately zero when interaction is disabled, although the unedited margin is already close to zero. On Qwen2.5-VL-7B, reversal still causes substantial margin loss without interaction, confirming that baseline negative terms can also be functionally important. Because the unedited margins differ between conditions, these comparisons do not isolate the contribution of the induced increment.

(ii) Separating baseline and interaction-induced negative coefficients. In the fixed frame of Section 2.2, we further decompose the negative coefficients on LLaVA-OV-7B. The interaction-induced portion accounts for 19% of the negative coefficient mass for Keys and 11% for Values. Reversing this portion alone gives margin losses of 0.063 and 0.008, respectively, compared with 0.265 and 0.092 for the baseline portion. Thus, both portions can affect prediction, but the induced increment alone does not account for the full effect of editing the current negative terms.

Cross-image interaction makes negative coefficients more prevalent, and PCD edits the negative reconstruction terms in the current representation. We clarify this distinction in the Abstract and Sections 1, 2.2, 2.3, and 6. Section 2.3 points to the expanded diagnostics in Sections S5.1 and S5.2.

W1(c). What do the random and coefficient-editing controls establish?

These controls test whether the benefit comes from generic representation perturbation or from the selected negative reconstruction terms. Across four model–site settings on Mantis (Table 6), negative-coefficient editing achieves the highest average accuracy (67.63%), compared with 65.90% for positive-only editing, 66.36% for norm-matched random editing, and 67.17% for reversing all coefficients.

Negative-coefficient editing outperforms positive-only and random editing in all four settings. The random control matches per-token edit norms and layer coverage at each candidate α\alpha, with configurations independently selected on the same selection subset. Table S12 additionally transfers PCD's selected α\alpha and β\beta to the norm-matched random control, and PCD outperforms it in all six comparisons.

These results indicate that the benefit is not explained by perturbation magnitude alone, and that negative-term selection provides an additional advantage over generic perturbation and non-selective coefficient editing.

W1(d). The margin diagnostic uses only 64 examples per model.

We expanded the diagnostic from 64 to all 199 eligible Mantis examples per model, using identical question IDs across LLaVA-OV-7B and Qwen2.5-VL-7B and all editing conditions.

Each cell below reports candidate accuracy (%) / mean margin loss. The strongest incorrect competitor is selected under the original branch and held fixed across edits.

SiteOperationα\alphaLLaVA-OV-7BQwen2.5-VL-7B
--Original062.31 / 0.00074.37 / 0.000
KStrengthen-0.560.30 / 0.04065.83 / 0.370
KRemove0.559.80 / 0.27768.34 / 0.634
KReverse1.045.73 / 0.75446.23 / 1.949
VStrengthen-0.561.81 / -0.07974.37 / 0.122
VRemove0.555.78 / 0.49069.35 / 0.565
VReverse1.047.74 / 0.78137.69 / 1.971

Across both models and both sites, removing negative terms reduces the mean correct-answer margin, and reversing them produces a larger reduction. The expanded experiment therefore preserves the original margin trend. These results are included in Section S5.1 of the revised manuscript.

W1(e). How does the margin diagnostic connect to final decoding performance?

The connection follows directly from the contrastive decoding rule. For a fixed correct–incorrect candidate pair, let m0m_{0} and mαm_{\alpha} denote the score margins under the original and perturbed branches. Then

=m_{0}+\beta(m_{0}-m_{\alpha}).$$ When $m_{0}-m_{\alpha} > 0$, contrastive decoding increases the correct candidate's score margin against that competitor by $\beta(m_{0}-m_{\alpha})$. This identity holds for a fixed token pair at each decoding position. **The diagnostic measures this margin degradation at the first answer position**, while our benchmark evaluations separately measure final generation accuracy. We do not equate first-token margin changes with guaranteed improvements in complete decoding. The original submission states this identity alongside the decoding equation in Section 3. **Q1. Which controls share examples, layers, and perturbation magnitudes?** The matching conditions are summarized below. | Comparison | Examples | Layers | Perturbation matching | |---|---|---|---| | Table 2/S4 and expanded margin diagnostic | Same examples across operations; expanded analysis uses identical 199 question IDs across both models | All decoder layers | $\alpha=-0.5/0.5/1.0$; no equal-norm constraint across operations | | Table 6 random control | Same Mantis evaluation examples within each model | Matched layer coverage | Per-token edit norms matched at each candidate $\alpha$; independently tuned on the same selection subset | | Table S12 random control | Same Mantis evaluation examples within each model | Matched layer coverage | Per-token norm matching; PCD-selected $\alpha$ and $\beta$ transferred to random editing | **W2(a) / Q2. How do K-PCD and V-PCD compare with A-PCD across benchmarks?** We agree that A-PCD is an important reference for assessing the additional benefit of directional editing. Beyond the full-set results in Table 5, we provide **held-out comparisons** after excluding the fixed 10% selection subset. All three methods are evaluated on the same held-out examples within each model–benchmark setting. The table reports seven-model average accuracies (%). | Benchmark | A-PCD | K-PCD | V-PCD | |---|---:|---:|---:| | Mantis | 56.49 | 57.22 | **58.16** | | MIRB | 47.83 | **47.88** | 47.85 | | MuirBench | 33.78 | 34.00 | **34.02** | | QBench2 | **65.65** | 65.00 | 65.49 | | **Average** | **50.94** | **51.03** | **51.38** | On held-out examples, K-PCD and V-PCD outperform A-PCD by **+0.09 and +0.44 percentage points** on average, respectively. The additional gains are concentrated on Mantis (+0.73/+1.67 pp). Differences are small on MIRB (+0.05/+0.02 pp) and MuirBench (+0.22/+0.24 pp), while A-PCD performs better on QBench2. **W2(b) / Q2. How is A-PCD selected?** A-PCD uses the cross-image-attention-disabled run as its contrastive reference. Its contrast strength $\beta$ is selected on the same fixed 10% selection subset used for the other methods and then frozen for evaluation. Since A-PCD does not edit reconstruction terms, it has no perturbation-strength parameter $\alpha$. These details are provided in Section S6.3.

W2(c) / Q2. Does resampling preserve shared-question and paired-item dependencies?

The revised Table S13 uses held-out per-question correctness and dependence-preserving resampling. Question draws are shared across models within each benchmark, and MuirBench items are grouped with their counterparts. In preparing it, we corrected the held-out K-PCD accuracy from 51.00% to 51.03% (gain over FOCUS from +0.80 to +0.818 pp). The original value was computed from an earlier K-PCD log scored with a different answer-matching rule; the revised table scores all three methods from the same per-question logs. The FOCUS and V-PCD accuracies are unchanged, and no other table is affected.

Both K-PCD and V-PCD remain significantly better than FOCUS under this procedure. The gains are +0.818 pp for K-PCD (95% CI [+0.268, +1.360], Holm-adjusted p=0.00297) and +1.174 pp for V-PCD (95% CI [+0.590, +1.763], Holm-adjusted p=0.00016). Section S10 describes the revised analysis.

W3. Can the evaluation cover additional models?

We add three models in six new model–benchmark settings, evaluating InternVL3.5-14B, Qwen3-VL-32B-Instruct, and GLM-4.5V on Mantis and MIRB. The table also includes the three InternVL3-14B settings from Table S14 of the original submission. GLM-4.5V uses a mixture-of-experts language backbone with 106B total parameters and 12B activated per token.

All experiments follow the same evaluation protocol. The table reports accuracy (%) for unmodified greedy decoding, K-PCD, and V-PCD.

ModelBenchmarkVanilla (%)K-PCD (%)V-PCD (%)
InternVL3-14BMantis72.875.677.0
InternVL3-14BMIRB53.456.156.4
InternVL3-14BMuirBench38.941.539.6
InternVL3.5-14BMantis73.775.176.0
InternVL3.5-14BMIRB59.060.260.5
Qwen3-VL-32B-InstructMantis74.776.075.6
Qwen3-VL-32B-InstructMIRB64.365.865.6
GLM-4.5VMantis77.478.378.3
GLM-4.5VMIRB65.866.866.9

Both K-PCD and V-PCD improve over vanilla decoding in all nine model–benchmark settings, with gains ranging from +0.7 to +4.2 percentage points. The new results extend the evidence for PCD to newer architectures and substantially larger models. Section S10.1 of the revised manuscript reports all nine settings together.

We thank the reviewer again for these suggestions, which helped us clarify the scope of our mechanism claims, strengthen the diagnostic evidence, and expand the evaluation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion

Dear Area Chair and Reviewers,

Thank you for your careful reading of our paper and for the constructive discussion. We especially thank Reviewer UJTW and Reviewer byVK for their positive feedback and for following up with us during the discussion period. Your feedback helped us strengthen the empirical evidence, clarify the link between our analysis and the decoding methods, and improve the presentation. As the discussion concludes, we would like to summarize the main additions suggested by reviewers and the revisions.

  • We clarified that Eq. 9 edits negative reconstruction terms in the current representation, including terms that can be present when direct cross-image attention is disabled. In response to Reviewer UJTW, we added diagnostics with cross-image interaction enabled and disabled and expanded the margin analysis from 64 to all 199 eligible Mantis examples per model. We also added held-out comparisons with A-PCD and revised the statistical analysis to account for shared questions across models and paired items in MuirBench.
  • We extended the evaluation to InternVL3.5-14B, Qwen3-VL-32B-Instruct, and GLM-4.5V in response to Reviewer Fxrk’s suggestion. Both K-PCD and V-PCD improved over vanilla decoding in all six new model–benchmark settings. We also revised the manuscript to show more clearly how the controlled Key and Value interventions motivate the geometric analysis and how the margin results inform the decoding method.
  • We clarified how K-PCD and V-PCD construct the contrastive reference and, following Reviewer byVK’s suggestion, added a held-out comparison with DEGAP across three models and four benchmarks. V-PCD improved on DEGAP by 0.81 percentage points on average. For K-PCD, the estimated gain was 0.34 percentage points, with a confidence interval that included zero. We also added an independently tuned joint Key–Value variant and found that strengths selected on MIRB achieved similar average accuracy to target-specific tuning across the nine transfer settings.

We would like to highlight three contributions of the paper.

  1. We establish the functional role of cross-image changes in visual Keys and Values. Under the original causal attention mask, replacing either component with its counterpart from a run with direct cross-image attention disabled reduces answer accuracy.
  2. We characterize the geometry of these changes and test its relevance to prediction. Cross-image interaction tends to make negative reconstruction coefficients along earlier-image directions more prevalent. Removing the corresponding terms from the current Keys or Values reduces the correct answer’s mean margin, and reversing them produces a larger reduction.
  3. We use this finding to design two training-free decoding methods. K-PCD and V-PCD construct contrastive reference predictions by selectively perturbing visual Keys or Values at inference time. Both improve average accuracy over the evaluated baselines in the main seven-model, four-benchmark evaluation.

If you have any further or last-minute questions, please let us know. We would be glad to address them before the discussion period closes. Thank you again for the time and care you devoted to reviewing our work.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.