- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors investigate how images interact in an MLLM and how it changes the KV representations of visual tokens. Based on this, they propose two training-free contrastive decoding methods to improve multi-image reasoning.
First, the authors discover that cross-image interactions introduce negative shifts in later image representations, along directions defined by earlier images. Removing such a shift weakens the model's performance. Thus, they propose K-PCD and V-PCD, which selectively reverse these negative components to introduce degradation into the prediction for contrastive decoding.
Results show K-PCD and V-PCD achieve overall accuracies of 51.89% and 51.69%, respectively, and outperform the strongest baseline by 1% consistently on multiple benchmarks.
Critical strengths
-
Interesting analysis of geometric phenomena in multi-image interactions. The negative shifts in later images are an interesting discovery and raise a hypothesis about how MLLMs process visual information.
-
A training-free contrastive decoding method with decent performance, as demonstrated by experiments.
-
Extensive evaluations across seven models from three model families and four benchmarks support the effectiveness of the proposed decoding methods across different architectures.
Critical weaknesses
-
Insufficient baseline comparison. The authors compare against baseline methods, including VCD, FOCUS, and DelimScaling. However, the authors should add a comparison to a very closely related method: [1] directly perturbs the visual attention by masking the visual tokens to produce contrastive logits, which is potentially very similar to perturbing the KV representations.
-
The improvement over the strongest baseline method is rather limited, with only a 1% advantage.
Questions for authors
-
How does KV perturbation differ from DEGAP [1]? In particular, what are the advantages of perturbing K and V over directly modifying the visual attention? An empirical comparison would help establish the benefit of the proposed perturbation mechanism.
-
The joint KV perturbation in Section 4.3.2 performs worse than individual KV perturbation. The authors attribute the worse performance of joint KV-PCD to interference between the two methods. Could they clarify whether the Key and Value perturbation strengths are tuned independently? Would allowing separate, potentially smaller strengths for Keys and Values mitigate the degradation?
-
How do the hyperparameters generalize to new tasks? The authors selected their hyperparameters using a 10% subset from each of the benchmarks, meaning they are adapted to these specific tasks. How do the methods perform on some held-out tasks? Could the authors evaluate cross-benchmark hyperparameter transfer, where parameters selected on one benchmark are applied directly to another without retuning?
[1] Seo, H., Jeong, Y., Kang, M., Park, J., & Choi, Y. S. (2026). DEGAP: Dynamic Entropy-Guided Attention Perturbation for Contrastive Decoding in Large Vision-Language Models. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 109025–109044. https://proceedings.mlr.press/v306/seo26b.html
5 comments
Q2(a). Were Key and Value strengths tuned independently in the original joint variant?
The original joint variant selects one shared perturbation strength for Keys and Values. For LLaVA-OV-7B on Mantis, its selected configuration is and .
Q2(b). Can separate, potentially smaller strengths mitigate the degradation?
We search and on the same 21-example selection subset. The selected configuration is , , and . Evaluation on the same 196 held-out examples gives:
| Method | Held-out accuracy (%) |
|---|---|
| Original joint | 63.78 |
| Independently tuned joint | 63.78 |
| K-PCD | 62.24 |
| V-PCD | 64.29 |
The independently tuned joint variant remains at 63.78%, below V-PCD's 64.29% in this setting. Changing Keys also changes the attention weights used to aggregate the edited Values, which could explain the result. We have added this control in Section S6.1 of the revised manuscript.
Q3(a). How is held-out evaluation separated from parameter selection?
Our held-out comparisons exclude the fixed 10% subset used for parameter selection and evaluate the selected configurations on the remaining examples. Cross-benchmark transfer, which tests strengths on benchmarks other than the one used for their selection, is addressed separately below.
Q3(b). Can parameters selected on one benchmark transfer without target-benchmark retuning?
For each model and intervention site, we select and on MIRB and apply the same two strengths to Mantis, MuirBench, and QBench2 without using target-benchmark examples to select them. The transferred variants and target-tuned variants are evaluated on the same held-out target examples:
| Model | Target benchmark | K-PCD target-tuned (%) | K-PCD MIRB-transfer (%) | V-PCD target-tuned (%) | V-PCD MIRB-transfer (%) | FOCUS (%) |
|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B | Mantis | 71.43 | 71.43 | 70.92 | 69.90 | 70.41 |
| Qwen2.5-VL-7B | MuirBench | 46.37 | 46.15 | 46.20 | 46.45 | 45.64 |
| Qwen2.5-VL-7B | QBench2 | 75.00 | 75.11 | 75.56 | 76.00 | 75.11 |
| InternVL3-8B | Mantis | 68.37 | 67.35 | 69.39 | 68.88 | 66.33 |
| InternVL3-8B | MuirBench | 37.22 | 37.31 | 38.93 | 38.93 | 36.75 |
| InternVL3-8B | QBench2 | 76.33 | 76.33 | 76.44 | 75.56 | 75.33 |
| LLaVA-OV-7B | Mantis | 62.24 | 63.78 | 64.29 | 65.82 | 62.24 |
| LLaVA-OV-7B | MuirBench | 35.30 | 35.34 | 35.30 | 35.30 | 34.36 |
| LLaVA-OV-7B | QBench2 | 74.11 | 74.22 | 74.00 | 74.22 | 73.89 |
| Average | 9 settings | 60.71 | 60.78 | 61.22 | 61.23 | 60.01 |
MIRB-transfer accuracy averages 60.78% versus 60.71% with target-benchmark selection for K-PCD, and 61.23% versus 61.22% for V-PCD. Both transferred variants also exceed FOCUS's 60.01% average on these targets. We have added this transfer experiment in Section S10.3 of the revised manuscript.
[1] Seo et al. DEGAP: Dynamic Entropy-Guided Attention Perturbation for Contrastive Decoding in Large Vision-Language Models. ICML 2026.
Brief Follow-Up on Our Response
Dear Reviewer,
Thank you again for your specific suggestions. We added the requested held-out comparison with DEGAP across three models and four benchmarks. V-PCD improved on DEGAP by 0.81 percentage points on average, and K-PCD performed comparably. On LLaVA-OneVision-7B with Mantis, tuning the Key and Value strengths independently yielded the same accuracy as the original joint variant. We also found that strengths selected on MIRB gave nearly the same average accuracy as target-tuned strengths across three other benchmarks.
As the discussion period draws to a close, we would be grateful to hear whether these experiments address your questions about the DEGAP comparison, joint tuning, and hyperparameter transfer. If any questions remain, we would be happy to follow up.
Thank you again for your time and consideration.
Best regards,
The Authors
Thank you for the constructive review and for recognizing our geometric analysis and evaluation across model families.
Revisions addressing your comments are highlighted in purple in the revised manuscript.
W1 / Q1(a). How does K/V perturbation differ from DEGAP?
Thank you for pointing us to DEGAP. The distinction lies in how the reference branch is constructed. DEGAP directly perturbs visual attention and dynamically selects the affected layers according to model confidence [1]. Our methods select negative reconstruction terms along earlier-image directions and edit these terms within visual Keys or Values, while retaining the original attention mask. K-PCD changes the Keys used to compute attention scores, whereas V-PCD changes the content vectors aggregated by attention. This provides a geometrically defined choice of what to perturb and allows the two intervention sites to be examined separately. The contrastive decoding rule is shared; our contribution is the representation-level reference construction and the analysis motivating it.
W1 / Q1(b). Empirical comparison with DEGAP.
We evaluate DEGAP alongside K-PCD and V-PCD across three models and four benchmarks, excluding the fixed 10% selection subset. All three methods are compared on the same held-out examples within each model–benchmark setting.
Averaged equally over the 12 settings, K-PCD and V-PCD achieve 59.07% and 59.53%, compared with 58.72% for DEGAP. Using unrounded scores, the mean gains are +0.34 percentage points for K-PCD (95% CI [−0.20, +0.88], Holm-adjusted p=0.216) and +0.81 percentage points for V-PCD (95% CI [+0.35, +1.26], Holm-adjusted p=0.00098). The paired bootstrap and randomization test keep each question together across models and group MuirBench counterparts. Holm correction covers both comparisons against DEGAP. Section S10.2 of the revised manuscript reports these results.
W2. The improvement over the strongest baseline is approximately one percentage point.
Thank you for raising this point. With DEGAP added, the strongest baseline is no longer the same in every setting, so we report held-out comparisons against both DEGAP and FOCUS. All numbers exclude the fixed 10% selection subset.
Against DEGAP (3 models × 4 benchmarks; table in W1 / Q1(b)). On the nine held-out target settings in Table S17, DEGAP averages 60.68%, compared with 60.01% for FOCUS. V-PCD improves over DEGAP by +0.81 pp (95% CI [+0.35, +1.26], Holm-adjusted p=0.00098). K-PCD is on par with DEGAP: +0.34 pp (95% CI [−0.20, +0.88], Holm-adjusted p=0.216).
Against FOCUS (7 models × 4 benchmarks; revised Table S13, where the held-out K-PCD accuracy is corrected from 51.00% to 51.03%).
The revised analysis in Section S10 uses held-out per-question correctness, shares question draws across models within each benchmark, and groups MuirBench items with their counterparts. Both gains over FOCUS remain statistically significant.
We agree that the gains are moderate in size. Beyond accuracy, the paper contributes the analysis behind the method: controlled interventions show that cross-image-induced changes in both visual Keys and Values contribute to answer accuracy, the geometric analysis characterizes how cross-image interaction changes their coefficients along earlier-image directions, and the editing diagnostic shows that removing or reversing negative reconstruction terms weakens the correct answer's margin.