acceptodds
Under review as a conference paper at ICLR 2027

GeoPID: Decomposing and Steering Visual Information in Vision-Language Models

Abstract

While recent vision-language models (VLMs) have shown outstanding performance across diverse applications, they tend to under-use visual information and over-rely on textual context. In this work, we propose GeoPID, a training-free framework that analyzes multimodal information within VLMs from a geometric perspective. GeoPID decomposes information into Redundant, Modality-Unique, and Synergistic components through the geometric relationships between visual and textual representation subspaces. Through an extensive analysis across 22 VLMs and 14 benchmarks, we confirm that correct predictions exhibit stronger vision-unique components when questions strongly require visual grounding. Building on this geometric analysis, we introduce a targeted intervention technique that selectively amplifies visual representations along the vision-unique subspace during inference. As a result, visual grounding capabilities were enhanced without any additional model parameter updates, achieving an average relative accuracy gain of 7.63%.

Primary area:
generative models
Keywords:
Vision-Language Models (VLMs), Partial Information Decomposition (PID), Inference-time Steering

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_iDUKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
1(−1)
Read review
Paper summary

This paper studies how vision and text information are represented inside vision-language models. GeoPID defines shared, modality-specific, and matched-pair-dependent measures from hidden-state geometry, then amplifies a vision-unique subspace during inference. The paper reports gains across many models and benchmarks without updating model parameters.

Critical strengths
  1. This paper studies an important problem: the underuse of image evidence. Studying hidden-state geometry offers a concrete way to analyze it.

  2. The vision-unique subspace gives the intervention a clear target. The edit changes only a selected part of the visual representation and needs no model training.

  3. Extensive experiments. The analysis covers many model and task settings, and calibration examples are kept separate from final evaluation. The paper also checks alternative intervention subspaces in the appendix.

Critical weaknesses
  1. Evaluation.
    • No confidence intervals or significance tests are reported. Given that most evaluation sets contain only about 400 items, the reported numbers cannot reflect how good the method is.
    • The gain comes almost entirely from the three ViGoRL models and the two T2I-evaluation benchmarks (Rap4o, GenAI); excluding them, the mean absolute gain under the fixed rule (Table 5) drops from +5.38 to +0.25 pp.
    • A doubtful result. +91.2 pp for ViGoRL-MCTS-3B on Rap4o implies a baseline below 9%, which points to answer-parsing failures rather than better grounding, and the fixed rule beats per-cell calibration by 25 pp on POPE (+26.4 vs. +0.7).
  2. Practicality concerns.
    • The gains become marginal as models get stronger. Qwen3-VL-8B +0.49, InternVL3-8B +0.15 under the fixed rule), and all tested models are ≤8B, so the benefit for current frontier-scale VLMs is doubtful.
    • The method requires access to hidden states, so it cannot be applied to closed-source frontier models.
    • Deployment cost is nontrivial: each model×task pair needs about 100 labeled questions and a grid search over depth and α, which assumes the task distribution is known in advance.
  3. Presentation is bad. Figures and tables are hard for readers to understand and are imprecise. For example, the “geometry score” plotted in Fig. 2(a) and thresholded in Sec. 3.2 is never defined. The Mean column of Table 5 also does not match the row averages (e.g., OpenVLThinker: reported +0.09 vs. −0.45 computed).
Questions for authors
  1. How does GeoPID compare with end-to-end optimization in terms of efficiency and performance?
  2. How do the main results change when the ViGoRL family is excluded? Please also report absolute baseline accuracies and answer-extraction failure rates for ViGoRL.
  3. OOD performance. Can the geometry-based rule be selected on some model families and benchmarks, then evaluated on entirely held-out families and tasks?
7 comments
AuthorsOpenReview ↗

W1. Evaluation

We thank the reviewer for the detailed feedback, which helped us improve the rigor of the paper. We have addressed each of your concerns through additional experiments and clarifications, with changes highlighted in blue in the revised pdf.

W1-1-(a): No confidence intervals or significance tests are reported.

Response: We revised Tables 2 and 5. We explicitly report per-model 95% bootstrap confidence intervals.

ModelTable 2: Mean gain [95% CI], ppTable 5: Mean gain [95% CI], pp
General-purpose instruction-tuned VLMs
InternVL3-2B+0.7 [−0.0, +1.5]+0.45 [−0.0, +0.9]
SmolVLM-2B+1.8 [+0.8, +2.7]+0.32 [−0.3, +1.0]
Qwen2.5-VL-7B+1.6 [+1.2, +2.0]+0.90 [+0.5, +1.3]
Qwen3-VL-8B+1.1 [+0.5, +1.7]+0.46 [+0.1, +0.8]
InternVL3-8B+1.5 [+0.8, +2.2]+0.19 [−0.1, +0.4]
LLaVA-OV-7B+2.6 [+2.0, +3.2]+1.66 [+1.0, +2.3]
Idefics2-8B−0.1 [−0.7, +0.6]+0.09 [−0.3, +0.5]
Reasoning- and task-oriented post-trained VLMs
VLAA-Thinker-Q2VL-2B+0.8 [+0.2, +1.4]+0.36 [−0.2, +0.9]
VLAA-Thinker-Q2.5VL-3B+2.5 [+1.9, +3.2]+1.21 [+0.7, +1.7]
VLM-R1-OVD-3B+3.2 [+2.5, +3.8]+1.18 [+0.7, +1.6]
VLM-R1-Math-3B+2.2 [+1.6, +2.9]+0.59 [+0.1, +1.1]
MMR1-Math-7B+1.1 [+0.7, +1.5]+0.89 [+0.6, +1.2]
VL-Rethinker-7B+1.2 [+0.7, +1.7]+0.19 [−0.2, +0.5]
OpenVLThinker-7B+0.3 [−0.2, +0.9]−0.45 [−0.9, −0.1]
Orsta-7B+1.2 [+0.6, +1.8]+0.81 [+0.4, +1.2]
ThinkLite-VL-7B+1.1 [+0.7, +1.6]+0.76 [+0.4, +1.1]
VLAA-Thinker-Q2VL-7B+1.4 [+0.8, +1.9]+0.93 [+0.5, +1.4]
VLAA-Thinker-Q2.5VL-7B+1.6 [+1.1, +2.0]+0.80 [+0.5, +1.1]
Vision-R1-7B+5.5 [+4.8, +6.3]+4.44 [+3.8, +5.1]
Explicitly grounded reasoning models
ViGoRL-3B (Choice)+24.8 [+23.4, +26.0]+26.74 [+25.4, +28.2]
ViGoRL-3B (Generation)+2.2 [+1.4, +2.9]+0.91 [+0.0, +1.8]
ViGoRL-MCTS-3B (Choice)+21.2 [+20.2, +22.4]+26.02 [+24.7, +27.3]
ViGoRL-MCTS-3B (Generation)+4.2 [+3.1, +5.3]+2.87 [+1.7, +4.0]
ViGoRL-7B (Choice)+47.7 [+46.1, +49.2]+49.76 [+48.1, +51.5]
ViGoRL-7B (Generation)+2.8 [+1.9, +3.8]−0.30 [−1.3, +0.8]
34B model
LLaVA-v1.6-34B+2.6 [+1.8, +3.3]+0.84 [+0.3, +1.3]

W1-1-(b): Given that most evaluation sets contain only about 400 items, the reported numbers cannot reflect how good the method is.

Response: Please see Table 10, which explicitly reports the benchmark sample counts. 12 of the 14 benchmarks contain 700 items, while RealWorldQA and HallusionBench contain 425 and 447 items, respectively.

Benchmark poolNumber of benchmarksItems per benchmark
All benchmarks other than the two below12700
RealWorldQA1425
HallusionBench1447

Thus, the approximately 400-item counts apply to only two benchmarks, rather than most of the benchmark pools.

W1-2: The gain comes almost entirely from the three ViGoRL models and the two T2I-evaluation benchmarks (Rap4o, GenAI); excluding them, the mean absolute gain under the fixed rule (Table 5) drops from +5.38 to +0.25 pp.

Response: Please see the revised Tables 2 and 5. To address this concern, we expanded the evaluation in two directions: full-response Generation for all three ViGoRL models and evaluation of LLaVA-v1.6-34B. Across the evaluation spanning 23 models and 14 benchmarks, average gains remain positive consistently after excluding ViGoRL models and both T2I-evaluation benchmarks

Model / evaluationTable 2: Mean gain [95% CI], ppTable 5: Mean gain [95% CI], pp
ViGoRL-3B (Generation)+2.2 [+1.4, +2.9]+0.91 [+0.0, +1.8]
ViGoRL-MCTS-3B (Generation)+4.2 [+3.1, +5.3]+2.87 [+1.7, +4.0]
ViGoRL-7B (Generation)+2.8 [+1.9, +3.8]−0.30 [−1.3, +0.8]
LLaVA-v1.6-34B+2.6 [+1.8, +3.3]+0.84 [+0.3, +1.3]

W1-3-(a): A doubtful result. +91.2 pp for ViGoRL-MCTS-3B on Rap4o implies a baseline below 9%, which points to answer-parsing failures rather than better grounding,

Response: The +91.2 pp Rapidata-4o gain is explicitly a Choice-scoring result. We additionally report baseline accuracies and extraction failure rates in the new Appendix E.11 and Table 22:

ModelBaseline accuracy: Choice / Generation (%)Extraction failure: baseline (%)Extraction failure: Table 2 (%)Extraction failure: Table 5 (%)
ViGoRL-3B39.4 / 73.02.12.22.2
ViGoRL-MCTS-3B36.7 / 67.25.95.96.1
ViGoRL-7B38.0 / 74.75.76.07.4
AuthorsOpenReview ↗

W1. Evaluation - continued

W1-3-(b): and the fixed rule beats per-cell calibration by 25 pp on POPE (+26.4 vs. +0.7).

Response: Please see L430–433, where we clarify how Table 2’s settings are selected. For each model–benchmark pair, we select the intervention layer and strength using 100 labeled calibration questions disjoint from the final evaluation set.

ProtocolSetting selectionFinal evaluation
Table 2Maximize accuracy on the calibration subsetApply the selected settings without further tuning
Table 5Apply the established geometry-based ruleEvaluate on the held-out set
AuthorsOpenReview ↗

W2. Practicality concerns.

W2-1: The gains become marginal as models get stronger. Qwen3-VL-8B +0.49, InternVL3-8B +0.15 under the fixed rule), and all tested models are ≤8B, so the benefit for current frontier-scale VLMs is doubtful.

Response: We revise Tables 2 and 5 to include LLaVA-v1.6-34B (Liu et al, 2024). This model achieves positive absolute average gains under both calibration-selected settings and the fixed rule, with both 95% confidence intervals above zero. These results extend the evidence for GeoPID’s effectiveness beyond the previously evaluated 2-8B models to a 34B model.

ModelCalibration-selected gain [95% CI], ppFixed-rule gain [95% CI], pp
LLaVA-v1.6-34B+2.6 [1.8, 3.3]+0.84 [0.3, 1.3]

W2-2: The method requires access to hidden states, so it cannot be applied to closed-source frontier models.

Response: GeoPID requires access to internal representations because our central contribution is to reveal how VLMs represent and use visual information (L126-L128), and to turn these geometric insights into targeted, training-free interventions (L129-L132).

W2-3: Deployment cost is nontrivial: each model×task pair needs about 100 labeled questions and a grid search over depth and α, which assumes the task distribution is known in advance.

Response: We added Appendix E.12 and Table 23. Deployment requires a one-time basis computation taking 30 seconds using 100 unlabeled samples. Applying the established geometry-based rule requires no new labeled grid search for each target model-task pair, as we clarify in L366-L370 and L429-L432. The average inference latency increases by only 16 ms (1121->1137ms), on a single NVIDIA A40 GPU. These measurements demonstrate GeoPID’s lightweight deployment cost without model parameter updates.

ComponentMeasurement
Reference samples for basis estimation100
Labels required for basis estimation0
One-time basis computation30 s
Basis storage0.22 MB
Average inference latency: baseline → GeoPID1121 → 1137 ms
Additional inference latency16 ms, approximately 1.4%
Reviewer_QpJhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GeoPID proposes a training-free framework to analyze and improve visual information utilization in VLMs. It uses representation geometry and mutual information to characterize redundant, modality-unique, and synergistic information, and selectively amplifies vision-unique features during inference to enhance visual grounding. Experiments across 22 VLMs and 14 benchmarks report an average relative accuracy improvement of 7.63%.

Critical strengths
  1. Practical and interpretable intervention. The paper connects geometric representation analysis with a simple training-free intervention to improve visual grounding.

  2. Extensive experimental evaluation. The method is evaluated across 22 VLMs and 14 benchmarks, with additional analyses and ablation studies supporting its effectiveness.

Critical weaknesses
  1. Ambiguous interpretation of the proposed geometric information components.

    • Figure 2 compares the geometric information components across different VLMs and observes that models with weaker vision-unique geometry tend to exhibit lower vision-unique information UvU_v and synergy SS, which motivates the proposed intervention. However, since visual and textual tokens are identified by their input positions, the extent to which these geometric components reflect their claimed information content may vary substantially across models.

      • For example, a model with highly integrated visual and textual representations may exhibit a smaller measured UvU_v, even if it effectively utilizes vision-specific information.
      • Similarly, increasing shared geometry GRG_R may reflect increasingly mixed representations rather than more effective cross-modal integration.
    • The interpretation of the synergy component is even less clear. Although the matched/mismatched experiments demonstrate correspondence-dependent information in GSG_S, they do not establish whether this information represents genuine cross-modal synergy rather than redundancy shared between modalities. Moreover, Table 1 does not explicitly evaluate questions requiring redundant information, and the measured SS for joint-reasoning questions JJ is slightly lower than for vision-dependent questions (V), which seems counterintuitive and warrants further explanation.

  2. Limited effectiveness and generalizability across models. Although the paper reports promising overall performance improvements for a training-free approach, these gains are largely driven by the ViGoRL model family, while improvements on most other VLMs are relatively limited. This raises concerns about the effectiveness and generalizability of the proposed method across different models.

Questions for authors
  1. As mentioned in weakness #1, in Table 1, the measured synergy (S) for joint-reasoning questions (J) is slightly lower than that for vision-dependent questions (V). Could the authors explain this counterintuitive result?

  2. Table 3 shows that amplifying vision-unique features also improves performance on text-sufficient questions (T and T−). Since these questions do not necessarily require visual information, could the authors explain why strengthening vision-unique features leads to such improvements?

  3. A recent preprint [1], released outside the two-month contemporaneous-work window, appears highly similar to this work in both motivation and methodology, particularly in its use of geometric subspace analysis and manipulation of visual representations. Although this is currently a non-peer-reviewed preprintand should not constitute grounds for rejection, clarifying the key differences and contributions of GeoPID relative to this work would help readers better understand its novelty.

[1] When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models. arXiv:2605.08245.

7 comments

We thank the reviewer for the detailed feedback, which helped us improve the rigor of the paper. We have addressed each of your concerns through additional experiments and clarifications, with changes highlighted in blue in the revised pdf.

W1-1: Figure 2 compares the geometric information components across different VLMs and observes that models with weaker vision-unique geometry tend to exhibit lower vision-unique information (U_v) and synergy (S), which motivates the proposed intervention. However, since visual and textual tokens are identified by their input positions, the extent to which these geometric components reflect their claimed information content may vary substantially across models. For example, a model with highly integrated visual and textual representations may exhibit a smaller measured (U_v), even if it effectively utilizes vision-specific information. Similarly, increasing shared geometry (G_R) may reflect increasingly mixed representations rather than more effective cross-modal integration.

Response: We appreciate the reviewer’s point that visual information may also be represented in shared directions, and that geometric separation alone does not establish how a model uses visual evidence. To examine the connection between GeoPID’s geometric analysis and actual visual information use, we conducted three complementary tests: behavioral sensitivity to image removal, dependence of intervention gains on the correct image, and comparisons across intervention directions. Together, these tests connect our measurements to observable visual-input dependence and support the selected vision-unique directions as an effective intervention target.

First, please see Appendix E.7 (L1449–1457) and Tables 17–18, where we test whether the information measurements reflect models’ reliance on visual input. Table 17 reports the following Spearman correlations with the accuracy drop after image removal:

BenchmarkUvU_v vs. image-removal accuracy dropSS vs. image-removal accuracy drop
POPE+0.59+0.46
MME+0.31+0.57
MMBench+0.25+0.65
HallusionBench+0.37+0.80
Rapidata-4o+0.26+0.33
GenAI-Bench+0.57+0.83
SEED-Bench+0.68+0.83
AI2D+0.47+0.91
ScienceQA+0.60+0.65
A-OKVQA+0.22+0.45
CV-Bench+0.33+0.83
NaturalBench+0.52+0.66
RealWorldQA+0.43+0.78
Reefknot+0.13+0.13
Pooled+0.63+0.73

Both measures correlate positively with image-removal sensitivity in every evaluated benchmark, connecting the information measurements to observable reliance on visual input.

To examine whether the association is driven by ViGoRL, Table 18 repeats the analysis for SS after excluding this family:

AggregationAll models: Spearman correlationExcluding ViGoRL: Spearman correlation
Model level+0.79+0.68
Model–benchmark level+0.73+0.67

The positive association persists after excluding ViGoRL, supporting the connection between the measured correspondence-dependent information and visual-input reliance beyond this model family.

Second, please see Appendix F.2 and Table 25 (L1782–1797), where we test whether the intervention gains require the correct visual evidence. For V and J questions, gains are substantially larger with the correct image than with mismatched or blank images:

Question typeCorrect-image gain (pp)Mismatched-image gain (pp)Blank-image gain (pp)
V+13.1+1.0−0.2
J+13.9−3.3+0.3

These results show that the benefits on visually dependent questions require the image to match the question, rather than merely the presence of an image.

Third, please see Appendix G.1 and Table 28 (L1877–1908), where we test whether the intervention direction matters. On the same 15 model–benchmark pairs, we fix the normalized intervention depth at 0.6 and amplification strength at α=1\alpha=1, and compare alternative targets:

Amplification targetMean accuracy gain (pp)
Selected vision-unique directions, GUvG_{U_v}+0.47
Shared directions, GRG_R−1.67
Text-unique directions, GUtG_{U_t}−0.47
Synergy-evaluation space, GSG_S−0.27
Random hidden-space directions+0.07
Random text-complement directions+0.07
Full text-orthogonal residual+0.13

The selected vision-unique directions yield the highest mean gain among the tested amplification targets, supporting their usefulness beyond orthogonality to text alone. Our interpretation therefore rests on complementary behavioral and intervention evidence, without assuming that all useful visual information must reside exclusively in the vision-unique subspace.

AuthorsOpenReview ↗

W1-2 + Questions for authors - Q1

W1-2: “The interpretation of the synergy component is even less clear. Although the matched/mismatched experiments demonstrate correspondence-dependent information in (G_S), they do not establish whether this information represents genuine cross-modal synergy rather than redundancy shared between modalities. Moreover, Table 1 does not explicitly evaluate questions requiring redundant information, and the measured (S) for joint-reasoning questions (J) is slightly lower than for vision-dependent questions (V), which seems counterintuitive and warrants further explanation.”

Q1: “As mentioned in weakness #1, in Table 1, the measured synergy (S) for joint-reasoning questions (J) is slightly lower than that for vision-dependent questions (V). Could the authors explain this counterintuitive result?”

Response: To strengthen the evidence beyond image mismatch, we added Table 14, where we have added the new controls analysis to test the roles of visual and textual evidence separately. The reason why we selected NaturalBench and ScienceQA in this experiment is because their structures enable complementary controls on visual and textual evidence. (Also described in L1362-L1365).

BenchmarkMeasureInput conditionQwen2.5-VL-7BInternVL3-8BLLaVA-OV-7B
NaturalBenchAnswer information, I2(HGS;Y)I_2(HG_S;Y)Original0.0830.0770.060
NaturalBenchAnswer information, I2(HGS;Y)I_2(HG_S;Y)Random image0.00010.00010.0001
ScienceQAAccuracy (%)Original87.797.396.0
ScienceQAAccuracy (%)Image replaced73.776.075.7
ScienceQAAccuracy (%)Hint replaced82.791.389.7
ScienceQAAnswer-information loss within GSG_S (%)Image replaced525756
ScienceQAAnswer-information loss within GSG_S (%)Hint replaced403843

Across three models on ScienceQA, replacing the image reduces answer information within GSG_S by 52–57%, while replacing the textual hint reduces it by 38–43%. The latter occurs with the original image unchanged, supporting dependence on both modalities. Table 13 further shows that information decreases even when the replacement image preserves the original category, supporting sensitivity to the specific image–text pairing.

Model / benchmarkMatched imageSame-category replacementDifferent-category replacementBlank image
InternVL3-8B / MMBench0.41500.03230.02740.0149
Qwen2.5-VL-7B / POPE0.18820.01060.00550.0010
ViGoRL-3B / MMBench0.00980.00330.00310.0019

Additionally, We have revised the discussion of the V–J comparison in L344–L350. In Table 1, V and J questions, both of which require visual evidence, show nearly identical normalized S values (3.56 and 3.55), both exceeding T (2.24). We emphasize this shared pattern of higher S for questions requiring visual evidence.

W2: Limited effectiveness and generalizability across models. Although the paper reports promising overall performance improvements for a training-free approach, these gains are largely driven by the ViGoRL model family, while improvements on most other VLMs are relatively limited. This raises concerns about the effectiveness and generalizability of the proposed method across different models.

Response: Please see the expanded Tables 2 and 5 and the newly added Section 3.3 and Table 6. We address this concern through broad model–benchmark coverage, an additional larger model, two evaluation protocols for ViGoRL, and explicit transfer experiments on held-out models and benchmark groups.

First, our updated evaluation covers 23 models and 14 benchmarks, comprising 322 model–benchmark pairs (L424–425). The models include general-purpose instruction-tuned, reasoning-oriented, and explicitly grounded reasoning VLMs. Even after remove ViGoRL models, we still have the positive average gains.

Second, we expanded the original 22-model evaluation by adding LLaVA-v1.6-34B, extending the tested model sizes beyond 2–8B. Please see Tables 2 and 5 (L403–404 and L511):

Added modelTable 2: Mean gain [95% CI], ppTable 5: Mean gain [95% CI], pp
LLaVA-v1.6-34B+2.6 [+1.8, +3.3]+0.84 [+0.3, +1.3]

The 34B model improves under both intervention protocols, with both confidence intervals above zero.

Third, we evaluated all three ViGoRL models under both Choice and Generation, as separately reported in Tables 2 and 5. Choice scores answer-choice tokens before reasoning, whereas Generation generates a full response and extracts the final answer. The Generation results are:

ModelTable 2: Generation gain [95% CI], ppTable 5: Generation gain [95% CI], pp
ViGoRL-3B+2.2 [+1.4, +2.9]+0.91 [+0.0, +1.8]
ViGoRL-MCTS-3B+4.2 [+3.1, +5.3]+2.87 [+1.7, +4.0]
ViGoRL-7B+2.8 [+1.9, +3.8]−0.30 [−1.3, +0.8]

All three models achieve positive Generation gains under calibration-selected settings, with confidence intervals above zero, demonstrating benefits beyond pre-reasoning answer-choice scoring. Appendix E.11 and Table 22 additionally report baseline accuracies and extraction failure rates, with all failed extractions counted as incorrect.

Fourth, we added explicit OOD transfer experiments in Section 3.3 and Table 6 (L483–485 and L515–528). We select intervention settings on source splits and apply them to target splits containing both held-out models and held-out benchmark groups.

OOD transfer outcomeResult
Evaluations with positive gains8 / 8
Range of accuracy gains+0.13 to +3.72 pp

These transfer results directly support applicability beyond the models and benchmark groups used for setting selection. Together, the expanded model coverage, positive non-ViGoRL averages, native Generation evaluation, and held-out transfer experiments provide complementary evidence for GeoPID’s effectiveness across model sizes, evaluation protocols, and tasks.

Reviewer_fh1hOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents GeoPID, a training-free framework for analyzing multimodal information in VLMs from a geometric perspective. It characterizes shared, modality-unique, and complementary subspaces within visual and textual representations. Building on this analysis, the authors propose selectively amplifying visual representations along the vision-unique subspace to improve performance on questions requiring visual grounding. Experiments across 22 VLMs demonstrate broad improvements in vision-grounded understanding without updating model parameters.

Critical strengths

The paper is well structured, combining extensive empirical observations, explicit mathematical constructions, and comprehensive experimental validation.

  • Clear and meaningful motivation. Understanding how information is shared and differentiated across modalities is an important problem in multimodal learning. Distinguishing shared, modality-unique, and complementary components provides a reasonable perspective for investigating visual information use. Figure 1 illustrates this intuition clearly and effectively motivates the proposed intervention.

  • Interpretable geometric construction and concise intervention. The construction of GRG_R, GUvG_{U_v}, GUtG_{U_t}, and GSG_S is mathematically explicit and easy to follow. Orthogonal projection and PCA provide interpretable operations for identifying the corresponding geometric structures. These constructions lead naturally to a simple intervention that selectively amplifies vision-unique directions while preserving their orthogonal complement.

  • Comprehensive and informative experiments. The experiments reveal how multimodal geometry vary across decoder depth, model backbones, and question types. The intervention produces consistent benefits across 22 VLMs. The appendix provides substantial ablation and robustness evidence.

Critical weaknesses
  • The choice of decision representation needs stronger justification. The paper uses the last input token’s hidden state as the decision representation, but provides limited explanation or empirical validation for this choice. The authors should clarify why projecting this representation onto the geometric bases is appropriate for measuring RR, UtU_t, UvU_v, and SS. In particular, it remains unclear how sensitive the resulting measurements and conclusions are to alternative representations, such as pooled text-token representations.

  • The substantial variation in intervention benefits across VLMs deserves deeper analysis. Models such as ViGoRL exhibit much larger improvements than most other evaluated models. Although the paper relates intervention effectiveness to geometric profiles, the reasons for this unusually large performance gap remain insufficiently explained. A more detailed comparison of baseline visual dependence, answer-option bias, geometric structure, and responses to intervention could clarify why these models benefit disproportionately and better establish the method’s conditions of applicability.

  • The mismatched-image analysis should more carefully distinguish synergy from visual dependence. In equation 10, a reduction in answer-related information after replacing the image does not necessarily indicate the loss of information requiring joint visual and textual evidence. For questions answerable primarily from the image, replacing it while retaining the original answer label can also reduce this information, even when the text contributes no additional evidence. Therefore, the construction and evaluation of mismatched pairs should explicitly consider question types. In particular, analyzing joint-evidence questions, alongside image-dependent questions as a control, can help establish whether S captures cross-modal synergy beyond sensitivity to the correct image.

Questions for authors
  • Table 2 could include a mean column to facilitate comparison with Table 5.
  • The authors could strengthen the discussion by further explaining the choice of decision representation and the reasons for the substantial variation in performance gains across VLMs.
5 comments

We thank the reviewer for the detailed feedback, which helped us improve the rigor of the paper. We have addressed each of your concerns through additional experiments and clarifications, with changes highlighted in blue in the revised pdf.

W1: The choice of decision representation needs stronger justification. The paper uses the last input token’s hidden state as the decision representation, but provides limited explanation or empirical validation for this choice. The authors should clarify why projecting this representation onto the geometric bases is appropriate for measuring R, UtU_t, UVU_V, and S. In particular, it remains unclear how sensitive the resulting measurements and conclusions are to alternative representations, such as pooled text-token representations.

Response: Please see the newly added Appendix G.4 and Table 32 (L1974–1990), where we directly compare the last-input-token representation with the mean text-token representation across six models on MMBench and POPE, yielding 12 model–benchmark pairs. We found that U_v, U_t, and S retain positive rank correlations across the two representations. (Table 32 reports the following agreement between the rankings obtained under the two choices).

MeasureSpearman correlation between last-token and mean-pooled rankings
RR−0.10
UvU_v+0.46
UtU_t+0.62
SS+0.52
I2(H;Y)I_2(H;Y)+0.36

However, since replacing the last-input-token representation with mean text-token pooling reduces the measured answer information I2(H;Y)I_2(H;Y) by a factor of 10–30, we use the last input token for decision-level analysis because it retains substantially more measured answer information.

AuthorsOpenReview ↗

W2 + Questions for authors - Q2-2

W2: The substantial variation in intervention benefits across VLMs deserves deeper analysis. Models such as ViGoRL exhibit much larger improvements than most other evaluated models. Although the paper relates intervention effectiveness to geometric profiles, the reasons for this unusually large performance gap remain insufficiently explained. A more detailed comparison of baseline visual dependence, answer-option bias, geometric structure, and responses to intervention could clarify why these models benefit disproportionately and better establish the method’s conditions of applicability.

Q2-2: and the reasons for the substantial variation in performance gains across VLMs.

Response: Please see Appendices E.7, E.10–E.11, and F.2–F.3, where we examine the requested factors.

1. ViGoRL baseline performance and evaluation protocol. We revised Tables 2 and 5 to distinguish Choice scoring before reasoning from native full-response Generation followed by answer extraction. Table 22 reports markedly different baseline accuracies under these protocols:

ModelBaseline Choice accuracy (%)Baseline Generation accuracy (%)Table 2 Choice gain [95% CI], ppTable 2 Generation gain [95% CI], pp
ViGoRL-3B39.473.0+24.8 [+23.4, +26.0]+2.2 [+1.4, +2.9]
ViGoRL-MCTS-3B36.767.2+21.2 [+20.2, +22.4]+4.2 [+3.1, +5.3]
ViGoRL-7B38.074.7+47.7 [+46.1, +49.2]+2.8 [+1.9, +3.8]

As explained in Appendix E.11 (L1635–1640), Choice exhibits strong late-layer text dominance at the end of the prompt, whereas Generation exhibits weaker text dominance at the start of the answer span after reasoning. The lower Choice baseline leaves greater room for improvement, and the stronger text dominance is consistent with greater benefit from vision-unique amplification.

All three models nevertheless improve under Generation with calibration-selected settings, with confidence intervals above zero. For completeness, their fixed-rule Generation gains in Table 5 are +0.91% [0.0, 1.8], +2.87% [1.7, 4.0], and −0.30% [−1.3, 0.8], respectively.

2. Baseline visual dependence. Please see Appendix E.7 (L1449–1457) and Tables 17–18. The information measurements correlate positively with the accuracy drop after image removal:

Measurement / aggregationSpearman correlation with image-removal accuracy drop
UvU_v, pooled model–benchmark pairs+0.63
SS, pooled model–benchmark pairs+0.73
SS, model–benchmark pairs excluding ViGoRL+0.67
SS, model-level aggregation excluding ViGoRL+0.68

These results connect the measured information to observable visual-input reliance beyond the ViGoRL family.

3. Answer-option bias. Please see Appendix F.3 and Table 26 (L1803–1815). For models exhibiting strong text dominance, intervention reduces the concentration of predictions on a single answer option:

StatisticShare
Most frequent predicted option before intervention72%
Most frequent predicted option after intervention54%
Most frequent ground-truth option39%

The intervention reduces the tendency to repeatedly choose the same option, providing a behavioral explanation complementary to the geometric analysis.

4. Geometric predictors of intervention benefit. Please see L1593-L1614 and Table 21. We relate baseline profiles to intervention gains across the 308 model–benchmark combinations used in this analysis:

Baseline feature / predictorCorrelation with intervention gainCorrelation type
Baseline SS−0.54Spearman
Late-layer decline in shared geometry, ΔGR\Delta G_R+0.54Pearson
Combined geometric-change feature, Δ(GR,GUt,GUv)\Delta(G_R,G_{U_t},G_{U_v})+0.56Pearson
Combined feature including SS+0.61Pearson
Gradient-boosted predictor using geometric-change features+0.66Pearson
Gradient-boosted predictor using full layer-wise profiles+0.80Pearson

Lower baseline SS and the evolution of shared and modality-specific geometry are associated with larger gains. The full layer-wise profiles predict variation in gains more closely than the individual features, supporting the use of geometric profiles to guide intervention settings.

AuthorsOpenReview ↗

W2 - continued

W2-5: and responses to intervention could clarify why these models benefit disproportionately and better establish the method’s conditions of applicability.

Response: 5. Responses to intervention. Please see L1782-L1797 and Table 25.

Question typeCorrect-image gain (pp)Mismatched-image gain (pp)Blank-image gain (pp)
V+13.1+1.0−0.2
J+13.9−3.3+0.3

Table 27 further examines initially incorrect predictions that change after intervention:

StatisticResult
Eligible model–item pairs678
Correct after intervention74.0%
Expected correctness under random switching34.1%

In the tested strongly text-dominant models, V/J gains are substantially larger with the correct image than with mismatched or blank images. Among initially incorrect predictions that change, 74.0% become correct, compared with 34.1% expected under random switching. These controls show that improvements on visually demanding questions depend on the correct visual evidence and that changed predictions favor the correct answer.

Reviewer_m8gsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  • GeoPID provides a geometric "partial information decomposition" of vision/language representations into redundant, modality-unique, and synergistic components
  • analysis of this decomposition across 22 VLMs and 14 benchmarks implies that vision-unique components are more important tasks requiring stronger visual grounding
  • an inference-time intervention is proposed that builds upon this analysis (amplify vision-unique visual representation component directions), with reported gains concentrated in the three ViGoRL models (~1-2 pp for the other 19)
Critical strengths
  1. the geometric framework and motivation are interesting and clearly presented
  2. Fig. 1 is excellently done. provides great visual/geometric intuition for the paper's core ideas
  3. this lens on where shared vs modality-specific information lives is the main source of value in the paper
Critical weaknesses
  1. the intervention gains are almost entirely from the three ViGoRL models... i suspect it could be an eval artifact (see Q2)

    • without them, the mean gain drops 5.68 -> ~1.6 pp (Tab. 2) and 5.38 -> ~0.8 pp (Tab. 5)
    • the intervention is the main validation of the framework, so this undercuts the central claim
  2. the "PID" framing is loose and the analysis is thin

    • the components are measured separately rather than split from one total, so they aren't really a decomposition (eg S alone is 2-3.5x the total info in Tab. 1)
    • the correct-vs-incorrect finding (Fig. 3) is a single model, and its significance test (Tab. 17) compares question types, not correct vs incorrect
Questions for authors
  1. how does this relate to the Platonic Representation Hypothesis (Huh 2024)? I think discussion of this would greatly strengthen the paper!
  2. why does ViGoRL improve so much more than every other model?
    • one possible explanation is a simple eval artifact: ViGoRL is trained to reason before answering, so its first-token answer logits may be near-arbitrary. this would fit its near-chance baselines (eg ~29% on 4-way MMBench vs ~85% for base Qwen2.5-VL-7B, Fig. 7), ~95-100% answered-blind rates (Fig. 7), and the 72% single-option collapse (Tab. 21)
    • can you report ViGoRL under its native generate-then-parse protocol (vs its published numbers), and the main results without ViGoRL?
    • are the gains in Tabs. 2 and 5 absolute or relative?
  3. how sensitive is this to the choice of reference batch? any ablations on this?

suggestions:

  1. the math in sec 2.1 on p 4 got a bit dense with the inline math mode. might be best to expand these computations/equations fully in the appendix and reference them in the main text? it's currently two consecutive pages of math
  2. fig 2: increase the font size of axes and legend
  3. Tabs. 2, 5 are way too massive / hard to read. need to do something to make this easier (eg some groupings by family / size, etc)
6 comments
AuthorsOpenReview ↗

W1 + Questions for authors - Q2

We thank the reviewer for the detailed feedback, which helped us improve the rigor of the paper. We have addressed each of your concerns through additional experiments and clarifications, with changes highlighted in blue in the revised pdf.

W1: the intervention gains are almost entirely from the three ViGoRL models... i suspect it could be an eval artifact (see Q2)

Q2: why does ViGoRL improve so much more than every other model? one possible explanation is a simple eval artifact: ViGoRL is trained to reason before answering, so its first-token answer logits may be near-arbitrary. this would fit its near-chance baselines (eg ~29% on 4-way MMBench vs ~85% for base Qwen2.5-VL-7B, Fig. 7), ~95-100% answered-blind rates (Fig. 7), and the 72% single-option collapse (Tab. 21)

Response: Please see the revised Tables 2 and 5 and the newly added Appendix E.11 and Table 22 (L1619–1640). Following your suggestion, we evaluated all three ViGoRL models using native full-response generation followed by final-answer extraction ('Generation'), alongside the original answer-choice ('Choice') scoring protocol. Here is the detailed response:

Under 'Choice', ViGoRL’s baseline accuracies under this protocol are substantially lower than under native Generation, leaving greater room for improvement:

ModelBaseline Choice accuracy (%)Table 2 Choice gain [95% CI], ppTable 5 Choice gain [95% CI], ppBaseline Generation accuracy (%)
ViGoRL-3B39.4+24.8 [+23.4, +26.0]+26.74 [+25.4, +28.2]73.0
ViGoRL-MCTS-3B36.7+21.2 [+20.2, +22.4]+26.02 [+24.7, +27.3]67.2
ViGoRL-7B38.0+47.7 [+46.1, +49.2]+49.76 [+48.1, +51.5]74.7

Appendix E.11 (L1635–1640) explains the GeoPID's gain difference between Choice' and Generation' to the text-dominance patterns illustrated in Figure 4:

ProtocolRepresentation examinedObserved geometric pattern
ChoiceEnd of the prompt, before reasoningStrong late-layer text dominance
GenerationStart of the answer span, after reasoningWeaker text dominance

Under Choice, the decision-state representation is more strongly associated with text-unique than vision-unique directions. GeoPID selectively amplifies visual representations along the vision-unique subspace, making it particularly relevant to this text-dominant setting. After generated reasoning, the weaker text dominance and higher baseline accuracy are consistent with smaller intervention gains. This provides a geometric account of why ViGoRL benefits especially strongly under Choice.

Q2-2: can you report ViGoRL under its native generate-then-parse protocol (vs its published numbers), and the main results without ViGoRL?

Response: Following your suggestion, we generated full responses and parsed their final answers. The resulting gains are:

ModelBaseline Generation accuracy (%)Table 2 Generation gain [95% CI], ppTable 5 Generation gain [95% CI], pp
ViGoRL-3B73.0+2.2 [+1.4, +2.9]+0.91 [+0.0, +1.8]
ViGoRL-MCTS-3B67.2+4.2 [+3.1, +5.3]+2.87 [+1.7, +4.0]
ViGoRL-7B74.7+2.8 [+1.9, +3.8]−0.30 [−1.3, +0.8]

All three models improve under native Generation with calibration-selected settings, with 95% confidence intervals above zero. Thus, the intervention’s benefits extend beyond pre-reasoning answer-choice scoring, while their magnitude depends on the evaluation protocol.

To address potential answer-parsing artifacts, Table 22 additionally reports extraction failure rates:

ModelBaseline extraction failure (%)Table 2 extraction failure (%)Table 5 extraction failure (%)
ViGoRL-3B2.12.22.2
ViGoRL-MCTS-3B5.95.96.1
ViGoRL-7B5.76.07.4

All failed extractions are counted as incorrect. Failure rates remain unchanged or increase after intervention, so the accuracy gains cannot be attributed to fewer parsing failures.

Q2-3: are the gains in Tabs. 2 and 5 absolute or relative?

Response: We explicitly identify the gains in Tables 2 and 5 as absolute accuracy improvements in their captions (L378–385 and L486–493).

W2-1: the components are measured separately rather than split from one total, so they aren't really a decomposition (eg S alone is 2-3.5x the total info in Tab. 1)

Response: Please see L253–269 and Eqs. (9)–(10). We agree that these separately measured components do not form an additive decomposition of the full-state information. Our components quantify answer-related information associated with explicitly constructed geometric subspaces. For clarity, their definitions are:

ComponentOperational interpretation
RRAnswer-related information along visual directions aligned with the estimated text subspace.
UvU_vAnswer-related information along vision-specific directions orthogonal to the estimated text subspace.
UtU_tAnswer-related information along text-specific directions orthogonal to the estimated vision subspace.
SSDifference in answer-related information between matched and mismatched inputs within the complementary subspace.

These separately measured quantities are not constrained to sum to the full-state information I2(H;Y)I_2(H;Y). Our contribution is to connect shared and modality-specific geometry to answer-related information and use that diagnosis to identify an effective intervention target. Tables 2 and 5 test the resulting targeted amplification directly.

W2-2: the correct-vs-incorrect finding (Fig. 3) is a single model, and its significance test (Tab. 17) compares question types, not correct vs incorrect

Response: We added Appendix E.8 and Table 20 (L1533–1558). This shows comparisons between correct and incorrect predictions within each benchmark, separately for V and J questions. The following reproduces the results for two different models:

ModelTypeBenchmarkDifferencep
Qwen2.5-VL-7BVMMBench+0.660.005
Qwen2.5-VL-7BVReefknot+1.060.105
Qwen2.5-VL-7BVMME+0.440.373
Qwen2.5-VL-7BVA-OKVQA+0.400.079
Qwen2.5-VL-7BVScienceQA+0.350.224
Qwen2.5-VL-7BVPOPE+0.320.150
Qwen2.5-VL-7BVSEED+0.280.084
Qwen2.5-VL-7BVNaturalBench+0.220.319
Qwen2.5-VL-7BVAI2D+0.070.767
Qwen2.5-VL-7BVRealWorldQA+0.050.725
Qwen2.5-VL-7BVHallusionBench+0.020.966
Qwen2.5-VL-7BVCV-Bench−0.190.325
Qwen2.5-VL-7BJMME+0.460.038
Qwen2.5-VL-7BJHallusionBench+0.330.202
Qwen2.5-VL-7BJNaturalBench+0.290.249
Qwen2.5-VL-7BJRapidata-4o+0.160.270
Qwen2.5-VL-7BJGenAI-Bench−0.070.652
Qwen3-VL-8BVMME+0.940.036
Qwen3-VL-8BVScienceQA+0.920.218
Qwen3-VL-8BVA-OKVQA+0.85<0.001
Qwen3-VL-8BVPOPE+0.80<0.001
Qwen3-VL-8BVAI2D+0.750.004
Qwen3-VL-8BVSEED+0.63<0.001
Qwen3-VL-8BVMMBench+0.420.120
Qwen3-VL-8BVRealWorldQA+0.190.235
Qwen3-VL-8BVCV-Bench+0.160.594
Qwen3-VL-8BVNaturalBench+0.040.835
Qwen3-VL-8BVHallusionBench−0.080.764
Qwen3-VL-8BVReefknot−0.210.577
Qwen3-VL-8BJMME+0.660.001
Qwen3-VL-8BJHallusionBench+0.640.180
Qwen3-VL-8BJRapidata-4o+0.270.089
Qwen3-VL-8BJNaturalBench+0.020.940
Qwen3-VL-8BJGenAI-Bench−0.070.644

For Qwen2.5-VL-7B, 15 of 17 differences are positive, with two reaching nominal p < 0.05. For the newly added Qwen3-VL-8B, 14 of 17 differences are positive, with six reaching nominal p < 0.05. This provides within-type, within-benchmark evidence supporting the trend in Figure 3, with the table reporting the statistical uncertainty of each comparison explicitly.

AuthorsOpenReview ↗

Questions for authors - Q1

Q1: how does this relate to the Platonic Representation Hypothesis (Huh 2024)? I think discussion of this would greatly strengthen the paper!

Response: We thank the reviewer for suggesting this connection. Please see the added discussion of Huh et al. (2024) in L139–141 and Appendix B.2 (L906–910).

The Platonic Representation Hypothesis motivates studying shared structure across representations learned by different models and modalities. GeoPID asks a complementary question: within a VLM, how do shared and modality-specific directions relate to the information needed to answer a question, and can this structure guide an intervention?

AspectGeoPID analysis and evidence
Shared representational structureIdentify aligned directions between visual and textual principal subspaces within the same VLM layer.
Modality-specific informationMeasure answer-related information along vision-unique and text-unique directions, alongside shared and complementary components (Eqs. (9)–(10), Table 1).
Dependence on question requirementsCompare components across evidence-requirement categories and prediction correctness (Table 1, Figure 3, Table 20).
Practical use of the geometryAmplify vision-unique representations and evaluate prediction changes (Tables 2 and 5), including gains by question type (Table 3).

This connection motivates examining shared structure together with modality-specific directions: the latter provide additional information about visual evidence use and a concrete target for intervention.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revision Discussion for Area Chair

Dear Area Chair,

Thank you for your time and consideration. We briefly summarize GeoPID’s contributions and how we addressed the main concerns raised during review. Changes are highlighted in blue in the revised paper.

Contributions

GeoPID connects the geometry of visual and textual representations to answer-related information and uses this analysis to guide a training-free intervention without updating model parameters. Our revised main evaluation covers 23 models and 14 benchmarks, with additional tests of native Generation, bigger model (LLaVA-v1.6-34B), and transfer to held-out models and benchmarks.

Main concerns and our responses

1. Geometric interpretation and visual information use.

Reviewers asked whether the geometric components reflect meaningful visual information use. We addressed this through image-removal correlations (Tables 17–18), image-substitution controls (Table 13), separate image- and text-replacement controls (Table 14), question-type-specific image controls (Table 25), and intervention-direction comparisons (Table 28). Together, these tests connect the measurements to model behavior and support the usefulness of the selected intervention directions.

In their follow-up, reviewer QpJh explicitly stated that these experiments substantially addressed their W1 concerns and appreciated the clarification that V and J have nearly identical normalized synergy.

2. Effectiveness beyond the largest ViGoRL gains.

Reviewers questioned the concentration of gains in ViGoRL and the Choice evaluation protocol. In Table 2 and Table 5, we added native Generation results, and added baseline/extraction-failure statistics in Table 22. All three ViGoRL models achieve positive Generation gains with 95% confidence intervals above zero.

We also added LLaVA-v1.6-34B, which improves under both Table 2 and 5, and eight held-out transfer evaluations, all with positive gains (Table 6). Positive average gains remain after excluding ViGoRL, Rap4o and GenAI. These show that the evidence for GeoPID’s effectiveness extends beyond the settings responsible for its largest gains.

3. Statistical support and variation across models.

In response to concerns about the correct-versus-incorrect analysis, we expanded Table 20 to two models. It shows positive differences in 14 of 17 comparisons, with six reaching nominal p < 0.05.

We also explained variation in gains using baseline geometric profiles (Table 21), protocol-specific text dominance (Appendix E.11), reduced answer-option concentration after intervention (Table 26), and correction of initially incorrect predictions among answers changed by the intervention (Table 27). These analyses complement the direct intervention evaluations.

4. Robustness, cost, and prior work.

We added a comparison of decision representations (Table 32), reference-batch robustness tests (Table 35), and deployment measurements (Table 23). Average inference overhead is 16ms, with no model parameter updates.

To clarify the distinction from the prior work[1], we added text-PC removal and full text-orthogonal residual amplification ablations (Table 28). GeoPID achieves the highest mean. Reviewer QpJh stated that this distinction was adequately clarified.

5. Statistical uncertainty and calibration.

Reviewers requested clearer reporting of average gains, uncertainty, and setting selection. We revised Tables 2 and 5 to report absolute percentage-point gains, means across 14 benchmarks, and per-model 95% bootstrap confidence intervals. We clarified that Table 2 settings are selected on 100 calibration questions disjoint from final evaluation and Table 5 uses the fixed geometry-based rule. We also pointed to Table 10, which shows that 12 of the 14 benchmark pools contain 700 items, addressing concerns about small evaluation pools.

Outcome of the discussion

Reviewer QpJh remained weakly positive, stating that the rebuttal improved soundness and clarity, and reviewer iDUK raised their score to 2. We responded with the expanded correctness analysis, native Generation evaluation, and image-condition controls.

We also added Section I (Limitations) to clearly state the scope of the synergy interpretation, variation in effect sizes, and the post-hoc nature of the proposed explanation for text-sufficient questions.

The remaining concerns qualify the interpretation and size of the effects, while the revised paper provides direct evidence of useful interventions across multiple models, protocols, and held-out settings. We believe this supports GeoPID’s contribution as both an empirical framework for studying visual information use and a practical training-free intervention.

Thank you for considering our revised paper and the review discussion.

[1] When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models. arXiv:2605.08245.

Dear Reviewers,

The rebuttal period is coming to an end. Please read the authors' responses and update your reviews if appropriate. It would be helpful to share your thoughts as soon as possible to allow enough time for a productive discussion. Thank you!

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.