- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 1(−1)
Read review
Paper summary
This paper studies how vision and text information are represented inside vision-language models. GeoPID defines shared, modality-specific, and matched-pair-dependent measures from hidden-state geometry, then amplifies a vision-unique subspace during inference. The paper reports gains across many models and benchmarks without updating model parameters.
Critical strengths
-
This paper studies an important problem: the underuse of image evidence. Studying hidden-state geometry offers a concrete way to analyze it.
-
The vision-unique subspace gives the intervention a clear target. The edit changes only a selected part of the visual representation and needs no model training.
-
Extensive experiments. The analysis covers many model and task settings, and calibration examples are kept separate from final evaluation. The paper also checks alternative intervention subspaces in the appendix.
Critical weaknesses
- Evaluation.
- No confidence intervals or significance tests are reported. Given that most evaluation sets contain only about 400 items, the reported numbers cannot reflect how good the method is.
- The gain comes almost entirely from the three ViGoRL models and the two T2I-evaluation benchmarks (Rap4o, GenAI); excluding them, the mean absolute gain under the fixed rule (Table 5) drops from +5.38 to +0.25 pp.
- A doubtful result. +91.2 pp for ViGoRL-MCTS-3B on Rap4o implies a baseline below 9%, which points to answer-parsing failures rather than better grounding, and the fixed rule beats per-cell calibration by 25 pp on POPE (+26.4 vs. +0.7).
- Practicality concerns.
- The gains become marginal as models get stronger. Qwen3-VL-8B +0.49, InternVL3-8B +0.15 under the fixed rule), and all tested models are ≤8B, so the benefit for current frontier-scale VLMs is doubtful.
- The method requires access to hidden states, so it cannot be applied to closed-source frontier models.
- Deployment cost is nontrivial: each model×task pair needs about 100 labeled questions and a grid search over depth and α, which assumes the task distribution is known in advance.
- Presentation is bad. Figures and tables are hard for readers to understand and are imprecise. For example, the “geometry score” plotted in Fig. 2(a) and thresholded in Sec. 3.2 is never defined. The Mean column of Table 5 also does not match the row averages (e.g., OpenVLThinker: reported +0.09 vs. −0.45 computed).
Questions for authors
- How does GeoPID compare with end-to-end optimization in terms of efficiency and performance?
- How do the main results change when the ViGoRL family is excluded? Please also report absolute baseline accuracies and answer-extraction failure rates for ViGoRL.
- OOD performance. Can the geometry-based rule be selected on some model families and benchmarks, then evaluated on entirely held-out families and tasks?
7 comments
W1. Evaluation - continued
W1-3-(b): and the fixed rule beats per-cell calibration by 25 pp on POPE (+26.4 vs. +0.7).
Response: Please see L430–433, where we clarify how Table 2’s settings are selected. For each model–benchmark pair, we select the intervention layer and strength using 100 labeled calibration questions disjoint from the final evaluation set.
| Protocol | Setting selection | Final evaluation |
|---|---|---|
| Table 2 | Maximize accuracy on the calibration subset | Apply the selected settings without further tuning |
| Table 5 | Apply the established geometry-based rule | Evaluate on the held-out set |
W2. Practicality concerns.
W2-1: The gains become marginal as models get stronger. Qwen3-VL-8B +0.49, InternVL3-8B +0.15 under the fixed rule), and all tested models are ≤8B, so the benefit for current frontier-scale VLMs is doubtful.
Response: We revise Tables 2 and 5 to include LLaVA-v1.6-34B (Liu et al, 2024). This model achieves positive absolute average gains under both calibration-selected settings and the fixed rule, with both 95% confidence intervals above zero. These results extend the evidence for GeoPID’s effectiveness beyond the previously evaluated 2-8B models to a 34B model.
| Model | Calibration-selected gain [95% CI], pp | Fixed-rule gain [95% CI], pp |
|---|---|---|
| LLaVA-v1.6-34B | +2.6 [1.8, 3.3] | +0.84 [0.3, 1.3] |
W2-2: The method requires access to hidden states, so it cannot be applied to closed-source frontier models.
Response: GeoPID requires access to internal representations because our central contribution is to reveal how VLMs represent and use visual information (L126-L128), and to turn these geometric insights into targeted, training-free interventions (L129-L132).
W2-3: Deployment cost is nontrivial: each model×task pair needs about 100 labeled questions and a grid search over depth and α, which assumes the task distribution is known in advance.
Response: We added Appendix E.12 and Table 23. Deployment requires a one-time basis computation taking 30 seconds using 100 unlabeled samples. Applying the established geometry-based rule requires no new labeled grid search for each target model-task pair, as we clarify in L366-L370 and L429-L432. The average inference latency increases by only 16 ms (1121->1137ms), on a single NVIDIA A40 GPU. These measurements demonstrate GeoPID’s lightweight deployment cost without model parameter updates.
| Component | Measurement |
|---|---|
| Reference samples for basis estimation | 100 |
| Labels required for basis estimation | 0 |
| One-time basis computation | 30 s |
| Basis storage | 0.22 MB |
| Average inference latency: baseline → GeoPID | 1121 → 1137 ms |
| Additional inference latency | 16 ms, approximately 1.4% |
W1. Evaluation
We thank the reviewer for the detailed feedback, which helped us improve the rigor of the paper. We have addressed each of your concerns through additional experiments and clarifications, with changes highlighted in blue in the revised pdf.
Response: We revised Tables 2 and 5. We explicitly report per-model 95% bootstrap confidence intervals.
Response: Please see Table 10, which explicitly reports the benchmark sample counts. 12 of the 14 benchmarks contain 700 items, while RealWorldQA and HallusionBench contain 425 and 447 items, respectively.
Thus, the approximately 400-item counts apply to only two benchmarks, rather than most of the benchmark pools.
Response: Please see the revised Tables 2 and 5. To address this concern, we expanded the evaluation in two directions: full-response Generation for all three ViGoRL models and evaluation of LLaVA-v1.6-34B. Across the evaluation spanning 23 models and 14 benchmarks, average gains remain positive consistently after excluding ViGoRL models and both T2I-evaluation benchmarks
Response: The +91.2 pp Rapidata-4o gain is explicitly a Choice-scoring result. We additionally report baseline accuracies and extraction failure rates in the new Appendix E.11 and Table 22: