- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EvidenceFlow, a training-free inference-time intervention for mitigating context-supported false affirmations in vision-language models. The method uses counterfactual image pairs to identify visual cues associated with incorrect object-existence predictions, applies activation restoration and blocking to locate the relevant intervention site, and suppresses a learned activation direction during inference. Experiments on three VLMs and three object-hallucination benchmarks show consistent accuracy improvements. The paper also includes direction controls and additional analyses intended to support the proposed mechanism. While the approach is promising, questions remain regarding the reliability of the calibration procedure, the specificity of the intervention, and whether the gains can be attributed to the proposed representation-space method rather than a simpler decision-level adjustment.
Critical strengths
The paper investigates a meaningful problem in VLM hallucination: contextual visual cues may support incorrect object-existence predictions even when the target object is absent. The use of counterfactual comparisons and activation interventions provides a potentially useful way to investigate this failure mode without updating model parameters. The method is training-free and achieves accuracy improvements across the evaluated model–benchmark combinations. The direction and strength controls provide preliminary evidence that the intervention is not entirely arbitrary. The paper is generally well organized, and its attempt to connect internal representations with observable prediction changes is valuable.
Critical weaknesses
- The intervention direction is estimated from only 12–17 image pairs per model. Activation changes may reflect occlusion artifacts or contextual differences rather than semantic cue contributions. Its stability across calibration subsets and cue-removal methods remains unclear.
- EvidenceFlow suppresses all baseline Yes predictions without distinguishing hallucinations from correct affirmations. This may correct false positives but also damage true positives. Further analysis of this trade-off is needed.
- Accuracy gains may partly result from a more conservative decision rule. Comparisons with calibrated Yes-logit thresholds or margin shifts are needed to establish the advantage of representation-space intervention.
- Experiments focus on binary object-existence decisions at the first answer position. Whether the method generalizes to free-form generation and other hallucination types remains unclear.
Questions for authors
- How stable are the learned intervention directions when the calibration pairs are resampled or replaced? Could the authors report sensitivity analyses using different calibration subsets and alternative cue-removal procedures?
- Since the intervention is applied to every baseline Yes prediction, how does the method balance false-positive correction against the loss of correct affirmative predictions? Could the authors provide more detailed results for false-positive correction, true-positive retention, and the resulting accuracy trade-off?
- Could the authors compare EvidenceFlow with a calibrated Yes-logit threshold or fixed margin-shift baseline under matched true-positive retention or prediction-change rates? Such a comparison would help determine whether the proposed internal intervention provides benefits beyond a more conservative decision rule.
- How well does the method generalize beyond binary object-existence questions? Could the authors provide evidence that the learned intervention remains useful for free-form multimodal generation or other types of hallucination?
9 comments
Response to Reviewer 6d1U - Weakness 1 and Question 1 - Calibration stability and cue-removal controls
We are very encouraged by your feedback and sincerely appreciate your comments and insights, which are invaluable in helping us improve the paper. We regret that some supplementary experiments are still in progress, so several numerical entries in the responses below are currently marked with “—” as placeholders.
Weakness 1
The intervention direction is estimated from only 12–17 image pairs per model. Activation changes may reflect occlusion artifacts or contextual differences rather than semantic cue contributions. Its stability across calibration subsets and cue-removal methods remains unclear.
Question 1
How stable are the learned intervention directions when the calibration pairs are resampled or replaced? Could the authors report sensitivity analyses using different calibration subsets and alternative cue-removal procedures?
Response
-
Calibration size. We evaluated ten calibration-set sizes; Table R3.1 shows three representative settings, . Across the six completed model–dataset combinations, the mean accuracy changes by at most 0.38 percentage points between and . Increasing the number of pairs mainly reduces sampling variability, without a consistent monotonic improvement in mean accuracy. For example, InternVL3 on POPEv2 changes from 81.27 ± 1.74 to 81.34 ± 0.19. These results indicate that useful downstream performance can be obtained with a small calibration set, while larger sets improve reproducibility. The DeepSeek calibration experiments are still running and are marked with “—”.
-
Calibration composition. At fixed , we evaluated ten randomly sampled calibration subsets (Table R3.2). Every repeat exceeds the corresponding manuscript baseline in all six completed combinations. Using the manuscript full-benchmark baselines as a common reference, the mean accuracy differences range from +0.68 to +8.54 percentage points. Variation across subsets is dataset-dependent: Qwen3 on POPEv2 ranges from 87.05 to 91.29, whereas its Causal-HalBench range is 86.72–87.51. Table R3.2 summarizes the same ten runs used for the entries in Table R3.1.
Table R3.1. Calibration-size sensitivity on three models and three datasets.
| Model | Dataset | N = 4 | N = 16 | N = 128 |
|---|---|---|---|---|
| InternVL3-8B | POPEv2 | 81.27 ± 1.74 | 81.34 ± 0.77 | 81.34 ± 0.19 |
| InternVL3-8B | Causal-HalBench | 86.04 ± 0.66 | 86.07 ± 0.41 | 85.88 ± 0.15 |
| InternVL3-8B | RePOPE | 92.31 ± 0.72 | 92.34 ± 0.39 | 92.24 ± 0.13 |
| Qwen3-VL-8B | POPEv2 | 89.98 ± 1.22 | 90.00 ± 1.44 | 90.36 ± 0.14 |
| Qwen3-VL-8B | Causal-HalBench | 87.32 ± 0.23 | 87.26 ± 0.24 | 87.34 ± 0.05 |
| Qwen3-VL-8B | RePOPE | 94.78 ± 0.24 | 94.67 ± 0.29 | 95.00 ± 0.04 |
| DeepSeek-VL2-small | POPEv2 | — | — | — |
| DeepSeek-VL2-small | Causal-HalBench | — | — | — |
| DeepSeek-VL2-small | RePOPE | — | — | — |
Note. Accuracy (%), mean ± SD across calibration-subset repeats. Three representative sizes are shown from the ten tested settings. DeepSeek results are pending.
Table R3.2. Calibration-subset sensitivity at fixed N = 16.
| Model | Dataset | Accuracy, mean ± SD | Accuracy range | F1-Yes, mean ± SD | Manuscript baseline (%) | Mean accuracy − manuscript baseline (pp) |
|---|---|---|---|---|---|---|
| InternVL3-8B | POPEv2 | 81.34 ± 0.77 | 79.69–82.14 | 84.08 ± 0.51 | 72.80 | +8.54 |
| InternVL3-8B | Causal-HalBench | 86.07 ± 0.41 | 85.09–86.46 | 83.94 ± 0.29 | 82.55 | +3.52 |
| InternVL3-8B | RePOPE | 92.34 ± 0.39 | 91.43–92.62 | 91.43 ± 0.37 | 89.30 | +3.04 |
| Qwen3-VL-8B | POPEv2 | 90.00 ± 1.44 | 87.05–91.29 | 90.46 ± 1.14 | 86.20 | +3.80 |
| Qwen3-VL-8B | Causal-HalBench | 87.26 ± 0.24 | 86.72–87.51 | 84.80 ± 0.22 | 86.58 | +0.68 |
| Qwen3-VL-8B | RePOPE | 94.67 ± 0.29 | 94.11–94.91 | 93.82 ± 0.27 | 93.50 | +1.17 |
| DeepSeek-VL2-small | POPEv2 | — | — | — | 80.50 | — |
| DeepSeek-VL2-small | Causal-HalBench | — | — | — | 88.87 | — |
| DeepSeek-VL2-small | RePOPE | — | — | — | 92.97 | — |
Note. Accuracy and F1-Yes are percentages. Statistics summarize ten calibration subsets and reuse the N = 16 runs in Table R3.1. SD describes variation across calibration subsets. DeepSeek results are pending.
Response to Reviewer 6d1U - Weakness 1 and Question 1 - Calibration stability and cue-removal controls
Weakness 1
The intervention direction is estimated from only 12–17 image pairs per model. Activation changes may reflect occlusion artifacts or contextual differences rather than semantic cue contributions. Its stability across calibration subsets and cue-removal methods remains unclear.
Question 1
How stable are the learned intervention directions when the calibration pairs are resampled or replaced? Could the authors report sensitivity analyses using different calibration subsets and alternative cue-removal procedures?
Response
-
Cue-category coverage. Keeping , we also varied the number of cue categories , allocating pairs as evenly as possible across categories (Table R3.3). Broader coverage generally reduces variability for InternVL3, with modest changes in mean accuracy. For example, its POPEv2 SD decreases from 2.67 at to 1.69 at . Qwen3 has similar mean performance across the available settings. This experiment evaluates sensitivity to category coverage within the calibration pool.
-
Alternative cue-removal methods. We compared LaMa and Telea using the same saved cue masks and the original model-specific calibration sets. For each editing method, we refitted the direction and reference center while keeping the intervention window and strength fixed (Table R3.4). Both methods produce positive accuracy gains in all nine model–dataset combinations, with larger gains from LaMa in this comparison. We also evaluated a hidden-state sham-adjusted direction, whose accuracy gains are equal to or slightly above the original LaMa configuration.
-
Area-matched removal controls. Section 4.2, Figure 3(a), and Appendix C.1–C.2 compare each diagnostic example with eight same-image area-matched control removals. The mean sham-adjusted cue effect is positive in all three models. These controls assess cue-specific support relative to generic removal effects. The LaMa–Telea comparison in Table R3.4 separately evaluates downstream performance when the cue-removal method changes.
Table R3.3. Cue-category coverage at fixed N = 16.
| Model | Dataset | K = 2 | K = 4 | K = 8 | K = 16 |
|---|---|---|---|---|---|
| InternVL3-8B | POPEv2 | 79.93 ± 2.67 | 79.93 ± 2.16 | 80.54 ± 1.60 | 80.56 ± 1.69 |
| InternVL3-8B | Causal-HalBench | 85.46 ± 1.08 | 85.33 ± 0.84 | 85.50 ± 0.61 | 85.61 ± 0.65 |
| InternVL3-8B | RePOPE | 91.92 ± 0.89 | 91.76 ± 0.71 | 91.93 ± 0.58 | 92.00 ± 0.51 |
| Qwen3-VL-8B | POPEv2 | 90.45 ± 0.46 | 90.22 ± 1.06 | 90.16 ± 0.83 | — |
| Qwen3-VL-8B | Causal-HalBench | 87.38 ± 0.12 | 87.32 ± 0.15 | 87.23 ± 0.13 | — |
| Qwen3-VL-8B | RePOPE | 94.87 ± 0.15 | 94.80 ± 0.19 | 94.83 ± 0.17 | — |
| DeepSeek-VL2-small | POPEv2 | — | — | — | — |
| DeepSeek-VL2-small | Causal-HalBench | — | — | — | — |
| DeepSeek-VL2-small | RePOPE | — | — | — | — |
Note. Accuracy (%), mean ± SD. K denotes the number of cue categories represented among the 16 pairs, with approximately balanced allocation. “—” denotes an unavailable result. This table uses category-balanced sampling, whereas Table R3.2 uses unrestricted sampling at N = 16.
Table R3.4. Editing-backend and hidden-state sham-adjustment comparisons.
| Model | Dataset | LaMa | Telea | Sham-adjusted |
|---|---|---|---|---|
| InternVL3-8B | POPEv2 | +8.26 | +2.90 | +8.26 |
| InternVL3-8B | Causal-HalBench | +3.87 | +2.13 | +4.05 |
| InternVL3-8B | RePOPE | +3.53 | +1.99 | +3.55 |
| Qwen3-VL-8B | POPEv2 | +4.69 | +2.68 | +4.69 |
| Qwen3-VL-8B | Causal-HalBench | +0.93 | +0.85 | +1.05 |
| Qwen3-VL-8B | RePOPE | +1.44 | +0.94 | +1.44 |
| DeepSeek-VL2-small | POPEv2 | +1.79 | +0.22 | +2.23 |
| DeepSeek-VL2-small | Causal-HalBench | +0.91 | +0.48 | +0.97 |
| DeepSeek-VL2-small | RePOPE | +0.60 | +0.35 | +0.60 |
Note. Accuracy improvement over the corresponding unmodified baseline, in percentage points. These comparisons use the original model-specific calibration sets and fixed image-group test folds: 448 examples for POPEv2, 4,836 for Causal-HalBench, and 4,026 for RePOPE. The intervention window and strength remain fixed. The sham-adjusted column reports a hidden-state contrast adjusted for the mean sham displacement. These are separate experiments from the expanded-pool sensitivity analyses in Tables R3.1–R3.3.
I wanted to follow up because I would still be interested in hearing your thoughts on the questions I raised. Even brief clarifications on some of the main points would help me better understand the method and experimental results.
I also want to emphasize that I am open to reconsidering my evaluation. If your responses help address some of my main concerns, I would certainly consider increasing my score.
I look forward to hearing your thoughts!