acceptodds
Under review as a conference paper at ICLR 2027

EvidenceFlow: Inference-Time Suppression of Confounder Evidence in Vision-Language Models

Abstract

A vision-language model often affirms an object that is absent from an image when a related object is visible. Such errors are usually attributed to weak visual grounding, yet the visible content behind them is accurate and genuinely used: the failure is one of attribution rather than amount, so supplying more or less visual influence does not address it. We make this failure measurable per example. A visible cue is confounder evidence for an object query when removing it lowers the model's Yes margin more than every area-matched control removal, and the error it sustains is a context-supported false affirmation. In every evaluated model, removing co-occurring cues fixed in advance lowers the Yes margin more than the controls do on average, and restoration and blocking show that this support is concentrated in a narrow layer window at the prediction position. EvidenceFlow acts in that window at inference time: when the model answers Yes, it subtracts the positive component of the prediction state along a calibrated confounder direction, removing the cue's support for the queried claim while keeping the visual content. The direction and window come from 12-17 image pairs per model, one strength serves all models, and deployment uses only the original image and question with frozen weights. On POPEv2, EvidenceFlow improves InternVL3-8B accuracy by 8.7 percentage points, correcting 35.4% of false affirmatives while retaining 98.4% of correct ones, and accuracy improves in all nine model-benchmark cells. On Causal-HalBench with InternVL3-8B, replacing the calibrated direction with a random one at the same window and strength lowers correction from 46.2% to 1.1%.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, object hallucination, mechanistic interpretability, activation steering, counterfactual intervention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6d1UOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvidenceFlow, a training-free inference-time intervention for mitigating context-supported false affirmations in vision-language models. The method uses counterfactual image pairs to identify visual cues associated with incorrect object-existence predictions, applies activation restoration and blocking to locate the relevant intervention site, and suppresses a learned activation direction during inference. Experiments on three VLMs and three object-hallucination benchmarks show consistent accuracy improvements. The paper also includes direction controls and additional analyses intended to support the proposed mechanism. While the approach is promising, questions remain regarding the reliability of the calibration procedure, the specificity of the intervention, and whether the gains can be attributed to the proposed representation-space method rather than a simpler decision-level adjustment.

Critical strengths

The paper investigates a meaningful problem in VLM hallucination: contextual visual cues may support incorrect object-existence predictions even when the target object is absent. The use of counterfactual comparisons and activation interventions provides a potentially useful way to investigate this failure mode without updating model parameters. The method is training-free and achieves accuracy improvements across the evaluated model–benchmark combinations. The direction and strength controls provide preliminary evidence that the intervention is not entirely arbitrary. The paper is generally well organized, and its attempt to connect internal representations with observable prediction changes is valuable.

Critical weaknesses
  1. The intervention direction is estimated from only 12–17 image pairs per model. Activation changes may reflect occlusion artifacts or contextual differences rather than semantic cue contributions. Its stability across calibration subsets and cue-removal methods remains unclear.
  2. EvidenceFlow suppresses all baseline Yes predictions without distinguishing hallucinations from correct affirmations. This may correct false positives but also damage true positives. Further analysis of this trade-off is needed.
  3. Accuracy gains may partly result from a more conservative decision rule. Comparisons with calibrated Yes-logit thresholds or margin shifts are needed to establish the advantage of representation-space intervention.
  4. Experiments focus on binary object-existence decisions at the first answer position. Whether the method generalizes to free-form generation and other hallucination types remains unclear.
Questions for authors
  1. How stable are the learned intervention directions when the calibration pairs are resampled or replaced? Could the authors report sensitivity analyses using different calibration subsets and alternative cue-removal procedures?
  2. Since the intervention is applied to every baseline Yes prediction, how does the method balance false-positive correction against the loss of correct affirmative predictions? Could the authors provide more detailed results for false-positive correction, true-positive retention, and the resulting accuracy trade-off?
  3. Could the authors compare EvidenceFlow with a calibrated Yes-logit threshold or fixed margin-shift baseline under matched true-positive retention or prediction-change rates? Such a comparison would help determine whether the proposed internal intervention provides benefits beyond a more conservative decision rule.
  4. How well does the method generalize beyond binary object-existence questions? Could the authors provide evidence that the learned intervention remains useful for free-form multimodal generation or other types of hallucination?
9 comments

I wanted to follow up because I would still be interested in hearing your thoughts on the questions I raised. Even brief clarifications on some of the main points would help me better understand the method and experimental results.

I also want to emphasize that I am open to reconsidering my evaluation. If your responses help address some of my main concerns, I would certainly consider increasing my score.

I look forward to hearing your thoughts!

AuthorsOpenReview ↗

Response to Reviewer 6d1U - Weakness 1 and Question 1 - Calibration stability and cue-removal controls

We are very encouraged by your feedback and sincerely appreciate your comments and insights, which are invaluable in helping us improve the paper. We regret that some supplementary experiments are still in progress, so several numerical entries in the responses below are currently marked with “—” as placeholders.

Weakness 1

The intervention direction is estimated from only 12–17 image pairs per model. Activation changes may reflect occlusion artifacts or contextual differences rather than semantic cue contributions. Its stability across calibration subsets and cue-removal methods remains unclear.

Question 1

How stable are the learned intervention directions when the calibration pairs are resampled or replaced? Could the authors report sensitivity analyses using different calibration subsets and alternative cue-removal procedures?

Response

  1. Calibration size. We evaluated ten calibration-set sizes; Table R3.1 shows three representative settings, N=4,16,128N=4,16,128. Across the six completed model–dataset combinations, the mean accuracy changes by at most 0.38 percentage points between N=4N=4 and N=128N=128. Increasing the number of pairs mainly reduces sampling variability, without a consistent monotonic improvement in mean accuracy. For example, InternVL3 on POPEv2 changes from 81.27 ± 1.74 to 81.34 ± 0.19. These results indicate that useful downstream performance can be obtained with a small calibration set, while larger sets improve reproducibility. The DeepSeek calibration experiments are still running and are marked with “—”.

  2. Calibration composition. At fixed N=16N=16, we evaluated ten randomly sampled calibration subsets (Table R3.2). Every repeat exceeds the corresponding manuscript baseline in all six completed combinations. Using the manuscript full-benchmark baselines as a common reference, the mean accuracy differences range from +0.68 to +8.54 percentage points. Variation across subsets is dataset-dependent: Qwen3 on POPEv2 ranges from 87.05 to 91.29, whereas its Causal-HalBench range is 86.72–87.51. Table R3.2 summarizes the same ten runs used for the N=16N=16 entries in Table R3.1.

Table R3.1. Calibration-size sensitivity on three models and three datasets.

ModelDatasetN = 4N = 16N = 128
InternVL3-8BPOPEv281.27 ± 1.7481.34 ± 0.7781.34 ± 0.19
InternVL3-8BCausal-HalBench86.04 ± 0.6686.07 ± 0.4185.88 ± 0.15
InternVL3-8BRePOPE92.31 ± 0.7292.34 ± 0.3992.24 ± 0.13
Qwen3-VL-8BPOPEv289.98 ± 1.2290.00 ± 1.4490.36 ± 0.14
Qwen3-VL-8BCausal-HalBench87.32 ± 0.2387.26 ± 0.2487.34 ± 0.05
Qwen3-VL-8BRePOPE94.78 ± 0.2494.67 ± 0.2995.00 ± 0.04
DeepSeek-VL2-smallPOPEv2———
DeepSeek-VL2-smallCausal-HalBench———
DeepSeek-VL2-smallRePOPE———

Note. Accuracy (%), mean ± SD across calibration-subset repeats. Three representative sizes are shown from the ten tested settings. DeepSeek results are pending.

Table R3.2. Calibration-subset sensitivity at fixed N = 16.

ModelDatasetAccuracy, mean ± SDAccuracy rangeF1-Yes, mean ± SDManuscript baseline (%)Mean accuracy − manuscript baseline (pp)
InternVL3-8BPOPEv281.34 ± 0.7779.69–82.1484.08 ± 0.5172.80+8.54
InternVL3-8BCausal-HalBench86.07 ± 0.4185.09–86.4683.94 ± 0.2982.55+3.52
InternVL3-8BRePOPE92.34 ± 0.3991.43–92.6291.43 ± 0.3789.30+3.04
Qwen3-VL-8BPOPEv290.00 ± 1.4487.05–91.2990.46 ± 1.1486.20+3.80
Qwen3-VL-8BCausal-HalBench87.26 ± 0.2486.72–87.5184.80 ± 0.2286.58+0.68
Qwen3-VL-8BRePOPE94.67 ± 0.2994.11–94.9193.82 ± 0.2793.50+1.17
DeepSeek-VL2-smallPOPEv2———80.50—
DeepSeek-VL2-smallCausal-HalBench———88.87—
DeepSeek-VL2-smallRePOPE———92.97—

Note. Accuracy and F1-Yes are percentages. Statistics summarize ten calibration subsets and reuse the N = 16 runs in Table R3.1. SD describes variation across calibration subsets. DeepSeek results are pending.

AuthorsOpenReview ↗

Response to Reviewer 6d1U - Weakness 1 and Question 1 - Calibration stability and cue-removal controls

Weakness 1

The intervention direction is estimated from only 12–17 image pairs per model. Activation changes may reflect occlusion artifacts or contextual differences rather than semantic cue contributions. Its stability across calibration subsets and cue-removal methods remains unclear.

Question 1

How stable are the learned intervention directions when the calibration pairs are resampled or replaced? Could the authors report sensitivity analyses using different calibration subsets and alternative cue-removal procedures?

Response

  1. Cue-category coverage. Keeping N=16N=16, we also varied the number of cue categories KK, allocating pairs as evenly as possible across categories (Table R3.3). Broader coverage generally reduces variability for InternVL3, with modest changes in mean accuracy. For example, its POPEv2 SD decreases from 2.67 at K=2K=2 to 1.69 at K=16K=16. Qwen3 has similar mean performance across the available K=2,4,8K=2,4,8 settings. This experiment evaluates sensitivity to category coverage within the calibration pool.

  2. Alternative cue-removal methods. We compared LaMa and Telea using the same saved cue masks and the original model-specific calibration sets. For each editing method, we refitted the direction and reference center while keeping the intervention window and strength fixed (Table R3.4). Both methods produce positive accuracy gains in all nine model–dataset combinations, with larger gains from LaMa in this comparison. We also evaluated a hidden-state sham-adjusted direction, whose accuracy gains are equal to or slightly above the original LaMa configuration.

  3. Area-matched removal controls. Section 4.2, Figure 3(a), and Appendix C.1–C.2 compare each diagnostic example with eight same-image area-matched control removals. The mean sham-adjusted cue effect is positive in all three models. These controls assess cue-specific support relative to generic removal effects. The LaMa–Telea comparison in Table R3.4 separately evaluates downstream performance when the cue-removal method changes.

Table R3.3. Cue-category coverage at fixed N = 16.

ModelDatasetK = 2K = 4K = 8K = 16
InternVL3-8BPOPEv279.93 ± 2.6779.93 ± 2.1680.54 ± 1.6080.56 ± 1.69
InternVL3-8BCausal-HalBench85.46 ± 1.0885.33 ± 0.8485.50 ± 0.6185.61 ± 0.65
InternVL3-8BRePOPE91.92 ± 0.8991.76 ± 0.7191.93 ± 0.5892.00 ± 0.51
Qwen3-VL-8BPOPEv290.45 ± 0.4690.22 ± 1.0690.16 ± 0.83—
Qwen3-VL-8BCausal-HalBench87.38 ± 0.1287.32 ± 0.1587.23 ± 0.13—
Qwen3-VL-8BRePOPE94.87 ± 0.1594.80 ± 0.1994.83 ± 0.17—
DeepSeek-VL2-smallPOPEv2————
DeepSeek-VL2-smallCausal-HalBench————
DeepSeek-VL2-smallRePOPE————

Note. Accuracy (%), mean ± SD. K denotes the number of cue categories represented among the 16 pairs, with approximately balanced allocation. “—” denotes an unavailable result. This table uses category-balanced sampling, whereas Table R3.2 uses unrestricted sampling at N = 16.

Table R3.4. Editing-backend and hidden-state sham-adjustment comparisons.

ModelDatasetLaMaTeleaSham-adjusted
InternVL3-8BPOPEv2+8.26+2.90+8.26
InternVL3-8BCausal-HalBench+3.87+2.13+4.05
InternVL3-8BRePOPE+3.53+1.99+3.55
Qwen3-VL-8BPOPEv2+4.69+2.68+4.69
Qwen3-VL-8BCausal-HalBench+0.93+0.85+1.05
Qwen3-VL-8BRePOPE+1.44+0.94+1.44
DeepSeek-VL2-smallPOPEv2+1.79+0.22+2.23
DeepSeek-VL2-smallCausal-HalBench+0.91+0.48+0.97
DeepSeek-VL2-smallRePOPE+0.60+0.35+0.60

Note. Accuracy improvement over the corresponding unmodified baseline, in percentage points. These comparisons use the original model-specific calibration sets and fixed image-group test folds: 448 examples for POPEv2, 4,836 for Causal-HalBench, and 4,026 for RePOPE. The intervention window and strength remain fixed. The sham-adjusted column reports a hidden-state contrast adjusted for the mean sham displacement. These are separate experiments from the expanded-pool sensitivity analyses in Tables R3.1–R3.3.

Reviewer_LWf6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at the problem of hallucinations in vision language models, especially in the presence of spurious correlations (for example: saying that a laptop is present in the image when only a keyboard is present). They propose a method where internal states contributing to the spurious cue issue are identified and then used to suppress "Yes" output at test time. The method is compared with many interpretability baselines and some improvements are shown.

Critical strengths

The idea of using internal states to fix issues in the model seems interesting.

Critical weaknesses

Many methodological issues are present throughout. For example:

  1. Some very simple baselines are excluded, such are prompt engineering ( for instance, "only say yes if the whole object is in view" or "If you are not certain the object is visible, answer No." ), adding a negative bias to the "yes" token, describe the object, then answer, etc. Other baselines like random direction are extremely weak and uninformative

  2. It is not clear if the extracted direction is pointed towards the spurious correlation or a generic "Yes". If the question is rephrased as "Answer no if there is a laptop in the image", does the method still yield an improvement? If no, it seems a bit overfit to this particular case and prompt.

  3. Baselines like VTI and REVIS seem untuned, for example in appendix D.4 it is mentioned that the model answers are basically constant with these methods.

I also struggled with the dense phrasing of the paper, with non-standard terms like "sham edit" being introduced suddenly.

Questions for authors

Please fix the weaknesses listed above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer LWf6 - Weaknesses 1 and 2 - Simple prompt baselines, confounder support, and the generic Yes direction

We sincerely thank you for taking the time to engage so thoughtfully with our work. Your candid feedback has helped us reconsider important methodological choices and communicate our findings more clearly.

Weakness 1

Some very simple baselines are excluded, such are prompt engineering ( for instance, "only say yes if the whole object is in view" or "If you are not certain the object is visible, answer No." ), adding a negative bias to the "yes" token, describe the object, then answer, etc. Other baselines like random direction are extremely weak and uninformative

Weakness 2

It is not clear if the extracted direction is pointed towards the spurious correlation or a generic "Yes". If the question is rephrased as "Answer no if there is a laptop in the image", does the method still yield an improvement? If no, it seems a bit overfit to this particular case and prompt.

Weakness 3

Baselines like VTI and REVIS seem untuned, for example in appendix D.4 it is mentioned that the model answers are basically constant with these methods.

Comment

I also struggled with the dense phrasing of the paper, with non-standard terms like "sham edit" being introduced suddenly.

Response

1.1. Prompt baselines. We evaluated two fixed instructions appended to the object-existence question:

  • Visible-evidence prompt: “Answer Yes only if the queried object is actually visible; do not infer it from surrounding objects. Reply Yes or No only.”
  • Uncertainty-to-No prompt: “If you are uncertain whether the object is present, answer No. Reply Yes or No only.”

1.2. Results. On InternVL3-8B, EvidenceFlow achieves higher accuracy and F1-Yes than both prompts on POPEv2, Causal-HalBench, and RePOPE (Table R2.1). Its accuracy exceeds the visible-evidence prompt by 2.23, 0.99, and 1.82 percentage points, respectively. The current round of supplementary prompt evaluations for Qwen3-VL-8B and DeepSeek-VL2-small is still in progress.

Table R2.1. EvidenceFlow and two prompt baselines on InternVL3-8B.

DatasetTest examplesBaselineVisible-evidence promptUncertainty-to-No promptEvidenceFlow
POPEv244873.88 / 79.1479.91 / 83.2178.35 / 82.0782.14 / 84.62
Causal-HalBench4,83682.40 / 81.3585.28 / 83.7984.84 / 83.4186.27 / 83.96
RePOPE4,02689.37 / 88.7191.08 / 90.2790.91 / 90.1592.90 / 91.96

Note. Each performance cell reports accuracy / F1-Yes (%). All methods are evaluated on the same image-grouped test folds (split seed 20261106). Baseline denotes the model with the original question and no intervention. These are test-fold results; manuscript Table 1 reports full-benchmark results.

2.1. Counterfactual support and the Yes margin. The direction is estimated from paired images that hold the question and target absence fixed while changing a visible cue. Section 4.2, Figure 3(a) measures the resulting Yes-margin change relative to area-matched control removals. Since a confounder supplies misleading affirmative support for an absent target, its downstream effect can include a component along a generic Yes direction. A connection to the cue and an effect on the Yes decision are therefore compatible.

2.2. Overlap with a generic Yes direction. For a unit-norm generic Yes direction dyd_y, the learned direction can be written as

dc=⟨dc,dy⟩dy+d⊥.d_c=\langle d_c,d_y\rangle d_y+d_\perp.

If the aligned component dominates the output change, generic Yes suppression can produce similar behavior. Our norm-matched Yes-axis control gives similar correction–retention outcomes in several settings while reusing EvidenceFlow's sample-dependent edit magnitudes. This is consistent with overlap in their effects on the Yes decision. EvidenceFlow computes each edit from the input's hidden state relative to the counterfactual reference center, so the correction magnitude varies across samples.

2.3. Reversed answer mapping. In the existing reversed-mapping experiment, keeping the direction, reference center, intervention window, and strength frozen left accuracy unchanged relative to the reversed-prompt baseline. The counterfactual-support analysis and the Yes-axis comparison above concern the standard Yes/No answer mapping.

3.1.We are conducting supplementary VTI and REVIS experiments with method-specific intervention-strength calibration under a common evaluation protocol. These experiments are in progress, and we will add the results when complete.

4.1.We will define “area-matched control removal” at its first occurrence, before introducing the shorthand “sham edit”: it removes a control region matched in area to the cue-removal region, providing a reference for the effect of image editing. We will also shorten dense sentences and describe cue selection, counterfactual calibration, and inference-time intervention in separate steps.

AuthorsOpenReview ↗

Response to Reviewer LWf6 - Weaknesses 1 - Simple prompt baselines

Following your suggestion, we conducted further experiments to evaluate EvidenceFlow in combination with simple prompt-based strategies.

The labels Visible and Uncertain→No refer to our implementations of the two types of prompts you suggested:

  • Visible addresses the visible-evidence requirement in your suggestion, “only say yes if the whole object is in view.” Our implementation uses: “Answer Yes only if the queried object is actually visible; do not infer it from surrounding objects. Reply Yes or No only.”
  • Uncertain→No corresponds to your suggestion, “If you are not certain the object is visible, answer No.” Our implementation uses: “If you are uncertain whether the object is present, answer No. Reply Yes or No only.”

We recalibrated EvidenceFlow under each prompt and evaluated the resulting combinations on InternVL3-8B across POPEv2, RePOPE, and Causal-HalBench. In the new experiments, both combinations improve accuracy and F1-Yes over their corresponding prompt-only baselines on all three datasets.

DatasetOriginal EFVisibleVisible + recalibrated EF†Uncertain→NoUncertain→No + recalibrated EF†
POPEv282.14 / 84.6279.91 / 83.2188.17 / 89.1678.35 / 82.0786.83 / 88.27
RePOPE92.90 / 91.9691.08 / 90.2793.47 / 92.3590.91 / 90.1593.32 / 92.36
Causal-HalBench86.27 / 83.9685.28 / 83.7988.56 / 86.1684.84 / 83.4187.70 / 85.38

Note. Each cell reports accuracy / F1-Yes (%). † The two combined configurations come from the new experimental run; the standalone columns retain the previously reported results. The improvement claims above use the corresponding prompt-only baselines within the new run.

These results suggest that conservative prompting and EvidenceFlow can complement each other. The prompts encourage the model to rely on visible evidence and answer conservatively, while EvidenceFlow applies a sample-dependent intervention to the hidden states under the corresponding prompt. The additional improvements across all three datasets support the usefulness of combining EvidenceFlow with these simple prompting strategies.

Reviewer_yG2DOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

EvidenceFlow is a training-free, inference-time method designed to reduce object hallucination in Vision-Language Models (VLMs). VLMs often hallucinate absent objects because related items are present. EvidenceFlow uses a small set of calibration images to isolate the specific internal state ("confounder direction") where this misleading association occurs. During inference, if the model attempts to answer "Yes" to an object's presence, EvidenceFlow subtracts this misleading component from the model's prediction state, removing the faulty support. It significantly improves accuracy by successfully correcting false affirmatives while leaving correct answers and visual representations intact.

Critical strengths
  1. EvidenceFlow is lightweight and practical for deployment. By estimating a global confounder direction offline using just 12 image pairs, the method avoids the need for computationally expensive per-example image editing or counterfactual generation at test time.
  2. Unlike many mitigation methods that indiscriminately dampen visual influence, EvidenceFlow is highly targeted with demonstrated significant asymmetry in their intervention, flipping false affirmatives about 11 times as often as true affirmatives.
Critical weaknesses
  1. The first concern is in the way the confounder direction (dld_l) is computed. By averaging latent displacements across all calibration pairs including those where removing the cue did not alter the model's answer (Sec. 3.1), the method mixes causal and non-causal data. This raises the concern that dld_l might simply be capturing generic visual features of the removed object rather than strictly isolating the underlying hallucination mechanism. I suggest a confounder retention test: after the intervention fixes a false affirmative (e.g., "laptop"), probe that same edited state for the confounder itself (e.g., "keyboard"). A "No" would indicate the method is destructively erasing the cue's visual features.

  2. The second concern is the absence of analysis regarding the calibration set (C\mathcal{C}). The method relies on a very small, fixed set of just 12 to 17 image pairs to estimate the global confounder direction, yet provides no sensitivity analysis on how the (1) composition, (2) diversity, (3) size being calibrated of this set impacts downstream performance. This leaves it unclear whether the estimated direction is robust or heavily biased by the specific sample chosen.

Questions for authors
  1. Which tools or models were used to create the counterfactual images? Please clarify the pipeline for reproducibility, as editing artifacts could affect the VLM's performance.

  2. How are confounder cues (cc) paired with target object queries (qoq_o)? Is this based on dataset co-occurrence, language priors, or manual selection? Clarifying this would help explain how the method scales and how the evaluation sets were built.

3 comments
AuthorsOpenReview ↗

Response to Reviewer yG2D - Weaknesses 1 and 2 - Counterfactual calibration, cue retention, and calibration-set sensitivity

We sincerely appreciate the care and thought you have put into reviewing our work. Your insightful comments have encouraged us to examine our method more closely and have been valuable in helping us strengthen the paper.

Weakness 1

The first concern is in the way the confounder direction is computed. By averaging latent displacements across all calibration pairs including those where removing the cue did not alter the model's answer (Sec. 3.1), the method mixes causal and non-causal data. This raises the concern that might simply be capturing generic visual features of the removed object rather than strictly isolating the underlying hallucination mechanism. I suggest a confounder retention test: after the intervention fixes a false affirmative (e.g., "laptop"), probe that same edited state for the confounder itself (e.g., "keyboard"). A "No" would indicate the method is destructively erasing the cue's visual features.

Weakness 2

The second concern is the absence of analysis regarding the calibration set. The method relies on a very small, fixed set of just 12 to 17 image pairs to estimate the global confounder direction, yet provides no sensitivity analysis on how the (1) composition, (2) diversity, (3) size being calibrated of this set impacts downstream performance. This leaves it unclear whether the estimated direction is robust or heavily biased by the specific sample chosen.

Response

1.1. Counterfactual support. Section 3.1 estimates the direction from paired hidden-state displacements while holding the question and the target's absence fixed. As explained in Appendix A, “Why the cue condition is graded,” a cue can contribute affirmative support even when removing it leaves the answer above the Yes threshold. The cue-support analysis therefore measures the margin change relative to area-matched sham removals, capturing both changes in support and answer reversals.

1.2. Cue retention. We evaluated cue judgments by replaying the target-query edit vectors, layer by layer, during a query about the visible cue in the same image. Among examples excluded from direction fitting where the target error was corrected and the cue was initially recognized, cue retention is 100% for InternVL3-8B, 100% for DeepSeek-VL2-small, and 80% for Qwen3-VL-8B. We are considering expanding the calibration set for further experiments on cue retention.

2.1. Calibration size. We evaluated ten calibration-set sizes; Table R1.1 shows three representative settings, N=4,16,128N=4,16,128. Across the six completed model–dataset combinations, the mean accuracy changes by at most 0.38 percentage points between N=4N=4 and N=128N=128. Increasing the number of pairs mainly reduces sampling variability, without a consistent monotonic improvement in mean accuracy. For example, InternVL3 on POPEv2 changes from 81.27 ± 1.74 to 81.34 ± 0.19. These results indicate that useful downstream performance can be obtained with a small calibration set, while larger sets reduce variation across calibration subsets. The DeepSeek calibration experiments are still running and are marked with “—”.

Table R1.1. Calibration-size sensitivity on three models and three datasets.

ModelDatasetN = 4N = 16N = 128
InternVL3-8BPOPEv281.27 ± 1.7481.34 ± 0.7781.34 ± 0.19
InternVL3-8BCausal-HalBench86.04 ± 0.6686.07 ± 0.4185.88 ± 0.15
InternVL3-8BRePOPE92.31 ± 0.7292.34 ± 0.3992.24 ± 0.13
Qwen3-VL-8BPOPEv289.98 ± 1.2290.00 ± 1.4490.36 ± 0.14
Qwen3-VL-8BCausal-HalBench87.32 ± 0.2387.26 ± 0.2487.34 ± 0.05
Qwen3-VL-8BRePOPE94.78 ± 0.2494.67 ± 0.2995.00 ± 0.04
DeepSeek-VL2-smallPOPEv2———
DeepSeek-VL2-smallCausal-HalBench———
DeepSeek-VL2-smallRePOPE———

Note. Accuracy (%), mean ± SD across calibration-subset repeats. Three representative sizes are shown from the ten tested settings. DeepSeek results are pending.

AuthorsOpenReview ↗

Response to Reviewer yG2D - Weakness 2 - Calibration size, composition, and diversity

Response

2.2. Calibration composition. At fixed N=16N=16, we evaluated ten randomly sampled calibration subsets (Table R1.2). Every repeat exceeds the corresponding manuscript baseline in all six completed combinations. Using the manuscript full-benchmark baselines as a common reference, the mean accuracy differences range from +0.68 to +8.54 percentage points. Variation across subsets is dataset-dependent: Qwen3 on POPEv2 ranges from 87.05 to 91.29, whereas its Causal-HalBench range is 86.72–87.51. Table R1.2 summarizes the same ten runs used for the N=16N=16 entries in Table R1.1.

2.3. Cue-category coverage. Keeping N=16N=16, we also varied the number of cue categories KK, allocating pairs as evenly as possible across categories (Table R1.3). Broader coverage generally reduces variability for InternVL3, with modest changes in mean accuracy. For example, its POPEv2 SD decreases from 2.67 at K=2K=2 to 1.69 at K=16K=16. Qwen3 has similar mean performance across the available K=2,4,8K=2,4,8 settings. This experiment evaluates sensitivity to category coverage within the calibration pool.

2.4. Cue-removal controls. Section 4.2, Figure 3(a), and Appendix C.1–C.2 report area-matched control removals. Each diagnostic example is compared with eight same-image controls, and the mean sham-adjusted cue effect is positive in all three models. These controls assess cue-specific support relative to generic removal effects, while Tables R1.1–R1.3 quantify downstream performance across calibration sizes and compositions.

Table R1.2. Calibration-subset sensitivity at fixed N = 16.

ModelDatasetAccuracy, mean ± SDAccuracy rangeF1-Yes, mean ± SDManuscript baseline (%)Mean accuracy − manuscript baseline (pp)
InternVL3-8BPOPEv281.34 ± 0.7779.69–82.1484.08 ± 0.5172.80+8.54
InternVL3-8BCausal-HalBench86.07 ± 0.4185.09–86.4683.94 ± 0.2982.55+3.52
InternVL3-8BRePOPE92.34 ± 0.3991.43–92.6291.43 ± 0.3789.30+3.04
Qwen3-VL-8BPOPEv290.00 ± 1.4487.05–91.2990.46 ± 1.1486.20+3.80
Qwen3-VL-8BCausal-HalBench87.26 ± 0.2486.72–87.5184.80 ± 0.2286.58+0.68
Qwen3-VL-8BRePOPE94.67 ± 0.2994.11–94.9193.82 ± 0.2793.50+1.17
DeepSeek-VL2-smallPOPEv2———80.50—
DeepSeek-VL2-smallCausal-HalBench———88.87—
DeepSeek-VL2-smallRePOPE———92.97—

Note. Accuracy and F1-Yes are percentages. Statistics summarize ten calibration subsets and reuse the N = 16 runs in Table R1.1. SD describes variation across calibration subsets. DeepSeek results are pending.

Table R1.3. Cue-category coverage at fixed N = 16.

ModelDatasetK = 2K = 4K = 8K = 16
InternVL3-8BPOPEv279.93 ± 2.6779.93 ± 2.1680.54 ± 1.6080.56 ± 1.69
InternVL3-8BCausal-HalBench85.46 ± 1.0885.33 ± 0.8485.50 ± 0.6185.61 ± 0.65
InternVL3-8BRePOPE91.92 ± 0.8991.76 ± 0.7191.93 ± 0.5892.00 ± 0.51
Qwen3-VL-8BPOPEv290.45 ± 0.4690.22 ± 1.0690.16 ± 0.83—
Qwen3-VL-8BCausal-HalBench87.38 ± 0.1287.32 ± 0.1587.23 ± 0.13—
Qwen3-VL-8BRePOPE94.87 ± 0.1594.80 ± 0.1994.83 ± 0.17—
DeepSeek-VL2-smallPOPEv2————
DeepSeek-VL2-smallCausal-HalBench————
DeepSeek-VL2-smallRePOPE————

Note. Accuracy (%), mean ± SD. K denotes the number of cue categories represented among the 16 pairs, with approximately balanced allocation. “—” denotes an unavailable result. This table uses category-balanced sampling, whereas Table R1.2 uses unrestricted sampling at N = 16.

AuthorsOpenReview ↗

Response to Reviewer yG2D - Questions 1 and 2 - Counterfactual image generation, editing controls, and cue–target pairing

Question 1

Which tools or models were used to create the counterfactual images? Please clarify the pipeline for reproducibility, as editing artifacts could affect the VLM's performance.

Question 2

How are confounder cues () paired with target object queries ()? Is this based on dataset co-occurrence, language priors, or manual selection? Clarifying this would help explain how the method scales and how the evaluation sets were built.

Response

1.1. Counterfactual image generation. We generate cue-removed images with LaMa using the Big-LaMa weights. For the selected cue category, we take the union of its COCO instance-segmentation masks, dilate it by a 7-pixel radius, and inpaint the masked region. We retain candidate masks whose dilated area is 0.2%–35% of the image area. Following Appendix C.1, sham controls remove a same-image non-target object of matching area when available; otherwise, they use a translated equal-area region.

1.2. Editing controls. We compared the original LaMa calibration with Telea using the same saved cue masks, refitting the direction and reference center while keeping the intervention window and strength fixed. We also evaluated a hidden-state sham-adjusted direction (Table R1.4). Telea produces positive accuracy gains in all nine model–dataset combinations, while LaMa produces larger gains in this comparison. The sham-adjusted variant preserves the gains, with accuracy improvements equal to or slightly above the original configuration. Together with the area-matched controls in Section 4.2, these comparisons characterize the effect under alternative editing and calibration procedures.

Table R1.4. Editing-backend and hidden-state sham-adjustment comparisons.

ModelDatasetLaMaTeleaSham-adjusted
InternVL3-8BPOPEv2+8.26+2.90+8.26
InternVL3-8BCausal-HalBench+3.87+2.13+4.05
InternVL3-8BRePOPE+3.53+1.99+3.55
Qwen3-VL-8BPOPEv2+4.69+2.68+4.69
Qwen3-VL-8BCausal-HalBench+0.93+0.85+1.05
Qwen3-VL-8BRePOPE+1.44+0.94+1.44
DeepSeek-VL2-smallPOPEv2+1.79+0.22+2.23
DeepSeek-VL2-smallCausal-HalBench+0.91+0.48+0.97
DeepSeek-VL2-smallRePOPE+0.60+0.35+0.60

Note. Accuracy improvement over the corresponding unmodified baseline, in percentage points. These comparisons use the original model-specific calibration sets and fixed image-group test folds: 448 examples for POPEv2, 4,836 for Causal-HalBench, and 4,026 for RePOPE. The intervention window and strength remain fixed. The sham-adjusted column reports a hidden-state contrast adjusted for the mean sham displacement. These are separate experiments from the expanded-pool sensitivity analyses in Tables R1.1–R1.3.

2.1. Cue–target pairing. During offline diagnosis and calibration, we construct queries about target objects that are absent from the images and select candidate cues from visible object categories annotated in COCO for the same images. Before image editing, we select the highest-scoring cue among candidates satisfying the mask-area conditions, using a smoothed image-level co-occurrence score computed from COCO training annotations:

s(c,o)=log⁡(n(c,o)+1)(Ntrain+1)(n(c)+1)(n(o)+1)s(c,o)=\log\frac{(n(c,o)+1)(N_{\mathrm{train}}+1)}{(n(c)+1)(n(o)+1)},

where NtrainN_{\mathrm{train}} is the number of training images, n(c,o)n(c,o) counts images containing both categories, and n(c)n(c) and n(o)n(o) are their marginal image counts. Ties are resolved by the total annotated instance area of the cue category and its name. This gives a deterministic, annotation-based cue-selection procedure.

2.2. Model-specific calibration and deployment. For each selected cue, we measure the change in the target's Yes margin under cue removal relative to sham editing. At the initial construction stage, the model-specific calibration lists were selected using a Yes-to-No change under cue removal with Yes retained under a single sham edit. The eight-sham analysis reported in Section 4.2 and Appendix C.1–C.2 subsequently evaluates cue support in the diagnostic and calibration sets. Cue selection and image editing are performed offline. EvidenceFlow-Core then uses the original image, question, and frozen intervention parameters at inference, without requiring per-test-image cue selection or counterfactual generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.