acceptodds
Withdrawn from ICLR 2027

Verify Before Edit: Counterfactual Visual Evidence Verification for MLLM Hallucination Mitigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) often produce plausible responses unsupported by the visual input. Existing training-free mitigation methods typically suppress language priors or amplify visual representations, but rarely verify whether the targeted visual evidence is relevant to the current decision. We introduce **VBEdit**, a training-free **"verify before edit"** framework for correcting visually unsupported predictions in binary visual queries. Before editing the first-step answer representation, VBEdit evaluates the decision relevance of answer-conditioned visual evidence. It computes signed visual-token attributions with respect to the semantic margin between competing answer candidates, separating supporting from opposing evidence. It then counterfactually perturbs the most supportive tokens and measures the resulting margin change as a sample-specific test of decision relevance. This counterfactual response gates a contrastive edit of the answer representation, without requiring external detectors, reference responses, or auxiliary supervision. Extensive experiments across diverse MLLMs and benchmarks show that VBEdit improves over matched Vanilla and representative methods. VBEdit also improves accuracy and F1 on AMBER-D at both LLaVA scales and raises HOPE accuracy by and percentage points on LLaVA-1.5-7B and 13B, respectively, without HOPE-specific parameter tuning. Controlled ablations isolate the benefits of signed attribution and counterfactual verification beyond the number of edited examples alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Hallucination Mitigation, Counterfactual Verification, Representation Editing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TrhiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VBEDIT, a training-free method to reduce hallucinations in MLLMs. The method first computes the margin between different answer candidates and uses the gradient of this margin to select visual tokens that support or oppose the current answer. Then, it perturbs the most supportive visual tokens by replacing them with the mean visual representation and measures how much the margin changes. Based on this change, the method decides whether the answer needs to be edited or not. If the counterfactual response is larger than a threshold and the predicted answer is "Yes", the method constructs a vector using the positive and negative visual-token centroids and uses it to reduce the margin of the current prediction. The method is evaluated on several MLLMs and benchmarks, including POPE, MME, AMBER-D, and HOPE, and shows improvements over the original models and other hallucination mitigation methods. The paper also provides ablation studies on different components of the method.

Critical strengths

The method addresses an important problem, and it is training-free. The experiments are conducted across multiple benchmarks and models, with adequate ablations on their hyperparameters.

Critical weaknesses
  1. The main motivation of the paper is that, in the proposed method, the relevance of the visual evidence is verified, and then the model's answer only changes when it is not visually supported. But I cannot see how the proposed method relates to this motivation. The proposed counterfactual verification measures the sensitivity of the prediction to perturbations of selected visual tokens, not whether it needs to be corrected or whether it is supported by the image.

  2. The correction method is not sufficiently justified. It is not clear why reducing the margin should reduce hallucination. There can be multiple cases in which the model is confident about its answer, and the answer is also correct. Also, in the paper, only "Yes" answers are altered, but it is not clear why there should be a difference between the answers.

  3. The method is only evaluated on questions with "Yes" and "No" answers, and it is not clear whether it transfers to other types of questions. It is not clear whether reducing the margin works for other types of answers as well, or whether it can be a reliable detection method for hallucinated tokens.

  4. A vector is constructed based on the positive and negative centroids, but again, it is aligned with the direction of the margin gradient. So it is not justified why that particular vector is needed. Isn't it possible to use the gradient of the margin directly? The ablation study only excludes the negative centroid, but other update strategies need to be ablated to justify the role of this vector.

Questions for authors
  1. The method has multiple hyperparameters, but it is not clear how they were tuned. Please clarify this. If they were tuned on the POPE benchmark, then comparing the results with other baselines taken from their original papers may not be fair.

  2. Why should reducing the margin be the strategy for reducing hallucinations? How is it directly connected to hallucination reduction, and why is it limited to "Yes" answers?

Reviewer_Eq3LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VBEDIT, a training-free framework for mitigating visual hallucination in MLLM. VBEDIT aims to reduce harmful interventions on well-grounded predictions. More specifically, VBEDIT employs answer-conditioned signed visual-token attribution to distinguish supporting from opposing evidence, followed by counterfactual evidence verification to measure the decision relevance of the selected tokens. Based on the counterfactual response, a reliability gate selectively applies contrastive editing to the first-step answer representation. The authors evaluate the method across four MLLMs and four benchmarks, demonstrating its effectiveness through performance comparisons and ablation studies.

Critical strengths
  1. The proposed method requires no additional training, external detectors, or auxiliary supervision, making it easy to apply to pretrained MLLMs.
  2. The paper evaluates the proposed method with four MLLMs on four benchmarks. The results show that the method has certain advantages.
Critical weaknesses
  1. Tables 1 and 5 show that VBEDIT achieves different levels of improvement across models. And table 7 shows that Relation Accuracy on AMBER-D drops from 69.0% to 59.4% with LLaVA-1.5-7B, but the authors provide limited analysis of the reasons about above issue.
  2. In fact, VBEDIT increases inference latency to 1.81× that of matched Vanilla as shown in Table 14, which may limit its practicality in latency-sensitive applications.
Questions for authors
  1. Could the authors evaluate the reliability of the counterfactual response across different models and benchmarks?
  2. What accounts for the much smaller improvements on InternVL3.5-8B compared with the LLaVA models?
Reviewer_gbqsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VBEDIT, a training-free inference-time intervention method for mitigating visual hallucinations in multimodal large language models. It computes signed attribution scores from answer-margin gradients to identify visual tokens supporting or opposing the current answer. It then replaces selected tokens with the mean visual representation and measures the resulting margin changes to assess counterfactual sensitivity. These signals determine whether to edit the hidden state at the first answer position along a contrastive visual direction. Experiments on LLaVA-1.5, Qwen3-VL, and InternVL3.5 report improvements on POPE, MME, AMBER-D, and HOPE, with ablations examining the main components.

Critical strengths
  • Clear motivation for selective visual intervention. The paper recognizes that token importance or sensitivity to visual perturbations does not necessarily indicate hallucination. VBEDIT therefore assesses the current decision’s dependence on visual evidence before intervening, addressing the risk of damaging correct predictions through indiscriminate editing.

  • Complete framework with targeted empirical analysis. The experiments examine replacement targets, attribution strategies, reliability gating, and editing directions. In particular, the matched random-gating comparison controls the intervention budget. VBEDIT achieves a net increase of 109 correct predictions, compared with an average of 22.4 for matched random gating, providing evidence that its sample selection is more effective than random selection.

Critical weaknesses
  • Counterfactual sensitivity does not establish answer correctness. VBEDIT uses the counterfactual response signal R_cf to guide intervention. However, correct answers can depend strongly on critical visual evidence, while incorrect answers may be insensitive to the selected perturbations. A high R_cf therefore does not necessarily indicate hallucination. Although the ablations support the signal’s usefulness, its ability to reliably distinguish incorrect from correct answers remains insufficiently established. Table 3 reports 262 false positives corrected to true negatives, but also 153 true positives changed to false negatives, indicating a substantial risk of harmful intervention.

  • Limited applicability beyond predefined answer spaces. VBEDIT constructs semantic margins using a predefined candidate answer set. This is suitable for binary yes/no tasks, but extending it to multiple-choice questions or open-ended generation requires additional decisions about candidate construction and margin definition. Since the paper leaves these extensions to future work, the current evidence supports a narrower scope than general MLLM hallucination mitigation.

  • Unclear accuracy–efficiency trade-off. On LLaVA-1.5-7B, VBEDIT improves POPE accuracy from 86.42% to 87.64% and F1 from 86.96% to 87.58%, while increasing average inference latency from 223.66 ms to 405.78 ms, approximately 1.81 times the vanilla latency. Throughput decreases by approximately 44.89%. Without comparisons against alternative inference strategies under comparable computational budgets, it remains unclear whether these gains justify the additional cost of gradient attribution and counterfactual verification.

  • Insufficient clarification of the counterfactual diagnostics. Table 4 reports 100% First-order Agreement for signed attribution and an AUC_FP>TP of 83.16%. These results concern different properties and should be interpreted separately. The computation conditions for First-order Agreement need clearer explanation, as agreement in local directional prediction does not by itself demonstrate reliable hallucination detection. A clearer distinction between gradient-direction agreement, counterfactual sensitivity, and answer correctness would strengthen the interpretation.

Questions for authors
  1. Reliability of counterfactual sensitivity. How well does R_cf distinguish incorrect from correct predictions across models and benchmarks? What characterizes the correct predictions that are harmed by intervention?

  2. Scope of applicability. What assumptions about the candidate answer set are required by the current method, and which reported results establish applicability beyond binary yes/no decisions?

  3. Accuracy–efficiency trade-off. How does VBEDIT compare with alternative inference strategies under a comparable computational budget?

  4. Interpretation of the diagnostics. How is First-order Agreement computed, and does it evaluate finite counterfactual replacements or infinitesimal perturbations? How should this result be distinguished from the evidence that R_cf identifies incorrect answers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.