acceptodds
Under review as a conference paper at ICLR 2027

Not Blind but Silenced: Preserving Visual Evidence through Counter-Context Validation

Abstract

Multimodal large language models (MLLMs) can produce incorrect answers even when intermediate representations robustly favor the factual hypothesis. We characterize this failure through layer-wise evidence competition and identify strict silencing, in which established intermediate factual dominance is overturned before the final prediction. To preserve the influence of useful visual evidence, we propose Adaptive Counter-context Evidence (ACE), a training-free decoding framework that follows Challenge, Validate, Locate, and Preserve. ACE constructs a controlled counter-context to estimate contribution reliability, locates an evidence anchor, and selectively amplifies original-image contributions throughout the decoder suffix. The counter-context is discarded before answer generation, while reliability gates remain fixed and current contributions evolve with each decoding step. Across four MLLM backbones, ACE improves POPE-Adversarial accuracy by 6.37–8.97 percentage points and reduces CHAIR by 3.9–5.0 points, while maintaining MM-Vet performance within 0.1 point of greedy decoding. Functional diagnostics demonstrate stronger evidence identification, while ACE achieves four times as many strict-silencing rescues as uniform-suffix preservation on the fixed Qwen3 evaluation cohort.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, VLM, Hallucination, Inference-Time Hallucination Mitigation, Counter-Context Intervention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WKaLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates visual hallucination in multimodal large language models (MLLMs) from the perspective of layer-wise evidence competition. The authors identify a failure mode termed strict silencing, where intermediate layers favor the factual answer but the final prediction favors an incorrect alternative. To address this issue, the paper proposes Adaptive Counter-context Evidence (ACE), a training-free decoding framework that constructs controlled counter-context probes, estimates visual contribution reliability, locates an evidence anchor, and selectively amplifies visual contributions throughout the decoder suffix. Experiments across four MLLM backbones show substantial improvements on POPE, CHAIR, and FREAK while largely preserving MM-Vet performance. Additional functional diagnostics and ablation studies support the effectiveness of reliability-based evidence selection and sustained preservation.

Critical strengths
  1. Interesting motivation and evidence analysis. The paper provides a useful perspective on visual hallucination by examining how intermediate factual dominance can be lost during decoding. The strict-silencing formulation and layer-wise diagnostics help characterize this failure mode more precisely.

  2. Strong empirical results with extensive validation. ACE consistently improves hallucination-related benchmarks across four MLLM backbones. The functional evidence diagnostics, preservation ablations, and amplitude-matched comparisons provide meaningful support for the proposed reliability-based intervention.

Critical weaknesses
  1. The connection between strict silencing and the overall performance gains remains incomplete. Although strict silencing motivates the proposed framework, a substantial portion of ACE's corrections occur outside the strict-silencing cohort. For example, Table 14 shows that 120 of the 191 corrected errors on Qwen3-VL are outside this cohort. This raises questions about whether the observed improvements are primarily attributable to recovering silenced evidence.

  2. The relationship between reliability estimation and factual improvement remains insufficiently established. ACE identifies reliable visual contributions based on counter-context stability and visual specificity, but these properties do not necessarily imply factual correctness. Although the AFE diagnostics support the effectiveness of the proposed ranking, single-unit deletion effects may not directly reflect the outcomes of sustained amplification across multiple decoder layers. Therefore, it remains unclear how reliably the estimated scores distinguish beneficial visual contributions from stable but potentially misleading ones.

Questions for authors
  1. Regarding the relationship between strict silencing and overall improvements. Table 14 shows that a substantial fraction of corrected errors fall outside the strict-silencing cohort. Could the authors further characterize these cases and clarify whether ACE primarily addresses strict silencing or a broader visual-evidence underutilization problem?

  2. Regarding reliability estimation and functional evidence. The proposed reliability scores are validated using single-unit ablation effects, whereas ACE applies sustained amplification to multiple visual contributions. How well do these scores predict the actual effects of amplification? In particular, what prevents ACE from amplifying stable but factually incorrect visual contributions?

  3. Regarding prediction stability and generalization. Table 14 reports zero correct-to-incorrect transitions across all four POPE-Adversarial cohorts. Could the authors provide further analysis of this result and investigate whether similar prediction stability holds across other benchmarks, challenging visual conditions, and different preservation strengths?

  4. Regarding counter-context robustness. How sensitive is ACE to the size and semantic coverage of the counter-context bank and to the placement of contextual edits? In particular, how does the method behave when the edited region contains relevant visual information or when no explicit target concept is available?

  5. Regarding manuscript and figure quality. The manuscript would benefit from more careful editing and formatting. Several figures, particularly Figures 1 and 2, contain overly compressed elements, crowded annotations, and unclear visual connections. In addition, inconsistent spacing and excessive whitespace affect the overall presentation. Although the authors disclose the use of AI tools for manuscript and figure preparation, the current presentation still requires substantial manual refinement. Could the authors improve the figure layouts, arrow alignment, text readability, and document formatting?

Reviewer_eWumOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual hallucination in multimodal large language models (MLLMs). The authors introduce the concept of strict silencing, where intermediate decoder representations favor the factual hypothesis but subsequent layers overturn this preference. They propose Adaptive Counter-context Evidence (ACE), a training-free decoding method by constructing a counter-context to assess the reliability of visual contributions. Experiments on multiple MLLM backbones demonstrate improvements on visual hallucination benchmarks.

Critical strengths
  • The authors investigate how useful visual evidence can become less influential during subsequent decoder computation. This is an interesting perspective.
  • The experiments are conducted on several backbones and show consistent improvements.
  • The limitations of the proposed method are frankly discussed.
Critical weaknesses
  • The authors should clarify in more details about the reason why some datasets should have bigger improvements while others do not. It would be helpful to provide a more detailed breakdown analysis for each dataset.
  • In Table 3-B, ACE achieves 52 Rescued, while the simpler anchor-frozen gating achieves 49. The difference is quite marginal, showing that the design is not very effective.
  • Although ACE is training-free, it relies on architecture-specific visual-token mappings, access to attention contributions, and additional counter-context processing. Through the reported latency in Table 2, the computational and extra implementation requirements may limit practical value.
  • It is not clear what the sensitivity of the visual assets is, e.g., the diversity, number, and whether it is generalizable.
Questions for authors

See weaknesses.

Reviewer_Mvv5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose "NOT BLIND BUT SILENCED", a training-free method to reduce hallucinations in MLLMs. The main motivation is that correct visual evidence may exist in intermediate layers but lose its influence in later layers, which the authors refer to as silencing. The proposed method, ACE, consists of multiple parts to identify reliable contributions from the image. First, they challenge the process by constructing a counter-context image, then they estimate the reliability of visual contributions based on their magnitude, stability, and visual specificity. Next, they locate the appropriate layer to start the intervention, and finally, they amplify the contributions from reliable visual tokens. They are able to substantially improve performance on the POPE-Adversarial benchmark. They also evaluate their method on the FREAK and CHAIR benchmarks while maintaining comparable performance on MM-Vet.

Critical strengths
  1. The proposed method addresses an important problem in the field, and the authors are able to substantially improve performance on the POPE-Adversarial benchmark compared to other methods.

  2. The experimental section is comprehensive. The authors evaluate their method across multiple MLLMs and benchmarks. They also provide extensive ablation studies and additional analyses to investigate the effectiveness of different components of their method and whether their reliability scores are aligned with the factual contributions of visual tokens.

Critical weaknesses
  1. The method has multiple ambiguities when applied to open-ended tasks. First, for the Challenge stage, it is not clear how the counter-context images are constructed without affecting the relevant visual information when there is no specific target object. Moreover, the reliability scores and the anchor layer are determined at the final prompt position before generating the first token. In open-ended tasks, different generated tokens may require different visual information, and it is not clear why the reliability scores and the anchor layer determined at this position should be useful for the entire generation process.

  2. The novelty of the proposed method is not clear to me. Previous works have already investigated using auxiliary samples to reduce hallucinations or to identify reliable visual tokens. VCD [1] uses distorted images to contrast the output logits, and CIPHER [2] uses counterfactual images to investigate changes in internal representations. Moreover, ILVAD [3] identifies reliable visual tokens and amplifies their contributions during generation. The proposed method seems to be a combination of already investigated ideas.

  3. The definition of silencing is not clear to me. The authors use the logits from intermediate layers to determine whether the model favors the factual or competing hypothesis. However, these intermediate representations are not directly trained to predict the next token, and it is not clear why these logits should provide useful information about the model's decision at those layers. Moreover, why should a positive margin necessarily mean that the model has correct visual evidence, and a negative margin mean that this evidence has been suppressed? I think this part needs more justification.

  4. The proposed method is too complex and consists of multiple components and steps. This complexity may limit its generalizability to other benchmarks or tasks, and the method should be altered for them. In the current benchmarks, for exmaple it is not transferring to open-grounded tasks clearly. Different prompts can also change the applicability of the method.

  5. The anchor layer selection is not justified enough. The authors select the layer with the highest aggregated evidence score, but it is not clear why this layer should be the best one to start the intervention. Having the highest score does not necessarily mean that starting the amplification from this layer would be more effective than starting from earlier or later layers.

[1] Leng et al., "Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding" CVPR 2024.

[2] Dastmalchi et al., "Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression" 2026.

[3] Xie et al., "Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy" ICML 2026

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.