acceptodds
Withdrawn from ICLR 2027

When Visual Signals Mislead: A Mechanistic Study of Attribute Hallucination in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Attribute hallucination—where vision-language models (VLMs) correctly identify an object but mischaracterize its properties—is prevalent yet mechanistically poorly understood. The dominant explanation, language-prior dominance, has motivated prior-suppression methods, but this explanation has not been directly tested at the attribute level. We present VISOR (Visual-Operational Remediation), a unified framework that couples null-image-based diagnosis with routed remediation. Its VSNR diagnostic compares a real-image logit margin with a null-image reference margin. Across the three model families and three attribute types, the real-image margin tracks the false-positive decision while the null-image reference carries little information. VISOR uses this diagnosis to separate two failure modes: low-margin but directionally correct signals in color/state attributes, and low-SNR or late-stage readout-misaligned signals in material attributes. The same diagnosis routes each query to the appropriate operator: calibration for threshold-placement errors, abstention for training-free low-SNR handling, or targeted visual adaptation for material failures that prior suppression cannot correct. On closed-vocabulary Yes/No attribute probes from VAW and GQA, VISOR attains the lowest false-positive rate across all nine model–attribute settings, and on material it does so without increasing the false-negative rate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attribute hallucination, Routed remediation, Visual signal decomposition, Vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_H8FtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper looks into the problem of attribute hallucination in vision-language models, focusing on cases where the model can correctly recognize objects but gets their color, material, or state wrong. The authors propose the VISOR framework, which diagnoses prediction errors for different attributes by comparing the Yes/No logit margin under real versus blank image conditions and building an attribute-level signal-to-noise ratio (VSNR). Based on this analysis, the paper identifies two error mechanisms: one where the visual signal direction is correct but the decision margin is insufficient, and another where there’s a mismatch in the mapping from late-stage visual features to output space for material attributes. To fix different types of errors, VISOR applies decision calibration (Calib), abstaining (Abstain), and attribute-wise visual LoRA adaptation (Adapt). Experiments on three vision-language models and the VAW attribute classification task show reduced attribute misreporting, and layer-wise probing and attribute-level analysis support their proposed mechanisms.

Critical strengths
  1. The research question has some value. The paper focuses on fine-grained attribute hallucinations, which have been less studied compared to object existence hallucinations, and tries to distinguish errors in different attributes instead of attributing all mistakes to language priors. This perspective is quite meaningful.

  2. It provides a fairly complete diagnostic and repair framework. The paper integrates output distribution analysis, hierarchical representation probing, and targeted repair strategies into a unified framework, and validates it on three different visual language model architectures. The experiments report both FPR and FNR, which helps analyze the trade-offs in performance when reducing hallucinations.

Critical weaknesses
  1. The method isn't very innovative, and the unique value of the core framework hasn't been proven. The main components of VISOR are all closely related to existing technologies. Comparing outputs of real images and blank images is similar to contrast methods like VCD, Calib is basically logit margin calibration, the intermediate layer analysis uses something like Logit Lens, and both Abstain and attribute-wise LoRA are well-established techniques. The overall framework mainly relies on offline attribute statistics to choose existing correction operators, lacking clear new algorithm design, and it hasn't been shown that this combination has independent advantages over simpler alternatives.
  2. The independent diagnostic value of VSNR hasn’t been fully validated. The paper defines SNR as the ratio of the mean absolute value to the standard deviation of attribute-level logit margins and reports a strong correlation with FPR in Table 4 (ρ = −0.916). However, both SNR and FPR are calculated from the same negative sample margin distribution, so they are mathematically linked. This means the correlation doesn’t independently prove that SNR actually captures visual signal quality or underlying hallucination mechanisms. The paper also lacks additional experiments to show whether VSNR provides diagnostic information beyond simple output statistics, and whether such information really helps distinguish different error mechanisms.
  3. The explanations for the two attribute hallucination mechanisms lack solid validation. For Mechanism A, the authors infer that the visual signal points in the right direction based on the fact that the average margin of attribute words is negative. However, the average result at the attribute level doesn’t prove that the visual evidence in specific wrong cases also points to the correct answer. For Mechanism B, the authors interpret the later SNR drop as a mismatch in mapping from vision to output space, by fixing the final LM head while reading intermediate layer representations. But this analysis can’t distinguish between the degradation of visual information and changes in alignment between the intermediate representation and the output head. Without independent probing or causal interventions, the explanations for both mechanisms are still not very convincing.
  4. Key baselines are missing, and the independent contribution of the fix is unclear. VISOR-Calib mainly reduces FPR by adjusting the Yes/No decision boundary, but the paper lacks a direct comparison with simple threshold calibration. Table 1 shows that while FPR for color attributes drops, FNR rises significantly at the same time. For instance, for Qwen2.5-VL-3B, FPR goes from 33.8% down to 22.2%, while FNR goes from 16.1% up to 28.4%. Moreover, Adapt involves extra per-attribute supervised training, but there’s no LoRA control using the same data and parameter budget. So, the current experiments can’t fully distinguish the contributions of simple calibration, additional supervised training, or mechanism-driven fixes.
  5. There are potential issues with data construction and the evaluation protocol. The paper uses instance-level attribute annotations from VAW but asks questions that don’t point to specific instances, like ‘Is the [object] [attribute]?’. When there are multiple similar objects in the image, this can lead to a mismatch between the question and the attribute labels, affecting the reliability of hallucination evaluation. Also, VISOR estimates attribute statistics, sets calibration parameters, and selects target material words using labeled offline data, but it’s not clearly explained whether this data is strictly independent from the final test set. So, data ambiguity and the potential risk of evaluation information leakage still need clarification.
Questions for authors

See weakness

Reviewer_ALfjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies attribute hallucination through closed-vocabulary Yes/No probes, mainly on VAW, using Qwen2.5-VL-3B, InternVL3.5-4B-Flash, and LLaVA-1.5-7B. VISOR measures the real-image and blank-image Yes/No margins and uses a per-word statistic, |mean margin|/standard deviation, to describe failure patterns. The proposed remedies are null-margin-based calibration for color/state, abstention for low-SNR material words, and supervised per-word visual LoRA adapters when a definitive answer is needed. Routing is offline and per attribute word, rather than per query.

Calibration reduces FPR with an accompanying FNR increase. Targeted material adaptation reduces the reported FPR without increasing aggregate material FNR, but covers only six words. The paper also discusses annotation ambiguity and intermediate-layer Yes/No readouts, and reports a GQA margin-sign consistency check.

Critical strengths
  1. The paper examines a failure mode that object-existence checks can miss. Breaking results down by color, material, and state across three model families is useful, and Table 1 reports FNR as well as FPR. This makes the cost of more conservative answers visible. The authors also explicitly distinguish the blank-image reference from a language-only prior, and describe Table 2's margin-sign correlation as a consistency check rather than independent causal evidence.

  2. The targeted material results are promising within their stated scope. In Table 6, the mean FPR over six Qwen target words falls from 38.1% to 23.9% with per-word adapters, compared with 31.1% for the shared hard-negative adapter. Leather improves with the per-word adapter even though the shared adapter makes it worse. Table 1 reports unchanged material FNR for Qwen and lower FNR for the other two models. The annotation audit in Table 3 is also a useful reminder that some apparent hallucinations depend on how labels are interpreted.

Critical weaknesses
  1. The central negative claim — that attribute false positives are not driven by the language prior — is not established by the evidence presented. The prior is operationalized only as the Yes/No margin under a blank gray image, which still passes through the vision tower; stronger operationalizations (a text-only forward pass, instruction-level priors) are not tested. Moreover, the paper's own numbers cut against a clean negative conclusion: the null reference alone reaches 58–59% accuracy, and at the word level delta_null significantly predicts color FPR. At best the evidence shows the gray-image margin does not dominate these decisions. The supporting layer-wise analysis is similarly limited: the intermediate-layer LM-head readout measures alignment with the final Yes/No readout rather than the presence of attribute information, and the word-level SNR is derived from the same margin distribution that defines FPR, so the SNR–FPR association is close to circular. Without causal interventions, "late-stage readout misalignment" remains a suggestive description rather than a mechanism.

  2. The evaluation is a narrow proxy for hallucination, and the main training-free gains are error trade-offs rather than improved discrimination. All primary results come from closed-vocabulary Yes/No probes over a known word list from a single annotation family; hallucination in practice arises in open-ended generation, and the remedy designs (a Yes/No margin shift, binary abstention, word-keyed adapters) offer no tested path to that setting. Within the probe setting, Calib's color FPR reduction is bought with a near doubling of FNR, and no threshold-independent measure or matched-FNR comparison against a plain threshold on delta_vis is reported to show what the null-image term adds. Several cells show no effect at all (LLaVA state is numerically identical to baseline; material Calib is exactly 0.0 pp on Qwen and InternVL), and the main table reports no confidence intervals.

  3. The remedies are narrow and do not compose into a practical mitigation method. Calib is per-word, per-model threshold tuning on labeled negatives — case-by-case and addressing the decision boundary rather than the diagnosed visual deficiency. Adapt requires one supervised LoRA per attribute word (with ground-truth data and an oracle router keyed to the queried word), covers only six material words, and shows no transfer beyond them; the natural consolidation, a shared adapter, actively hurts (leather worsens by 8 pp where per-word Adapt improves it by 12 pp), and the weighted full-material estimate improves only from 15.7% to 13.5%. Finally, the routed framework in Table 1 effectively applies the best operator per column; no ablation shows that VSNR routing outperforms simply applying each operator everywhere, so the claimed diagnosis-to-remedy link remains asserted rather than demonstrated. A zero FPR change from this particular calibration rule on material also does not imply the errors are not threshold-correctable in general, since a null-margin-based rule gives no shift when the null margin is near zero.

Questions for authors
  1. Please identify the image-disjoint data used for offline diagnosis, gamma/tau selection, selecting the six adapted words, adapter training, and evaluation. What numerical routing thresholds are used, and how does the Abstain operator turn the word-level diagnosis into a decision on a particular query? If these details are in the separate supplement, please point to the relevant sections.

  2. Have you tested stronger operationalizations of the language prior (e.g., a text-only forward pass)? At matched FNR, how does Calib compare with a global or per-word threshold on delta_vis alone, and can you report a threshold-independent separation measure (e.g., AUC) for Table 1?

  3. For Table 6, what data and total adapter capacity are used for shared versus per-word LoRA? Does adaptation transfer to held-out material words, and can you show held-out positive/negative score distributions before and after adaptation to distinguish improved discrimination from a shift toward answering No?

  4. Can you provide any evidence that the diagnosis or remedies transfer beyond Yes/No probes, e.g., attribute-error rates in free-form captions before and after Calib/Adapt? The caption of Table 1 also says the 14.2% Qwen material FPR and the 13.5% weighted estimate use different protocols — please state both explicitly, along with how the annotation audit denominator was computed.

Reviewer_1zyPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates attribute hallucination in Vision-Language Models (VLMs). They apply the simiilar idea from Visual Contrastive Decoding to model the null-image answer, capturing the language prior. Based on that application, they VISOR (Visual-Operational Remediation), a framework that uses a diagnostic metric, VSNR (Visual Signal-to-Noise Ratio), to classify errors into two distinct failure modes.

Critical strengths

Well-written papers.

Critical weaknesses
  1. The paper has critical hallucinated AI references.
  • Wrong title - InternVL2.5: An open-source multimodal LLM for competitive performance with GPT-4o → No papers with such titles exist
  • Wrong title - VASR: Visual analogies of situation and relation.
  • "Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens" - Hallucinated author list.
  • "Zhiyuan Li, Hangyu Shi, Yu Gao, Ding Liu, Zhen Wang, Yufan Chen, Tao Liu, Li Zhao, Haoran Wang, and Dimitris N. Metaxas. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering." - Hallucinated author list.
  • "Mohammad Asadi, Jack W. O’Sullivan, Fang Cao, Tahoura Nedaee, Kamyar Rajabalifardi, Fei-Fei Li, Ehsan Adeli, and Euan Ashley.
  • Mirage: The illusion of visual understanding. arXiv preprint arXiv:2501.09905, 2025". Wrong arxiv ID and year
  1. The proposed framework lacks originality. It is a combination of existing techniques repackaged with new names:
  • VISOR-Calib: This is simple logit-bias calibration (vis-null). Adjusting decision thresholds based on null prompts or marginal likelihoods is a standard technique in zero-shot classification (e.g., "Calibrate Before Use") and has low novelty.
  • VISOR-Abstain: This is standard uncertainty-based selective classification (refusing to predict when confidence is below a threshold), which is a long-established paradigm.
  • VISOR-Adapt (Per-Word LoRA): Training a dedicated LoRA adapter for each individual attribute word (e.g., one for "rubber", one for "leather", one for "metal") is highly unpractical:
  • Infeasible Scaling: In a real-world system with thousands of attributes, storing and dynamically loading a custom LoRA adapter for every single attribute is computationally and architecturally prohibitive.
  • Unfair Comparison: The baseline models are evaluated in a zero-shot setting. VISOR-Adapt, however, is trained using supervised negative and positive examples of the specific target attributes. It is unsurprising that supervised, attribute-specific fine-tuning outperforms zero-shot baselines; this does not represent a truly comparable "zero-shot remediation."

[1] Calibrate Before Use: Improving Few-Shot Performance of Language Models, ICML'21

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.