- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper looks into the problem of attribute hallucination in vision-language models, focusing on cases where the model can correctly recognize objects but gets their color, material, or state wrong. The authors propose the VISOR framework, which diagnoses prediction errors for different attributes by comparing the Yes/No logit margin under real versus blank image conditions and building an attribute-level signal-to-noise ratio (VSNR). Based on this analysis, the paper identifies two error mechanisms: one where the visual signal direction is correct but the decision margin is insufficient, and another where there’s a mismatch in the mapping from late-stage visual features to output space for material attributes. To fix different types of errors, VISOR applies decision calibration (Calib), abstaining (Abstain), and attribute-wise visual LoRA adaptation (Adapt). Experiments on three vision-language models and the VAW attribute classification task show reduced attribute misreporting, and layer-wise probing and attribute-level analysis support their proposed mechanisms.
Critical strengths
-
The research question has some value. The paper focuses on fine-grained attribute hallucinations, which have been less studied compared to object existence hallucinations, and tries to distinguish errors in different attributes instead of attributing all mistakes to language priors. This perspective is quite meaningful.
-
It provides a fairly complete diagnostic and repair framework. The paper integrates output distribution analysis, hierarchical representation probing, and targeted repair strategies into a unified framework, and validates it on three different visual language model architectures. The experiments report both FPR and FNR, which helps analyze the trade-offs in performance when reducing hallucinations.
Critical weaknesses
- The method isn't very innovative, and the unique value of the core framework hasn't been proven. The main components of VISOR are all closely related to existing technologies. Comparing outputs of real images and blank images is similar to contrast methods like VCD, Calib is basically logit margin calibration, the intermediate layer analysis uses something like Logit Lens, and both Abstain and attribute-wise LoRA are well-established techniques. The overall framework mainly relies on offline attribute statistics to choose existing correction operators, lacking clear new algorithm design, and it hasn't been shown that this combination has independent advantages over simpler alternatives.
- The independent diagnostic value of VSNR hasn’t been fully validated. The paper defines SNR as the ratio of the mean absolute value to the standard deviation of attribute-level logit margins and reports a strong correlation with FPR in Table 4 (ρ = −0.916). However, both SNR and FPR are calculated from the same negative sample margin distribution, so they are mathematically linked. This means the correlation doesn’t independently prove that SNR actually captures visual signal quality or underlying hallucination mechanisms. The paper also lacks additional experiments to show whether VSNR provides diagnostic information beyond simple output statistics, and whether such information really helps distinguish different error mechanisms.
- The explanations for the two attribute hallucination mechanisms lack solid validation. For Mechanism A, the authors infer that the visual signal points in the right direction based on the fact that the average margin of attribute words is negative. However, the average result at the attribute level doesn’t prove that the visual evidence in specific wrong cases also points to the correct answer. For Mechanism B, the authors interpret the later SNR drop as a mismatch in mapping from vision to output space, by fixing the final LM head while reading intermediate layer representations. But this analysis can’t distinguish between the degradation of visual information and changes in alignment between the intermediate representation and the output head. Without independent probing or causal interventions, the explanations for both mechanisms are still not very convincing.
- Key baselines are missing, and the independent contribution of the fix is unclear. VISOR-Calib mainly reduces FPR by adjusting the Yes/No decision boundary, but the paper lacks a direct comparison with simple threshold calibration. Table 1 shows that while FPR for color attributes drops, FNR rises significantly at the same time. For instance, for Qwen2.5-VL-3B, FPR goes from 33.8% down to 22.2%, while FNR goes from 16.1% up to 28.4%. Moreover, Adapt involves extra per-attribute supervised training, but there’s no LoRA control using the same data and parameter budget. So, the current experiments can’t fully distinguish the contributions of simple calibration, additional supervised training, or mechanism-driven fixes.
- There are potential issues with data construction and the evaluation protocol. The paper uses instance-level attribute annotations from VAW but asks questions that don’t point to specific instances, like ‘Is the [object] [attribute]?’. When there are multiple similar objects in the image, this can lead to a mismatch between the question and the attribute labels, affecting the reliability of hallucination evaluation. Also, VISOR estimates attribute statistics, sets calibration parameters, and selects target material words using labeled offline data, but it’s not clearly explained whether this data is strictly independent from the final test set. So, data ambiguity and the potential risk of evaluation information leakage still need clarification.
Questions for authors
See weakness