- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether symbolic labels attached to visual candidates (e.g., A/B labels on points, boxes, or subimages) influence large vision-language model predictions. The authors propose Symbol Sensitivity Analysis (SSA), which uses activation patching to identify attention heads sensitive to label changes, and Sensitivity-Aware Activation Debiasing (SAAD), which adjusts head activations according to their measured sensitivity.
Critical strengths
- The paper addresses a specific and relevant problem: models may rely on labels that are intended only to identify visual candidates.
- It combines behavioral evaluation with internal activation interventions to investigate how label information may affect predictions.
Critical weaknesses
- The empirical evidence is insufficient to support the proposed theory. The experimental analysis is underdeveloped, occupying barely one page.
Questions for authors
The introduction reports Relative Depth accuracy of 44.2% for A/B and 51.3% for X/Y, while Table 2 reports 49.63% and 46.92%, respectively. These may refer to different subsets or settings?