acceptodds
Withdrawn from ICLR 2027

Seeing Through Symbols: Tracing and Mitigating Label Symbol Bias in Large Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated strong capabilities in multimodal reasoning, yet their predictions can be sensitive to the symbolic identifiers assigned to visual candidates. We investigate this phenomenon as Label Symbol Bias, where changing the labels of candidate regions, points, or sub-images can alter model predictions despite leaving the underlying visual content and reasoning task unchanged. To uncover the mechanisms underlying this bias, we introduce Symbol Sensitivity Analysis (SSA), a path-patching-based framework that identifies and quantifies the sensitivity of attention heads to changes in label symbols. By tracing the attention patterns of these symbol-sensitive heads, we uncover how label information associated with visual candidates is propagated toward the final prediction, revealing a mechanism through which symbolic identifiers can influence multimodal reasoning beyond the visual content they denote. Building on these findings, we propose Sensitivity-Aware Activation Debiasing (SAAD), a training-free intervention that estimates the label symbol bias component of head activations and selectively amplifies their task-relevant component according to each head's measured label symbol sensitivity. Extensive experiments across diverse domains demonstrate that SAAD promotes more balanced predictions and improves performance by up to 15% over the underlying model. These results demonstrate that reducing the influence of label symbol bias enables LVLMs to rely more faithfully on task-relevant visual evidence, thereby improving their reasoning performance.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Large Vision-Language Models, Multimodal Multiple-Choice Reasoning, Symbolic Bias, Path Patching, Activation Debiasing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Sv7tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether symbolic labels attached to visual candidates (e.g., A/B labels on points, boxes, or subimages) influence large vision-language model predictions. The authors propose Symbol Sensitivity Analysis (SSA), which uses activation patching to identify attention heads sensitive to label changes, and Sensitivity-Aware Activation Debiasing (SAAD), which adjusts head activations according to their measured sensitivity.

Critical strengths
  • The paper addresses a specific and relevant problem: models may rely on labels that are intended only to identify visual candidates.
  • It combines behavioral evaluation with internal activation interventions to investigate how label information may affect predictions.
Critical weaknesses
  • The empirical evidence is insufficient to support the proposed theory. The experimental analysis is underdeveloped, occupying barely one page.
Questions for authors

The introduction reports Relative Depth accuracy of 44.2% for A/B and 51.3% for X/Y, while Table 2 reports 49.63% and 46.92%, respectively. These may refer to different subsets or settings?

Reviewer_NdT5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies label symbol bias in LVLMs, where changing the symbolic identifiers assigned to visual candidates can alter model predictions despite unchanged visual content. The authors use Symbol Sensitivity Analysis (SSA) to identify attention heads that causally propagate label-symbol information from image regions to the final prediction. They further propose Sensitivity-Aware Activation Debiasing (SAAD), which selectively amplifies task-dependent activation components in symbol-sensitive heads to reduce this bias and improve visual reasoning accuracy.

Critical strengths

This work goes beyond text modality by providing a cross-modality analysis of how visual symbols propagate through LVLMs and influence their predictions.

Critical weaknesses
  1. This work largely extends Option Symbol Matters [R1] from LLMs to the vision-language setting. Both SSA and SAAD closely resemble the prior work’s path-patching analysis and sensitivity-weighted activation intervention, limiting the novelty and orthogonality. Moreover, Table 2 and Figure 3 do not support label symbol bias as a distinct new phenomenon, since the experiments simultaneously change the image-embedded labels and the option symbols in the prompt. The observed effects may therefore arise from the existing option-symbol bias. Finally, the cross-modality analysis in Section 3.2 relies heavily on a case study, leaving the generality of the identified propagation pathway unclear.

  2. The proposed SAAD is evaluated using only one LVLM, LLaVA-NeXT-Vicuna-7B, and one benchmark, BLINK. This limited evaluation raises concerns about the reliability of the SAAD across model architectures and data distributions.

[R1] Yang, Z., Jian, P., and Li, C. “Option Symbol Matters: Investigating and Mitigating Multiple-Choice Option Symbol Bias of Large Language Models.” NAACL 2025.

Questions for authors

Please refer to the Critical Weaknesses.

Reviewer_hruDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies label symbol bias in large vision-language models, where changing the symbolic identifiers attached to visual candidates can alter model predictions while leaving the underlying visual reasoning problem unchanged. The authors introduce SSA, which uses path patching under label-symbol perturbations to identify symbol-sensitive attention heads and trace a proposed propagation pathway from visual symbols to textual symbols, candidate representations, and final predictions. Based on this analysis, the paper proposes SAAD, a parameter-update-free intervention that amplifies mean-centered head activations according to their measured symbol sensitivity. Experiments on six BLINK domains show improved accuracy and reduced sensitivity to different symbol sets.

Critical strengths
  • The paper identifies an interesting source of brittleness in multimodal reasoning and goes beyond purely behavioral analysis.
  • SAAD produces consistent behavioral improvements across the evaluated domains and symbol sets.
Critical weaknesses
  1. The current perturbation does not cleanly disentangle visual label-symbol bias from textual symbol bias. SSA changes the label symbols both in the image and in the textual prompt. Therefore, the resulting prediction difference can reflect a combination of visual-symbol sensitivity, textual token priors, and previously studied multiple-choice symbol bias. Although the causal tracing provides evidence that visual label regions participate in the computation, it does not quantify how much of the behavioral effect originates specifically from symbols embedded in the image. A controlled decomposition of visual-only and text-only symbol perturbations would be important for establishing the paper's central claim.
  2. The key assumptions behind SAAD are insufficiently validated. SAAD assumes that the dataset-level mean head activation represents a systematic symbol-bias component and that the deviation from this mean represents task-relevant information. This decomposition is not directly validated, and Eq. 10 shows that SAAD retains the estimated bias component while amplifying the presumed task-dependent component rather than explicitly removing the bias. Moreover, the current ablations do not isolate whether the symbol-sensitivity weights obtained by SSA are necessary: a uniform or shuffled-weight mean-centered activation baseline would help determine whether the gains specifically arise from sensitivity-aware intervention.
Questions for authors
  1. How does SAAD compare with uniform or randomly shuffled sensitivity weights while keeping the same mean-centered activation intervention? This would clarify whether SSA-derived sensitivity is actually responsible for the improvement.
  2. Are the samples used to estimate head sensitivity and dataset-level mean activations strictly disjoint from the evaluation samples?
  3. Do the identified sensitive heads and the effectiveness of SAAD generalize to LVLM families beyond LLaVA-NeXT-Vicuna-7B?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.