Seeing Through Symbols: Tracing and Mitigating Label Symbol Bias in Large Vision-Language Models

Large Vision-Language Models (LVLMs) have demonstrated strong capabilities in multimodal reasoning, yet their predictions can be sensitive to the symbolic identifiers assigned to visual candidates. We investigate this phenomenon as Label Symbol Bias, where changing the labels of candidate regions, points, or sub-images can alter model predictions despite leaving the underlying visual content and reasoning task unchanged. To uncover the mechanisms underlying this bias, we introduce Symbol Sensitivity Analysis (SSA), a path-patching-based framework that identifies and quantifies the sensitivity of attention heads to changes in label symbols. By tracing the attention patterns of these symbol-sensitive heads, we uncover how label information associated with visual candidates is propagated toward the final prediction, revealing a mechanism through which symbolic identifiers can influence multimodal reasoning beyond the visual content they denote. Building on these findings, we propose Sensitivity-Aware Activation Debiasing (SAAD), a training-free intervention that estimates the label symbol bias component of head activations and selectively amplifies their task-relevant component according to each head's measured label symbol sensitivity. Extensive experiments across diverse domains demonstrate that SAAD promotes more balanced predictions and improves performance by up to 15% over the underlying model. These results demonstrate that reducing the influence of label symbol bias enables LVLMs to rely more faithfully on task-relevant visual evidence, thereby improving their reasoning performance.