Bias-Sensitive but Not Bias-Specific: Supervision-Free Fairness Intervention in Vision-Language Models

Vision-language models inherit predictive shortcuts from the language priors they are trained on, and when those priors correlate with demographic or social categories, the resulting associations surface directly in multimodal predictions. Existing training-free methods reduce this bias effectively, but each relies on some form of bias-specific prior knowledge constructed before the intervention is applied, leaving open the question of whether meaningful reduction is achievable without such construction. We introduce SAAV (Supervision-free Attention Attenuation for VLMs), a single inference-time rule that attenuates internal representations in late language-model layers, leaving every model parameter unchanged, and evaluate its transfer across four VLM families and five bias benchmarks that each define fairness differently. Paired bootstrap analysis over the resulting 20 model-benchmark comparisons confirms bias reductions in 9 settings and degradations in 4. These outcomes are not distributed randomly across models or benchmarks but track what each benchmark's bias signal represents, suggesting SAAV is sensitive to whether a social association is spurious or task-relevant. Despite this selectivity, the aggregate fairness index rises for every model and capability on four held-out benchmarks is retained at 98.52–99.91% of baseline, and SAAV remains competitive against existing training-free methods despite using none of their bias-specific prior knowledge. Together, these results suggest that a bias-sensitive representation is not automatically a bias-specific one and whether attenuation helps or harms a given prediction depends on whether the pathway it targets carries a spurious correlation or evidence the task genuinely needs.