- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ABCDE-Bench, a counterfactual VQA benchmark for auditing fairness in black-box LVLMs. It covers five categories of visual attributes and uses human-validated image pairs to evaluate decision consistency. The authors also propose refusal-aware metrics to distinguish inconsistent answers from refusal behavior. Experiments across 11 LVLMs reveal lower consistency under non-demographic changes, while prompting-based interventions mainly improve consistency by increasing symmetric refusals.
Critical strengths
1.The benchmark extends fairness evaluation beyond standard demographic attributes. The broad attribute coverage and human validation make it a useful resource for studying contextual sensitivity in LVLMs. 2.The evaluation covers a wide range of proprietary and open-weight models. The refusal-aware analysis is particularly useful, as it shows that higher consistency does not necessarily mean more stable substantive answers.
Critical weaknesses
1.I'm not fully convinced by the comparison between demographic and non-demographic attributes. The categories involve different questions and decision targets, so the consistency gap may partly reflect differences in task difficulty rather than sensitivity to the manipulated attributes. A more controlled comparison using matched questions would help clarify this. 2.The connection between decision consistency and fairness remains somewhat unclear. Many questions ask models to infer subjective properties such as trustworthiness, income, or competence from images, where no reliable answer may exist. In these cases, inconsistent responses do not necessarily indicate unfair treatment. The paper would benefit from a clearer distinction between general response instability and fairness-related failures. 3.The human comparison needs stronger support. Human performance is measured on only 30% of the benchmark, while the reported model results cover the full dataset. The claimed opposite trend between humans and models would be more convincing with results on the same subset and uncertainty estimates.
Questions for authors
1.Does the demographic versus non-demographic consistency gap remain when question types and difficulty are controlled across categories? 2.Can the authors compare human and model consistency on exactly the same evaluation subset, with confidence intervals?
3 comments
Follow-up on Our Rebuttal
Dear Reviewer wuhG,
Thank you again for your time and constructive feedback on our submission. We have carefully considered your comments and addressed them in our rebuttal.
We understand that you may be busy, but we would appreciate it if you could review our response before the discussion period ends. If any points remain unclear, we would be happy to clarify them.
Best regards,
The Authors
Thank you for the detailed response and additional experiments. They directly address my concerns, so I will maintain my positive rating and increase my confidence.
Response to Reviewer wuhG
Dear Reviewer wuhG,
Thank you for the positive assessment of our benchmark and refusal-aware analysis. We address the concerns through additional controlled comparisons and a matched human–model analysis.
W1 and Q1. Question composition and task difficulty
To examine whether question composition explains the observed gap, we grouped the benchmark questions into 15 question-target types and standardized the demographic and non-demographic groups to a common target distribution. The model-average gap remains +6.5 points after adjustment, compared with +6.4 points before adjustment. For example, the gaps within income/SES, education, and competence are +5.0 (positive for all 11 models), +6.0 (10/11 models), and +6.4 points (10/11 models), respectively.
We also conducted a controlled experiment using 60 base images, each independently edited to change either a demographic or a non-demographic attribute, yielding 120 counterfactual pairs in total. For each base image, both counterfactual pairs were evaluated using the same question and answer options. Non-demographic edits were balanced across the four categories, with questions covering the three targets above. The gap remains positive for both GPT-5 (+4.4 points) and Claude Sonnet 5 (+7.2 points), averaging +5.8 points (95% CI: [0.6, 11.4]).
These results further support our main finding when base images and VQA tasks are held fixed, suggesting that differences in question composition alone do not explain the observed gap. Both analyses have been added to Appendix E.7 (highlighted in magenta).
W2. Decision consistency and fairness
Counterfactual invariance has a well-established role in fairness evaluation [1], and related counterfactual probing approaches have also been used in NLP and vision-language fairness research [2–4]. Our work extends this invariance-based perspective to black-box LVLMs, examining changes to attributes that human annotators judge inappropriate as evidence for the specific question (Sec. 3.3, Step 2). Unlike general robustness tests under arbitrary visual perturbations, our evaluation is grounded in human judgments about which visual attributes should not influence a particular decision.
For subjective questions such as income or competence, a uniquely verifiable answer cannot be established from an image alone, and our benchmark does not prescribe a desirable response to each question. Counterfactual invariance can be evaluated without determining whether an individual prediction is correct. We therefore adopt a ground-truth-free evaluation that measures consistency across counterfactual image pairs, without judging individual answers as correct or refusals as incorrect (Sec. 3.3, L249–250).
[1] Kusner+, Counterfactual Fairness, NeurIPS2017.
[2] Zhao+, Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods, NAACL2018.
[3] Howard+, SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples, CVPR2024.
[4] Xiao+, GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing, IJCV2025.
W3 and Q2. Human-model comparison
To ensure a direct comparison, we recomputed model consistency on exactly the same 30% subset used for human evaluation and calculated pair-level bootstrap confidence intervals. On these identical pairs, the model-average demographic–non-demographic gap remains positive at +7.7 points (95% CI: [3.5, 11.9]), whereas the human gap is −3.8 points (95% CI: [−14.1, 6.6]). The paired model–human difference in gaps is +11.5 points (95% CI: [0.4, 22.3]). We have added the matched comparison results to Finding 1 (highlighted in magenta).
The matched comparison thus supports a larger demographic–non-demographic gap in models than in humans. The human point estimate shows an opposite tendency, although its confidence interval includes zero. We therefore interpret this trend descriptively rather than as a statistically established reversal. We have revised the corresponding statements in the abstract, introduction, and Finding 1 (highlighted in magenta) to avoid implying a statistically established opposite trend. As noted in our Limitations (L1510–1511), human evaluation serves as a behavioral reference rather than a normative fairness baseline.