- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a multilingual evaluation framework to analyze the sensitive-attribute judgments in VLMs under multilingual variation. Based on VisBias multiple-choice settings, the proposed framework utilizes semantically aligned prompts in three different languages to evaluate across bias-related domains. The evaluation results reveal substantially varied cross-lingual behaviors in VLMs.
Critical strengths
- The evaluation protocol is comprehensive: it includes both cross-lingual consistency and demographic differentiation within a single language. Metrics such as agreement for cross-lingual consistency and nJSD also enable intuitive interpretation of models' biases.
- The evaluation dataset has a wide and balanced coverage of different demographic groups, providing a holistic evaluation of the social biases. Various families of open-source VLMs of different families are also evaluated in this paper, and the results highlight the multilingual biases as a common problem across open-source VLMs
Critical weaknesses
- Under the current evaluation framework, for certain languages and tasks, VLMs produce very few valid responses due to abstention (e.g., Gemma-4, Politics, ZH in Table 1, line 257). Metrics calculated based on these small samples could be unstable and unrepresentative. The missing treatment of abstention could also conflate VLM's social biases with other factors (e.g., VLM's refusal behavior or instruction-following capability), weakening the current interpretation of social biases in this paper.
- While various open-source VLMs have been evaluated, no commercial VLMs are covered in this paper. These models may differ in their training pipeline and thus may demonstrate bias patterns. Analyzing their biases could further strengthen this paper's central claim about cross-lingual biases in VLMs.
Questions for authors
- It remains unclear how Section 4.3, visual-textual consistency analysis, contributes to the paper's evaluation of cross-lingual biases in VLMs. What additional insights does the comparison with a text-only setup provide?
- How sensitive are the cross-lingual bias findings to the prompts used? For example, will changing the wording or order of the options lead to significant changes to the results?