acceptodds
Withdrawn from ICLR 2027

What Language Makes Models See: Cross-Lingual Bias in Sensitive Judgments by Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bias evaluations of vision–language models (VLMs) are usually conducted in a single language, leaving open whether sensitive judgments remain stable when only the prompt language changes. We evaluate five VLMs on the VisBias dataset using semantically aligned English, Spanish, and Chinese prompts across education, salary, politics, and religion. Under a controlled forced-choice setting, we distinguish cross-lingual consistency from within-language differences across ethnicity, gender, and occupation, including directional rank contrasts for the ordinal salary and education tasks. Cross-lingual behavior varies substantially by task and model: political orientation shows pronounced instability, although low valid-response coverage limits some comparisons, whereas salary is comparatively consistent. However, high agreement can partly reflect concentrated default responses rather than robust cross-lingual consistency. Demographic analysis reveals recurring group-conditioned patterns, including higher predicted salary ranks for male images in most model–language configurations and systematic ethnicity-related shifts. Overall, our results show that single-language evaluations can obscure both language-conditioned variation and demographic patterns in sensitive VLM judgments, motivating their joint assessment in multilingual bias evaluation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Social Bias, Multimodal Bias, Vision-Language Models, Multilingual Evaluation, Cross-Lingual Consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VrvyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multilingual evaluation framework to analyze the sensitive-attribute judgments in VLMs under multilingual variation. Based on VisBias multiple-choice settings, the proposed framework utilizes semantically aligned prompts in three different languages to evaluate across bias-related domains. The evaluation results reveal substantially varied cross-lingual behaviors in VLMs.

Critical strengths
  • The evaluation protocol is comprehensive: it includes both cross-lingual consistency and demographic differentiation within a single language. Metrics such as agreement for cross-lingual consistency and nJSD also enable intuitive interpretation of models' biases.
  • The evaluation dataset has a wide and balanced coverage of different demographic groups, providing a holistic evaluation of the social biases. Various families of open-source VLMs of different families are also evaluated in this paper, and the results highlight the multilingual biases as a common problem across open-source VLMs
Critical weaknesses
  • Under the current evaluation framework, for certain languages and tasks, VLMs produce very few valid responses due to abstention (e.g., Gemma-4, Politics, ZH in Table 1, line 257). Metrics calculated based on these small samples could be unstable and unrepresentative. The missing treatment of abstention could also conflate VLM's social biases with other factors (e.g., VLM's refusal behavior or instruction-following capability), weakening the current interpretation of social biases in this paper.
  • While various open-source VLMs have been evaluated, no commercial VLMs are covered in this paper. These models may differ in their training pipeline and thus may demonstrate bias patterns. Analyzing their biases could further strengthen this paper's central claim about cross-lingual biases in VLMs.
Questions for authors
  • It remains unclear how Section 4.3, visual-textual consistency analysis, contributes to the paper's evaluation of cross-lingual biases in VLMs. What additional insights does the comparison with a text-only setup provide?
  • How sensitive are the cross-lingual bias findings to the prompts used? For example, will changing the wording or order of the options lead to significant changes to the results?
Reviewer_BEobOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper empirically investigates how different prompt languages affect the social bias in VLMs. The paper offers quantitative evaluations of the agreement of multiple-choice questions regarding some demographic attributes, like salary and religion, with different prompt languages (English, Spanish, and Chinese) and answer distributions across demographic groups, etc. The results demonstrate that for the same images, prompts in different languages result in different answers, and the trends are different for different types of questions.

Critical strengths

The paper empirically shows that query languages can change models’ output. This reconfirms the social problems in VLMs and potentially motivates future efforts to address such problems.

Critical weaknesses

I find the contribution of this paper somewhat unclear. I believe that many researchers are already aware that VLMs and LLMs exhibit biases inherited from their training data. Furthermore, since different languages are likely associated with different biases, it is reasonably predictable that changing the language of a prompt would lead to biases reflecting those present in the training data for that language. Therefore, I do not consider demonstrating this phenomenon alone to constitute a significant contribution.

Regarding the quantification of agreement and differences in response distributions, it is also unclear from the paper what value these measurements provide beyond enabling comparisons. While the quantitative results reveal the extent to which responses differ across languages, the paper does not discuss what the desired behavior should be. In this respect, measuring the degree of bias within each language individually, as done in Section 4.2, seems more useful, as it provides concrete targets for bias mitigation. For this reason, I find it difficult to see substantial value in the cross-lingual comparisons themselves.

As mentioned above, I believe the contribution of this paper could be significantly strengthened by discussing the implications of these findings, how they could inform the design of VLMs and related models, and what actions the research community should take in light of the results. However, the current version lacks such discussions and remains largely limited to quantitative evaluations.

In summary, the paper does not appear to go beyond experimentally demonstrating that biases differ depending on the language used. I therefore find it difficult to regard the contribution as sufficiently substantial in its current form.

Questions for authors

Is it possible to provide some discussions mentioned in the weakness section?

Reviewer_PNffOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the bias in VLMs' judgements introduced due to prompt language. By using English VisBias dataset, the authors translate multiple choice prompts to Spanish and Chinese, and evaluate biases induced due to language on five open-source VLMs. By looking at agreement between three languages, the authors find that cross-lingual consistency depends largely on the task and the model. The sensitive attributes evaluated are education, salary, political orientation, and religion, and the paper finds political orientation to be least stable in cross-lingual analysis. The authors also compare answers across gender, ethnicity, and occupation groups within each language, and run a text-only version as well.

Critical strengths
  1. The idea for experimentation is grounded and well-motivated. I like what the authors are trying to do, as it's important to understand the effects that languages have in inferences and evaluations.
Critical weaknesses

There are some critical issues in the paper, that I feel raises concerns about the results and how to interpret them.

  1. The differences cannot just be attributed to languages, because the authors don't account for other confounding factors like positional biases, and the prompt variations themselves. The option order for MCQs is same throughout. LLaVA picks salary option B ($20k-40k) in almost all cases, which could be model playing safe or positional preference.
  2. The political orientation results for Gemma-4 and Ministral contradict the paper's appendix. Ministral-3 in Spanish has 324 valid answers but 99.7% abstention, so almost all of its "valid" answers are abstentions according to the paper's classifier. This also happens in English and Chinese, and also for Gemma-4. So, the language shift and agreement for Ministral-3 mostly come from refusals.
  3. The demographic findings are also not tested statistically. There are no CIs or permutation baselines. The scope is also narrow, with only three high-resource languages evaluated.
  4. The contribution over VisBias is also small, as VisBias already uses the same 700 images, questions, and JSD between groups to measure bias. The new parts are mainly the two translated languages.

The motivation is good, but the experimental design currently has many confounding variables, that needs to be addressed first before any conclusions can be drawn from the paper.

Questions for authors
  1. Can the authors explain the mismatch between the valid counts in Table 1 and the abstention rates in Tables 6-9?

Some suggestions for authors:

  1. Four models pick "other religion" a lot like an abstention option. Please discuss this in the high agreement on religion.
  2. Please add a related work section and position the paper accordingly.
  3. Ordinal alpha would be better for education and salary instead of nominal alpha for Krippendorff.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.