acceptodds
Under review as a conference paper at ICLR 2027

Whose Values Are Generated? Measuring Visual Value Leakage in Multilingual Text-to-Image Generation

Abstract

Text-to-image (T2I) systems serve users across languages and cultural contexts. However, evaluations of cultural representation focus mainly on recognizable elements such as landmarks, clothing, and activities, while overlooking the cultural values conveyed by generated images. We introduce **VisionValueBench**, a multilingual benchmark that operationalizes six cultural value dimensions from Hofstede’s framework, such as individualism versus collectivism, to investigate how T2I systems frame neutral social situations in a particular cultural orientation. We find that value-neutral prompts allowing diverse cultural interpretations nevertheless produce images that favor particular value orientations, a phenomenon we call *visual value leakage*. Across 960 social scenarios and six T2I configurations, we evaluate prompts varying in language and geographic cues. Generated images consistently favor low power distance, individualism, care and quality of life, long-term orientation, high uncertainty avoidance and indulgence. Prompt language has limited influence on value orientations, whereas geographic cues shift some dimensions toward associated cultural values while still preserving the prevailing orientations. These findings reveal substantial homogeneity in the cultural values conveyed by T2I systems worldwide.

Primary area:
datasets and benchmarks
Keywords:
Text-to-image Generation, Cultural values, Multilinguality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7ycPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether T2I models express particular cultural values when the prompt leaves those values open, a phenomenon the authors call “visual value leakage.” They introduce VisionValueBench, a multilingual benchmark based on Hofstede’s six cultural dimensions. Scenarios are derived from CDEval questions and rewritten as value-neutral prompts (e.g., “a teacher and students communicating in a classroom”), then translated into Chinese, Thai, and Hindi and optionally paired with a country cue. To score the generated images, the authors propose two evaluation strategies, SAE and IAE, in which three vision-language models propose and verify visual evidence for each value orientation. Evaluating six T2I systems on 960 scenarios, they find that all systems express similar values: low power distance, individualism, care and quality of life, long-term orientation, high uncertainty avoidance, and indulgence. Prompt language has little effect on these orientations, while country cues shift some dimensions but mostly preserve the overall pattern.

Critical strengths
  1. I find the topic very timely and important. While human values preferences have been long studied in LLMs, relatively little work has explored them in the context of image generation. I therefore consider this paper a valuable step towards addressing this gap.
  2. The prompt construction process is careful and well motivated. I liked how the authors turned CDEval questions into prompts for T2I models.
  3. I have some reservations about using AI models as judges, especially models from the same developers as some of the systems being evaluated. Still, I appreciate that the authors use three different models and require majority agreement instead of relying on a single judge.
Critical weaknesses
  1. One issue I had was with the presentation. I find the paper very dense, and at times hard to read. Many important details appear only in the appendix, so the main text is hard to follow on its own. For example, the abstract mentions 960 scenarios while Section 3 describes 2,773, and the reason is explained only in Appendix A.3.

  2. Even though I know that Hofstede have been used to study value orientation in LLMs, I am not entirely convinced that it's approrpiate for images. Some dimensions are visible in an image (e.g. power distances and individualism show up in how people are arranged and interact). Others, especially uncertainty avoidance or long-term orientation are about innter attitutes, or indirect visual cues which are more ambiguous. The papers results also reflect this, e.g. SAE and IAE disagree on UAI.

The authors acknowledge some of these limitations in Appendix H, but I believe dimension-specific validation is needed. A potential idea is a positive control with prompts that explicitly specify an orientation, to test whether the pipeline recovers it (e.g. high UAI: “a passenger in an autonomous car, gripping the seat and anxiously checking the safety manual”, low UAI: “a passenger in an autonomous car, relaxed and curious, happily letting the car choose the route”).

  1. I see in the "reproducibility statement" that the authors intend to release the benchmark data. However, I believe that the full set of prompts should already be available for review given that the benchmarks relies heavility on the prompt construction process. Even though some prompt examples are provided, I think that these are insufficient to assess whether the prompts satisfy the criteria described in Section 3.

  2. The authors generate only one image per scenario/model/experimental condition. However, given the stochastic nature of t2i models, it is unclear whether repeated generations of the same prompt would express the same cultural orientation or produce different ones. While aggregating results across 960 scenarios, captures overall tendencies, evaluating multiple generations, even for a subset of scenarios, would provide a more complete picture of the model's behavior.

Questions for authors

I am missing a discussion of what the desired model behavior would be (this relates to point 4 in critical weaknesses section). An LLM can offer several valid interpretations of the same question within one response, but an image generator must commit to a single image. So what should we expect? Should a model produce different interpretations across generations? Should it adapt its outputs to each country? Aligning per country is also a simplification, since values vary within countries and not everyone in a country shares the same orientation. Finally, the authors briefly suggest letting users specify the value orientation they intend, but it is unclear how this would work in practice, since the relevant values differ from scenario to scenario. Without a clear notion of the target behavior, it is hard to interpret whether the observed homogeneity is a problem and what a better outcome would look like.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7ycP (1/2)

Dear Reviewer 7ycP,

Thank you for your thoughtful review. We are glad that you find the topic timely and important, the prompt construction careful and well motivated, and our use of three judges with majority agreement a reasonable safeguard. We address each point below.

W1. The paper is dense, and key details appear only in the appendix.

The full VisionValueBench contains 2,773 scenarios, and our experiments evaluate a random sample of 960 of them (160 per dimension). We use this subset for two reasons. First, evaluating every scenario across six systems and up to 14 prompt conditions is costly. Second, the results are already stable at this size: profiles computed on 240 and 480 scenarios are essentially the same as those on 960. The revised paper states this in Sections 1 and 3. Using the additional page allowed in the revision, we also moved key details from the appendix into the main text: the evaluation protocol and the human-validation design to Sections 3 and 4. The paper is updated.

W2. Is Hofstede’s framework appropriate for images? Dimension-specific validation.

We agree that the six dimensions are not equally visible. Our method does not infer the inner values of the people depicted. We ask whether the image shows visible evidence, such as behavior, expressions or how options are treated, that expresses an orientation. Following your suggestion, we validated each dimension in two ways. Both show that all six dimensions can be measured from images, and that UAI is the hardest.

Positive control. For ten scenarios per dimension plus the autonomous-car example you mentioned, we wrote pairs of prompts that differ only in one visually concrete detail expressing one orientation as “high UAI: “a passenger in an autonomous car, gripping the seat and anxiously checking the safety manual””, generated images with four systems (488 images), and scored them with the same pipeline, whose verifiers see only the original value-neutral scenario text. Recovery is the share of images in which the depicted orientation is supported. As a base rate, we apply the same measurement to the image generated from the original value-neutral prompt for the same scenario and system, which shows how often that orientation is found when the prompt does not request it:

DimensionRecoveryNeutral base rate
PDI0.790.33
IDV0.780.33
MAS0.900.51
UAI0.560.17
LTO0.740.36
IVR0.640.31

Recovery exceeds the base rate by 0.33–0.46 in every dimension, and the autonomous-car example is recovered in all eight images. Because the verifiers don’t see the orientation-bearing prompt, this gap between the recovery scores and neutral base rate can only come from the generated image. UAI and IVR have the lowest recovery rate, because some of their cues concern a motivational state that an image can suggest but not establish.

Human agreement per dimension. On the 60 human-reviewed images (expanded in the rebuttal phase), IAE reaches κ = 0.62 on LTO against the human majority, within the range of the more visible dimensions (0.52–0.72), and κ = 0.53 on UAI, close to the lowest. LTO is therefore not harder to evaluate than the other dimensions, while UAI is among the hardest.

SAE–IAE disagreement on UAI. This disagreement reflects the method rather than missing evidence: annotators did find UAI evidence in the images, and IAE recovered 15 of the 33 human-supported UAI orientations, while SAE recovered only 2 of 16. This is why the paper uses IAE and treats SAE’s UAI results with caution. In Appendix A.4, we added benchmark images of high and low UAI and of long- and short-term LTO.

W3. The full prompt set should be available for review.

We have released the full benchmark anonymously: https://anonymous.4open.science/r/visionvaluebench-materials-7AF2. It contains all 2,773 scenarios with their source CDEval items and the rendered prompts in every language and cue condition, with the 960 evaluated scenarios marked. We remove 180 scenarios, each with the quoted wording that fixes an orientation.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer 7ycP (2/2)

W4. Only one image per scenario.

Yes, the paper uses one image per prompt. Since our conclusions aggregate over many scenarios, they are stable across samples. For better illustration, we generated four more images per prompt and added this experiment to Appendix E.8.

Setup. We used three systems that we run on our own GPUs: HiDream with its refiner, FLUX.2 with its upsampler, and Ideogram with Magic Prompt. We used 240 scenarios (a fixed quarter of the benchmark, 40 per dimension) and English prompts with no cue and with the China cue. For each prompt, sample 1 is the image in the paper and samples 2–5 are new. Each new sample redraws both the diffusion seed and the output of the prompt rewriter. We scored all 5,760 new images with the same judges and protocol.

Results. The table gives each IAE score averaged over the five samples, with the minimum and maximum across samples (no cue).

SystemPDIIDVMASUAILTOIVR
HiDream + Refiner−0.91 [−0.98, −0.88]+0.30 [+0.18, +0.38]−0.25 [−0.34, −0.17]+0.19 [+0.05, +0.33]+0.56 [+0.48, +0.68]+0.81 [+0.76, +0.88]
FLUX.2 + Upsampler−0.87 [−0.93, −0.76]+0.37 [+0.28, +0.48]−0.24 [−0.28, −0.18]+0.02 [−0.03, +0.10]+0.66 [+0.53, +0.73]+0.73 [+0.63, +0.87]
Ideogram + Magic Prompt−0.87 [−0.92, −0.83]+0.43 [+0.37, +0.48]−0.26 [−0.31, −0.21]+0.26 [+0.00, +0.49]+0.60 [+0.57, +0.63]+0.82 [+0.75, +0.87]

It’s shown that value directions are stable. Across the five samples, 87 of 90 IAE scores and 89 of 90 SAE scores have the same direction as in the paper. The exceptions are UAI scores close to zero (|S| ≤ 0.04), where the two UAI orientations appear about equally often. The China-cue effect is also stable. Under IAE, adding the China cue raises the PDI score and lowers the IDV score for all three systems in all five samples. Individual images vary, but rarely in direction. For a given prompt and dimension, the new samples include opposite orientations in only 3–4.5% of cases.

Judges from the same developers as some evaluated systems.

Our evaluation is based on multi-agent voting. Our three judges come from different developers (Gemini from Google, Muse from Meta, and Luna from OpenAI), and a label requires a majority of the three, so no single developer's model decides an orientation. For each evaluated system, at most one of the three judges shares its developer, so the other two still vote independently.

We also validate the reliability of VLM judges against humans: on 60 images, IAE reaches κ = 0.65 against the human majority, indicating strong agreement with human evaluation.

Q. What is the desired model behavior?

We agree that homogeneity is not a flaw in itself, and we do not claim that any value orientation is better or worse. The main problem we want to highlight is that users can barely change an image’s value orientation by changing the prompt language or changing the model. This means models implicitly generate images with a particular value orientation, even when serving users from different cultural backgrounds. Our project aims to make these values measurable, so that people can see which values a model’s images reflect by default. Looking ahead, we hope for models that adapt easily to different users: a foundation model should be able to generate a more diverse range of values and adjust its outputs when given a cultural cue, such as the prompt language or a country. Our results show that current models do this only partially. Country cues shift the values, while changing the prompt language barely does. We have revised the implications section to state this more clearly.

Moreover, we do not take any country’s scores as the target, since, as you note, values vary within countries; a country cue is one signal a model can follow, instead of a target to directly align with.

Thank you again for your careful and constructive review. We hope the new experiments and revisions address your concerns, and if so, we would greatly appreciate your considering a higher score.

Dear Reviewer 7ycP,

Thank you again for your thoughtful feedback. As the discussion period is coming to a close, we would greatly appreciate it if you could take a moment to review our responses and the revised manuscript.

If any questions remain, we would be glad to address them in the time remaining. If your concerns have been resolved, we would be grateful if you would update your score accordingly, if necessary.

Best regards,

The Authors

Reviewer_p14zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors present VisionValueBench, a benchmark for measuring which cultural values T2I models implicitly express when generating images from value neutral prompts. The authors also introduce evaluation metrics (SAE and IAE) to extract and evaluate the values from the generated images. The main finding is that all the T2I models surprisingly express similar cultural values across their generation even when prompts are written in different languages or specify different countries.

Critical strengths

The paper moves beyond the usual concept centric evaluation prevalent for cultural evaluations of T2I models to value centric evaluations. This is a very interesting, relevant and relatively underexplored area and the benchmark and evaluation method proposed by the authors can be a nice starting point to look deeper into this.

Critical weaknesses
  1. My major concern is the claim of value neutrality of constructed scenarios in the benchmark. The authors claim that they remove value specific mentions from the prompts. However, this does not mean that depictions of different orientations of a particular value are also equally easy to depict. For instance, taking the example from Fig.1, for the prompt - a person shopping in the department store - it feels more natural for AI systems to depict people having items than just staring at something in the store due to the explicit mention of shopping in the prompt. Similarly, for the prompt - A farmer is harvesting apples in orchards - it feels very unnatural to cut a tree to harvest apples. The natural way feels like just plucking them. Hence, it might be easier and more natural for the generative model to depict one value over the other as a consequence of scenario construction which might confound with the actual values represented by the T2I models. In short, I feel there is semantic neutrality which the authors try to ensure in their prompt but there are no checks for generative neutrality (whether generations of different orientations of a value are equally likely) which might bias generations in categories like indulgence.

  2. The authors introduce an elaborate evaluation framework which is one of their methodological contributions. However, they do not compare it with simpler methods like directly asking the VLMs directly (value and the reason) and systematically compare with their proposed method. This is a critical weakness because the authors proposed multi-agent method is significantly more expensive than simpler methods. Moreover human validation is limited to 20 images. This is not sufficient to draw any claims or do any significance analysis. I would encourage the authors to expand this to at least 50-100 images. Also, if possible it would be good to see diverse human annotators evaluate these tasks since these images are culturally very diverse and might lead to different responses from different people and it would be interesting to see the agreement.

Questions for authors

Questions:

  1. Please address the points raised in the critical weakness section
  2. Have the authors generated only 1 image for a prompt? This is what I got from reading the paper. If true, then the authors should consider generating more images per prompt. I know that this is expensive for closed models but an analysis of even open models could be very useful.
  3. I have concerns about the threshold of 0.1. Have the authors tested if this change is a perceptually or culturally meaningful change in the generated images? One way to test this would be to ask people to judge images at this threshold and see whether they feel there is a meaningful difference in value orientations above this threshold and vice versa.

Suggestions:

  1. Although the authors don’t claim this directly in the paper, the subsection on adding US as a geographic identifier not changing anything needs to be more comprehensive. Is this specific to the US? This experiment needs to be tested with more countries. Otherwise I don't think it adds anything meaningful but leaves open the interpretation that these models are US centric.

  2. The authors could also discuss to which countries the dominant values shown by the models correspond to? Is it more similar to the US and how different is it to the countries being probed?

  3. This is more of a discussion point rather than an explicit weakness. The authors highlight the same cultural values being depicted by the models as a limitation. But it is my opinion that given how underspecified the prompts are, why is such homogeneity a limitation? Earlier when T2I models depicted one set of imagery for a particular country it was called stereotyping. What I see now is an improvement where across cultures when the prompt is underspecified similar values are shown (at least from a broader deployment point of view). Also, I would argue that automatically changing social values based just on geographic cues could be considered/introduce stereotypes. It would be great if the authors could better motivate the implications section and give their thoughts on why this homogeneity in such underspecified and value neutral prompts is concerning. What would be more interesting is if models showed such biases even after explicitly instructed.

Overall, I like the broad direction of the paper. If the authors address the experimental concerns and have a meaningful discussion on other points I would lean towards acceptance.

4 comments
AuthorsOpenReview ↗

Response to Reviewer p14z (1/3)

Dear Reviewer p14z,

Thank you for your detailed and constructive review. We are glad that you find the move from concept-centric to value-centric evaluation interesting, relevant and underexplored, and our benchmark a useful starting point. We address each weakness below and have revised the paper accordingly. Revisions are marked in blue in the updated PDF.

W1. Can both orientations be depicted naturally?

We agree that the orchard example in Figure 1 is unnatural. It was only an illustration for explaining the dimension, not a scenario from the benchmark. We have replaced it, and all examples in Figure 1 now come from the benchmark.

We think the key question is whether both orientations can be depicted naturally in a scenario, not whether they are equally likely to be generated. If both can be depicted naturally and a model still consistently chooses one, that preference is exactly the visual value leakage we measure. A scenario is flawed only if one orientation cannot be depicted naturally. Our construction pipeline already targets this criterion: we require each scenario to naturally admit both orientations through the image’s main visible content, and a filter removes scenarios whose wording already determines one orientation. For example, “a person confronting someone of higher social status” was removed, because confronting a superior already implies low power distance.

The statistics generated images can also confirm it. Each scenario has 36 images: six systems, each under the six prompt conditions. Among scenarios whose images express the dimension, 73% have generated images from both orientations. If we restrict the analysis to these scenarios, the average statistics of all 36 system–dimension profiles still keep the same as our conclusion, indicating a consistent value of homogeneity across models, prompt languages and geographic cues.

W2. Comparison with a simpler evaluation; size and diversity of the human validation.

We choose a multi-agent evaluation framework so that our results do not depend on the biases of any single judge model. On the 60 human-reviewed images (expanded during the rebuttal phase), we compare our pipeline with simpler variants in which each model proposes and verifies evidence on its own with the same prompts (single model), and with a majority vote over these three single-model labels. The table reports Cohen’s κ against the human majority in the image-adaptive review:

Methodκ
Pooled evidence → three-judge vote (ours)0.65
Single model → three-judge vote0.62
Single model: Gemini0.42
Single model: Luna0.60
Single model: Muse0.68

Our pipeline agrees with humans more closely than the single-model vote and two of the three single models. Muse alone is slightly higher (0.68), but single models vary widely (κ 0.42–0.68), and we think just one evaluation model will be biased. Pooling evidence and voting keeps agreement close to the best single model without depending on any one model

We have expanded the human study from 20 to 60 images with 4 more annotators, covering every source dimension and system family, reviewed by 9 annotators whose cultural backgrounds span China, India, the United States and Korea. IAE reaches κ = 0.65 against the human majority, and SAE κ = 0.58 (per-dimension results in our response to Reviewer FfMe, W2). Both methods therefore agree with human evidence review on a larger and more diverse set. We have updated our paper’s human annotation results at Section 4.3.

DimensionAnnotator vs annotator (κ)IAE vs IAE-H (κ)
All0.620.65
PDI0.680.66
IDV0.540.52
MAS0.570.72
UAI0.560.53
LTO0.490.62
IVR0.740.72

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer p14z (2/3)

W2 & Q2. Only one image per prompt.

Yes, the paper uses one image per prompt. Since our conclusions aggregate over many scenarios, they are stable across samples. For better illustration, we generated four more images per prompt and added this experiment to Appendix E.8.

Setup. We used three systems that we run on our own GPUs: HiDream with its refiner, FLUX.2 with its upsampler, and Ideogram with Magic Prompt. We used 240 scenarios (a fixed quarter of the benchmark, 40 per dimension) and English prompts with no cue and with the China cue. For each prompt, sample 1 is the image in the paper and samples 2–5 are new. Each new sample redraws both the diffusion seed and the output of the prompt rewriter. We scored all 5,760 new images with the same judges and protocol.

Results. The table gives each IAE score averaged over the five samples, with the minimum and maximum across samples (no cue).

SystemPDIIDVMASUAILTOIVR
HiDream + Refiner−0.91 [−0.98, −0.88]+0.30 [+0.18, +0.38]−0.25 [−0.34, −0.17]+0.19 [+0.05, +0.33]+0.56 [+0.48, +0.68]+0.81 [+0.76, +0.88]
FLUX.2 + Upsampler−0.87 [−0.93, −0.76]+0.37 [+0.28, +0.48]−0.24 [−0.28, −0.18]+0.02 [−0.03, +0.10]+0.66 [+0.53, +0.73]+0.73 [+0.63, +0.87]
Ideogram + Magic Prompt−0.87 [−0.92, −0.83]+0.43 [+0.37, +0.48]−0.26 [−0.31, −0.21]+0.26 [+0.00, +0.49]+0.60 [+0.57, +0.63]+0.82 [+0.75, +0.87]

It’s shown that value directions are stable. Across the five samples, 87 of 90 IAE scores and 89 of 90 SAE scores have the same direction as in the paper. The exceptions are UAI scores close to zero (|S| ≤ 0.04), where the two UAI orientations appear about equally often. The China-cue effect is also stable. Under IAE, adding the China cue raises the PDI score and lowers the IDV score for all three systems in all five samples. Individual images vary, but rarely in direction. For a given prompt and dimension, the new samples include opposite orientations in only 3–4.5% of cases.

Q3. Is the 0.10 threshold perceptually or culturally meaningful?

The 0.10 threshold is not for a perceptual threshold for single images. Our score S is aggregated over hundreds of images: a change of 0.10 means that, among 100 images expressing a dimension, the net count favoring one orientation changes by 10 (Figure 4b). We use 0.10 only as a rule of thumb to describe how large language and cue effects are. Whether an effect is real is judged by paired confidence intervals, and our main findings (the directions of the value profiles, and the China cue raising PDI and lowering IDV) do not use the threshold.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer p14z (3/3)

S1. Is the US-cue result specific to the US?

We already test more countries. Besides the US and China, English prompts are also paired with India and Thailand cues, and Hindi and Thai prompts with their own countries. We have added the results of Thai and India to Section 5.

Change in IAE score when adding each country cue to English prompts (mean over the three commercial systems) are shown below:

CuePDIIDVMASUAILTOIVR
US−0.00−0.06+0.02+0.08−0.01−0.05
China+0.26−0.25+0.10+0.23−0.04−0.18
India+0.12−0.21+0.04+0.11−0.04−0.09
Thailand+0.15−0.22−0.05+0.14−0.08−0.02

Section 5 also notes that no-cue profiles are closer to US-cue profiles than to China-cue profiles. The result is specific to the US: the China, India and Thailand cues all raise PDI and lower IDV in all three systems, while the US cue changes scores little. Without a cue, the models therefore produce outputs closest to what they produce for the United States cue. However, we’d like to clarify that we can't draw a conclusion that the model is U.S centric. As you mentioned, adding a geographic cue can also lead to stereotypes. Our conclusions here can just show adding a US cue in the prompt is closest to the English prompt’s value orientation in our evaluation framework. We’d like to avoid generalizing these conclusions to the model’s alignment with a certain culture.

S2. Which countries do the dominant values correspond to?

In our paper, we do not take the dominant values to correspond to any single country. Instead, we compare them with the country reference scores from the Hofstede survey (Appendix A, Table 2). Among the four countries we probe, they are closest to the United States. Low power distance, individualism and indulgence match the US (PDI 40, IDV 60, IVR 68) and differ from China, India and Thailand (PDI 64–80, IDV 19–43, IVR 24–45). On other dimensions the profile departs from the US: it leans toward care rather than achievement (MAS; US 62) and toward long-term orientation (LTO; US 50). The cue results agree: the China, India and Thailand cues raise PDI and lower IDV, toward their countries’ scores (S1). We have updated our implication discussion with this analysis in Section 5.

S3. Why is homogeneity under underspecified prompts a limitation?

We agree that homogeneity is not a flaw in itself, and we do not claim that any value orientation is better or worse. The main problem we want to highlight is that users can barely change an image’s value orientation by changing the prompt language or changing the model. This means models implicitly generate images with a particular value orientation, even when serving users from different cultural backgrounds. Our project aims to make these values measurable, so that people can see which values a model’s images reflect by default. Looking ahead, we hope for models that adapt easily to different users: a foundation model should be able to generate a more diverse range of values and adjust its outputs when given a cultural cue, such as the prompt language or a country. Our results show that current models do this only partially. Country cues shift the values, while changing the prompt language barely does. We have revised the implications section to state this more clearly.

Thank you again for your careful review. We hope the new experiments and revised paper address your concerns, and we would be grateful if you would consider updating your score accordingly.

Dear Reviewer p14z,

Thank you again for your thoughtful feedback. As the discussion period is coming to a close, we would greatly appreciate it if you could take a moment to review our responses and the revised manuscript.

If any questions remain, we would be glad to address them in the time remaining. If your concerns have been resolved, we would be grateful if you would update your score accordingly, if necessary.

Best regards,

The Authors

Reviewer_FfMeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VisionValueBench benchmark based on six Hofstede’s cultural dimensions. They collect specific scenarios that can target each of these dimensions and without explicitly mentioning in the prompt they generate the scenario and translate them to 3 other languages and given the scenarios in each language T2I models generate the images. They further introduce evaluation framework including two variations of SAE (scenario anticipated evaluation) and IAE (image adaptive evaluation) and finally validate it with human evaluations.

Critical strengths
  • This is an interesting, and under explored problem and study of T2I models’ cultural value orientations in generating the scenarios.
  • The scenario and prompt construct is carefully designed to prevent representing dimension while keeping the scenario relevant to the dimension.
  • Careful design for different languages separating the language and country cues in the study.
Critical weaknesses
  • Some evidence pieces can more strongly represent some dimensions and some others weaker, but both are considered equal in evaluation (binary) and human validation. To make human validation stronger annotators can compare two images and use the metric to compare the images as well and get the agreement.
  • The human validation is small. 5 annotators from the author’s lab annotate 20 images making it challenging to validate the findings.
  • While precision is high for VLMs is high and recall is low, which can make validations challenging without knowing how the missing orientations are distributed between A and B.
  • The human agreement across different dimension is not provided, given the low kappa agreement on PDI and IDV, without human agreement on each dimension it would be challenging to tell if the low agreement is because of metric error or the ambiguity in the images and evidence.
Questions for authors
3 comments
AuthorsOpenReview ↗

Response to Reviewer FfMe (1/2)

Dear Reviewer FfMe,

Thank you for your careful review. We are glad that you find the study of T2I models’ value orientations interesting and under-explored, and that you find our scenario construction and the separation of language and geographic cues careful. We address each weakness below and have revised the paper accordingly. Revisions are marked in blue in the updated PDF.

W1. Binary decisions treat strong and weak evidence the same.

We agree that the evidence for each cultural orientation can vary in strength. In our work, we chose a binary decision design to prioritize evaluation accuracy, accepting reduced granularity as the tradeoff. Our evaluation metrics are aggregated across all images within a dimension, so they measure how often generated images express an orientation, not how strongly any single image expresses it. Our design is therefore better suited to capturing the value trend of a T2I model than the value strength of an individual image.

We agree that evaluating per-image evidence strength would be valuable supplementary metrics, and we ran one additional analysis for that to verify if the current VLM judges can measure strength: We test whether VLM judges can compare the strength of evidence. We sample 60 image pairs that cover every dimension, every system and four prompt languages; each pair shows two images generated from the same prompt by different systems. Three annotators and three VLM judges answer which image expresses a given orientation more strongly or about the same.

We found that human annotators agree moderately with each other. Their pairwise agreement is only 54% (κ = 0.30), which means it’s hard to get the common sense for the strength of evidence from humans. In comparison, VLM judges agree with each other more closely than annotators do. Their pairwise agreement is 74% (κ = 0.55). But VLM judges agree far less with humans. The best judge, Gemini, reaches κ = 0.30 with the human majority, the other two judges 0.12–0.14, and the VLM majority 0.15.

These results show that VLM judges are internally consistent but share a criterion that differs from human judgment because they rarely treat two images as equally strong. It’s indicated that evaluating the strength of value is an ambiguous task for humans and pretty challenging for LLM-as-a-judge. This is why we evaluate whether an orientation is present rather than how strongly. For our evaluation, IAE scores of VLM judges reached κ = 0.65 with human reviewers. A strength-aware score would be a promising future work which requires calibrating VLM judges against human judgments. We added this discussion to Appendix C.4.

W2. The human validation is small.

We have expanded the human study from 20 to 60 images, and have 4 more annotators outside of the lab, covering each dimension and each system family ten times. The 60 images were reviewed by 9 annotators with cultural backgrounds spanning China, India, the United States and Korea. For each image, we have three annotators. The table reports Cohen’s κ between each method and the human majority:

DimensionSAE (κ)IAE (κ)
All0.580.65
PDI0.650.66
IDV0.460.52
MAS0.660.72
UAI0.200.53
LTO0.650.62
IVR0.830.72

Both methods therefore agree with human evidence review on a larger and more diverse set: IAE reaches κ = 0.65 overall and at least 0.52 in every dimension, and SAE reaches 0.58. IAE agrees with humans more closely than SAE overall, and the gap is largest on UAI (0.53 vs 0.20), where SAE recovers only 2 of the 16 orientations that annotators support. This is why the paper uses IAE for most analyses. We have updated our paper’s human annotation results at Section 4.3.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer FfMe (2/2)

W3. Precision is high but recall is low: how are the misses split between A and B?

High precision indicates that when our judges mark an orientation as supported, humans almost always agree. Low recall indicates the judges miss some orientations that humans see. In other words, the judges are more conservative than humans: they count an orientation only when the evidence is clearly linked to it. We chose this trade-off so that every supported orientation is reliable. Making VLM judges more sensitive to subtle cues is a direction for future work. Low recall can bias the direction of S only if A and B are missed at different rates, so we measured recall separately for A and B on the 60 images. Recall is similar for the two orientations: IAE misses 53 of the 149 human-supported A judgments (36%) and 46 of the 137 B judgments (34%). Missed evidence is therefore not concentrated on one side, so it does not systematically shift the direction of S.

W4. Human agreement per dimension; low κ on PDI and IDV.

We agree that some dimensions can be ambiguous even for humans. To separate metric error from ambiguous evidence, we report per-dimension agreement on the 60 images, all as Cohen’s κ: agreement among the three annotators of each image in the image review (pooled over all annotator pairs), and agreement of each method with its own human reference:

DimensionAnnotator vs annotator (κ)IAE vs IAE-H (κ)SAE vs SAE-H (κ)
All0.620.650.58
PDI0.680.660.65
IDV0.540.520.46
MAS0.570.720.66
UAI0.560.530.20
LTO0.490.620.65
IVR0.740.720.83

For PDI, the low κ in the submission (0.32) came from only 20 images. On 60 images, annotators agree well on PDI (0.68), and so do IAE (0.66) and SAE (0.65). For IDV, annotators themselves agree less (0.54), and both methods are at a similar level (IAE 0.52, SAE 0.46). The lower κ therefore reflects ambiguous IDV evidence rather than metric error.

SAE has the lowest κ on UAI (κ = 0.20), where SAE recovers only 2 of the 16 UAI orientations that annotators support, while IAE reaches κ = 0.53. Our paper already reported this gap in Section 5 and it is why the paper uses IAE for most analyses.

Thank you for your feedback on our work again! We hope that our responses address your concerns, and we are happy to discuss any remaining questions.

Dear Reviewer FfMe,

Thank you again for your thoughtful feedback. As the discussion period is coming to a close, we would greatly appreciate it if you could take a moment to review our responses and the revised manuscript.

If any questions remain, we would be glad to address them in the time remaining. If your concerns have been resolved, we would be grateful if you would update your score accordingly, if necessary.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We sincerely thank all reviewers for their time and insightful comments on our paper. Below, we summarize our key contributions and address the main concerns raised during the review process.

Key Contributions

  • A new problem and benchmark. All three reviewers find this problem interesting, important and under-explored. VisionValueBench moves the cultural evaluation of T2I models from objects to values. It contains 2,773 value-neutral scenarios over Hofstede's six dimensions, in 4 languages, with geographic cues that separate language from country.
  • Human-validated evaluation. We propose two evaluation methods, Scenario-Anticipated Evaluation (SAE) and Image-Adaptive Evaluation (IAE). Both ground every judgment in visible evidence that three VLM judges verify. On 60 images reviewed by nine culturally diverse annotators, IAE agrees with the human majority at κ = 0.65.
  • A striking new finding: different T2I systems express the same values. We provide the first systematic measurement of visual value leakage and find strong value homogeneity across systems: all six systems, commercial and open-weight, share one value profile (low power distance, individualism, care, long-term orientation, high uncertainty avoidance and indulgence), which stays stable across repeated samples. Switching the prompt language barely changes this profile, and geographic cues mainly change its strength while largely preserving its directions.

Replies to Key Concerns

  • The human validation is small (FfMe, p14z). The study is now 3× larger, with 60 images reviewed by nine annotators from China, India, the US and Korea. IAE reaches κ = 0.65 overall and ≥ 0.52 on every dimension.
  • Are the prompts value-neutral? (p14z, 7ycP) A filter removed the 180 scenarios whose wording fixes an orientation, and the full benchmark is public at: https://anonymous.4open.science/r/visionvaluebench-materials-7AF2.
  • Implications of our findings (p14z, 7ycP). We do not claim that homogeneity is bad. Our concern is that users can barely change the values an image expresses, whether they switch the model or the prompt language, so users from different cultures receive the same default values, often without noticing. VisionValueBench makes these default values visible and measurable. Looking ahead, we hope for models that adapt to different users, adjusting the values they express when given a cultural cue.

We greatly appreciate the reviewers' valuable feedback, which has helped us refine and strengthen our work. We have also updated our paper accordingly and hightlighted all the changes .

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.