- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies whether T2I models express particular cultural values when the prompt leaves those values open, a phenomenon the authors call “visual value leakage.” They introduce VisionValueBench, a multilingual benchmark based on Hofstede’s six cultural dimensions. Scenarios are derived from CDEval questions and rewritten as value-neutral prompts (e.g., “a teacher and students communicating in a classroom”), then translated into Chinese, Thai, and Hindi and optionally paired with a country cue. To score the generated images, the authors propose two evaluation strategies, SAE and IAE, in which three vision-language models propose and verify visual evidence for each value orientation. Evaluating six T2I systems on 960 scenarios, they find that all systems express similar values: low power distance, individualism, care and quality of life, long-term orientation, high uncertainty avoidance, and indulgence. Prompt language has little effect on these orientations, while country cues shift some dimensions but mostly preserve the overall pattern.
Critical strengths
- I find the topic very timely and important. While human values preferences have been long studied in LLMs, relatively little work has explored them in the context of image generation. I therefore consider this paper a valuable step towards addressing this gap.
- The prompt construction process is careful and well motivated. I liked how the authors turned CDEval questions into prompts for T2I models.
- I have some reservations about using AI models as judges, especially models from the same developers as some of the systems being evaluated. Still, I appreciate that the authors use three different models and require majority agreement instead of relying on a single judge.
Critical weaknesses
-
One issue I had was with the presentation. I find the paper very dense, and at times hard to read. Many important details appear only in the appendix, so the main text is hard to follow on its own. For example, the abstract mentions 960 scenarios while Section 3 describes 2,773, and the reason is explained only in Appendix A.3.
-
Even though I know that Hofstede have been used to study value orientation in LLMs, I am not entirely convinced that it's approrpiate for images. Some dimensions are visible in an image (e.g. power distances and individualism show up in how people are arranged and interact). Others, especially uncertainty avoidance or long-term orientation are about innter attitutes, or indirect visual cues which are more ambiguous. The papers results also reflect this, e.g. SAE and IAE disagree on UAI.
The authors acknowledge some of these limitations in Appendix H, but I believe dimension-specific validation is needed. A potential idea is a positive control with prompts that explicitly specify an orientation, to test whether the pipeline recovers it (e.g. high UAI: “a passenger in an autonomous car, gripping the seat and anxiously checking the safety manual”, low UAI: “a passenger in an autonomous car, relaxed and curious, happily letting the car choose the route”).
-
I see in the "reproducibility statement" that the authors intend to release the benchmark data. However, I believe that the full set of prompts should already be available for review given that the benchmarks relies heavility on the prompt construction process. Even though some prompt examples are provided, I think that these are insufficient to assess whether the prompts satisfy the criteria described in Section 3.
-
The authors generate only one image per scenario/model/experimental condition. However, given the stochastic nature of t2i models, it is unclear whether repeated generations of the same prompt would express the same cultural orientation or produce different ones. While aggregating results across 960 scenarios, captures overall tendencies, evaluating multiple generations, even for a subset of scenarios, would provide a more complete picture of the model's behavior.
Questions for authors
I am missing a discussion of what the desired model behavior would be (this relates to point 4 in critical weaknesses section). An LLM can offer several valid interpretations of the same question within one response, but an image generator must commit to a single image. So what should we expect? Should a model produce different interpretations across generations? Should it adapt its outputs to each country? Aligning per country is also a simplification, since values vary within countries and not everyone in a country shares the same orientation. Finally, the authors briefly suggest letting users specify the value orientation they intend, but it is unclear how this would work in practice, since the relevant values differ from scenario to scenario. Without a clear notion of the target behavior, it is hard to interpret whether the observed homogeneity is a problem and what a better outcome would look like.
3 comments
Response to Reviewer 7ycP (2/2)
W4. Only one image per scenario.
Yes, the paper uses one image per prompt. Since our conclusions aggregate over many scenarios, they are stable across samples. For better illustration, we generated four more images per prompt and added this experiment to Appendix E.8.
Setup. We used three systems that we run on our own GPUs: HiDream with its refiner, FLUX.2 with its upsampler, and Ideogram with Magic Prompt. We used 240 scenarios (a fixed quarter of the benchmark, 40 per dimension) and English prompts with no cue and with the China cue. For each prompt, sample 1 is the image in the paper and samples 2–5 are new. Each new sample redraws both the diffusion seed and the output of the prompt rewriter. We scored all 5,760 new images with the same judges and protocol.
Results. The table gives each IAE score averaged over the five samples, with the minimum and maximum across samples (no cue).
| System | PDI | IDV | MAS | UAI | LTO | IVR |
|---|---|---|---|---|---|---|
| HiDream + Refiner | −0.91 [−0.98, −0.88] | +0.30 [+0.18, +0.38] | −0.25 [−0.34, −0.17] | +0.19 [+0.05, +0.33] | +0.56 [+0.48, +0.68] | +0.81 [+0.76, +0.88] |
| FLUX.2 + Upsampler | −0.87 [−0.93, −0.76] | +0.37 [+0.28, +0.48] | −0.24 [−0.28, −0.18] | +0.02 [−0.03, +0.10] | +0.66 [+0.53, +0.73] | +0.73 [+0.63, +0.87] |
| Ideogram + Magic Prompt | −0.87 [−0.92, −0.83] | +0.43 [+0.37, +0.48] | −0.26 [−0.31, −0.21] | +0.26 [+0.00, +0.49] | +0.60 [+0.57, +0.63] | +0.82 [+0.75, +0.87] |
It’s shown that value directions are stable. Across the five samples, 87 of 90 IAE scores and 89 of 90 SAE scores have the same direction as in the paper. The exceptions are UAI scores close to zero (|S| ≤ 0.04), where the two UAI orientations appear about equally often. The China-cue effect is also stable. Under IAE, adding the China cue raises the PDI score and lowers the IDV score for all three systems in all five samples. Individual images vary, but rarely in direction. For a given prompt and dimension, the new samples include opposite orientations in only 3–4.5% of cases.
Judges from the same developers as some evaluated systems.
Our evaluation is based on multi-agent voting. Our three judges come from different developers (Gemini from Google, Muse from Meta, and Luna from OpenAI), and a label requires a majority of the three, so no single developer's model decides an orientation. For each evaluated system, at most one of the three judges shares its developer, so the other two still vote independently.
We also validate the reliability of VLM judges against humans: on 60 images, IAE reaches κ = 0.65 against the human majority, indicating strong agreement with human evaluation.
Q. What is the desired model behavior?
We agree that homogeneity is not a flaw in itself, and we do not claim that any value orientation is better or worse. The main problem we want to highlight is that users can barely change an image’s value orientation by changing the prompt language or changing the model. This means models implicitly generate images with a particular value orientation, even when serving users from different cultural backgrounds. Our project aims to make these values measurable, so that people can see which values a model’s images reflect by default. Looking ahead, we hope for models that adapt easily to different users: a foundation model should be able to generate a more diverse range of values and adjust its outputs when given a cultural cue, such as the prompt language or a country. Our results show that current models do this only partially. Country cues shift the values, while changing the prompt language barely does. We have revised the implications section to state this more clearly.
Moreover, we do not take any country’s scores as the target, since, as you note, values vary within countries; a country cue is one signal a model can follow, instead of a target to directly align with.
Thank you again for your careful and constructive review. We hope the new experiments and revisions address your concerns, and if so, we would greatly appreciate your considering a higher score.
Dear Reviewer 7ycP,
Thank you again for your thoughtful feedback. As the discussion period is coming to a close, we would greatly appreciate it if you could take a moment to review our responses and the revised manuscript.
If any questions remain, we would be glad to address them in the time remaining. If your concerns have been resolved, we would be grateful if you would update your score accordingly, if necessary.
Best regards,
The Authors
Response to Reviewer 7ycP (1/2)
Dear Reviewer 7ycP,
Thank you for your thoughtful review. We are glad that you find the topic timely and important, the prompt construction careful and well motivated, and our use of three judges with majority agreement a reasonable safeguard. We address each point below.
W1. The paper is dense, and key details appear only in the appendix.
The full VisionValueBench contains 2,773 scenarios, and our experiments evaluate a random sample of 960 of them (160 per dimension). We use this subset for two reasons. First, evaluating every scenario across six systems and up to 14 prompt conditions is costly. Second, the results are already stable at this size: profiles computed on 240 and 480 scenarios are essentially the same as those on 960. The revised paper states this in Sections 1 and 3. Using the additional page allowed in the revision, we also moved key details from the appendix into the main text: the evaluation protocol and the human-validation design to Sections 3 and 4. The paper is updated.
W2. Is Hofstede’s framework appropriate for images? Dimension-specific validation.
We agree that the six dimensions are not equally visible. Our method does not infer the inner values of the people depicted. We ask whether the image shows visible evidence, such as behavior, expressions or how options are treated, that expresses an orientation. Following your suggestion, we validated each dimension in two ways. Both show that all six dimensions can be measured from images, and that UAI is the hardest.
Positive control. For ten scenarios per dimension plus the autonomous-car example you mentioned, we wrote pairs of prompts that differ only in one visually concrete detail expressing one orientation as “high UAI: “a passenger in an autonomous car, gripping the seat and anxiously checking the safety manual””, generated images with four systems (488 images), and scored them with the same pipeline, whose verifiers see only the original value-neutral scenario text. Recovery is the share of images in which the depicted orientation is supported. As a base rate, we apply the same measurement to the image generated from the original value-neutral prompt for the same scenario and system, which shows how often that orientation is found when the prompt does not request it:
Recovery exceeds the base rate by 0.33–0.46 in every dimension, and the autonomous-car example is recovered in all eight images. Because the verifiers don’t see the orientation-bearing prompt, this gap between the recovery scores and neutral base rate can only come from the generated image. UAI and IVR have the lowest recovery rate, because some of their cues concern a motivational state that an image can suggest but not establish.
Human agreement per dimension. On the 60 human-reviewed images (expanded in the rebuttal phase), IAE reaches κ = 0.62 on LTO against the human majority, within the range of the more visible dimensions (0.52–0.72), and κ = 0.53 on UAI, close to the lowest. LTO is therefore not harder to evaluate than the other dimensions, while UAI is among the hardest.
SAE–IAE disagreement on UAI. This disagreement reflects the method rather than missing evidence: annotators did find UAI evidence in the images, and IAE recovered 15 of the 33 human-supported UAI orientations, while SAE recovered only 2 of 16. This is why the paper uses IAE and treats SAE’s UAI results with caution. In Appendix A.4, we added benchmark images of high and low UAI and of long- and short-term LTO.
W3. The full prompt set should be available for review.
We have released the full benchmark anonymously: https://anonymous.4open.science/r/visionvaluebench-materials-7AF2. It contains all 2,773 scenarios with their source CDEval items and the rendered prompts in every language and cue condition, with the 960 evaluated scenarios marked. We remove 180 scenarios, each with the quoted wording that fixes an orientation.
(continued in the next comment)