acceptodds
Under review as a conference paper at ICLR 2027

Seeing Culture When Words Fall Short: Uncertainty-Gated Visual Grounding of Cultural Cues for LLMs

Abstract

Large Language Models (LLMs) have achieved strong performance across a wide range of tasks, yet they remain vulnerable to cultural bias due to the uneven distribution of cultural knowledge in training corpora. This limitation often leads to inaccurate understanding and generation in cross-cultural contexts, particularly when textual cues alone are insufficient. To address this issue at inference time, we propose V-CUE (Visual-Cultural Understanding Enhancement), a visual-space-assisted framework that improves culturally grounded reasoning by incorporating generated visual cues. Specifically, V-CUE first employs an uncertainty detection mechanism to identify potentially unreliable responses. Based on this signal, a text-to-image model generates culturally relevant visual content, including symbols, clothing, and contextual elements. These visual signals are incorporated into the generation process to refine model predictions and enhance cultural grounding. Evaluated on understanding task (CulturalBench) and generation task (CARE) across LLM-text, LLM-reasoning, and LLM-VL systems, V-CUE yields consistent accuracy gains of up to +19 points, while uncertainty gating reduces token cost by 46% with negligible accuracy loss. Further analyses confirm that gains scale with visual–query alignment quality and that V-CUE remains complementary to retrieval-augmented approaches. Without requiring additional training data, V-CUE offers a practical and generalizable multimodal pathway for enhancing the cultural robustness of LLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
cultural bias, cultural grounding, visual imagination, text-to-image generation, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x3x3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes V-CUE, which uses image generation models to provide visual hints for answering cultural questions. It first performs Uncertainty Detection (UD). If the LLM is uncertain, it extracts the cultural region, object, and symbol from the question as conditions for a diffusion-based image generation model (VG). Finally, Visual-based Generation Reconstruction (VR) uses the original question and generated image to produce the final answer for multimodal LLMs. For text-only LLMs, a vision-language model first generates a caption of the image, which is then combined with the original question to generate the answer.

Critical strengths
  • The overall pipeline is easy to understand.
Critical weaknesses
  1. Including CulFiT in Table 1 is not meaningful without evaluating Qwen2.5-7B and Llama3.1-8B with V-CUE, as there is no directly comparable baseline.
  2. Figure 3 lacks a color legend. I can only infer that purple, green, and orange represent LLM-VL, LLM-reasoning, and LLM-text systems respectively based on Table 1.
  3. Section 6.3 does not explicitly identify the models discussed, requiring readers to refer to the table. It also does not justify why different rows are selected for different comparisons. Since the results show consistent trends, an overall analysis would be clearer. Otherwise, the authors should explain why you select these results to compare.
  4. In Section 6.4, UD-gated achieves the lowest accuracy. To demonstrate a favorable accuracy–efficiency trade-off, the authors should also include the original model without V-CUE as the fastest baseline.
  5. Introducing an additional image generation model may introduce the knowledge from its, producing more comprehensive information. Thus, the improvements do not enough to demonstrate that visual representations could provide more information. Section 6.1 also shows that stronger image generators lead to better downstream performance.
  6. The paper repeatedly refers to "supplementary material," but none is provided. If the authors mean the appendix, they should specify the corresponding sections.
Questions for authors
  1. In Section 3, what is the difference between (Q) and (C)? If they refer to the same thing, please use consistent notation.
  2. Why are DeepSeek-V3/R1 and GPT-3.5-turbo/o3-mini used to distinguish LLM-text and LLM-reasoning systems? A more consistent comparison would use the same reasoning model with reasoning disabled (DeepSeek-R1) or reduced thinking effort(o3-mini), as the authors done with Qwen3-32B and Qwen3-235B-A22B. Additionally, why not include reasoning VL model to form a 2×2 comparison?
  3. In Table 1, CultureLLM and CulturePark require training, but their original papers did not use Qwen3-32B. How were these methods reproduced? Did the authors follow the original training setting while only replacing the backbone model? Also, why not apply V-CUE to these models? Although you describe them as “training-time reference points”, V-CUE is an inference-time method and should still be applicable for them.
  4. In Table 1, enabling reasoning reduces the accuracy of both Qwen3-235B-A22B and Qwen3-32B. Are these really generate wrong answers, or do the models reach the maximum token length before finishing? The max token length is not specified in the experiment setting. If it is limited to around 8,192 tokens or fewer, the reasoning models may simply exceed it. While, V-CUE may simply help models answers with lower token length rather than help the model answer correctly.
  5. In Section 6.3, why is the performance drop without UD attributed to " LLMs retain limited inherent capability for cultural attribute recognition"? Doesn't it simply show that UD filtering is useful? One possible explanation is that generating images without filtering introduces additional noise. However, this is only a hypothesis and requires further verification. Moreover, if the LLM has limited ability to recognize cultural elements, as you concluded, why do you use the same model to extract culture-aware condition units?
  6. In Section 6.3, could the authors replace generated images with retrieved similar images as a VG ablation to demonstrate the benefit of image generation? Also, when VG or VR is removed, what role does UD still play, given that it originally determines whether visual generation should be triggered?
  7. In Section 6.3, what exactly does "removes VR for text/reasoning systems" mean? Without VR, how is the final answer generated, and how does the language model use the image produced by VG?
  8. What is the difference between "Always-on" in Section 6.4 and "w/o UD" in Section 6.3?
  9. Which specific models are used in the experiments in Section 6.5?
Reviewer_sHJVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Summary

  1. Research gap: LLMs remain vulnerable to cultural bias due to the uneven distribution of cultural knowledge in training corpora.
  2. This limitation often leads to inaccurate understanding and generation in cross-cultural contexts.
  3. V-CUE: A visual-space-assisted framework that improves culturally grounded reasoning by incorporating generated visual cues.
  • First employs an uncertainty detection mechanism to identify potentially unreliable responses.
  • Second, a text-to-image generates culturally relevant signals; these visual signals are incorporated into the generation process to refine model predictions and enhance cultural grounding.

Authors attempted to reduce the cultural bias in LLMs without additional training.

Critical strengths

Strength

  1. A genuinely novel and well-motivated angle.
  2. The uncertainty gate is a practical, honest contribution.
  3. The analysis section is unusually thorough and anticipates weak points.
  4. Robust human validation of the automatic judge.
Critical weaknesses

Weakness

  1. For non-VL backbones the pathway of Eq. 7 is complemented by the caption generation by Qwen2.5-VL-72B. So, if I am not wrong, then the reported improvements, therefore, are results of a 72B VL model’s cultural knowledge. Also, Table 8 shows a monotonic scaling with the captioner size. It is a signature of a larger model’s knowledge being distilled into text. It seems improvements are independent of a modality-specific visual effect. Also, Table 2 has entries on the VL setting only. As a result, it fails to rule out the confusion for the text setup.

  2. Qwen3-32B as a reasoning system has a base multiple-choice accuracy of 64.2%, which is far below its own text-mode base (79.2%) and below the much smaller GPT-3.5-turbo. Note that a reasoning model scoring below its non-reasoning counterpart on the same items is surprising and indicates a potential parsing issue. So, the term “+19.1 points” partially measures recovery from a degraded baseline rather than cultural correction. Kindly check this issue.

  3. Authors claim that V-CUE reduces cultural errors by enhancing the salience of existing cultural cues through visual signals, rather than introducing new knowledge (see lines 67-69, 496-470, and 482-483). But from Fig. 2, the required concept 'Sanggyeonrye' is not in the given question or in the extracted cues. So, 'Sanggyeonrye' is a new concept rather than an implicit one, and the generator supplies it. Need more clarification on it.

  4. This paper is on cultural robustness, but the evidence skews to high-resource settings. Note that CulturalBench covers 45 global regions, and it reports only aggregate accuracy with no per-region breakdown. So we cannot understand whether its low-resource regions gained, stalled, or regressed. Again, CARE covers only Chinese, Arabic, and Japanese (lines 189-192), all high-resource, and gains are largest on Chinese-based models (lines 297-298). It is important because the paper’s own results suggest V-CUE should help least exactly where it’s needed most. The method works better when the generated image closely matches the question (lines 338-343). But for less-represented cultures, less accurate images are generated. As a result, the match is poor. When the image is a poor match, V-CUE actually scores below the no-image baseline (Table 2, lines 378-380), i.e., it hurts rather than helps. Low-resource cultures where the method, V-CUE, is most likely to backfire. It needs more testing on that scenario.

Questions for authors

Please resolve the issues mentioned in the 'Weakness' section.

Reviewer_fxwLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes V-CUE framework, which generates culturally relevant images when an LLM answer is wrong and uses them to refine its answer. Across CulturalBench and CARE, V-CUE improves the performance text, reasoning, LLMs and VLMs. The authors argue V-CUE mitigate the cultural bias and make the LLMs more robust in culture domain.

Critical strengths
  1. V-CUE is a training-free framework, which can be applicable to LLMs and VLMs.
  2. The experiments cover several model families, cultural understanding tasks, and open-ended generation tasks.
  3. The paper evaluates different image generators, input modalities, cultural relevance, individual components, RAG integration, and inference cost.
Critical weaknesses
  1. The contribution of vision is not convincingly. Image inputs outperform image captions by only about 0.2%–0.6% in figure 5. Although there are ablations for RAG on wikipedia knowledge, the comparison is not fair enough. There should be an experiments about the change the questions to open-end questions and then use the response as the context input for the models.The
  2. cultural-bias claim is insufficiently supported. The benchmarks primarily measure cultural knowledge and generation quality, not bias reduction. The paper does not evaluate whether generated cultural imagery reinforces stereotypes.
  3. Cultural diversity is oversimplified. The benchmark like CARE only have three countries, which is not big enough to support the claim. Representing culture through region, object, and symbol fields may overlook within-culture variation, mixed identities, and contextual nuances.
  4. No explorations about the culture bias across modalities. There are several papers discuss the biases in VLMs, like CVQA[1], FoodieQA[2], RAVENEA[3], there is no quality-check in the image captioning step in the paper.
  5. No supplementary material.
  6. Saving cost is insufficiently supported. V-CUE frameworks will at least run one more round for the model itself to check the uncertainty. Compare to reading image every time, it will save tokens, but this is added by V-CUE framework, not from the baseline.

[1]Romero, David, et al. "Cvqa: Culturally-diverse multilingual visual question answering benchmark."

[2]Li, Wenyan, et al. "FoodieQA: A multimodal dataset for fine-grained understanding of Chinese food culture."

[3Li, Jiaang, et al. "Ravenea: A benchmark for multimodal retrieval-augmented visual culture understanding."

Questions for authors
  1. From line 40 to line 45, I didn’t really see the correlation between the models and their developed countries. Is there any evidence to support this claim?
  2. What’s the std for the five times running in table 1? Why does the baseline models are better than culturellm and culture park in multi-choice?
  3. What if use the unified MLLMs, like Bagel? It has image generation and understanding in one model.
  4. Why is the always-on method worse than oracle-gated in table 4? Does it mean the V-CUE may make the model change the correct answer to the wrong one?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.