acceptodds
Withdrawn from ICLR 2027

CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs), trained primarily on English-centric data, frequently generate culturally inappropriate or misaligned responses in cross-cultural settings. To mitigate this, we introduce the task of cross-cultural knowledge insertion, which focuses on adapting models to specific cultural contexts while preserving their original behavior in other cultures. To facilitate research in this area, we introduce CrossCult-KIBench, a comprehensive evaluation benchmark for assessing both the effectiveness of knowledge insertion and its unintended side effects on non-target cultures. The benchmark includes 9,800 image-grounded cases covering 49 culturally relevant visual scenarios across English, Chinese, and Arabic language-culture groups. It supports evaluation in both single-insert and sequential-insert settings. We also propose Memory-Conditioned Knowledge Insertion (MCKI) as a baseline method. MCKI retrieves relevant cultural knowledge from an external memory using frozen MLLM representations, prepending matched entries as conditional prompts when applicable. Extensive experiments on CrossCult-KIBench reveal that current approaches struggle to balance effective cultural adaptation with behavioral preservation, highlighting a key challenge in developing culturally-aware MLLMs. Our work thus underscores an important research direction for developing more culturally adaptive and responsible MLLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Knowledge Editing, Cross-Cultural Knowledge Insertion

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cUE3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CrossCult-KIBench, a benchmark for evaluating cross-cultural knowledge insertion in MLLMs. It contains 9,800 multilingual cases across 49 cultural scenarios and supports both single-insert and sequential-insert evaluation. The authors also propose MCKI, a memory-based insertion method using frozen MLLM representations and a trainable router. While the research problem is meaningful, concerns remain regarding task novelty, benchmark validity, evaluation design, and experimental rigor.

Critical strengths

S1. Meaningful research direction. The paper addresses an important problem in culturally adaptive MLLMs, emphasizing both targeted knowledge insertion and preservation of non-target behavior.

S2. Structured benchmark design. CrossCult-KIBench covers multiple cultural topics and introduces both single-insert and sequential-insert settings, providing complementary perspectives on insertion effectiveness and knowledge interference.

S3. Comprehensive baseline comparisons. The experiments include representative parameter-based and memory-based knowledge editing methods, two MLLM backbones, and supplementary analyses of retention, human evaluation, and efficiency.

Critical weaknesses

W1. Insufficient distinction from existing knowledge editing.

The proposed task closely resembles conventional knowledge editing, where targeted model behavior is modified while unrelated behavior is preserved. The main distinction is that the inserted knowledge is cultural rather than factual. However, the benchmark does not verify whether the target knowledge is actually missing before insertion. It is therefore unclear whether the task evaluates knowledge insertion, correction, or reinforcement of existing knowledge. The authors should better justify the need for a separate task formulation.

W2. Questionable validity of culture-specific ground truth.

Reference answers are generated from manually constructed scenario metadata and subsequently checked for consistency with that metadata. This does not independently establish cultural correctness. Some examples involve subjective societal judgments, such as public attitudes toward surveillance (Figure 15), which may vary considerably within the same cultural community. Moreover, associating Arabic with a broad Arab cultural region risks overlooking substantial cultural and religious diversity. Independent validation by culturally knowledgeable annotators and clearer treatment of ambiguous cases are needed.

W3. Limitations in the evaluation metrics.

Locality measures similarity to the original model's outputs, even when those outputs may already be culturally inappropriate. Thus, preserving incorrect behavior can receive a high score. Furthermore, the equal-weighted Overall score can obscure important trade-offs, which should be clarified more clearly. For instance, on Qwen3.5-9B, MCKI achieves higher Overall ROUGE-L than MSCKE (65.84 vs. 56.87), despite substantially lower Reliability (50.66 vs. 71.27).

W4. Insufficient differentiation from retrieval-augmented generation (RAG).

MCKI closely resembles a conventional RAG framework, where stored image-question-answer examples are retrieved through embedding similarity and incorporated into the generation prompt. Its main distinction appears to be the trainable multimodal router and threshold-based activation. However, the paper does not demonstrate whether these components provide meaningful improvements over simpler RAG alternatives. A controlled comparison with a frozen-embedding retrieval baseline using the same memory entries and prompting strategy is necessary to establish the specific contribution of MCKI.

W5. Concerns about baseline fairness and experimental robustness.

The FineTune baseline uses 20 optimization steps with a learning rate of 1e-3, potentially causing severe generation degradation. The repetitive and incoherent outputs shown in Figures 14–15 suggest that some performance drops may result from unstable optimization rather than knowledge interference alone. Stronger hyperparameter tuning is needed.

W6. Limited generalizability of the benchmark setting.

Sequential insertion involves only three updates on the same image, limiting its relevance to realistic settings involving heterogeneous scenarios and larger knowledge memories. The single-insert and sequential-insert tests also use different scenario subsets, complicating comparisons of task difficulty. Finally, the paper does not sufficiently isolate the contribution of visual information through text-only or image-perturbation controls, nor does it establish generalization to independently constructed cultural benchmarks.

Questions for authors
  1. How does the proposed task fundamentally differ from existing multimodal knowledge editing, and how are genuinely missing cultural knowledge cases identified?
  2. How are culture-specific reference answers independently validated, particularly for subjective or contested cultural judgments?
  3. Can the authors provide stronger retrieval baselines, MCKI ablations, and better-tuned parameter-editing baselines to substantiate the reported improvements?
  4. How does MCKI perform under longer, mixed-scenario insertion sequences, and are its gains statistically significant?
Reviewer_XFTnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CrossCult-KIBench, a benchmark for cross-cultural knowledge insertion in multimodal large language models (MLLMs). The task aims to incorporate culture-specific knowledge while preserving the model's behavior in non-target contexts. The benchmark contains 9,800 image-grounded cases across 49 visual scenarios and three language-culture partitions (English/U.S., Chinese/China, and Arabic/Arab region). It evaluates both single-insert and sequential-insert settings using Reliability, Generality, and Locality metrics. The authors also propose Memory-Conditioned Knowledge Insertion (MCKI), which uses a trained router over frozen MLLM representations to retrieve relevant knowledge from external memory. Experiments on InternVL3.5-8B and Qwen3.5-9B suggest that existing editing methods struggle to balance successful cultural adaptation and behavioral preservation, while MCKI achieves stronger aggregate performance.

Critical strengths
  • The paper makes a useful distinction between conventional factual knowledge editing and inserting missing culture-specific knowledge. Evaluating both insertion success and unintended effects on other cultural contexts is practically relevant. The inclusion of single-insert and sequential-insert settings also helps examine different aspects of knowledge insertion.

  • The benchmark covers 49 scenarios across three cultural topics, with a documented construction pipeline combining human-designed metadata, public and generated images, and multilingual QA generation. The experiments compare several representative editing approaches and reveal meaningful trade-offs between insertion effectiveness, generalization, and locality. These findings could be useful for future research on culturally adaptive MLLMs.

Critical weaknesses
  1. Insufficient validation of the cultural reference answers.

My main concern is whether the benchmark's reference answers reliably represent the cultural judgments they are intended to measure. The reference answers are generated from author-designed scenario metadata using Gemini 3.1 Flash-Lite and subsequently reviewed for consistency with that metadata. However, the paper does not provide sufficient information about independent cultural validation, annotators' cultural backgrounds, inter-annotator agreement, or the extent of disagreement and revision. This matters because cultural judgments are often heterogeneous and context-dependent. For example, the benchmark uses broad language-culture partitions, particularly Arabic/Arab region, and presents single reference judgments for potentially subjective or socially sensitive questions, such as same-sex marriage and alcohol consumption (Figure 2). The different descriptions of wearing green hats in Figures 2 and 3 also raise questions about how consistently cultural interpretations are represented. Although Appendix D.1 provides human validation of the LLM judge, this does not establish the cultural validity of the underlying reference answers. Since the benchmark is the paper's primary contribution, stronger evidence that its reference answers are culturally appropriate and sufficiently representative is important for supporting its conclusions.

  1. The Overall metric may overstate MCKI's advantages, and the comparison requires stronger validation.

The Overall score assigns equal weight to Reliability, Generality, Cross-Language Locality, and Cross-Scenario Locality. However, Locality is measured against the original base model's outputs rather than independently validated reference answers. As a result, an unchanged model obtains perfect or near-perfect Locality scores regardless of whether its original responses are culturally appropriate. This creates a potentially misleading aggregate comparison. For example, on Qwen3.5-9B in Table 2, the unedited Base Model achieves an Overall ROUGE-L score of 57.05, exceeding several editing methods, including MSCKE (56.87). MCKI achieves the highest Overall score (65.84), but its Reliability is only 50.66, substantially below MSCKE (71.27). Its advantage therefore comes primarily from preserving non-target behavior rather than achieving the strongest target insertion performance. Moreover, MCKI explicitly trains its router to distinguish Generality requests from Locality requests and falls back to the base model when retrieval is not activated. This naturally favors the benchmark's preservation metrics. The paper does not report routing precision/recall, activation rates, or threshold sensitivity, making it difficult to assess whether its gains reflect effective knowledge transfer or conservative routing. The comparison is further complicated by the severe degeneration of some parameter-editing baselines, particularly FineTune and MEND in the sequential setting. For example, FineTune uses 20 update steps with a learning rate of 1e-3 (Table 4), but the paper does not establish whether alternative hyperparameters could substantially improve its locality. These issues do not invalidate the reported trade-offs, but they weaken the evidence that MCKI is generally superior to existing approaches.

Questions for authors
  • Could the authors clarify who constructed and validated the cultural metadata and reference answers, including their relevant cultural or linguistic backgrounds? Were multiple independent annotators involved, and are there any agreement or disagreement statistics? How were subjective or potentially controversial cultural judgments handled?

  • Appendix B.1 states that the router uses paired Generality evaluation items as positives and Locality evaluation items as negatives. Could the authors clarify whether these are exclusively training-split items and whether any test images, questions, scenarios, or evaluation pairs were used during router training or threshold calibration? Reporting routing precision/recall and activation rates would also help interpret MCKI's performance.

  • Given that the unedited Base Model already achieves an Overall ROUGE-L score above 57 on Qwen3.5-9B, and that MCKI trails MSCKE substantially in Reliability, how should the aggregate score be interpreted as evidence of successful knowledge insertion? Would the conclusions remain similar under alternative metric weightings or when non-target cultural correctness is evaluated independently of the base model's outputs?

  • Were the FineTune and MEND hyperparameters tuned to balance Reliability and Locality, particularly given their severe degeneration in sequential insertion? Could the authors also clarify whether the single-insert and sequential-insert test sets contain disjoint scenarios (Table 1), and how memory entries and demonstrations are constructed for each method at test time?

Reviewer_R5txOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce the task of cross-cultural knowledge insertion and created a benchmark to evaluate MLLMs on that task. The task focuses on adapting models to specific cultural contexts while preserving their original behavior in other cultures. Their benchmark, CrossCult-KIBench includes 9,800 image-grounded cases covering 49 culturally relevant visual scenarios across English, Chinese, and Arabic language-culture groups. They also propose Memory-Conditioned Knowledge Insertion (MCKI) as a baseline method which retrieves relevant cultural knowledge from an external memory using frozen MLLM representations. They show that current approaches struggle to balance effective cultural adaptation with behavioral preservation.

Critical strengths
  • The task is timely and relevant to the current state of AI models.
  • Utilizing World Values Survey (WVS) greatly support the benchmark legitimicy. However, they don't fully utilize its data.
  • The benchmark size is good.
Critical weaknesses
  • The task definition is not clear. f' is the model after insertion (f' = insert(f,e^P)) is desired to produce an answer consistent with the cultural answer (a^p). However, the cultural answer is a part of the insertion process (e^p = (i, q^p, a^p)), so the model already has access to the answer.
  • The insertion process is not defined.
  • Figure 3 is very confusing with the arrows going in different directions in each subfigure.
  • The authors didn't make use of the rich WVS. They ask gemini to generate QA instead of using the reliable WVS QA
  • Many captions in the figures are not translated to English making the figures hard to understand. (Figure 3 for example)
  • The metdata construction step doesn't discuss how the scenarios were chosen and how the description were written and verified.
  • No details mentioned about the background of people who did manual work.
  • no ablations for the memory construction.
  • The MCKI approach is simply a conditioned few-shot prompting where the condition depends on cosine similarity. There is no need to introduce it as a novel methodology.
  • Where does the results in Figure 1B come from?
Questions for authors

Address the weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.