- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces CrossCult-KIBench, a benchmark for evaluating cross-cultural knowledge insertion in MLLMs. It contains 9,800 multilingual cases across 49 cultural scenarios and supports both single-insert and sequential-insert evaluation. The authors also propose MCKI, a memory-based insertion method using frozen MLLM representations and a trainable router. While the research problem is meaningful, concerns remain regarding task novelty, benchmark validity, evaluation design, and experimental rigor.
Critical strengths
S1. Meaningful research direction. The paper addresses an important problem in culturally adaptive MLLMs, emphasizing both targeted knowledge insertion and preservation of non-target behavior.
S2. Structured benchmark design. CrossCult-KIBench covers multiple cultural topics and introduces both single-insert and sequential-insert settings, providing complementary perspectives on insertion effectiveness and knowledge interference.
S3. Comprehensive baseline comparisons. The experiments include representative parameter-based and memory-based knowledge editing methods, two MLLM backbones, and supplementary analyses of retention, human evaluation, and efficiency.
Critical weaknesses
W1. Insufficient distinction from existing knowledge editing.
The proposed task closely resembles conventional knowledge editing, where targeted model behavior is modified while unrelated behavior is preserved. The main distinction is that the inserted knowledge is cultural rather than factual. However, the benchmark does not verify whether the target knowledge is actually missing before insertion. It is therefore unclear whether the task evaluates knowledge insertion, correction, or reinforcement of existing knowledge. The authors should better justify the need for a separate task formulation.
W2. Questionable validity of culture-specific ground truth.
Reference answers are generated from manually constructed scenario metadata and subsequently checked for consistency with that metadata. This does not independently establish cultural correctness. Some examples involve subjective societal judgments, such as public attitudes toward surveillance (Figure 15), which may vary considerably within the same cultural community. Moreover, associating Arabic with a broad Arab cultural region risks overlooking substantial cultural and religious diversity. Independent validation by culturally knowledgeable annotators and clearer treatment of ambiguous cases are needed.
W3. Limitations in the evaluation metrics.
Locality measures similarity to the original model's outputs, even when those outputs may already be culturally inappropriate. Thus, preserving incorrect behavior can receive a high score. Furthermore, the equal-weighted Overall score can obscure important trade-offs, which should be clarified more clearly. For instance, on Qwen3.5-9B, MCKI achieves higher Overall ROUGE-L than MSCKE (65.84 vs. 56.87), despite substantially lower Reliability (50.66 vs. 71.27).
W4. Insufficient differentiation from retrieval-augmented generation (RAG).
MCKI closely resembles a conventional RAG framework, where stored image-question-answer examples are retrieved through embedding similarity and incorporated into the generation prompt. Its main distinction appears to be the trainable multimodal router and threshold-based activation. However, the paper does not demonstrate whether these components provide meaningful improvements over simpler RAG alternatives. A controlled comparison with a frozen-embedding retrieval baseline using the same memory entries and prompting strategy is necessary to establish the specific contribution of MCKI.
W5. Concerns about baseline fairness and experimental robustness.
The FineTune baseline uses 20 optimization steps with a learning rate of 1e-3, potentially causing severe generation degradation. The repetitive and incoherent outputs shown in Figures 14–15 suggest that some performance drops may result from unstable optimization rather than knowledge interference alone. Stronger hyperparameter tuning is needed.
W6. Limited generalizability of the benchmark setting.
Sequential insertion involves only three updates on the same image, limiting its relevance to realistic settings involving heterogeneous scenarios and larger knowledge memories. The single-insert and sequential-insert tests also use different scenario subsets, complicating comparisons of task difficulty. Finally, the paper does not sufficiently isolate the contribution of visual information through text-only or image-perturbation controls, nor does it establish generalization to independently constructed cultural benchmarks.
Questions for authors
- How does the proposed task fundamentally differ from existing multimodal knowledge editing, and how are genuinely missing cultural knowledge cases identified?
- How are culture-specific reference answers independently validated, particularly for subjective or contested cultural judgments?
- Can the authors provide stronger retrieval baselines, MCKI ablations, and better-tuned parameter-editing baselines to substantiate the reported improvements?
- How does MCKI perform under longer, mixed-scenario insertion sequences, and are its gains statistically significant?