- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces Culture-MMT, a benchmark of 498 Chinese UGC translation samples categorized into Symbol, Visual, and Hybrid types. The authors propose a domain-specific Judger and evaluate 17 model configurations under different evidence conditions. While the benchmark addresses a relevant problem and offers useful diagnostic perspectives, I have concerns regarding its novelty, construction validity, evaluation reliability, and experimental conclusions.
Critical strengths
- The paper addresses a relevant problem in culturally aware multimodal translation.
- The evidence-dependency taxonomy provides an intuitive diagnostic framework.
- The evaluation covers diverse models and includes useful evidence-control experiments.
Critical weaknesses
W1. Limited novelty beyond existing multimodal translation research.
The novelty positioning appears overly broad. Visual disambiguation and underspecified translation have been extensively studied (Caglayan et al., 2019; CoMMuTE, Futeral et al., 2023), while CaMMT (Villa-Cueva et al., 2025) explicitly investigates visually grounded cultural translation. Although Culture-MMT introduces a UGC-specific taxonomy, its distinction from previous works remain insufficiently demonstrated.
W2. Concerns about benchmark construction and evidence-dependency validity.
The boundaries between evidence-dependency types are not always convincing. For example, “肥宅复健 Day 1,” labeled Hybrid in Table 1, appears largely interpretable through cultural knowledge alone, raising questions about whether visual evidence is necessary or merely helpful. Furthermore, Visual and Hybrid candidates undergo model-based differential filtering based on translation gains, potentially introducing selection bias. Since the main experiments subsequently measure gains from visual evidence, this procedure may preferentially select samples that already exhibit the expected behavior, potentially biasing the reported improvements. Extensive text rewriting and image regeneration may also alter the original dependencies, yet quantitative validation of label preservation and inter-annotator agreement is missing.
W3. Concerns about evaluation reliability and visual explicitation bias.
The Judger penalizes translations that do not explicitly include information from the image (L899-901), even when the translation is already accurate and natural without those details. Furthermore, the Human Caption condition uses the same task-relevant descriptions available to the Judger, making it difficult to separate genuine translation improvements from alignment with the evaluation reference. Independent human evaluation of the reported visual gains would strengthen the findings.
W4. Insufficient evidence for separating visual and cultural difficulties.
The experiments do not clearly demonstrate that visual understanding and cultural knowledge are separate sources of translation difficulty. In Section 4.3.2, the authors argue that Hybrid samples perform worse than Visual samples even when image descriptions are provided, suggesting that cultural knowledge remains a challenge. However, the two categories contain different samples, and Hybrid samples already have lower scores without images. Therefore, the performance gap cannot be attributed solely to cultural difficulties.
W5. Limitations of experimental controls and statistical robustness.
The Random Image condition may introduce misleading evidence rather than serving as a neutral placebo. Thus, the performance gap between real and random images may reflect both relevant visual grounding and interference from mismatched inputs. Additionally, the paper provides limited statistical significance testing, confidence intervals, or repeated-run analyses despite using stochastic decoding. These limitations weaken the robustness of several relatively small reported gains.
Questions for authors
Q1. What specific capabilities does Culture-MMT evaluate beyond CoMMuTE, CaMMT, CULTURE-MT and other similar previous works?
Q2. Do the main results persist on samples selected without model-based differential filtering, and how reliably are dependency labels preserved after rewriting?
Q3. Do independent human evaluators confirm the visual gains reported by the Judger?
3 comments
Rebuttal - Part II
W4. How do the experiments distinguish visual and cultural difficulty?
The Human Caption scores, 2.316 on Visual and 2.035 on Hybrid, show that the jointly dependent group remains challenging when visual facts are supplied explicitly (Section 4.3.2). Because the groups contain different notes, the comparison is a diagnostic observation rather than a numerical estimate of a purely cultural effect.
The within-note interventions provide a more specific question: does supplying the missing evidence resolve the translation point? In Appendix C, captions enable “keycaps” for “小黄帽,” while another note still renders “这颗蛋” as “this egg” despite the available visual description. The “rounded heel counter” example illustrates an English terminology problem after the fact is recovered.
Together, the controlled inputs and case analysis help identify errors associated with visual fact recovery, reference integration, and target-language expression. These observations show that providing visual facts does not necessarily resolve all translation difficulties, particularly when cultural interpretation is also required. We therefore interpret the results as diagnostic evidence of different failure mechanisms, rather than a strict causal decomposition of independent visual and cultural abilities.
W5. How should random images and statistical robustness be interpreted?
Real-image gains use text-only as the baseline. Random images separately test interference from mismatched evidence, so the real-minus-random difference is not treated as a pure estimate of grounding benefit.
| Type | Text-only | Random image | Real image |
|---|---|---|---|
| Symbol | 2.190 | 2.116 | 2.221 |
| Visual | 1.946 | 1.941 | 2.242 |
| Hybrid | 1.788 | 1.711 | 2.020 |
These four-configuration means (Table 6) show real-image benefits on Visual and Hybrid relative to text-only. Random images do not reproduce that pattern. We will call this a mismatched-image control.
We repeat translation experiments three times and report the averages, with matched translator prompts and decoding settings and deterministic judge decoding. We will specify this aggregation in the setup. The reported subset means describe consistency across configurations, without implying improvement on every note or a formal significance test. All 17 configurations have positive mean gains: +0.145 to +0.446 on Visual and +0.042 to +0.349 on Hybrid (Table 5).
References
[1] Probing the Need for Visual Context in Multimodal Machine Translation.
[2] Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation.
[3] CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation.
[4] Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC.
Thank you for your comments!
Thank you for the comprehensive rebuttal and the additional clarifications! The authors have adequately addressed my questions regarding the paper's novelty, evaluation criteria, and interpretation of the experimental results.
However, I still keep the perspective that the model-based differential filtering risks introducing selection bias, and the sensitivity of the reported gains to this filtering step remains unverified.
I will maintain my score of 2. Thank you again for the constructive clarifications.
Rebuttal - Part I
We thank Reviewer CxEf for recognizing the relevance of the problem, the intuitive dependency taxonomy, and the breadth of the evidence-controlled evaluation.
W1 & Q1. What does Culture-MMT evaluate beyond earlier multimodal and cultural translation benchmarks?
Probing the Need for Visual Context in Multimodal Machine Translation [1] and CoMMuTE [2] study visual context and disambiguation in translation. CaMMT [3] focuses on culturally aware multimodal translation, while CULTURE-MT [4] evaluates cultural effectiveness in text-based social-media translation.
Culture-MMT adds a joint evaluation of required evidence and its use in open-ended UGC translation. Symbol provides an image-independent cultural control, Visual tests image-grounded meaning, and Hybrid tests their combination. Models translate titles and bodies whose accuracy, register, emotion, and cultural fit are assessed together.
The four input conditions then vary access to matching images, unrelated images, or explicit visual facts while keeping the evaluator’s context fixed. Together with the concrete translation points in Appendix C, these comparisons support diagnosis of fact recovery, reference integration, and English expression. The added value is this combination of evidence controls and pragmatic translation assessment, rather than the taxonomy alone.
W2 & Q2. How are dependency labels established, candidates selected, and dependencies preserved?
In “肥宅复健 Day 1,” “肥宅” requires a self-mocking internet register, while the image grounds whether “复健” refers to exercise or injury rehabilitation (Table 1). The Hybrid label concerns both requirements; a plausible text-only inference does not uniquely establish the post-specific meaning.
The reported Visual and Hybrid results use differential filtering followed by human review; a separate evaluation on candidates collected without this filter is not reported. Humans determine final inclusion and labels and can override model pre-labels. Symbol follows its own human-confirmed route (Figure 2). The filtering threshold concerns collection-stage drafts, not a required score gap for all final translations. The gains therefore characterize the human-reviewed diagnostic set. A differently selected set would test selection sensitivity, while independent re-scoring would test evaluator sensitivity.
Rewriting follows a ledger of expressions, intended meanings, evidence carriers, required clues, and forbidden shortcuts (Appendix E.3). Text and images are updated jointly. An acceptance step separate from generation uses the ledger to check cultural implicitness for Symbol, image dependence for Visual, and both for Hybrid. Failed instances are revised and re-accepted or removed. Appendix E.4 describes retained evidence-preservation and privacy-check records. Table 8 measures surface separation across 498 notes; semantic preservation is assessed through the item-level dependency checks.
W3 & Q3. Does the evaluator require unnecessary image details, and what does the human validation establish?
The rubric assesses the intended source meaning. Translating “单反” as a one-handed backhand resolves the tennis example without describing the court or clothing. Appendix D.3 penalizes incorrect, omitted, or overly generic image-grounded meanings. We will make this semantic-disambiguation criterion explicit.
The human caption supplies visual facts needed to interpret the post. For example, a tennis-scene description provides evidence for interpreting “单反”; it need not prescribe an English translation. Thus, withholding the expression-to-answer ledger does not mean withholding relevant facts. Captions are marked “do NOT translate,” and the judge’s source context stays fixed (Section 3.2). The task remains translating the title and body, not reproducing the caption. Table 6 also shows that captions need not yield higher scores for a given model: Gemini-3.1-pro (high) scores 2.412 with captions and 2.424 with real images. This comparison is descriptive and does not independently validate the observed translation-quality difference.
Human validation covers 1,160 outputs from 58 notes and 10 configurations, including both translator input conditions. Binary agreement is 0.820, with κ = 0.459 and four-level weighted κ = 0.455. These results measure agreement with human judgments, but do not directly validate the reported visual gains. In the subsequent revision, we will add a separate human translation and evaluation study. Human translators will translate the same posts with the relevant evidence, providing a human performance reference. We will also use independent human ratings of paired text-only and image-assisted model outputs to directly assess visual gains and better characterize the model–human performance gap.