acceptodds
Under review as a conference paper at ICLR 2027

Beyond Words Alone: Disentangling Distributed Evidence for Visually Grounded Cultural Translation

Abstract

Social media enables cross-lingual communication, yet the meaning of user-generated content (UGC) is often distributed across text, images, and cultural knowledge, making translation challenging. Existing evaluations typically treat images as optional and assume textual information is sufficient, obscuring whether errors stem from missing visual evidence or inadequate cultural adaptation. We introduce Culture-MMT, a diagnostic benchmark for distributed-evidence pragmatic reconstruction in social-media translation. It contains 498 curated, de-identified UGC instances across 20 domains, categorized into Symbol, Visual, and Hybrid by their required evidence. We also develop a human-aligned domain-specific judge, Judger, and a controlled evaluation framework that systematically varies the evidence available to translators. Across 17 model configurations, real images improve performance by +0.280 on Visual and +0.197 on Hybrid, but only +0.050 on Symbol, while random images provide no comparable gains. Human-authored descriptions further show that jointly dependent cases remain more challenging even when visual facts are explicit. These results characterize culturally effective social-media translation as meaning reconstruction from distributed evidence, highlighting visual grounding and cultural-pragmatic adaptation as distinct model bottlenecks.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Machine Translation, Vision-Language Models, Multimodal Learning, Visual Grounding, Image-Text Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CxEfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Culture-MMT, a benchmark of 498 Chinese UGC translation samples categorized into Symbol, Visual, and Hybrid types. The authors propose a domain-specific Judger and evaluate 17 model configurations under different evidence conditions. While the benchmark addresses a relevant problem and offers useful diagnostic perspectives, I have concerns regarding its novelty, construction validity, evaluation reliability, and experimental conclusions.

Critical strengths
  • The paper addresses a relevant problem in culturally aware multimodal translation.
  • The evidence-dependency taxonomy provides an intuitive diagnostic framework.
  • The evaluation covers diverse models and includes useful evidence-control experiments.
Critical weaknesses

W1. Limited novelty beyond existing multimodal translation research.

The novelty positioning appears overly broad. Visual disambiguation and underspecified translation have been extensively studied (Caglayan et al., 2019; CoMMuTE, Futeral et al., 2023), while CaMMT (Villa-Cueva et al., 2025) explicitly investigates visually grounded cultural translation. Although Culture-MMT introduces a UGC-specific taxonomy, its distinction from previous works remain insufficiently demonstrated.

W2. Concerns about benchmark construction and evidence-dependency validity.

The boundaries between evidence-dependency types are not always convincing. For example, “肥宅复健 Day 1,” labeled Hybrid in Table 1, appears largely interpretable through cultural knowledge alone, raising questions about whether visual evidence is necessary or merely helpful. Furthermore, Visual and Hybrid candidates undergo model-based differential filtering based on translation gains, potentially introducing selection bias. Since the main experiments subsequently measure gains from visual evidence, this procedure may preferentially select samples that already exhibit the expected behavior, potentially biasing the reported improvements. Extensive text rewriting and image regeneration may also alter the original dependencies, yet quantitative validation of label preservation and inter-annotator agreement is missing.

W3. Concerns about evaluation reliability and visual explicitation bias.

The Judger penalizes translations that do not explicitly include information from the image (L899-901), even when the translation is already accurate and natural without those details. Furthermore, the Human Caption condition uses the same task-relevant descriptions available to the Judger, making it difficult to separate genuine translation improvements from alignment with the evaluation reference. Independent human evaluation of the reported visual gains would strengthen the findings.

W4. Insufficient evidence for separating visual and cultural difficulties.

The experiments do not clearly demonstrate that visual understanding and cultural knowledge are separate sources of translation difficulty. In Section 4.3.2, the authors argue that Hybrid samples perform worse than Visual samples even when image descriptions are provided, suggesting that cultural knowledge remains a challenge. However, the two categories contain different samples, and Hybrid samples already have lower scores without images. Therefore, the performance gap cannot be attributed solely to cultural difficulties.

W5. Limitations of experimental controls and statistical robustness.

The Random Image condition may introduce misleading evidence rather than serving as a neutral placebo. Thus, the performance gap between real and random images may reflect both relevant visual grounding and interference from mismatched inputs. Additionally, the paper provides limited statistical significance testing, confidence intervals, or repeated-run analyses despite using stochastic decoding. These limitations weaken the robustness of several relatively small reported gains.

Questions for authors

Q1. What specific capabilities does Culture-MMT evaluate beyond CoMMuTE, CaMMT, CULTURE-MT and other similar previous works?

Q2. Do the main results persist on samples selected without model-based differential filtering, and how reliably are dependency labels preserved after rewriting?

Q3. Do independent human evaluators confirm the visual gains reported by the Judger?

3 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We thank Reviewer CxEf for recognizing the relevance of the problem, the intuitive dependency taxonomy, and the breadth of the evidence-controlled evaluation.

W1 & Q1. What does Culture-MMT evaluate beyond earlier multimodal and cultural translation benchmarks?

Probing the Need for Visual Context in Multimodal Machine Translation [1] and CoMMuTE [2] study visual context and disambiguation in translation. CaMMT [3] focuses on culturally aware multimodal translation, while CULTURE-MT [4] evaluates cultural effectiveness in text-based social-media translation.

Culture-MMT adds a joint evaluation of required evidence and its use in open-ended UGC translation. Symbol provides an image-independent cultural control, Visual tests image-grounded meaning, and Hybrid tests their combination. Models translate titles and bodies whose accuracy, register, emotion, and cultural fit are assessed together.

The four input conditions then vary access to matching images, unrelated images, or explicit visual facts while keeping the evaluator’s context fixed. Together with the concrete translation points in Appendix C, these comparisons support diagnosis of fact recovery, reference integration, and English expression. The added value is this combination of evidence controls and pragmatic translation assessment, rather than the taxonomy alone.

W2 & Q2. How are dependency labels established, candidates selected, and dependencies preserved?

In “肥宅复健 Day 1,” “肥宅” requires a self-mocking internet register, while the image grounds whether “复健” refers to exercise or injury rehabilitation (Table 1). The Hybrid label concerns both requirements; a plausible text-only inference does not uniquely establish the post-specific meaning.

The reported Visual and Hybrid results use differential filtering followed by human review; a separate evaluation on candidates collected without this filter is not reported. Humans determine final inclusion and labels and can override model pre-labels. Symbol follows its own human-confirmed route (Figure 2). The filtering threshold concerns collection-stage drafts, not a required score gap for all final translations. The gains therefore characterize the human-reviewed diagnostic set. A differently selected set would test selection sensitivity, while independent re-scoring would test evaluator sensitivity.

Rewriting follows a ledger of expressions, intended meanings, evidence carriers, required clues, and forbidden shortcuts (Appendix E.3). Text and images are updated jointly. An acceptance step separate from generation uses the ledger to check cultural implicitness for Symbol, image dependence for Visual, and both for Hybrid. Failed instances are revised and re-accepted or removed. Appendix E.4 describes retained evidence-preservation and privacy-check records. Table 8 measures surface separation across 498 notes; semantic preservation is assessed through the item-level dependency checks.

W3 & Q3. Does the evaluator require unnecessary image details, and what does the human validation establish?

The rubric assesses the intended source meaning. Translating “单反” as a one-handed backhand resolves the tennis example without describing the court or clothing. Appendix D.3 penalizes incorrect, omitted, or overly generic image-grounded meanings. We will make this semantic-disambiguation criterion explicit.

The human caption supplies visual facts needed to interpret the post. For example, a tennis-scene description provides evidence for interpreting “单反”; it need not prescribe an English translation. Thus, withholding the expression-to-answer ledger does not mean withholding relevant facts. Captions are marked “do NOT translate,” and the judge’s source context stays fixed (Section 3.2). The task remains translating the title and body, not reproducing the caption. Table 6 also shows that captions need not yield higher scores for a given model: Gemini-3.1-pro (high) scores 2.412 with captions and 2.424 with real images. This comparison is descriptive and does not independently validate the observed translation-quality difference.

Human validation covers 1,160 outputs from 58 notes and 10 configurations, including both translator input conditions. Binary agreement is 0.820, with κ = 0.459 and four-level weighted κ = 0.455. These results measure agreement with human judgments, but do not directly validate the reported visual gains. In the subsequent revision, we will add a separate human translation and evaluation study. Human translators will translate the same posts with the relevant evidence, providing a human performance reference. We will also use independent human ratings of paired text-only and image-assisted model outputs to directly assess visual gains and better characterize the model–human performance gap.

AuthorsOpenReview ↗

Rebuttal - Part II

W4. How do the experiments distinguish visual and cultural difficulty?

The Human Caption scores, 2.316 on Visual and 2.035 on Hybrid, show that the jointly dependent group remains challenging when visual facts are supplied explicitly (Section 4.3.2). Because the groups contain different notes, the comparison is a diagnostic observation rather than a numerical estimate of a purely cultural effect.

The within-note interventions provide a more specific question: does supplying the missing evidence resolve the translation point? In Appendix C, captions enable “keycaps” for “小黄帽,” while another note still renders “这颗蛋” as “this egg” despite the available visual description. The “rounded heel counter” example illustrates an English terminology problem after the fact is recovered.

Together, the controlled inputs and case analysis help identify errors associated with visual fact recovery, reference integration, and target-language expression. These observations show that providing visual facts does not necessarily resolve all translation difficulties, particularly when cultural interpretation is also required. We therefore interpret the results as diagnostic evidence of different failure mechanisms, rather than a strict causal decomposition of independent visual and cultural abilities.

W5. How should random images and statistical robustness be interpreted?

Real-image gains use text-only as the baseline. Random images separately test interference from mismatched evidence, so the real-minus-random difference is not treated as a pure estimate of grounding benefit.

TypeText-onlyRandom imageReal image
Symbol2.1902.1162.221
Visual1.9461.9412.242
Hybrid1.7881.7112.020

These four-configuration means (Table 6) show real-image benefits on Visual and Hybrid relative to text-only. Random images do not reproduce that pattern. We will call this a mismatched-image control.

We repeat translation experiments three times and report the averages, with matched translator prompts and decoding settings and deterministic judge decoding. We will specify this aggregation in the setup. The reported subset means describe consistency across configurations, without implying improvement on every note or a formal significance test. All 17 configurations have positive mean gains: +0.145 to +0.446 on Visual and +0.042 to +0.349 on Hybrid (Table 5).


References

[1] Probing the Need for Visual Context in Multimodal Machine Translation.

[2] Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation.

[3] CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation.

[4] Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC.

Reviewer_CxEfOpenReview ↗

Thank you for your comments!

Thank you for the comprehensive rebuttal and the additional clarifications! The authors have adequately addressed my questions regarding the paper's novelty, evaluation criteria, and interpretation of the experimental results.

However, I still keep the perspective that the model-based differential filtering risks introducing selection bias, and the sensitivity of the reported gains to this filtering step remains unverified.

I will maintain my score of 2. Thank you again for the constructive clarifications.

Reviewer_ggriOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces Culture-MT, a diagnostic machine translation benchmark focusing on translating social media posts from Chinese to English. The authors observe that users from specific platforms often speak in their own langauge of jargons and neologisms, thus general-purpose MT models often fail to capture the correct meanings. Across 17 MT models and four translation configurations, the authors find most MT models struggle to obtain the meanings of jargons from text alone (Symbol), showing a clear bottleneck for culture adaptation.

Critical strengths

One aspect that I find commendable:

The privacy rewriting step in dataset construction. This is a cheap but clever design to minimize the risk of containing privacy-leaking information, while still retaining the original semantics for as much as possible.

Critical weaknesses

I will focus on the data quality issues I find in the design of the Culture-MT benchmark/dataset.


Weakness 1 - All Dataset Instances May Come from a Single Source, Which May Result in a Heavily Biased Translator.

The authors position the diagnostic benchmark as a generic representation of Chinese social media contents. However, merely out of instinct, I find the included examples are most likely sampled from just a single source platform that is Xiaohongshu 小红书 (Red Note), which is further given away by the actual filtering prompt attached in Appendix E.2. This, unfortunately, may lead to two main problems w.r.t introducing biased data.

A. Culture-MT focuses on capturing the meanings behind generic Internet slangs, rather than simply explaining ‘insider jokes or niche memes’. Unfortunately, sourcing all dataset instances after one online community’s posts contradicts with the very purpose - MT models may adapt to only speak in Xiaohongshu’s tone.

B. This also explains the apparently biased domain distribution in Figure 3, where topics such as fashion and beauty, gaming, anime, and celebrity take a clear dominance over others, reflecting the known user demographics of Xiaohongshu in general (see the link in the Reference).

Both aforementioned problems are ultimately at odds with the authors’ intended offering that Culture-MT is a standardized diagnosis for social media translation in large.

My suggestion: The authors should at least collect or synthesize more diversified UGC samples from additional Chinese social media platforms, such as Bilibili B站 or Weibo 微博.

P.S. For those who are not too familar with the landscape of Chinese social media, as a coarse comparison, imagine all of the benchmark's data are sampled from just one Subreddit forum.


Weakness 2 - The Three Dataset Subsets (”Types”) Are Not Selected on the Same Standards.

In addition to the single-data-source issue in Weakness 1, I find the three subsets - namely the Symbol, Visual, and Hybrid types - are curated under uneven standards. The text-only Symbol subset is collected purely by human, while both Visual and Hybrid are additionally filtered through the GRPO-post-trained Judger [sic.] model. The same Judger is also the one that produces all culture-effectiveness scores in Tables 4 and 5.

This data curation setup unfortunately undermines the integrity of one main finding that images lead to higher scores in the Visual and Hybrid subsets, because the examples under these two subsets in the dataset are already pre-determined with a positive score gap via the Judger. I'd suggest the author re-score at least the Visual and Hybrid subsets using a different judge model that is not involved in data selection efforts, such as Gemini Pro 3.1 pro from Table 3.


Reference

Questions for authors

Questions.

Q1. Internet slangs are known to shift in meanings over time. How would the authors think to keep MT models' abilities up to date?

Q2. The scope of the entire study only focuses on Chinese-English translation. Have the authors tried sampling Instagram-like posts for English-Chinese translation in comparison?

Q3. Out of curiosity, I'm in particular interested in the translation quality of 种草 and 拔草, under the different domains' contexts in the current Culture-MT setup, IYKYK.

Misc.

  • ‘Judger’ is a grammatically incorrect word. I’d rather called it ‘Culture-MT Judge’.
3 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We thank Reviewer ggri for recognizing the privacy-preserving rewriting design and for the detailed questions about data construction and cultural interpretation.

W1. Platform coverage, topic distribution, and translation style

We collect real social-media image–text posts containing cultural expressions, implicit references, or translation-relevant visual evidence. Approximately 56% of the 498 instances come from Xiaohongshu (about 278 notes); the remaining 44% come from other platforms, such as Bilibili, Weibo, Douyin, and Kuaishou. Across sources, annotators check for a concrete translation point, the evidence needed to resolve it, and consistency between the text and image. The Xiaohongshu prompt in Appendix E.2 describes one collection route; we will clarify the broader composition in the revision.

The benchmark spans 20 domains, with 166 instances per dependency type. The four highlighted topics account for 253/498 notes (50.8%), while the remaining 16 contribute 245/498 (49.2%):

DomainInstancesShare
Fashion and beauty90/49818.1%
Gaming60/49812.0%
Anime and ACG57/49811.4%
Celebrities and fandom46/4989.2%
Remaining 16 domains245/49849.2%

The additional domains include pets, daily life, arts and crafts, food, technology and cars, work, parenting, and sports. General internet expressions and community-specific references are both included. We evaluate existing translators without benchmark-specific adaptation. Appropriate English follows the source’s communicative purpose: product recommendations preserve enthusiasm, fandom exchanges preserve their social tone, and ordinary descriptions remain descriptive.

W2. Why do the three subsets use different filtering routes, and how is the judge used?

The judge assists candidate discovery; humans decide final inclusion and dependency labels. Differential scores prioritize Visual and Hybrid candidates for human re-review. Annotators inspect the post and translation point, override model pre-labels when needed, and review the rewritten instance (Appendices E–F). Acceptance depends on the annotated evidence requirement, so the judge’s preferred translation or score is not itself the final label.

Symbol follows the same annotation principle but requires cultural interpretation without necessary visual evidence. A positive visual-gain gate would exclude the intended control, hence its human-confirmed “No visual-gain gate” route (Figure 2).

The threshold of at least 1 point concerns one collection-stage draft-translation pair. It is not a minimum gain required of every evaluated model on every rewritten instance. The final means, +0.280 on Visual and +0.197 on Hybrid, aggregate results from 17 configurations on the human-reviewed release. Different translators can infer a sense from text, recover different visual facts, or express the result differently, so their gains need not reproduce the collection-stage gap.

The reported gains characterize this human-reviewed diagnostic set rather than an unbiased estimate over all social-media posts. Human review determines final inclusion and dependency labels, but we acknowledge that model-based differential filtering may influence the magnitude of the observed gains. Importantly, the filtering threshold applies to collection-stage drafts, not to the final translations generated by the 17 evaluated configurations. Independent re-scoring with a separate judge would provide a useful additional check of evaluator sensitivity; Table 3 currently provides a complementary validation through agreement with human judgments.

Q1. How can the benchmark and MT models stay current as internet expressions evolve?

The existing pipeline collects UGC in successive, non-overlapping rounds and verifies meanings and evidence requirements through human review (Section 2.2; Appendix E.1). It annotates expressions in context rather than relying on a fixed slang dictionary, so new usage can be collected and checked through the same process.

For future maintenance, we could retain a fixed comparison set and add dated releases of emerging or changed expressions. For MT models, these slices would test whether updated knowledge, retrieval, or further training improves current usage while preserving established meanings. This links collection updates to measurable model progress. Tracking meaning shifts across communities and time is a further direction to explore.

AuthorsOpenReview ↗

Rebuttal - Part II

Q3. How should “种草” and “拔草” be translated in different domains?

The evaluation follows the intended action in context, rather than assigning each expression a fixed English equivalent. Appendix D.3 explicitly names “种草” as a culturally loaded expression requiring idiomatic interpretation. Two concrete contexts illustrate the distinction:

  • Parenting-product recommendations. For “被狠狠种草了” in a recommendation about busy books, a natural rendering could be “I’m completely sold on these busy books.” This preserves both purchase interest and enthusiasm.
  • Car-purchase advice. For “来拔草或者推一把” in a discussion of whether to buy a car, a suitable rendering could be “can anyone talk me out of it, or give me the final push?” This preserves the invitation for advice in either direction.

These illustrative renderings preserve domain-specific intent and tone, as required by the cultural-effectiveness rubric.

Naming

We appreciate the naming suggestion and will use “Culture-MMT Judge” in the revised presentation.

I appreciate all the updates. That being said, even though the authors reveal the dataset is sourced from multiple social media platforms beyond Xiaohongshu (which they should have done so explicitly), this does not adequately address the core underlying dataset quality issue. The authors have rather confirmed that their benchmark's domains are a long-tailed biased distribution of various topics. In addition, I didn't see any answer to my original Q2. I decide to keep my original rating.

Reviewer_vqFsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of cross-lingual translation (Chinese to English) in a pragmatic social-media setting using real user-generated content (UGC). Specifically, the authors study the effects of image and cultural context on translation quality. To this end, they propose a novel diagnostic benchmark called Cultural-MMT that contains 498 samples and is categorized into three types, depending on whether image, cultural context or just text is needed for effective translation. They train a judge (called Judger) using the Qwen3.5-27B model and evaluate several open-source and closed models for translation quality under different settings. The results show that having access to images can help in improving the translation scores.

Critical strengths
  1. The authors study the important problem of cross-lingual translation in a real-world social-media setting. The curation of Culture-MMT benchmark was done in several rounds and was finally manually curated by Humans.

  2. The authors conduct various controlled experiments that help to understand the different aspects affecting the translation quality. For example, the Random-Image control experiment is a nice way to isolate the effects of the influence of image on the overall translation. The authors do a good job in evaluating a large number of open-source and closed models.

Critical weaknesses
  1. In the Data Construction (Sec. 2.2), it is mentioned that only the samples whose gap is greater than or equal to 1 is kept (L171-173). This implies that the translation of all the samples of the benchmark, require visual input to be accurate. But then, how do you get the samples of the type “symbols”, which just require textual context and are not dependent on the visual context? This is an important category of the benchmark (33%) and several claims of the paper are dependent on this category. Hence, it is important to clarify this. Also, a related question: How is the categorization done? It might be good to provide details for this.

  2. Regarding the Judge Validity. The proposed Judge (Judger) is compared against the human judge on a subset of samples. Although Judger seems to have better agreements compared to the other models, the agreement still seems to be quite less (just 78%). Especially, even after consolidating the scores to just two buckets (0-1 and 2-3) instead of the actual four values used in the rating. If the agreement were calculated on the actual values, it would have been even lesser. Also, it would be helpful to report the confidence intervals and segregate the agreement between text-only and multimodal outputs.

  3. Main Results. a) The main results (Tab. 4) are mostly expected. Samples that require visual context will definitely achieve better scores when given the visual context when compared to the samples that only require text context. So, the visual gain is not very surprising. Although the gain seems to be less (just +0.28) for the visual category. Please, report the confidence intervals to help understand whether the gains are statistically significant or not. b) From Tab. 6 (row 1) it looks like even for the visual category, text-only context can give a very high score (2.277 on a scale of 0-3). This is a little concerning and is worth investigating if all the samples in this category genuinely need visual context or not.
    c) A human study is needed (at least on a representative subset) to get a sense of the ceiling. This will also give an idea about the possible errors in the benchmark. d) Although the idea of having the Random-Image ablation experiment is nice, the scores don't seem very convincing (Tab. 6). Even for samples of type “symbol” (which just requires textual context), providing a random (unrelated) image improves the performance when compared to giving just the text inputs. This seems counter-intuitive because one would imagine the random image to act as a distractor and reduce the score. e) The gap between “human caption” and “real multimodal” is negligible. So, where do the existing gaps come from? It might be good to disentangle the effects of culture, language and visual perception in a principled manner.

  4. Although not a major weakness, this work is just limited to one language. The data construction strategies might not transfer well to other languages.

Questions for authors
  1. What exactly are we translating here? Is it the title, body or image too? The translator prompt in Appendix D2 just says it as CONTENT. It is important to understand whether the output remains the same when translating with or without images.

  2. How is the Ground Truth Translation obtained? Is it manually annotated by humans?

  3. Regarding Judge Training (Appendix D3, L862). It is mentioned that “multiple translation models were used to construct the source-translation pairs” Can you mention the specific details of the dataset used for training? This is important because, the human-aligned Judger is claimed to be one of core contributions of the paper

3 comments
AuthorsOpenReview ↗

Rebuttal - Part I

W1. How is Symbol constructed if the filtering criterion requires a visual gain? How are the types assigned?

The visual-gain threshold applies to Visual and Hybrid. Symbol follows the human-confirmed “No visual-gain gate” route (Figure 2; Appendix E.2). The labels are assigned by the evidence needed to resolve the translation point: Symbol needs cultural knowledge without necessary visual evidence; Visual needs the image to determine the intended sense; Hybrid needs both cultural interpretation and visual grounding. Each type contains 166 instances.

Annotators inspect the title, body, and image against these criteria and can override model pre-labels (Appendix F.1). Thus, “避雷” requires internet-cultural knowledge, whereas “单反” in a tennis post requires visual disambiguation. We will state the threshold’s scope and these decision rules together in Section 2.2.

W2. How reliable is the judge, including on the original four-level scale?

Section 3.3 reports two related evaluations, with distinct sampling and metrics. The 1,160 human-scored translations from 58 notes and 10 configurations cover text-only and multimodal outputs. On this pool, binary agreement is 0.820, binary Cohen’s κ is 0.459, and four-level weighted κ is 0.455.

Table 3 uses a class-balanced comparison subset of 498 human-scored translations. Its unit is a translation output, distinct from the benchmark’s 498 source notes. Our judge achieves 0.781 binary agreement and κ = 0.562, compared with Gemini-3.1-pro’s 0.767 and 0.534. Section 3.3 also compares four-level agreement against the same Qwen3.5-27B base:

Four-level metricQwen3.5-27B baseCulture-MMT Judge
Exact accuracy ↑0.3980.522
Cohen’s κ ↑0.2250.342
Mean absolute error ↓0.7450.586

The κ of 0.342 belongs to the four-level base-model comparison; the 0.455 value is explicitly reported as weighted κ on the broader human-scored pool. These values should be read with their respective evaluation definitions. The improvements in exact agreement and absolute error show the benefit of human alignment for the score distinctions used in the main experiments.

All translator conditions share the same source, human-verified caption, rubric, and deterministic judge settings. The reported agreement estimates pool both text-only and multimodal outputs. We will clarify the sampling relationship and metric definitions in the revision. We acknowledge that condition-specific confidence intervals would provide a more detailed assessment of evaluation reliability, and the current pooled results should not be interpreted as establishing identical agreement across input conditions.

W3(a). What is learned beyond the expected benefit of images, and how should the magnitude be interpreted?

The results distinguish evidence dependence, absolute translation quality, and effective meaning transfer. Across 17 configurations, gains average +0.280 on Visual, +0.197 on Hybrid, and +0.050 on Symbol. Every configuration improves on Visual and Hybrid, while 5 show no improvement or a decrease on Symbol (Tables 4–5). This identifies where matching visual evidence is useful across different model families.

Evidence use and final quality are also distinct: Gemini-3.1-pro (high) reaches 2.424 with a +0.108 gain, while Qwen3.5-27B reaches 2.171 with a +0.267 gain. A model can benefit more from images yet still translate less effectively overall. The gains also move outputs across the effective-translation threshold: overall effective rates increase by 4.62 and 11.45 percentage points, respectively (Table 4). These complementary measurements reveal more than whether images help on average. They describe average effects under the evaluation protocol; consistency in their direction is not treated as a significance test.

W3(b). Why can a Visual instance receive a high text-only score?

The cited 2.277 is a mean across 166 Visual notes, not the score or target-expression accuracy of every individual instance. The Visual label describes the evidence grounding a translation point; it does not require every model to fail without the image. Two situations can produce a high aggregate score. A model may translate most of the note well while leaving the target expression unresolved. Alternatively, it may select the intended sense from linguistic priors even though the text does not uniquely establish that interpretation.

For “单反,” a tennis image establishes the intended sense as a one-handed backhand rather than a DSLR camera. Text-only inference may select the intended sense, while the image supplies the post-specific evidence for that choice. Gemini improves from 2.277 to 2.452 on Visual with matching images; across the four configurations in Table 6, the corresponding means rise from 1.946 to 2.242.

AuthorsOpenReview ↗

Rebuttal - Part II

W3(c). What human evaluation is available, and does it estimate a human ceiling?

The current study measures agreement with humans using 1,160 human-scored model translations from both input conditions (Section 3.3; Appendix F.3). Construction-stage human review also checks dependency labels, captions, and rewritten image–text consistency.

In the subsequent revision, we will add a separate human translation and evaluation study. Human translators will translate the same posts with the relevant evidence, and their outputs will be assessed under the same rubric to provide a human performance reference. We will also obtain independent human ratings of paired text-only and image-assisted model outputs to directly assess visual gains. This will complement the existing human–judge agreement analysis and help characterize the model–human performance gap.

W3(d). Do random images improve Symbol performance?

Table 6 shows that random images decrease Symbol scores in all four configurations:

ConfigurationText-onlyRandom image
Gemini-3.1-pro (high)2.5302.458
Qwen3.5-397B-A17B2.2172.120
Qwen3.5-27B2.0602.024
Qwen3.5-35B-A3B1.9521.861
Mean2.1902.116

The small random-image increase for Gemini occurs on Visual, from 2.277 to 2.307. Averaged across the four configurations, random-image changes are −0.074 on Symbol, −0.005 on Visual, and −0.077 on Hybrid. Matching images instead yield gains of +0.297 on Visual and +0.232 on Hybrid (Section 4.3.1). Unrelated images therefore do not reproduce the real-image benefit.

W3(e). What does the small caption–image gap reveal about the remaining errors?

Across the four configurations, overall means are 2.201 with human captions and 2.161 with real images; on Visual they are 2.316 and 2.242. Explicit visual facts thus provide a modest additional average benefit. The caption condition tests whether making those facts accessible changes the translation, while Appendix C examines what errors remain:

  1. For “小黄帽,” captions enable “keycaps.” This shows successful use of explicitly supplied object information after the direct-image condition leaves the reference unresolved.
  2. For “这颗蛋,” the caption condition still produces “this egg” instead of the intended Eggy character, suggesting a problem linking available facts to the textual reference.
  3. “Round closed heel” versus “rounded heel counter” illustrates a target-language terminology problem after the relevant fact is recovered.

These within-note comparisons support diagnostic interpretation of fact use, reference integration, and English expression. Separately, Hybrid reaches 2.035 under captions. Its comparison with Visual is a cross-group observation (Section 4.3.2); the case analysis identifies which translation errors remain when visual facts are explicit.

W4. How does the construction strategy relate to other languages?

The reusable component is annotation by required evidence: cultural knowledge, visual facts, or both. Other language pairs can use the same controls with locally annotated expressions, meanings, and target-language register. Our experiments establish the protocol for Chinese-to-English translation.

Q1. What is translated, and what stays the same across input conditions?

The translator outputs the English title and body, preserving the structural tags. Images and captions provide context for interpreting those fields. Visible image text may supply necessary evidence, but the task does not require translating every word in an image. Appendix D.2 explicitly marks the human caption “do NOT translate.”

The output fields stay the same across conditions. Their English wording can change when added evidence resolves an ambiguity in the source post.

Q2. How is the ground-truth translation obtained?

The test set is evaluated without manually annotated gold English reference translations. Instead, human annotations establish the intended meaning, evidence dependency, and task-relevant visual facts. The judge assesses each candidate against the Chinese source and human-verified image description for semantic accuracy, naturalness, and cultural fit.

This source-based approach accommodates multiple faithful English renderings. Its scoring standard is anchored to human judgments: Section 3.3 reports 0.820 binary agreement on 1,160 outputs and improved four-level agreement over the same base model. Image descriptions supply context, while collection-stage draft translations do not enter the released benchmark (Appendix E.1).

AuthorsOpenReview ↗

Rebuttal - Part III

Q3. What data are used to train the judge?

We collect source content from public social-media platforms, generate translations with multiple models, and label their quality using Gemini and human annotation. The collection covers different quality levels under the same 0–3 cultural-effectiveness rubric (Appendix D.3).

Training first uses score-balanced Gemini-labeled examples for supervised fine-tuning, then score-balanced human-labeled examples for human alignment, followed by GRPO on human-labeled data. This connects scalable supervision with human judgments of meaning, naturalness, and cultural appropriateness. Appendix D.3 gives the pool sizes and stage-wise sampling quantities. We will make the training-generator configurations and training/validation split construction explicit there; Table 2 separately lists the generators used for validation outputs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.