- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CULTURE-CMT, a benchmark dataset for evaluating contextual understanding and cultural effectiveness in social-media comment translation. The benchmark focuses on Chinese-to-English translation and comprises 572 examples that preserve the interaction context, including original posts and parent comments. It also provides three categories based on translation difficulty: Slang, Context, and Both. Furthermore, the authors develop a retrieval-augmented Judger to automatically evaluate cultural effectiveness and validate its agreement with human annotations. In their experiments, the authors evaluate 33 model configurations and analyze how interaction context and several other factors affect translation quality. The results show that providing context improves translation quality for many models, while examples involving both slang and contextual dependence remain challenging.
However, the descriptions of methodological details, particularly the dataset construction process, remain too abstract. As a result, it is difficult to assess the technical validity of the proposed benchmark in detail, and the evaluation is largely limited to its conceptual aspects. Therefore, I believe the paper's clarity needs improvement.
Critical strengths
- Introduced CULTURE-CMT, a benchmark dataset for evaluating contextual understanding and cultural effectiveness in social-media comment translation.
- The authors define three difficulty categories, Slang, Context, and Both, to analyze different translation challenges. They also conduct benchmarking from multiple perspectives using 33 model configurations.
- The authors develop a retrieval-augmented Judger and evaluate its agreement with human annotations using 432 human-annotated translation pairs. However, the details of the dataset construction process remain unclear. In particular, the paper does not provide sufficient information on how these translation pairs were constructed or collected, making it difficult to reproduce the dataset construction process. The description remains overly abstract and lacks the methodological details necessary to ensure reproducibility.
Critical weaknesses
- W1: Insufficient details and reproducibility of dataset construction.
- Although this paper provides an overview of the dataset construction process, it lacks sufficient methodological details, raising concerns about reproducibility and validity. For example, it is unclear where the 40K comments were collected from, how low-information comments were defined and filtered, and how the effectiveness of model-assisted annotation was validated.
- Regarding human review, the paper also lacks quantitative assessments of dataset quality, such as inter-annotator agreement for difficulty labels and context relevance. Furthermore, the construction of the 200K source-translation pairs used for Judger training is insufficiently explained, particularly regarding how the parallel data were collected and how their quality was controlled. Given that this paper emphasizes the challenges of context-aware translation, it is particularly important to clarify how these parallel data were constructed.
- Since dataset construction is a central contribution of this paper, I would like the authors to provide sufficient details in the main text so that readers can understand the construction procedure and assess its reproducibility and validity without relying on supplementary materials.
- W2: Unclear motivation and practical significance of the benchmark.
- It seems that only 572 examples were selected from 40K comments. This suggests that the benchmark may focus on relatively rare phenomena in real-world social-media translation. However, the paper does not sufficiently discuss how frequently these phenomena occur in practice or what impact they have on real-world applications.
- Furthermore, Figure 1 provides only one example, making it difficult to understand what other types of translation challenges are included in the benchmark. In particular, culturally situated expressions may not have a single correct translation, and further discussion is needed to clarify what constitutes an effective translation and how its validity is established.
- While I understand the research value of constructing a benchmark consisting of difficult examples, I would like the authors to explain more clearly why these phenomena need to be evaluated through a separate benchmark and what limitations of existing translation benchmarks, such as CULTURE-MT, this benchmark addresses.
- W3: Limited novelty and insufficient depth of experimental analysis.
- Although this paper conducts benchmarking using 33 model configurations, the analyses are largely limited to comparisons of aggregate performance and average scores across difficulty categories. These results do not provide sufficient insights into which linguistic phenomena are particularly challenging for models or what types of errors can be resolved by providing context. In particular, the authors should discuss the differences from CULTURE-MT in greater detail and demonstrate what new findings can be obtained specifically through this benchmark.
- Moreover, rather than focusing primarily on model rankings, I recommend conducting more detailed item-level and qualitative error analyses to better understand the relationship between contextual understanding and cultural effectiveness.
Questions for authors
Rather than questions, I would like to offer the following suggestions for the authors to address:
- Please address the three major weaknesses raised above (W1-W3). I have included specific suggestions for improvement in each bottom item.
- Improve the clarity and reproducibility of dataset construction. Although the overall dataset construction process is understandable, the paper lacks sufficient details regarding specific procedures and quality control. Since these details are essential for assessing the validity of this paper, I would like the authors to provide a self-contained explanation in the main text rather than relying solely on the Appendix or prior work. In particular, more detailed descriptions of data sources, filtering criteria, annotation procedures, inter-annotator agreement, and parallel data construction are needed. I believe that clarifying these aspects would allow for a more appropriate assessment of the dataset quality and technical validity.
- I suggest that discussing the differences from CULTURE-MT and existing context-aware translation benchmarks more clearly in the main text rather than relegating related work to the Appendix. These are not merely supplementary details but essential information for assessing the novelty and contributions of this paper.
- Discuss the limitations and practical implications. I recommend explicitly discussing the real-world frequency of the phenomena covered by the benchmark, its generalizability, and the limitations of cultural-effectiveness evaluation. Such discussions would help clarify the positioning of this research and its implications for future work.
4 comments
Clarifications and Responses to Reviewer 1uvp Comments - 2
- Limitations of the cultural-effectiveness evaluation:
We candidly note several limitations of our evaluation protocol that the revision will discuss explicitly:
- The 0–3 rubric ultimately rests on a human judgment of "whether communicative intent is conveyed," which is coarser than a fine-grained error taxonomy and cannot fully disambiguate cases with multiple defensible translations;
- As shown in our response to Reviewer D6Q6 (Q3), the Agent Judger exhibits an asymmetric error profile — Sensitivity 48.2% vs. Specificity 83.8% — meaning it is more reliable at confirming valid translations than at catching invalid ones, so Judger-reported Invalid Rates likely underestimate the true error rate;
- However, model-level agreement between the Judger and humans is strong for ranking (ρ=0.807).
- Generalizability:
We acknowledge that this is not discussed in the current submission. All experiments in this paper are conducted on Chinese-to-English translations of comments from several social-media platforms, under one cultural-effectiveness rubric. We have not validated whether the benchmark-construction methodology, the Judger, or the empirical Scale/Reasoning/Adaptation findings transfer to other language pairs. We will add this as an explicit limitation in the revision and frame cross-lingual validation as a direction for future work.
3. Differences from CULTURE-MT, and Depth of Analysis (W3, Q3)
- Distinction from CULTURE-MT:
CULTURE-MT evaluates the culturally situated translation of isolated comments. The core contribution of CULTURE-CMT is restoring the "post + parent comment" interaction structure in which a comment occurs (Figure 1), and decomposing difficulty into the three Slang/Context/Both subsets to enable diagnostic analysis — allowing us to separate "purely linguistic-cultural difficulty" from "purely contextual-dependence difficulty" and their combination, which CULTURE-MT's single-dimension design cannot do. The revision will foreground this distinction more prominently in the main text, rather than relegating it to the Appendix.
- Fine-grained error analysis:
We agree that aggregate difficulty-group average scores (Figure 6) are insufficient to reveal how "contextual understanding" concretely affects "cultural effectiveness," and that model-ranking comparisons alone do not surface which linguistic phenomena are specifically challenging. Using the existing 25×33 per-example error-type labels in our human annotations, we re-analyzed the error-type distribution by difficulty type (Slang/Context/Both), normalized per sample-model unit to control for the uneven number of samples across categories:
| Primary error type | Slang (per unit) | Context (per unit) | Both (per unit) |
|---|---|---|---|
| Literal translation | 1.226 | 0.715 | 0.897 |
| Semantic deviation (addition/omission) | 0.574 | 0.343 | 0.345 |
| Proper nouns | 0.381 | 0.071 | 0.235 |
| Special terms/forms of address | 0.110 | 0.087 | 0.358 |
The three difficulty types show distinct error profiles, not merely differences in severity:
-
The Slang subset has the highest error density, concentrated in literal-translation errors (1.226 per unit, about 1.7× that of Context) — consistent with the expectation that, for pure slang/internet-language difficulty, the model's most common failure mode is mechanical word-for-word translation.
-
Proper-noun errors occur almost exclusively in the Slang subset (0.381 vs. 0.071 for Context) — indicating this error type is tied to culturally loaded expressions themselves, rather than to missing context.
-
Special-term/address errors show a super-linear amplification in the Both subset (0.358, markedly higher than Slang's 0.110 and Context's 0.087, exceeding their simple sum) — indicating that when slang/cultural expressions and contextual dependence co-occur, the model's failure rate rises sharply on forms of address and in-group slang that require double decoding via both cultural background knowledge and situational cues.
This provides a concrete, error-mechanism-level explanation for why the Both subset is consistently the hardest in Figure 6, rather than leaving the observation at the level of aggregate scores.
Clarifications and Responses to Reviewer 1uvp Comments - 3
Representative cases:
- Case 1 (bench_0305, Both, a discussion about the mobile game Honor of Kings):
The source mentions "小王怎么算账" ("how is Xiao Wang doing the math") and "无双池子" ("the Wushuang pool"). Nearly all 33 model configurations failed on this example, with errors concentrated on two pieces of in-group slang: "小王" (a personifying nickname players use for the game's official account/developer) was almost universally transliterated or mistaken for a personal name (e.g., rendered as "Wang Xiao," following an English given-name/surname convention); "无双" (an in-game skin-quality tier, officially translated as "Flawless") was transliterated as "Wushuang" or mistranslated literally as "Unparalleled"/"Peerless." The available context (the post body mentions the "Xiaoqiao Wedding" skin and the Wushuang pool) does not by itself supply the cues needed to decode "小王," because resolving this reference requires gaming-community background knowledge unavailable to an outsider, rather than information recoverable from the text of the context alone.
- Case 2 (bench_0041, Both, a discussion about a local light-meal brand):
The title and body repeatedly mention two local brand names, "超模厨房" (officially "Super Model Kitchen") and "超级碗" (the brand "FoodBowl"); despite ample contextual information, models still commonly mistranslate these as "the supermodel" ("超级模特") and "Super Bowl" (the American football event), systematically distorting the brand meaning.
Both cases concretely illustrate that the difficulty of the Both subset is not a simple addition of "slang + context," but rather that correctly decoding certain in-group slang terms or brand names inherently depends on cultural background knowledge the model lacks — restoring interactive context cannot substitute for this missing knowledge. This is precisely the mechanism behind the super-linear amplification of special-term/address errors we observe in the error-type statistics.
We will incorporate this fine-grained analysis and the representative cases into the main text of the revision and use them to revisit our novelty argument relative to CULTURE-MT — whose single-dimension difficulty design cannot reveal this kind of super-linear amplification arising from difficulty interaction.
Thank you for your responses and additional analyses.
However, I noticed that the manuscript has not been updated. Since many of my concerns relate to the paper's soundness and central claims, it is difficult to assess whether they have been adequately addressed without examining the corresponding revisions in the manuscript. For these reasons, I will maintain my original score.
Furthermore, considering the evaluations from the other reviewers, I believe the paper is unlikely to be accepted in this cycle. Therefore, I have not examined the rebuttal in detail.
Clarifications and Responses to Reviewer 1uvp Comments - 1
We thank the reviewer for pointing out that the data-construction details are scattered and the analysis lacks depth. There are our responses:
1. Data-Construction Details and Reproducibility (W1, Q2)
We acknowledge that the current main text describes the data-construction pipeline too tersely, with relevant details scattered across Appendices B.1-B.4, D, and E.1-E.2. The revision will add a self-contained description in the main text that consolidates the following:
40,000 raw comments (with full original post title/body/parent-comment context) → rule-based filtering yields 10,000 candidates (removing malformed, too-short/too-long, templated, and low-information comments) → quality screening and preliminary difficulty labeling yields 5,000 (LLM-assisted judgment of whether a genuine translation challenge exists, Appendix B.1) → difficulty-type and topic-domain annotation yields 3,000 (Appendix B.3 translation-difference verification, B.4 context-dependence verification, excluding examples that are "purely stylistic variation" or "topically related but without substantive dependence") → deduplication and balancing yields 2,205 candidates → final human review by 3 graduate-student annotators, removing sensitive/private content, yields the final 572-example benchmark. Inter-annotator agreement (IAA, previously missing, now supplied):
Both figures fall within an acceptable range of annotation reliability, supporting the idea that our difficulty-tier labels and context-dependence judgments reflect good annotator agreement rather than arbitrary subjective classification, and that they are not adopted directly from model-produced labels. The revision will include this data and a description of the third-party adjudication mechanism in the main text.
160K labeled by Gemini 3.1 Pro, 40K manually annotated, following the four-level cultural-effectiveness rubric established by CULTURE-MT; this set is disjoint from (does not overlap with) the 40K-comment source pool used to build the benchmark.
2. Motivation, Practical Significance, Limitations, and Generalizability (W2, Q4)
The retention rate at each filtering stage, relative to the 40,000 original comments, is: 25% after rule-based filtering (10,000) → 12.5% after quality screening (5,000) → 7.5% after difficulty annotation (3,000) → 5.5% after deduplication/balancing (2,205) → 1.43% after human review (572).
We note that this overall retention rate conflates several distinct filtering motivations — comment-quality rejection, context-availability requirements, confirmed translation difficulty, deduplication, and deliberate balancing across 21 topical domains and three difficulty types — and therefore should not be read as a clean estimate of how often culturally risky mistranslation occurs in naturally distributed social-media comments; the balancing step in particular is designed to equalize representation across difficulty types rather than preserve their natural proportions.
What this funnel does support is a narrower, more defensible claim: genuinely challenging translation cases are a small minority even among comments already pre-filtered for basic quality and context availability, and such cases are easily diluted by aggregate scoring over a majority of 'easy' examples — which is the practical problem CULTURE-CMT is designed to surface through targeted, diagnostic sampling rather than through claims about population-level prevalence.
Our scoring rubric follows the already-validated four-level cultural-effectiveness scale from CULTURE-MT (Appendix E.2), whose design is explicitly intended to accommodate the reality that culturally situated expressions may admit multiple acceptable translations — scoring is based on "whether communicative intent is conveyed," not on exact match to a single reference. We offer three independent lines of evidence supporting the reliable application of this criterion:
Binary agreement between the Agent Judger and human annotation reaches 77.31% (κ=0.5498, Figure 5);
Independent human evaluation (detail in response 3 of Review D6Q6, N=200, paired McNemar tests, p=0.0104 non-think / p=0.0080 think) confirms that the Adaptation effect identified under this criterion is statistically significant;
The rank correlation between human and Judger scores across 33 model configurations reaches Spearman's ρ=0.807 (p<0.0001).