acceptodds
Under review as a conference paper at ICLR 2027

CULTURE-CMT: Benchmarking Contextualized Social Media Translation

Abstract

Social-media comments are interactions, yet they are often translated and evaluated as isolated text. This mismatch is especially problematic when meaning depends on surrounding discussion, community-specific slang, or culturally situated expressions. We introduce **CULTURE-CMT**, a Chinese–English benchmark for contextualized social-media comment translation that restores the post and conversational context naturally available to users. Its 572 manually verified examples are organized into **Slang**, **Context**, and **Both**, enabling diagnostic analysis of distinct and interacting sources of translation difficulty. To support scalable evaluation, we further develop a retrieval-augmented cultural-effectiveness Judger that combines a calibrated scoring model with dynamically retrieved cultural knowledge. CULTURE-CMT reveals several non-trivial patterns. Providing interaction context improves most model configurations, but does not eliminate culturally ineffective translations. Examples requiring both contextual grounding and culturally situated language remain substantially more difficult than either source of difficulty alone. Moreover, larger models are not consistently better, and thinking produces highly model-dependent gains, including occasional degradation. In contrast, a 27B model adapted on benchmark-disjoint social-media data achieves the lowest observed invalidity rate among the evaluated configurations under thinking inference with context. Together, these results suggest that realistic social-media translation is not simply a matter of scaling models or reasoning longer, but of learning to translate meaning as it is constructed within evolving online interactions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Social Media Translation, Context-Aware Machine Translation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1uvpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CULTURE-CMT, a benchmark dataset for evaluating contextual understanding and cultural effectiveness in social-media comment translation. The benchmark focuses on Chinese-to-English translation and comprises 572 examples that preserve the interaction context, including original posts and parent comments. It also provides three categories based on translation difficulty: Slang, Context, and Both. Furthermore, the authors develop a retrieval-augmented Judger to automatically evaluate cultural effectiveness and validate its agreement with human annotations. In their experiments, the authors evaluate 33 model configurations and analyze how interaction context and several other factors affect translation quality. The results show that providing context improves translation quality for many models, while examples involving both slang and contextual dependence remain challenging.

However, the descriptions of methodological details, particularly the dataset construction process, remain too abstract. As a result, it is difficult to assess the technical validity of the proposed benchmark in detail, and the evaluation is largely limited to its conceptual aspects. Therefore, I believe the paper's clarity needs improvement.

Critical strengths
  • Introduced CULTURE-CMT, a benchmark dataset for evaluating contextual understanding and cultural effectiveness in social-media comment translation.
  • The authors define three difficulty categories, Slang, Context, and Both, to analyze different translation challenges. They also conduct benchmarking from multiple perspectives using 33 model configurations.
  • The authors develop a retrieval-augmented Judger and evaluate its agreement with human annotations using 432 human-annotated translation pairs. However, the details of the dataset construction process remain unclear. In particular, the paper does not provide sufficient information on how these translation pairs were constructed or collected, making it difficult to reproduce the dataset construction process. The description remains overly abstract and lacks the methodological details necessary to ensure reproducibility.
Critical weaknesses
  • W1: Insufficient details and reproducibility of dataset construction.
    • Although this paper provides an overview of the dataset construction process, it lacks sufficient methodological details, raising concerns about reproducibility and validity. For example, it is unclear where the 40K comments were collected from, how low-information comments were defined and filtered, and how the effectiveness of model-assisted annotation was validated.
    • Regarding human review, the paper also lacks quantitative assessments of dataset quality, such as inter-annotator agreement for difficulty labels and context relevance. Furthermore, the construction of the 200K source-translation pairs used for Judger training is insufficiently explained, particularly regarding how the parallel data were collected and how their quality was controlled. Given that this paper emphasizes the challenges of context-aware translation, it is particularly important to clarify how these parallel data were constructed.
    • Since dataset construction is a central contribution of this paper, I would like the authors to provide sufficient details in the main text so that readers can understand the construction procedure and assess its reproducibility and validity without relying on supplementary materials.
  • W2: Unclear motivation and practical significance of the benchmark.
    • It seems that only 572 examples were selected from 40K comments. This suggests that the benchmark may focus on relatively rare phenomena in real-world social-media translation. However, the paper does not sufficiently discuss how frequently these phenomena occur in practice or what impact they have on real-world applications.
    • Furthermore, Figure 1 provides only one example, making it difficult to understand what other types of translation challenges are included in the benchmark. In particular, culturally situated expressions may not have a single correct translation, and further discussion is needed to clarify what constitutes an effective translation and how its validity is established.
    • While I understand the research value of constructing a benchmark consisting of difficult examples, I would like the authors to explain more clearly why these phenomena need to be evaluated through a separate benchmark and what limitations of existing translation benchmarks, such as CULTURE-MT, this benchmark addresses.
  • W3: Limited novelty and insufficient depth of experimental analysis.
    • Although this paper conducts benchmarking using 33 model configurations, the analyses are largely limited to comparisons of aggregate performance and average scores across difficulty categories. These results do not provide sufficient insights into which linguistic phenomena are particularly challenging for models or what types of errors can be resolved by providing context. In particular, the authors should discuss the differences from CULTURE-MT in greater detail and demonstrate what new findings can be obtained specifically through this benchmark.
    • Moreover, rather than focusing primarily on model rankings, I recommend conducting more detailed item-level and qualitative error analyses to better understand the relationship between contextual understanding and cultural effectiveness.
Questions for authors

Rather than questions, I would like to offer the following suggestions for the authors to address:

  • Please address the three major weaknesses raised above (W1-W3). I have included specific suggestions for improvement in each bottom item.
  • Improve the clarity and reproducibility of dataset construction. Although the overall dataset construction process is understandable, the paper lacks sufficient details regarding specific procedures and quality control. Since these details are essential for assessing the validity of this paper, I would like the authors to provide a self-contained explanation in the main text rather than relying solely on the Appendix or prior work. In particular, more detailed descriptions of data sources, filtering criteria, annotation procedures, inter-annotator agreement, and parallel data construction are needed. I believe that clarifying these aspects would allow for a more appropriate assessment of the dataset quality and technical validity.
  • I suggest that discussing the differences from CULTURE-MT and existing context-aware translation benchmarks more clearly in the main text rather than relegating related work to the Appendix. These are not merely supplementary details but essential information for assessing the novelty and contributions of this paper.
  • Discuss the limitations and practical implications. I recommend explicitly discussing the real-world frequency of the phenomena covered by the benchmark, its generalizability, and the limitations of cultural-effectiveness evaluation. Such discussions would help clarify the positioning of this research and its implications for future work.
4 comments
AuthorsOpenReview ↗

Clarifications and Responses to Reviewer 1uvp Comments - 1

We thank the reviewer for pointing out that the data-construction details are scattered and the analysis lacks depth. There are our responses:

1. Data-Construction Details and Reproducibility (W1, Q2)

We acknowledge that the current main text describes the data-construction pipeline too tersely, with relevant details scattered across Appendices B.1-B.4, D, and E.1-E.2. The revision will add a self-contained description in the main text that consolidates the following:

  • Construction pipeline (Figure 2):

40,000 raw comments (with full original post title/body/parent-comment context) → rule-based filtering yields 10,000 candidates (removing malformed, too-short/too-long, templated, and low-information comments) → quality screening and preliminary difficulty labeling yields 5,000 (LLM-assisted judgment of whether a genuine translation challenge exists, Appendix B.1) → difficulty-type and topic-domain annotation yields 3,000 (Appendix B.3 translation-difference verification, B.4 context-dependence verification, excluding examples that are "purely stylistic variation" or "topically related but without substantive dependence") → deduplication and balancing yields 2,205 candidates → final human review by 3 graduate-student annotators, removing sensitive/private content, yields the final 572-example benchmark. Inter-annotator agreement (IAA, previously missing, now supplied):

Annotation stepProcedureAgreement
Difficulty-type labeling (Slang/Context/Both)Two annotators independently, double-blind; a third adjudicates disagreements89.33%
Context-relevance judgment (Context/Both subsets; whether context has a substantive effect)Same procedure91.5%

Both figures fall within an acceptable range of annotation reliability, supporting the idea that our difficulty-tier labels and context-dependence judgments reflect good annotator agreement rather than arbitrary subjective classification, and that they are not adopted directly from model-produced labels. The revision will include this data and a description of the third-party adjudication mechanism in the main text.

  • 200K Judger training data:

160K labeled by Gemini 3.1 Pro, 40K manually annotated, following the four-level cultural-effectiveness rubric established by CULTURE-MT; this set is disjoint from (does not overlap with) the 40K-comment source pool used to build the benchmark.

2. Motivation, Practical Significance, Limitations, and Generalizability (W2, Q4)

  • Real-world prevalence:

The retention rate at each filtering stage, relative to the 40,000 original comments, is: 25% after rule-based filtering (10,000) → 12.5% after quality screening (5,000) → 7.5% after difficulty annotation (3,000) → 5.5% after deduplication/balancing (2,205) → 1.43% after human review (572).

We note that this overall retention rate conflates several distinct filtering motivations — comment-quality rejection, context-availability requirements, confirmed translation difficulty, deduplication, and deliberate balancing across 21 topical domains and three difficulty types — and therefore should not be read as a clean estimate of how often culturally risky mistranslation occurs in naturally distributed social-media comments; the balancing step in particular is designed to equalize representation across difficulty types rather than preserve their natural proportions.

What this funnel does support is a narrower, more defensible claim: genuinely challenging translation cases are a small minority even among comments already pre-filtered for basic quality and context availability, and such cases are easily diluted by aggregate scoring over a majority of 'easy' examples — which is the practical problem CULTURE-CMT is designed to surface through targeted, diagnostic sampling rather than through claims about population-level prevalence.

  • Validity of the evaluation criterion:

Our scoring rubric follows the already-validated four-level cultural-effectiveness scale from CULTURE-MT (Appendix E.2), whose design is explicitly intended to accommodate the reality that culturally situated expressions may admit multiple acceptable translations — scoring is based on "whether communicative intent is conveyed," not on exact match to a single reference. We offer three independent lines of evidence supporting the reliable application of this criterion:

  1. Binary agreement between the Agent Judger and human annotation reaches 77.31% (κ=0.5498, Figure 5);

  2. Independent human evaluation (detail in response 3 of Review D6Q6, N=200, paired McNemar tests, p=0.0104 non-think / p=0.0080 think) confirms that the Adaptation effect identified under this criterion is statistically significant;

  3. The rank correlation between human and Judger scores across 33 model configurations reaches Spearman's ρ=0.807 (p<0.0001).

AuthorsOpenReview ↗

Clarifications and Responses to Reviewer 1uvp Comments - 2

  • Limitations of the cultural-effectiveness evaluation:

We candidly note several limitations of our evaluation protocol that the revision will discuss explicitly:

  1. The 0–3 rubric ultimately rests on a human judgment of "whether communicative intent is conveyed," which is coarser than a fine-grained error taxonomy and cannot fully disambiguate cases with multiple defensible translations;
  2. As shown in our response to Reviewer D6Q6 (Q3), the Agent Judger exhibits an asymmetric error profile — Sensitivity 48.2% vs. Specificity 83.8% — meaning it is more reliable at confirming valid translations than at catching invalid ones, so Judger-reported Invalid Rates likely underestimate the true error rate;
  3. However, model-level agreement between the Judger and humans is strong for ranking (ρ=0.807).
  • Generalizability:

We acknowledge that this is not discussed in the current submission. All experiments in this paper are conducted on Chinese-to-English translations of comments from several social-media platforms, under one cultural-effectiveness rubric. We have not validated whether the benchmark-construction methodology, the Judger, or the empirical Scale/Reasoning/Adaptation findings transfer to other language pairs. We will add this as an explicit limitation in the revision and frame cross-lingual validation as a direction for future work.

3. Differences from CULTURE-MT, and Depth of Analysis (W3, Q3)

  • Distinction from CULTURE-MT:

CULTURE-MT evaluates the culturally situated translation of isolated comments. The core contribution of CULTURE-CMT is restoring the "post + parent comment" interaction structure in which a comment occurs (Figure 1), and decomposing difficulty into the three Slang/Context/Both subsets to enable diagnostic analysis — allowing us to separate "purely linguistic-cultural difficulty" from "purely contextual-dependence difficulty" and their combination, which CULTURE-MT's single-dimension design cannot do. The revision will foreground this distinction more prominently in the main text, rather than relegating it to the Appendix.

  • Fine-grained error analysis:

We agree that aggregate difficulty-group average scores (Figure 6) are insufficient to reveal how "contextual understanding" concretely affects "cultural effectiveness," and that model-ranking comparisons alone do not surface which linguistic phenomena are specifically challenging. Using the existing 25×33 per-example error-type labels in our human annotations, we re-analyzed the error-type distribution by difficulty type (Slang/Context/Both), normalized per sample-model unit to control for the uneven number of samples across categories:

Primary error typeSlang (per unit)Context (per unit)Both (per unit)
Literal translation1.2260.7150.897
Semantic deviation (addition/omission)0.5740.3430.345
Proper nouns0.3810.0710.235
Special terms/forms of address0.1100.0870.358

The three difficulty types show distinct error profiles, not merely differences in severity:

  • The Slang subset has the highest error density, concentrated in literal-translation errors (1.226 per unit, about 1.7× that of Context) — consistent with the expectation that, for pure slang/internet-language difficulty, the model's most common failure mode is mechanical word-for-word translation.

  • Proper-noun errors occur almost exclusively in the Slang subset (0.381 vs. 0.071 for Context) — indicating this error type is tied to culturally loaded expressions themselves, rather than to missing context.

  • Special-term/address errors show a super-linear amplification in the Both subset (0.358, markedly higher than Slang's 0.110 and Context's 0.087, exceeding their simple sum) — indicating that when slang/cultural expressions and contextual dependence co-occur, the model's failure rate rises sharply on forms of address and in-group slang that require double decoding via both cultural background knowledge and situational cues.

This provides a concrete, error-mechanism-level explanation for why the Both subset is consistently the hardest in Figure 6, rather than leaving the observation at the level of aggregate scores.

AuthorsOpenReview ↗

Clarifications and Responses to Reviewer 1uvp Comments - 3

Representative cases:

  1. Case 1 (bench_0305, Both, a discussion about the mobile game Honor of Kings):

The source mentions "小王怎么算账" ("how is Xiao Wang doing the math") and "无双池子" ("the Wushuang pool"). Nearly all 33 model configurations failed on this example, with errors concentrated on two pieces of in-group slang: "小王" (a personifying nickname players use for the game's official account/developer) was almost universally transliterated or mistaken for a personal name (e.g., rendered as "Wang Xiao," following an English given-name/surname convention); "无双" (an in-game skin-quality tier, officially translated as "Flawless") was transliterated as "Wushuang" or mistranslated literally as "Unparalleled"/"Peerless." The available context (the post body mentions the "Xiaoqiao Wedding" skin and the Wushuang pool) does not by itself supply the cues needed to decode "小王," because resolving this reference requires gaming-community background knowledge unavailable to an outsider, rather than information recoverable from the text of the context alone.

  1. Case 2 (bench_0041, Both, a discussion about a local light-meal brand):

The title and body repeatedly mention two local brand names, "超模厨房" (officially "Super Model Kitchen") and "超级碗" (the brand "FoodBowl"); despite ample contextual information, models still commonly mistranslate these as "the supermodel" ("超级模特") and "Super Bowl" (the American football event), systematically distorting the brand meaning.

Both cases concretely illustrate that the difficulty of the Both subset is not a simple addition of "slang + context," but rather that correctly decoding certain in-group slang terms or brand names inherently depends on cultural background knowledge the model lacks — restoring interactive context cannot substitute for this missing knowledge. This is precisely the mechanism behind the super-linear amplification of special-term/address errors we observe in the error-type statistics.

We will incorporate this fine-grained analysis and the representative cases into the main text of the revision and use them to revisit our novelty argument relative to CULTURE-MT — whose single-dimension difficulty design cannot reveal this kind of super-linear amplification arising from difficulty interaction.

Thank you for your responses and additional analyses.

However, I noticed that the manuscript has not been updated. Since many of my concerns relate to the paper's soundness and central claims, it is difficult to assess whether they have been adequately addressed without examining the corresponding revisions in the manuscript. For these reasons, I will maintain my original score.

Furthermore, considering the evaluations from the other reviewers, I believe the paper is unlikely to be accepted in this cycle. Therefore, I have not examined the rebuttal in detail.

Reviewer_SA5sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper works on an interesting task: contextualized machine translation that evaluates the cultural effectiveness of Chinese–English social-media comment translation. The benchmark has 572 manually verified examples, including not only slang but also contextual ambiguity, which is hard to translate without context. Experiments on LLMs show that the task is challenging and cannot be easily solved by increasing model scale or reasoning compute.

Critical strengths
  • The task is interesting to the community, especially contextual ambiguity beyond slang, which can be addressed using a dictionary. The experiments on LLMs show that the task is challenging for current SOTA models.
  • The data is reviewed by human annotators, which can guarantee its quality.
  • The authors provide a comprehensive selection of models and examine common routes for improving LLM-based social-media translation.
Critical weaknesses
  • There is no related work section. Slang translation is a classic machine translation task. It would be important to compare this dataset with existing slang datasets to show the novelty of this work.
  • This work discusses training a cultural-effectiveness judge through SFT and DPO using data generated by commercial LLMs. However, the performance of the trained judge is worse than that of the commercial LLMs. I am a bit confused about why the authors think the training is necessary.
  • This work provides very limited new insights to the community. The training uses SFT and DPO without any novelty. Although the authors explore some routes to improve performance, there is no in-depth analysis of the bottlenecks of these methods.
Questions for authors
  • Why do you use commercial models’ annotations to train a small model as the cultural-effectiveness judge instead of directly using the commercial models for evaluation?
  • Will you make the data public?
1 comment
AuthorsOpenReview ↗

Clarifications and Responses to Reviewer SA5s Comments

Thanks for your detailed reviews. Our responses to your weaknesses and questions are following:

1. Presentation of Related Work (W1)

The paper does include a Related Work discussion (Appendix A.1-A.3), but it is not presented as a standalone section in the main text — only briefly summarized in the Introduction. In the revision, we will: (1) add an explicit pointer from the Introduction to Appendix A; (2) expand the main text's comparison with existing slang/social-media translation resources such as SlangDIT, MTNT, and CSM-MTBench along concrete dimensions (whether native post/parent-comment context is retained, whether difficulty-stratified diagnosis is supported, whether the scoring criterion targets cultural communicative effectiveness), making explicit that our contribution lies in the combination of these three elements rather than in slang translation per se.

2. Necessity of Training the Judger (W2, Q1)

We first correct a factual point: the Judger is trained with SFT+GRPO, not DPO as stated in the review. More importantly, the reviewer's "worse performance" characterization is only half correct. If one compares only the static (non-retrieval-augmented) trained Judger against Gemini 3.1 Pro acting as a direct judge, the static version is indeed slightly weaker (SFT 74.54% vs. Gemini 74.77%). However, the main experiments actually use the retrieval-augmented Agent Judger, which achieves 77.31% binary accuracy and κ=0.5498 — clearly exceeding Gemini 3.1 Pro's 74.77%/κ=0.4986 (four-way accuracy is roughly tied). Training the Judger also brings cost and scalability benefits (avoiding repeated commercial API calls across 33 models × 572+ examples and avoiding scoring drift caused by commercial model version changes). We will present this comparison more prominently in the main text of the revision — it currently appears mainly in Figure 5 and can be easily overlooked.

3. Positioning of Novelty and Depth of Bottleneck Analysis (W3)

Our core contribution is not algorithmic novelty in the training method itself, but rather (1) a benchmark-construction methodology that jointly preserves interactive context and difficulty-stratified diagnosis, and (2) a systematic empirical characterization of Scale, Reasoning, and Adaptation under a unified cultural-effectiveness criterion — this is a descriptive assessment rather than a controlled decomposition of the three (already stated in Appendix A.3; we will restate this positioning more explicitly in the main text). In response to the specific concern about insufficient depth of analysis, we add two analyses directly targeting the Judger's limitations by human evaluation in Qwen3.6-27B and the SFT model (200samples, two models and think/nothink two modes, totaling 800), which can be viewed as a deeper characterization of the "bottleneck":

  • Sensitivity/Specificity asymmetry (N=800 human-annotated pairs): Sensitivity 48.2% vs. Specificity 83.8%, showing the Judger is markedly weaker at catching "invalid" translations than at confirming "valid" ones — this asymmetry is itself a bottleneck worth reporting;
  • Biased absolute scores but reliable ranking (33 model×mode configurations in 25 human-annotated pairs and their Judger score): the model-level Spearman correlation between human and Judger Invalid Rates is ρ=0.807 (p<0.0001), indicating that the Judger's practical value lies in reliable relative ranking rather than precise absolute scores — we will explicitly distinguish these two notions of reliability rather than claiming the Judger is simply "effective."

4. Open-Sourcing Commitment (Q2)

Consistent with our commitment to Reviewer D6Q6, we will open-source the benchmark, the evaluation score in the leaderboard, the details of evaluation to a personal email, and the retrieval evidence snapshot; license details will be confirmed in the camera-ready version.

Reviewer_D6Q6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CULTURE-CMT, a Chinese-to-English benchmark of social-media comments (each with a post title, body, replies, and a parent comment). Examples are labelled Slang, Context or Both. They are selected from 40K comments via rule-based filtering, synthetic GPT-5.5 annotation, LLM-based checks of translation difficulty and context dependence, plus review by three graduate students. The evaluation setting is reference-free, as is usual for user-generated content. A Qwen3.5 Judge is SFT-trained on 160K Gemini-labelled and 40K human-labelled pairs, then aligned with GRPO, and inference is done by a web-retrieval agent. It scores each translation from 0 to 3 under the CULTURE-MT rubric, and the main metric is the share of scores 0 or 1. The Judge reaches 77.3% binary agreement with human labels on 400 odd pairs. Across 30+ configurations, context lowers the invalid rate for most models; the Both subset is hardest, and a 27B Qwen3.6 model fine-tuned on 1.2M distilled examples has the lowest observed invalid rate.

Critical strengths

The task is framed the way readers meet it, and the resource does address a gap. Keeping the native post and parent-comment structure, and splitting difficulty into Slang, Context and Both, takes the vague idea of social-media difficulty turning it into something that can be measured per category. Fig. 1 shows a convincing case where the thread changes the referent of a nickname, and Fig. 6 shows that the Both subset is the hardest for every model in the figure, while models with similar overall rates differ in which single difficulty they handle worse. This matters for the assessment because context-aware MT evaluation has mostly targeted discourse phenomena such as anaphora and lexical cohesion (see Jwalapuram et al., 2020; Fernandes et al., 2023), and a benchmark built on interaction structure in user-generated content would be useful well beyond this paper.

The evaluation is broad, and its limits are well-stated. Many configs are scored in both context conditions and in both inference modes where available (Table 1), the human labels come from two annotation rounds with adjudication, and the paper itself notes that the cross-model comparisons are descriptive and that the Judge and the training data share one criterion. This matters because it makes the open questions easy to locate and, in principle, to resolve.

Critical weaknesses

The findings are on a single learned LLM Agent Judge whose agreement with people is only modest, and on differences that are small relative to the overall uncertainty. On the 400+ human-labelled pairs, this LLM Agent Judge reaches 77.3% binary accuracy and 44% four-way accuracy; which is below the 82.5% agreement between human rounds and close to untrained Gemini 3.1 Pro (frontier model) at 74.8. The evaluation set is balanced by score and pooled across 30 odd configs, so neither the Judge's error at the natural invalid rate of about 20% nor its bias on particular systems is known. Most comparisons in the abstract differ by 1-3 points across 570 odd items, with a 95% binomial interval of about (+-) 3 points. Also, no interval or paired test is reported. For instance, the claim that "larger models are not consistently better" rests on 18.9, 20.8, and 21.3 percent for the 27B, 122B, and 397B Qwen3.5 models, respectively. Context gain is also measured with a different Judge prompt in each condition, so it mixes the effect of context on translation with a change in how the translation is judged. In my opinion, this matters as the empirical claims are rankings and differences between configurations, and these are not yet separated from judge error and sampling noise.

The adaptation result is measured by a criterion that the adapted model was trained toward. The 1.2M SFT examples are Gemini translations refined with feedback from a static Judge under the same cultural-effectiveness criterion, and the evaluating Judge is itself trained on 160K Gemini labels App. E.1 notes that this shared criterion is not independent confirmation of the adapted model's ranking, yet the abstract and the contributions present the lowest invalid rate and "clear benefits from targeted adaptation" as findings. Benchmark selection has a similar dependency, since difficulty and context dependence are confirmed by disagreement among LLM translations (App. B.3/B.4) from unnamed models. Again, it matters because adaptation is one of the three headline findings, and a human comparison of the adapted and base Qwen3.6-27B on a subset would show whether it holds outside of the Judge.

Questions for authors
  • Could confidence intervals and paired tests over the same 570 odd items be added for the context gains and the comparisons? As an additional check, how do the context gains change if the comment-only translations are scored with the with-context Judge, so that both conditions are judged against the same interpretation?
  • Is any human evaluation of Qwen3.6-27B-SFT against the base Qwen3.6-27B available, even if on 100-200 items?
  • What are the judges' sensitivity and specificity for the invalid class, and does their agreement with humans differ across model families? I think a symmetric error would, on its own, produce an observed rate, so the error profile matters.
  • Will the full benchmark, the Judger weights and cached retrieval results be released, and under what licence?
  • Do the platform's terms of use permit redistribution of the comments? Since the agent uses live web search, how can scores be reproduced, if at all?
2 comments
AuthorsOpenReview ↗

Clarifications and Responses to Reviewer D6Q6 Comments - 1

We thank the reviewer for the careful attention to statistical rigor. Below, we address the concerns in five parts:

1. Confidence Intervals and Paired Significance Tests (first half of Q1)

We have added 95% bootstrap confidence intervals (10,000 resamples, N=572) for every reported Invalid Rate and paired McNemar tests for the core comparisons (samples are naturally paired: the same set of comments is translated and scored under different conditions).

  • Significance of Scale non-monotonicity: Qwen3.5-4B→9B (+12.4–18.0pp, p<10⁻⁶) and 9B→27B (+10.8–19.4pp, p<10⁻⁴) are highly significant; 27B→397B shows no significant difference (0.0pp and -2.4pp, n.s.). This statistically supports a "diminishing returns above 27B" reading rather than a simple "bigger is always better" claim.
  • Significance of Context Gain: 31 of 33 configurations (94%) show a positive direction, and 16 (48%) are significant at p<0.05. Configurations with non-significant direction (e.g., Gemini 3.1 Pro, thinking mode: 0.0pp, n.s.) are consistent with the already-small reported gains in Table 1, so the test results are credible. We will add full CIs and significance markers to Table 1 and Figure 7 in the revision, and qualify descriptive claims (e.g., "larger models are not necessarily better") with the appropriate statistical caveats.

2. Judging-Criterion Confound: Verification and Controlled Experiment (second half of Q1)

We first audited the actual inference code and confirmed the reviewer's concern: regardless of whether the translation received interactive context, our pipeline used the with-context Judger prompt during scoring. This is inconsistent with the description in Section 4.1 and Appendix E.3 ("each condition is scored with its corresponding Judger prompt"). We will correct this misstatement in the revision, and we apologize for the discrepancy. To quantify the actual impact, we re-scored no-context results using a without-context Judger prompt and compared these against the with-context Judger scores actually used.

  • Results:
  • The mean absolute shift across the 33 configurations is 1.75pp (median 1.75pp, max 7.0pp);
  • At the pooled level, the two Judger conventions yield a statistically significant difference in Invalid Rate (30.59% vs. 31.71%, +1.12pp, p=0.019) — confirming that the confound the reviewer identified is real.
  • Recomputing Context Gain with the strictly matched Judger raises the mean from the originally reported 4.23pp to 5.35pp (the two conventions' Gains are highly correlated, r=0.748, p<10⁻⁶). Re-evaluating under the judging-criterion convention not only leaves our core claim, that interactive context improves the cultural effectiveness of translation, but also shows the effect size was understated under the original setup. We will report results under both Judger conventions in the revision and correct the relevant description.

3. Independent Human Verification of Adaptation (Q2)

Using our existing double-annotation-and-adjudication pipeline, we conducted a two-stage independent human comparison between base Qwen3.6-27B and the SFT model:

StageNNon-think ΔThink ΔSignificance
Round 1 (difficulty-stratified)25+20.0pp+8.0ppUnderpowered (p=0.125 / 0.500)
Expanded (random sample, same distribution as the 572)200+9.0pp+8.1ppp=0.0104 / 0.0080 (both significant)
Reference: full Judger evaluation572+9.9pp+5.4pp—
The three independent lines of evidence (N=25 human, N=200 human, N=572 Judger) are highly consistent, forming a complete cross-validation chain showing that the Adaptation conclusion does not hinge on the Judger's own accuracy.

4. Judger Sensitivity and Specificity (Q3)

Based on an N=200 human-annotated set (800 human-vs-Judger pairs):

  • Overall binary agreement is 74.0%, with Sensitivity (correctly flagging "invalid" translations) at only 48.2% and Specificity (correctly confirming "valid" translations) at 83.8%; Cohen's κ=0.329. This asymmetry suggests the Judger is better at confirming that a translation is fine than at catching genuinely bad ones — meaning the Invalid Rates we report are more likely a conservative (under-)estimate rather than an inflated one. We additionally computed the model-level rank correlation between human and Judger Invalid Rates across 33 configurations between N=25 (scored by human) and N=572 (scored by Judger): Spearman's ρ=0.807 (p<0.0001) — indicating that while the Judger's absolute scores carry bias, its use for cross-model ranking (the basis for our Scale/Reasoning/Adaptation conclusions) is highly reliable.
AuthorsOpenReview ↗

Clarifications and Responses to Reviewer D6Q6 Comments - 2

5. Open-Sourcing and Reproducibility (Q4-Q5)

We will release the 572-example benchmark and the evaluation score on a leaderboard on Hugging Face; the details of the evaluation results can be sent to a personal email for their own use. The license (leaning toward CC BY-NC 4.0) will be finalized once our internal open-source approval process concludes and is confirmed in the camera-ready version. The release format for the comment text has been suited to platform terms and data-compliance review. Regarding reproducibility of retrieval-augmented evaluation, we will release a snapshot of retrieval evidence taken at experiment time; the default evaluation protocol will be based on this snapshot rather than live retrieval, ensuring exact reproducibility of scores. Live retrieval will remain available as an optional mode, with this limitation explicitly discussed in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.