- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
Paper studies study on whether translation naturalness can be attributed to specific attention heads in LLMs. The authors introduced Equi-DIFF, a sequence-level metric which compares sets of natural and literal translations instead of individual tokens or reference sequences. They identify heads associated with naturalness and validate their role through knockout and activation steering using path matching. They have experimented across 3 models, 12 translation directions, and 2 datasets which has shown that ablating these heads reduces natural-style preference, and that steering generally improves judged naturalness.
Critical strengths
The causal validation goes beyond differences in head rankings. I found the cross-ranking knockout experiment in Table 1 pretty useful: word-ranked heads affect the word-translation endpoint more, while naturalness-ranked heads affect the sequence-level naturalness endpoint more. The interaction persists under both zero and mean ablation and is supported by layer/type-matched controls. This is a meaningful evidence for different functional preferences, rather than simply observing that two metrics select different heads. The authors have appropriately distinguished this from claiming completely disjoint circuits.
2. Equi-DIFF addresses a relevant measurement problem. The motivation for scoring multiple complete translations is also convincing: neither a single token nor one particular wording adequately represents the intended behavior. The sequence-level, class-based formulation provides a useful way to study this choice, and the completion-choice analysis demonstrates that the selected wording can materially affect localization. This makes the readout a substantive part of the contribution.
Critical weaknesses
1. The interpretation of the natural/literal completion classes needs stronger validation. Appendix B constructs these classes using the model’s own generations, mechanical output filters, likelihood-based selection, and a prompt-conditioned score-separation criterion. These checks establish that the model distinguishes the two conditions, but do not by themselves establish that both classes preserve the source meaning and differ mainly in naturalness. I could not find an independent adequacy and style check of the discovery classes in the described protocol. Hence, the localized contrast may include semantic differences alongside stylistic ones. The held-out interventions remain informative, but the more specific interpretation of identifying components that choose between equally faithful translation strategies, is less firmly established.
2. The steering evaluation leaves the naturalness–adequacy trade-off difficult to assess. In Table 3, every reported Qwen direction, corpus setting favors the baseline among decisive adequacy judgments, despite favoring steering on naturalness. This does not establish a large overall adequacy loss: ties are excluded, MetricX improves, and COMET changes are small. However, it does require an explanation. Appendix E defers the underlying win/loss/tie counts to an evaluation artifact and states that they aggregate three prompt variants, making the aggregate results difficult to interpret from the manuscript alone. There is also a concrete reason to inspect the adequacy judgments: in Table 6, the steered translation of “Permission to relax” adds “watch some TV,” absent from the source as shown, yet receives a positive adequacy change. This example does not establish a systematic failure, but it makes independent validation of the adequacy results important.
Questions for authors
1.Clarification of the localization protocol: Were the natural and literal discovery completions independently checked for source-meaning preservation and the intended stylistic difference? If such checks were performed, please describe the procedure and summarize the results. This would directly affect my interpretation of what Equi-DIFF localizes.
2.Clarification of the existing evaluation: Could you provide the naturalness and adequacy win/loss/tie counts, together with paired adequacy-score changes, separately for neutral, natural, and literal prompts? Please also summarize the professional-translator assessment mentioned in Section 5 and clarify whether it covered adequacy or only naturalness. These results would help distinguish a limited trade-off from a broader problem with semantic preservation or judge reliability.
5 comments
Part 2/4
Weakness 2(a) & Question 2(a): Naturalness–Adequacy Trade-offs across Prompts.
Weakness 2(a):
- The steering evaluation leaves the naturalness–adequacy trade-off difficult to assess.
2 (a). In Table 3, every reported Qwen direction, corpus setting favors the baseline among decisive adequacy judgments, despite favoring steering on naturalness. This does not establish a large overall adequacy loss: ties are excluded, MetricX improves, and COMET changes are small. However, it does require an explanation.
Appendix E defers the underlying win/loss/tie counts to an evaluation artifact and states that they aggregate three prompt variants, making the aggregate results difficult to interpret from the manuscript alone.
Question 2(a):
2.Clarification of the existing evaluation:
2 (a). Could you provide the naturalness and adequacy win/loss/tie counts, together with paired adequacy-score changes, separately for neutral, natural, and literal prompts?
Response to Weakness 2(a) & Question 2(a):
We thank the reviewer for asking us to clarify the results by prompt. Steering improves naturalness across all three models, while its effect on adequacy varies by model and prompt. Below, we report the results by prompt and interpret the adequacy judgments alongside the naturalness gains.
Results by prompt
We use the Qwen family as an example, with Top-15 heads, α=1: the 15 highest-ranked heads, at steering strength α=1. The 144 settings cover 2 models × 2 corpora × 12 directions × 3 prompts. Complete results for Qwen and Gemma are in Appendix E.6, Tables 31–36.
The table shows selected language directions for Qwen3.5-4B on WMT24++, judged by Gemini-3.5-Flash, with 379 pairs per row. Neutral, Natural, and Literal use no style modifier, “naturally,” and “literally.” Nat. and Adeq. denote naturalness and adequacy. W/L/T means steering wins, baseline wins, and ties. Both dimensions use 1–7 scores; Δ is the mean steered-minus-baseline score difference, with a possible range of −6 to +6. Brackets show 95% confidence intervals.
| Direction | Prompt | Nat. W/L/T | Nat. Δ [95% CI] | Adeq. W/L/T | Adeq. Δ [95% CI] |
|---|---|---|---|---|---|
| En→Zh | Neutral | 210/82/87 | +0.605 [0.468, 0.748] | 80/129/170 | −0.134 [−0.214, −0.048] |
| Natural | 205/84/90 | +0.542 [0.417, 0.661] | 60/160/159 | −0.315 [−0.419, −0.208] | |
| Literal | 221/75/83 | +0.583 [0.454, 0.709] | 67/135/177 | −0.212 [−0.300, −0.132] | |
| En→Ja | Neutral | 161/127/91 | +0.164 [−0.001, 0.328] | 102/140/137 | −0.148 [−0.268, −0.034] |
| Natural | 167/122/90 | +0.197 [0.040, 0.357] | 117/132/130 | −0.082 [−0.196, 0.032] | |
| Literal | 191/96/92 | +0.425 [0.291, 0.570] | 130/109/140 | +0.079 [−0.029, 0.185] | |
| De→En | Neutral | 212/56/111 | +0.665 [0.543, 0.807] | 43/144/192 | −0.282 [−0.383, −0.180] |
| Natural | 190/60/129 | +0.504 [0.386, 0.649] | 40/165/174 | −0.394 [−0.486, −0.307] | |
| Literal | 227/44/108 | +0.786 [0.649, 0.934] | 55/95/229 | −0.084 [−0.182, 0.009] |
Averaging equally over the 12 directions, Literal gives the largest naturalness gain and smallest adequacy decrease for each Qwen model–corpus combination.
Part 3/4
Understanding the adequacy results
Steering encourages more natural expression. Adequacy evaluation helps identify when these changes also affect a detail of the source meaning, such as a qualifier or the speaker’s certainty.
For example, Qwen3.5-35B-A3B produced these WMT24++ translations under the Neutral prompt:
Source: Floor repair person is doing a vacuum, so hopefully that means he's finishing up.
Baseline: 地板维修人员正在吸尘,希望这意味着他即将完工。
Steered: 地板维修师傅正在吸尘,看来快收尾了。
The steered version uses more conversational phrasing, such as “维修师傅” and “快收尾了.” It also changes “希望” (“hopefully”) to “看来” (“it seems”), shifting a hope to an inference. This illustrates a specific trade-off accompanying the more natural expression: the average steered-minus-baseline adequacy difference was −1.00 under both sets of scoring instructions discussed below.
Human review also supports the naturalness gains. Four annotators independently assessed the same 200 English-to-Chinese pairs (100 sources), preferring steering for naturalness in 72% of comparisons on average. Their average baseline preference for adequacy was 34.88%, compared with 45.00% for the LLM on the same pairs. All percentages include ties in the denominator.
Scoring instructions also matter. The original instructions protected faithful translations from being penalized for awkward wording. The revised version also explicitly accepted meaning-preserving reformulations. On the same 70 pairs, evaluated in both presentation orders, the same Gemini judge’s preference for steering rose from 11.4% to 21.4%. The average steered-minus-baseline score difference increased by 0.117 points [95% CI: −0.001, 0.242]. This suggests that the original instructions were more conservative in accepting some meaning-preserving reformulations. Both versions retained penalties for omissions, mistranslations, and unsupported additions.
Interpreting COMET and MetricX
Both metrics learn from human quality assessments, but differ in their models, training data, and scoring scales. Our COMET model learns from direct quality ratings; MetricX-24 also learns from annotations of translation errors and their severity [1][2].
Across the four Qwen model–corpus combinations, MetricX scores decrease by 0.025–0.124 points, indicating improvement, while COMET changes range from −0.0016 to +0.0005. These overall-quality predictions complement the separate evaluations, which show the naturalness gains and accompanying adequacy trade-offs more directly.
Together, the results show that steering improves translation naturalness and that prompt choice matters for balancing this gain with adequacy.
[1] COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task
[2] MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task
Weakness 2(b): Source Context in the Translation Example.
2(b). There is also a concrete reason to inspect the adequacy judgments: in Table 6, the steered translation of “Permission to relax” adds “watch some TV,” absent from the source as shown, yet receives a positive adequacy change. This example does not establish a systematic failure, but it makes independent validation of the adequacy results important.
Response to Weakness 2(b):
We thank the reviewer for questioning the adequacy judgment in this example. The full source contains a television emoji (📺), which provides the context for “watch some TV.” The original judge received this complete source; the emoji was absent from the PDF example.
The recorded example is Qwen3.5-35B-A3B translating English into Chinese under the Literal (“literally”) prompt. Baseline and Steered denote outputs before and after intervention. The original Gemini-3.5-Flash adequacy scores use a 1–7 scale, with higher scores indicating better preservation of source meaning:
| Item | Original text | Original adequacy score |
|---|---|---|
| Source | Permission to relax. 📺 | — |
| Baseline | 放松许可。📺 | 3 |
| Steered | 放松一下,看个电视。📺 | 6 |
The original judge explained that “看个电视” (“watch some TV”) “naturally incorporates the TV emoji into the text”. It identified “放松许可” as an awkward noun phrase (“relaxation permit”), whereas “放松一下” conveys the intended invitation to relax. The television reference therefore has a basis in the source context. The revised appendix includes the complete source and translations (Table 21).
The separate four-annotator assessment described in our response to Weakness 2(a) & Question 2(a) covers 200 other English-to-Chinese translation pairs (100 source texts). Across its 800 paired judgments, 43.5% favored steering for naturalness while judging adequacy equal or better.
The complete source explains the television reference, and the separate human results support our interpretation: steering can improve natural expression while preserving the source’s intended meaning.
Part 1/4Weakness 1 & Question 1: Human Validation of Discovery Completions.
Weakness 1:
Question 1:
Response to Weakness 1 & Question 1:
We thank the reviewer for requesting linguistic validation of the discovery classes. Source-based human review supports both meaning preservation and the intended naturalness contrast in the reviewed text across all 15 discovery units. Reference regeneration provides separate evidence of stable localization of the core heads.
Our discovery pipeline establishes a controlled, model-specific contrast through matched natural/literal prompts and behavioral filtering. To make the linguistic evidence concrete, we report a focused analysis of Qwen3.5-4B English-to-Chinese. The source-based human assessment covers all 15 discovery units and examines the dimensions raised by the reviewer: source-meaning preservation, naturalness, and the types of linguistic differences.
Four Chinese-native annotators proficient in English contributed to the assessment. Each unit contains one English source and two five-member reference classes, generated under the natural (Yr) and literal (Yc) prompts. Reviewers recorded meaning relation and source faithfulness, checking for omissions, additions, and mistranslations. Naturalness judgments allowed ties, mixed preferences, and uncertainty.
The results are summarized below.
The final three rows are mutually exclusive difference types. 73.3% (11/15) of these units involve syntax, word order, or register differences, identifying linguistic choices our multi-token measure is designed to capture. These text-based judgments characterize meaning and expression, complementing the behavioral gate’s check of model responsiveness.
As a separate robustness check, we regenerated the model’s references and repeated localization. Fourteen of the original 15 sources passed the same behavioral criteria; old and new references were compared on this probability-selected subset. Top-15 heads denotes the 15 highest-ranked heads.
Ranks are positions in the regenerated head ranking; L and H denote layer and head/group indices.
Human review provides direct linguistic support for meaning-preserving naturalness differences in the reviewed text across all 15 discovery units, while reference regeneration recovers the stable core heads and provides complementary localization evidence.