- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors examine what happens during multilingual preference training when english examples have a much larger volume than examples in other languages. They construct translated opinion questions with either shared or opposing preferences and train four models using SFT+DPO. SFT retrains the input language while making preference errors and DPO introduces english fallback when preferences agree across languages. Conflicting preferences reduce this fallback under severe imbalance. The provides an examination of the effects by looking at the training objective, the preference update weights, and sparse-autoencoder features and interventions.
Critical strengths
S1. THe main result is a bit counterintuitive but seems consistent. It seems that conflicting preferences reduce the english fallback rate across all the models in the analysis quite significantly. The authors report that mean joint language and preference accuracy increases to 62%. Reporting these different outcomes helps show that preserving language and selecting the preferred answer involves some tradeoffs.
S2. The objective analysis gives a useful explanation of a vulnerability in DPO that doesn’t seem to have been discussed in related literature. Comparing two answers in the target language can improve the relative probabilities without directly requiring more probability on target language responses overall. This provides a good reason to examine the language behavior separately during preference training. The additional SAE interventions add even more evidence that features associated with a given language can affect answer selection.
Critical weaknesses
W1. The definition of multilingual disagreement being full preference reversal seems like a narrow test. In this experiments all four non-english languages oppose english but agree with one another, which gives the model a strong prior on what distinguishes english. It may be tricky but it would be helpful to expand this to focus on different preferences between all languages or to have a holdout where only particular topics conflict. Further definition of the disagreements expected and what their boundaries are could help.
Questions for authors
Q1. Why does aya lose joint accuracy under conflict even though its english fallback rate decreases?
Q2. For the SAE interventions what was hte split used to select layers and scales and how were the yes/no scored? It’d be helpful to know whether the reported preference cost was measured independently of the selection criteria.
2 comments
Continuation
Q2. SAE intervention selection, answer scoring, and preference cost.
(a) Layer and scale selection. We evaluated nine layers per model spanning early, middle, and late depths, chosen heuristically with reference to feature counts, and tested six normalized scales per layer and preference condition. Appendix G.2 defines the scale normalization, and Tables 3–4 report results across all tested layers. For the target-language amplification results summarized in Table 3, we selected the scale with the lowest English fallback at each layer, subject to at most a 3-percentage-point decrease in both raw and balanced preference correctness and a 5-point increase in unparsed outputs. Selection and reporting used the same evaluation set; there was no separate validation/test split for this scale selection. Thus, preference cost was part of the selection criterion and was not measured independently of selection. We will make this explicit in the intervention description. This constraint matters because stronger interventions can further reduce fallback while exceeding the preference-cost budget. For example, for Qwen under shared preferences at layer 24, increasing the scale from 0.6 to 1.6 reduces fallback from 20.33% to 11.80%, but increases the balanced-PC drop from 2.13 to 8.29 points. The stronger intervention is therefore excluded despite its lower fallback. The reported settings describe the observed tradeoff under this budget, rather than independently validated guarantees of preference preservation.
(b) Yes/No scoring. For target-language amplification, we rank the Yes/No answer forms across all five languages by their mean autoregressive answer-token log probability, excluding EOS, and select the highest-scoring candidate. For English-feature ablation, we use only the English and target-language Yes/No candidates, as specified in Table 5’s caption. We will clarify this distinction, since the candidate sets differ between the two experiments.
(c) Sensitivity to answer read-out. We additionally compared candidate ranking with greedy free generation, allowing up to eight new tokens, on the same 1,024 minority-language prompts. On unmodified checkpoints, fallback differs by at most 2.0 percentage points between the two read-outs. Using the intervention layers and scales from Table 1, SAE amplification also reduces fallback under free generation in all eight model–preference conditions. Across all 432 tested amplification settings, the mean fallback gap between read-outs is 0.3–1.2 points across models. These checks support the language-fallback finding beyond candidate ranking, while remaining limited to short answers; they do not provide an independent evaluation of the preference-cost budget.
Thank you again for your careful and positive review for our work. We hope the new experiments and clarifications address your concerns and make the scope and tradeoffs of our findings clearer.
Reply to Reviewer aRGE
Thank you for recognizing the counterintuitive finding and the usefulness of our objective analysis. To address your concerns, we provide new experiments with partial and language-specific preference conflict, report repeated-run results for Tiny-Aya, and clarify the SAE intervention selection and scoring procedures.
W1. Does the finding depend on full preference reversal across all minority languages?
We agree that full reversal is a controlled extreme. We use it to isolate the interaction between preference conflict and language imbalance, rather than to represent the full diversity of multilingual preferences. Here, conflict means that the preferred answer to the same question differs between a minority language and English. It does not require all minority languages to agree with one another. To test whether our finding depends on that agreement, we trained from the same shared-SFT checkpoint on a 50% shared / 50% conflicting mixture at 100:1 imbalance. Topics were assigned either commonly across minority languages (Same) or independently per language (Random). The independent assignment allows minority languages to disagree with one another on the same topic, removing the uniform English-versus-all-minority-languages structure. The table reports DPO English fallback, evaluated separately on shared and conflict topic subsets and averaged equally over the four minority languages (seed 0, values in %).
Conflict topics have lower fallback in all six model–assignment comparisons, although the magnitude varies. The reduction is substantial for BLOOMZ and modest for Llama and Qwen. Thus, the direction of the effect is not confined to full reversal or to all minority languages sharing the same conflict assignment. These results extend the controlled finding to partial and language-specific disagreement. We have added the mixed-setting results to Appendix K. We also clarify the boundary of our claim: these experiments still use constructed preference assignments and do not establish generalization to naturally occurring multilingual preference pairs.
Q1. Why does Tiny-Aya lose joint accuracy even when English fallback decreases? Joint accuracy TA requires both the preferred answer and the target language, whereas English fallback measures only whether the output is in English. Reducing fallback therefore does not guarantee higher TA: a response can avoid English and still express the wrong preference. Our additional runs confirm that Tiny-Aya is a boundary case. Across four training seeds at 100:1 imbalance, the conflict-minus-shared change in fallback is −1.73 ± 2.00 percentage points, while the change in TA is −7.57 ± 1.78 points (mean ± sample SD). The fallback reduction is small and inconsistent across seeds, whereas TA decreases in every seed. This is consistent with Figure 7, where Tiny-Aya exhibits a smaller shared–conflict difference in target-language-associated SAE activity, suggesting weaker sensitivity of its output-language behavior to preference structure. We have added the per-seed results and discussion to Appendix E. Our claim is that conflict can protect language under imbalance, not that it universally improves joint language-and-preference accuracy.