acceptodds
Under review as a conference paper at ICLR 2027

The Multilingual Alignment Paradox: Preference Conflict Hurts Alignment but Prevents Language Collapse

Abstract

Preserving language-specific preferences is challenging under language-imbalanced training, especially when preferences conflict across languages. We study this interaction using a controlled preference dataset derived from global opinion data, varying language imbalance and cross-language preference conflict. Across four multilingual LLMs, we compare supervised fine-tuning (SFT) and direct preference optimization (DPO). As imbalance increases, SFT largely preserves the input language but loses preference accuracy, whereas DPO increasingly produces English responses to minority-language inputs, especially under shared preferences. This reveals a **multilingual alignment paradox**: preference conflict reduces preference accuracy but can protect language fidelity. At an English-to-each-minority ratio of 100:1, conflict reduces English fallback across all four models and raises mean joint language-and-preference accuracy after DPO from 43.2% to 62.1%. Objective analysis shows that, unlike SFT, within-language DPO comparisons leave output-language probabilities unconstrained at the distribution level, while training dynamics indicate stronger residual minority preference signals under conflict. Sparse autoencoder analysis further identifies stronger target-language and weaker English-associated activity under conflict. Intervening on these features reduces English fallback with limited impact on preference accuracy. Together, these findings suggest that language-selection features retained with language-specific preferences even under severe imbalance in the alignment process.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Preference Alignment, Multilinguality, Conflicted Information

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aRGEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors examine what happens during multilingual preference training when english examples have a much larger volume than examples in other languages. They construct translated opinion questions with either shared or opposing preferences and train four models using SFT+DPO. SFT retrains the input language while making preference errors and DPO introduces english fallback when preferences agree across languages. Conflicting preferences reduce this fallback under severe imbalance. The provides an examination of the effects by looking at the training objective, the preference update weights, and sparse-autoencoder features and interventions.

Critical strengths

S1. THe main result is a bit counterintuitive but seems consistent. It seems that conflicting preferences reduce the english fallback rate across all the models in the analysis quite significantly. The authors report that mean joint language and preference accuracy increases to 62%. Reporting these different outcomes helps show that preserving language and selecting the preferred answer involves some tradeoffs.

S2. The objective analysis gives a useful explanation of a vulnerability in DPO that doesn’t seem to have been discussed in related literature. Comparing two answers in the target language can improve the relative probabilities without directly requiring more probability on target language responses overall. This provides a good reason to examine the language behavior separately during preference training. The additional SAE interventions add even more evidence that features associated with a given language can affect answer selection.

Critical weaknesses

W1. The definition of multilingual disagreement being full preference reversal seems like a narrow test. In this experiments all four non-english languages oppose english but agree with one another, which gives the model a strong prior on what distinguishes english. It may be tricky but it would be helpful to expand this to focus on different preferences between all languages or to have a holdout where only particular topics conflict. Further definition of the disagreements expected and what their boundaries are could help.

Questions for authors

Q1. Why does aya lose joint accuracy under conflict even though its english fallback rate decreases?

Q2. For the SAE interventions what was hte split used to select layers and scales and how were the yes/no scored? It’d be helpful to know whether the reported preference cost was measured independently of the selection criteria.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer aRGE

Thank you for recognizing the counterintuitive finding and the usefulness of our objective analysis. To address your concerns, we provide new experiments with partial and language-specific preference conflict, report repeated-run results for Tiny-Aya, and clarify the SAE intervention selection and scoring procedures.

W1. Does the finding depend on full preference reversal across all minority languages?

We agree that full reversal is a controlled extreme. We use it to isolate the interaction between preference conflict and language imbalance, rather than to represent the full diversity of multilingual preferences. Here, conflict means that the preferred answer to the same question differs between a minority language and English. It does not require all minority languages to agree with one another. To test whether our finding depends on that agreement, we trained from the same shared-SFT checkpoint on a 50% shared / 50% conflicting mixture at 100:1 imbalance. Topics were assigned either commonly across minority languages (Same) or independently per language (Random). The independent assignment allows minority languages to disagree with one another on the same topic, removing the uniform English-versus-all-minority-languages structure. The table reports DPO English fallback, evaluated separately on shared and conflict topic subsets and averaged equally over the four minority languages (seed 0, values in %).

ModelTopic assignmentShared topicsConflict topics
Llama-3.2-1BSame13.813.5
Llama-3.2-1BRandom35.833.7
BLOOMZ-1.1BSame62.946.8
BLOOMZ-1.1BRandom56.441.3
Qwen3-1.7BSame50.748.1
Qwen3-1.7BRandom44.842.8

Conflict topics have lower fallback in all six model–assignment comparisons, although the magnitude varies. The reduction is substantial for BLOOMZ and modest for Llama and Qwen. Thus, the direction of the effect is not confined to full reversal or to all minority languages sharing the same conflict assignment. These results extend the controlled finding to partial and language-specific disagreement. We have added the mixed-setting results to Appendix K. We also clarify the boundary of our claim: these experiments still use constructed preference assignments and do not establish generalization to naturally occurring multilingual preference pairs.

Q1. Why does Tiny-Aya lose joint accuracy even when English fallback decreases? Joint accuracy TA requires both the preferred answer and the target language, whereas English fallback measures only whether the output is in English. Reducing fallback therefore does not guarantee higher TA: a response can avoid English and still express the wrong preference. Our additional runs confirm that Tiny-Aya is a boundary case. Across four training seeds at 100:1 imbalance, the conflict-minus-shared change in fallback is −1.73 ± 2.00 percentage points, while the change in TA is −7.57 ± 1.78 points (mean ± sample SD). The fallback reduction is small and inconsistent across seeds, whereas TA decreases in every seed. This is consistent with Figure 7, where Tiny-Aya exhibits a smaller shared–conflict difference in target-language-associated SAE activity, suggesting weaker sensitivity of its output-language behavior to preference structure. We have added the per-seed results and discussion to Appendix E. Our claim is that conflict can protect language under imbalance, not that it universally improves joint language-and-preference accuracy.

AuthorsOpenReview ↗

Continuation

Q2. SAE intervention selection, answer scoring, and preference cost.

(a) Layer and scale selection. We evaluated nine layers per model spanning early, middle, and late depths, chosen heuristically with reference to feature counts, and tested six normalized scales per layer and preference condition. Appendix G.2 defines the scale normalization, and Tables 3–4 report results across all tested layers. For the target-language amplification results summarized in Table 3, we selected the scale with the lowest English fallback at each layer, subject to at most a 3-percentage-point decrease in both raw and balanced preference correctness and a 5-point increase in unparsed outputs. Selection and reporting used the same evaluation set; there was no separate validation/test split for this scale selection. Thus, preference cost was part of the selection criterion and was not measured independently of selection. We will make this explicit in the intervention description. This constraint matters because stronger interventions can further reduce fallback while exceeding the preference-cost budget. For example, for Qwen under shared preferences at layer 24, increasing the scale from 0.6 to 1.6 reduces fallback from 20.33% to 11.80%, but increases the balanced-PC drop from 2.13 to 8.29 points. The stronger intervention is therefore excluded despite its lower fallback. The reported settings describe the observed tradeoff under this budget, rather than independently validated guarantees of preference preservation.

(b) Yes/No scoring. For target-language amplification, we rank the Yes/No answer forms across all five languages by their mean autoregressive answer-token log probability, excluding EOS, and select the highest-scoring candidate. For English-feature ablation, we use only the English and target-language Yes/No candidates, as specified in Table 5’s caption. We will clarify this distinction, since the candidate sets differ between the two experiments.

(c) Sensitivity to answer read-out. We additionally compared candidate ranking with greedy free generation, allowing up to eight new tokens, on the same 1,024 minority-language prompts. On unmodified checkpoints, fallback differs by at most 2.0 percentage points between the two read-outs. Using the intervention layers and scales from Table 1, SAE amplification also reduces fallback under free generation in all eight model–preference conditions. Across all 432 tested amplification settings, the mean fallback gap between read-outs is 0.3–1.2 points across models. These checks support the language-fallback finding beyond candidate ranking, while remaining limited to short answers; they do not provide an independent evaluation of the preference-cost budget.

Thank you again for your careful and positive review for our work. We hope the new experiments and clarifications address your concerns and make the scope and tradeoffs of our findings clearer.

Reviewer_r5CZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the challenge of preserving minority language preferences and output language under language imbalance. Using a controlled multilingual preference dataset derived from GlobalOpinionQA, the authors vary language imbalance and preference conflict and compare SFT and DPO across four models and five languages. Under increasing imbalance, SFT largely preserves the input language but makes more preference selection errors, whereas DPO increasingly exhibits English fallback, especially under shared preferences. Conflicting preferences can make preference selection harder while reducing English fallback. The authors investigate this phenomenon through objective analysis, training dynamics, and SAE interventions.

Critical strengths

The research question is clearly defined. The paper distinguishes preference correctness from language preservation and shows that preference accuracy alone can hide English fallback.

Critical weaknesses
  1. The theoretical analysis does not sufficiently explain why preference conflict can preserve the target language. The key assumptions of the two-coordinate surrogate are not sufficiently validated against actual training dynamics, and larger residual minority DPO weights do not directly establish improved language preservation. SAE interventions provide causal evidence that language-associated representations influence output-language selection, but do not sufficiently establish how these representations mediate the effect of preference conflict on English fallback.
  2. The experimental setup and evaluation limit the generalizability of the findings and the strength of the causal interpretation. Fully reversed minority-language preferences, minority-language subsampling, and different SFT checkpoints make it difficult to isolate the effects of preference conflict, language imbalance, and initialization. The results primarily demonstrate reduced English fallback under controlled conditions. Additional controls, evaluation of freely generated responses, and repeated runs are needed to establish the robustness and broader applicability of the findings.
Questions for authors
  1. The cancellation of language selection factors shows that within language DPO comparisons do not directly constrain output language probabilities. What causes actual parameter updates to favor English under language imbalance?
  2. The two coordinate surrogate assumes that improving the English DPO margin helps minority languages under shared preferences but hurts them under conflicting preferences. What evidence supports this assumption in the trained models? How well does the local comparison at a common state explain training from different SFT checkpoints?
  3. Larger minority DPO weights could simply mean that these preference pairs remain harder to learn. Do the corresponding gradients and parameter updates actually help preserve the target language?
  4. The SAE interventions show that language associated features affect output language. Do they also explain why conflict reduces English fallback? Would removing the target language features retained under conflict reduce this advantage? Random directions with the same intervention dose would provide a useful control.
  5. In the conflict setting, every minority language reverses the English preference ordering. How much does the result depend on this construction? Does it hold when only some preferences conflict, or when preferences come from natural data? Could a task that requires distinguishing languages without conflicting preferences produce a similar benefit?
  6. Several factors change across conditions, including the number of minority language examples and the SFT checkpoint. Can the authors separate their effects by keeping the training examples fixed while varying sampling weights, and by starting DPO from the same SFT checkpoint? A continued SFT control with comparable training exposure would also help distinguish the effect of DPO from that of additional training.
  7. Do the SAE intervention results hold for freely generated responses, including longer answers, rather than only candidate ranking? Are they sensitive to the answer scoring method? Please clarify whether intervention layers and doses were selected on separate validation data. The discussion should also distinguish reduced English fallback from improved joint language and preference accuracy, which can remain below the SFT result.
  8. How much do the SFT and DPO training sets overlap? Are all rewrites and translations of each original question kept in the same data split? Please also report whether the main results hold across different training seeds and different samples of minority language training data.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer r5CZ

Thank you for the careful review of our work. To address your concerns, we provide new experimental results, clarify how our paper explains why preference conflict can preserve the target language, and discuss the generalizability of our findings and interpretation.

Our key contribution. We position our key contribution as uncovering the multilingual alignment paradox, a counterintuitive phenomenon that reviewer aRGE also highlighted as a strength. To our knowledge, this is the first work to study cross-lingual conflicting preferences in preference alignment, and to show that such conflict can reduce language fallback to English, a failure mode widely reported in recent multilingual agentic benchmarks such as BabelArena [1].

W1: Why preference conflict reduces language fallback

(a) Within-language DPO leaves output-language mass unidentified. As shown in Section 4, language-selection factors cancel in the DPO objective for within-language preference pairs, so nothing in the loss constrains the policy's language mass. This predicts that adding an NLL term on the chosen response, which supplies the language anchor that SFT provides, should prevent English fallback. We tested this on all four models under both preference structures at a 100:1 ratio, using an auxiliary NLL term normalized by chosen-response length with λ=1:

ModelShared: DPOShared: DPO + NLLConflict: DPOConflict: DPO + NLL
Llama-3.2-1B21.680.9810.550.59
BLOOMZ-1.1B61.040.398.501.95
Qwen3-1.7B60.741.4623.245.47
Tiny-Aya-3.35B30.270.6826.764.10

The prediction holds for all four models. Average fallback drops from 43.43% to 0.88% under shared preferences and from 17.26% to 3.03% under conflict preferences. We have added these results to Section 4.

(b) Conflict changes the learning signal in a way that protects target-language output (Q1–Q3). Figure 5 of our paper shows that under shared preferences, DPO training enlarges the implicit reward gap toward dominant-language responses, whereas conflict preferences suppress this gap. We replicated the experiment by training both preference settings from the same SFT checkpoint, and the conclusion is unchanged. This directly supports that conflicting preferences better preserve the target language, and we have added the same-checkpoint results to Section 4.

We previously used the coordinate surrogate to illustrate stronger residual minority preference signals under conflict. We agree that stronger signals alone do not establish language-preserving updates. Our revised explanation therefore starts from the original DPO loss, showing that conflict reverses the optimization pressure on the target-language preference margin. We test how this opposition transfers to language expression from a common Shared-SFT initialization. After one large-batch update, conflict produces **smaller increases in English-over-target reward gaps for the meanings across all three tested models, primarily through less suppression of target-language responses. During sustained training, conflict also weakens the English-preferred answer’s margin among English responses and strengthens the target-language advantage for the minority-preferred answer. These results provide empirical evidence linking preference opposition to reduced English dominance.

AuthorsOpenReview ↗

Continuation

(c) Conflict preserves language through identified SAE features (Q4). We agree that our original steering experiment shows only two things: conflict preferences yield more target-language features, and those features can control output language. It does not test whether conflict acts through them, as we noted in Appendix A. To test this directly, we ran a conflict-associated SAE feature ablation. For each minority language, we selected the 10 target-language SAE features with the largest increase in activation frequency from the shared to the conflict checkpoint, and ablated the same feature IDs in both checkpoints. As a control, we ablated 10 target-language features with matched activation frequency across the two checkpoints (five random draws where enough features exist). We evaluated Llama, BLOOMZ, and Qwen on 896 prompts per language. Fallback rates, averaged over the four minority languages, are:

ModelShared: native → ablated (%)Conflict: native → ablated (%)
Llama-3.2-1B24.6 → 27.79.7 → 19.3
BLOOMZ-1.1B61.4 → 61.77.4 → 60.3
Qwen3-1.7B62.2 → 63.125.4 → 31.7

After ablation, the shared–conflict fallback gap narrows from 14.9 to 8.4 points for Llama, 54.0 to 1.4 for BLOOMZ, and 36.8 to 31.4 for Qwen. This extends our steering analysis by testing the behavioral contribution of features selected for their differential activity between the two training conditions. Control ablations change the conflict checkpoints much less: fallback stays at 9.7% for Llama and rises only from 25.4% to 26.7% for Qwen. For BLOOMZ, enough control features exist only for Chinese, so we compare within that language: ablating the selected features raises conflict-checkpoint fallback from 14.5% to 78.9%, whereas control ablation yields 14.9%. Together, these results indicate that conflict-associated target-language features contribute to language preservation in these checkpoints. We will add results on the full dataset to the paper.

AuthorsOpenReview ↗

W2: The generalizability of our findings and interpretation

W2: The generalizability of our findings and interpretation

(a) Motivation for the synthetic dataset. There’s no existing training data that captures cross-lingual preference conflict. Cultural benchmarks such as BLEND[3] and Global MMLU[4] show that correct answers can differ across languages, and GlobalOpinionQA[2] shows that people's opinions on the same question vary across countries. Consistent with this, prior work finds that language models' generations are biased toward Western culture and that preference learning can cause significant language collapse[5], while test-time steering can partially recover local culture[6]. This motivated us to study the setting with a controlled dataset, so that we can isolate how models learn multilingual conflicting preferences during DPO training.

(b) The findings hold under a mixed preference setting.(Q5) We trained from the same shared-SFT checkpoint on a 50% shared / 50% conflicting mixture at 100:1 imbalance. Topics were assigned either commonly across minority languages (Same) or independently per language (Random). Table R1 reports fallback and joint language-and-preference accuracy (TA), evaluated separately on the shared and conflict topic subsets and averaged equally over the four minority languages (seed 0, values in %).

Table R1. Mixed-preference training at 100:1 imbalance.

ModelTopic assignmentTopic subsetSFT fallbackDPO fallbackSFT TADPO TA
Llama-3.2-1BSameShared0.013.863.554.5
Llama-3.2-1BSameConflict0.013.544.246.6
Llama-3.2-1BRandomShared0.235.864.039.9
Llama-3.2-1BRandomConflict0.233.757.135.5
BLOOMZ-1.1BSameShared0.062.975.922.3
BLOOMZ-1.1BSameConflict0.246.853.630.8
BLOOMZ-1.1BRandomShared0.056.473.229.1
BLOOMZ-1.1BRandomConflict0.241.367.642.3
Qwen3-1.7BSameShared0.250.764.227.2
Qwen3-1.7BSameConflict0.448.152.631.0
Qwen3-1.7BRandomShared0.044.861.731.2
Qwen3-1.7BRandomConflict1.442.857.231.1

Conflict topics have lower DPO fallback in all six model–assignment comparisons, though the magnitude varies. Conflict-minus-shared differences are −0.3 / −16.1 / −2.6 pp (Llama / BLOOMZ / Qwen) under the common assignment and −2.1 / −15.1 / −2.0 pp under independent assignments. The effect is strong for BLOOMZ and modest for Llama and Qwen, so the direction is not confined to full reversal or to a common conflict assignment. The implicit reward gap analysis on the mixed data also shows a gap toward the dominant language, smaller than in the fully conflicting setting because the weaker conflict signal comes from less conflict data. We use the extreme setup in the main paper by design, to isolate compounding factors such as imbalance and conflict rate. All mixed-setting results are added to the Appendix K.

A language-discrimination task without preference conflict falls outside this study and would be a useful control for whether language-conditioned supervision in general gives the same benefit. Our results do not establish that conflict is the only way to retain such information.

(c)Separating initialization from data exposure (Q6) We reran the experiments from a single shared SFT checkpoint, and all conclusions hold as before. Conflicting preferences yield fewer language fallbacks than shared preferences. In the implicit reward analysis, shared preferences show a reward gap toward the dominant language, and the conflict setting mitigates it. Since the trends match our original results and do not change any conclusion, we report the full results in the Appendix for reference.

(d) Free generation and sensitivity to answer read-out (Q7) We compared candidate ranking with greedy free generation (up to eight new tokens) on the same 1,024 minority-language prompts at 100:1 imbalance. English fallback differs by at most 2.0 pp between the two read-outs on the unmodified checkpoints, and conflict yields lower fallback in all four models under both methods.

SAE amplification also reduces fallback under free generation in all eight model–preference conditions, using the layers and doses from the paper's Table 1:

ModelShared: baseline → amplified (%)Conflict: baseline → amplified (%)
Llama-3.2-1B23.3 → 1.49.9 → 0.2
BLOOMZ-1.1B60.2 → 0.07.8 → 0.0
Qwen3-1.7B60.9 → 17.822.4 → 1.6
Tiny-Aya-3.35B29.8 → 0.326.9 → 3.2

These effects are similar to candidate ranking on the same prompts. Across all 432 tested amplification settings, the mean fallback gap between the two read-outs is 0.3–1.2 pp across models, supporting agreement beyond the selected settings.

Reviewer_TA8BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multilingual alignment, where one training objective aligns a model in several languages at once. It asks how imbalance between languages affects preference learning. English is the majority language (14,350 training prompts), and Chinese, French, Hindi and Arabic are subsampled to give ratios of 1:1, 10:1 and 100:1. In the "shared" condition every language prefers the same answer; in the "conflict" condition the four minority languages reverse the English preference. Four 1-3.35B models are trained with SFT and then with DPO.

An answer counts as correct only if it holds the preferred view and is written in the language of the question. A minority-language question that receives an English answer is therefore wrong; the paper calls this English fallback, or language collapse. SFT largely keeps the output in the input language, so it is a control for the DPO results. After DPO, outputs drift to English under severe imbalance. The main finding is that, under DPO, conflicting minority preferences reduce this drift: at 100:1, conflict lowers English fallback in all four models (Table 1) and raises joint language-and-preference accuracy (TA, averaged over models and minority languages) after DPO from 43.2% to 62.1% (l.263-266).

Critical strengths
  1. SFT controls and DPO across several models. Each condition is trained with SFT and then DPO on four model families, with the comparisons and test sets held fixed. Because SFT keeps the output language while DPO does not, the SFT runs isolate the language drift as a property of the DPO stage. This control is what makes the main finding interpretable. The SFT/DPO contrast and the effect of conflict on language fallback under DPO are worth knowing.

  2. The effect is isolated well across results. The design varies imbalance and preference structure separately, and scores language and preference separately. The reduction in English fallback under conflict appears in all four models (Table 1 baselines, e.g. BLOOMZ 60.7% to 9.4%, Qwen3 62.4% to 22.6% at 100:1). The finding is new to me. The setup and Figs. 2-3 are clear.

Critical weaknesses
  1. The explanation of the effect is neither shown thoroughly nor backed by an intervention. The main contribution is the discovery of this behaviour. To make it a solid contribution, the paper needs either a thorough account of why it happens, or an intervention derived from its explanation that changes the behaviour.

    The SAE experiments are shallow and do not explain the effect. Fig. 6 compares counts of active language features between the two conditions, which the authors call descriptive (l.455-456). Steering (Table 1) adds target-language feature directions to the activations at inference time, with the reported layer and dose chosen from a sweep (App. G). It reduces fallback in both conditions, for BLOOMZ to 0.0% in each (from 60.7% shared and 9.4% conflict). This shows that language features control output language. It does not show that conflict acts through these features (as l.655-657 concedes), so it does not test the proposed mechanism.

    An intervention on this effect by altering DPO is missing. Eqs. 5-7 suggest that adding an NLL term on y⁺ to DPO (as in Iterative RPO; Pang et al., 2024) would restore the language anchor that SFT has, so this variant should avoid language collapse under shared preferences at 100:1. More generally, the paper trains only standard DPO, so it is unknown whether DPO variants show the same behaviour. Without this, the effect may be an artifact of standard DPO rather than a property of preference conflict.

  2. One run per condition, so the effect itself is not shown thoroughly. The paper reports one result per cell, with no seeds or run-to-run variance for any main result. The direction of the fallback reduction is consistent across models, but the change in TA after DPO at 100:1 varies widely, from +51.3 points for BLOOMZ to −6.3 for Aya (averaged over the four minority languages, Table 2). While the effect is measured carefully, each result comes from one reported run, and the explanation of the effect is not tested. At least three seeds for the 100:1 cells are needed to confirm the effect is not run-to-run noise.

The effect is interesting and the controls are good. However, the paper neither shows the effect thoroughly (one reported run per cell, with gains uneven across models) nor backs its explanation with an intervention; the SAE steering works in both conditions and so does not test the mechanism. I would raise my score to weak accept if the authors (a) run multiple seeds, and (b) run a DPO variant derived from their analysis, such as DPO with an NLL term on y⁺, and show that it avoids language collapse under shared preferences.

Questions for authors
  1. (Experiment; could change my rating.) Can you report at least three seeds for the 100:1 cells?
  2. (Experiment; could change my rating.) Does DPO with an added NLL term on y⁺ avoid language collapse under shared preferences at 100:1, as Eqs. 5-7 suggest? Does the effect of conflict hold for other DPO variants?
3 comments
AuthorsOpenReview ↗

Reply to reviewer TA8B

Thank you for recognizing that the effect is interesting and the experiment controls are good. We also appreciate that you stated concrete conditions under which you would lean toward acceptance. We have run both suggested experiments : (a) additional seeds (4 seeds in total for the 100:1 cells, i.e., three beyond the original run), and (b) adding NLL term to DPO loss function to strengthen our objective analysis. Moreover, we also added a new SAE ablation targeting your concern about model representation analysis. Details below.

1. Robustness across training seeds. We ran three additional seeds (22, 26, and 37) for the shared and conflicting DPO conditions at 100:1 imbalance. The table below reports mean ± SD across all four seeds.

ModelShared fallback (%)Conflict fallback (%)Δ fallbackΔ TA
Llama 1B24.93 ± 4.4610.03 ± 1.58−14.89 ± 5.73+6.23 ± 7.82
BLOOMZ 1.1B49.98 ± 16.5710.67 ± 1.54−39.31 ± 17.47+40.84 ± 14.93
Qwen3 1.7B62.87 ± 5.9319.78 ± 3.13−43.09 ± 7.86+32.28 ± 7.32
Tiny-Aya 3.35B25.71 ± 3.1223.97 ± 2.68−1.73 ± 2.00−7.57 ± 1.78

Conflict reduces English fallback in every seed for Llama, BLOOMZ, and Qwen, confirming that the direction of the effect is consistent across runs, although its magnitude varies. Tiny-Aya is a boundary case: its fallback reduction is small and inconsistent across seeds. This is consistent with Figure 7, where Tiny-Aya exhibits a smaller shared–conflict difference in target-language-associated SAE activity, suggesting weaker sensitivity of its output-language behavior to preference structure.

Additionally, the joint accuracy score TA (target-aligned) measures joint success in preference prediction and output-language selection, and therefore reflects both preference errors and language collapse. Across 4 seeds, conflict consistently improves TA for BLOOMZ and Qwen, has mixed effects for Llama, and consistently reduces TA for Tiny-Aya. Thus, reduced fallback does not necessarily imply improved joint accuracy. We have added per-seed results and mean ± SD to Appendix E, and explicitly discuss this heterogeneity there.

2. Intervention derived from the objective analysis (DPO + NLL). Our analysis in Section 4 shows that for within-language preference pairs, language-selection factors cancel in the DPO objective, so the language mass of the policy is left unidentified. This predicts that adding an NLL term on the chosen response, which supplies the language anchor that SFT has, should prevent English fallback. We tested this prediction on all four models under both preference structures at 100:1, using an auxiliary NLL term normalized by the chosen response length with =1. The results are as below:

ModelShared: DPOShared: DPO + NLLConflict: DPOConflict: DPO + NLL
Llama-3.2-1B21.680.9810.550.59
BLOOMZ-1.1B61.040.398.501.95
Qwen3-1.7B60.741.4623.245.47
Tiny-Aya-3.35B30.270.6826.764.10

Our prediction holds for all 4 models. Average fallback drops for both shared preferences (from 43.43% to 0.88%) and for conflict preferences (from 17.26% to 3.03%). As what have predicted by the loss function analysis, language collapse is therefore not an inevitable consequence of imbalance, but a consequence of the missing language anchor in standard DPO, and restoring that anchor removes it. We added this experiment to Section 4. For our research scope, we agree that the protective effect of conflict for language collapse is objective-dependent. We interpret our findings as an interaction between preference conflict, language imbalance, and standard DPO’s lack of an explicit language anchor, rather than as an objective-independent effect of conflict.

AuthorsOpenReview ↗

Continuation

3. Do conflict-associated SAE features contribute to language preservation? We agree that the original steering experiment shows language features control output language but does not test whether conflict acts through them, as we noted in the Appendix A. To address this directly, we ran a conflict-associated SAE feature ablation. For each minority language, we select the 10 target-language SAE features with the largest increase in activation frequency from shared to conflict checkpoints and ablate the same feature IDs in both. As a control, we ablate 10 target-language features with matched activation frequency across the two checkpoints (five random draws where enough features exist). We evaluate Llama, BLOOMZ and Qwen on 896 prompts per language. The fallback rate is averaged over the four minority languages:

ModelShared: native → ablated (%)Conflict: native → ablated (%)
Llama-3.2-1B24.6 → 27.79.7 → 19.3
BLOOMZ-1.1B61.4 → 61.77.4 → 60.3
Qwen3-1.7B62.2 → 63.125.4 → 31.7

We found that the shared–conflict fallback gap narrows from 14.9 to 8.4 for Llama, 54.0 to 1.4 for BLOOMZ, and 36.8 to 31.4 for Qwen. This extends our original steering analysis by directly testing the behavioral contribution of features selected for their differential activity between the two training conditions. In contrast, control ablations produce smaller changes in the conflicting checkpoints: fallback remains at 9.7% for Llama and rises only from 25.4% to 26.7% for Qwen. For BLOOMZ, sufficient control features are available only for Chinese, so we compare within that language: selected-feature ablation raises conflict-checkpoint fallback from 14.5% to 78.9%, whereas control ablation yields 14.9%. These results provide us a further interpretation that conflict-associated target-language features contribute to language preservation in these checkpoints. We’ll add the experiment results on the full dataset to our paper.

Thank you again for your careful review. We hope the new experiments and revised paper address your concerns, and we would be grateful if you would consider updating your score accordingly.

AuthorsOpenReview ↗

Reminder for the discussion

Dear reviewer TA8B,

Thank you again for your constructive feedback, which has greatly helped us improve our work. As the discussion period closes soon, we wanted to follow up briefly. We hope our additional experiments and clarifications have addressed your concerns. We’d be grateful for any further feedback, and we’re happy to clarify anything that remains unclear.

Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank all reviewers for their thoughtful and constructive feedback, which helped us strengthen both the explanation and the empirical evidence.

  • Key contribution. To our knowledge, this is the first work to systematically study cross-lingual preference conflict in multilingual preference alignment. We uncover the multilingual alignment paradox: under language imbalance, conflicting preferences can reduce English fallback during DPO, even while making preference learning harder. Reviewer TA8B recognized the finding as new, while reviewer aRGE highlighted its counterintuitive nature and the value of our objective analysis.

  • Dataset motivation. Existing cultural and opinion benchmarks motivate cross-lingual disagreement but lack the controlled training comparisons needed to isolate its effects. There's no existing preference alignment dataset for cross-lingual conflicts. We therefore constructed a dataset that independently varies preference structure and language imbalance.

  • Strengthening the mechanistic explanation. Our results support the following explanation: within-language DPO leaves output-language probability unconstrained, while preference conflict sustains minority learning signals that counteract English dominance and help retain target-language features. The new experiments test this account at complementary levels. Adding a chosen-response NLL term restores an explicit language anchor, reducing average shared-preference fallback from 43.43% to 0.88% across four models. Training from the same SFT initialization confirms that conflict suppresses English-over-target reward gaps. Ablating conflict-associated language features further narrows BLOOMZ’s shared–conflict fallback gap from 54.0 to 1.4 points, with substantially smaller effects from control ablations.

  • Establishing broader robustness. With 50% shared / 50% conflicting preferences, conflict topics have lower fallback in all six model–assignment comparisons, including independent assignments across languages. The reduction also holds across all four seeds for Llama, BLOOMZ, and Qwen, and the fallback pattern persists under short free generation.

  • Together, these new experiments validate our explanation and establish the robustness of the finding and interpretation, directly addressing the reviewers’ core concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.