acceptodds
Withdrawn from ICLR 2027

Bug or Feature? Cross-Lingual Preference Distillation, Optimization Dynamics, and Capacity Contention in Multilingual LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multilingual Large Language Models project diverse languages into shared representations to transfer task capabilities, yet standard post-training alignment predominantly leverages English-centric preference data, imposing Western-centric defaults or triggering language confusion and English drift on non-Western queries. We hypothesize that an unaligned model's own greedy continuation modes provide effective negative supervision to purge degenerative artifacts and default priors without requiring additional human preference curation. To test this, we evaluate Cross-Lingual Preference Routing and Self-Contrastive Mode Suppression (CP-SMS), a post-training framework that pairs localized native completions () with the base model's unaligned greedy continuation modes () alongside general capability data. To benchmark performance without proprietary judge drift, we formalize a triangulated protocol anchored by multi-task language competence (Belebele), an open-weights Contrastive Perplexity Probability probe (CPP, ) measuring relative conditional likelihood under localized versus Western conditioning prefixes, and zero-shot frontier evaluation. Across five foundational model families and typologically diverse languages, our experiments reveal that supervised fine-tuning acts solely as a positive syntax booster, whereas self-contrastive preference tuning repels unaligned default modes, with sequential alignment (SFT_to_DPO) achieving the most favorable empirical balance between reading comprehension and preference win rates. Crucially, we uncover that in low-resource languages, DPO acts primarily as a generation-stabilization operator, purging repetitive loops, prompt echoing, and English drift, and in high-resource languages, it steers pragmatic completion stance away from English-dominant pre-training defaults. Furthermore, mixture scaling demonstrates that preference tuning acts as a latent steering operator rather than synthesizing absent representations as low-resource languages collapse into Negative Preference Transfer due to the compounding interaction of pre-training token deficits and extreme subword fragmentation. Finally, there is a significant trade-off between the subspace rank and the number of languages in post-training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Bias Distillation, Post-training, Preference Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xyYbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies multilingual preference tuning by pairing native Aya responses with greedy completions from the unaligned base model as rejected answers. It compares SFT, DPO, and sequential training, and examines data mixtures, language count, and LoRA rank. Evaluation combines Belebele, a persona-conditioned likelihood probe called CPP, and a generative judge. The authors interpret the results as evidence of generation stabilization, cultural preference steering, and capacity contention.

Critical strengths

The comparisons across training objectives, model families, and resource levels provide a useful empirical investigation of multilingual post-training. Evaluating reading comprehension separately from preference scores is particularly valuable: the results show that improvements in one do not necessarily imply improvements in the other. Using existing native responses and self-generated negatives is also a practical recipe worth examining.

Critical weaknesses
  1. The cultural-alignment conclusions are not sufficiently validated. CPP measures compatibility with selected collectivist versus individualist personas, rather than preferences established by the relevant language communities. More importantly, the claimed improvements are highly evaluator-dependent. Counting trained-versus-base changes in Table 10 gives positive probe win-rate changes in 20/25 cases under Aya, but only 9/25 under Gemma. This disagreement affects the direction of the measured gains. The coherent-subset analysis is useful, but without comparisons using fluent, length-matched rejected responses, it remains unclear how much the method learns cultural preferences beyond suppressing poor base-model continuations.
  2. The proposed mechanisms are stronger than the evidence supports. Section 2.1 correctly states that, when policy and reference coincide at initialization, both implicit rewards are zero and the DPO sigmoid weight is 0.5. Section 6 then explains larger gains through initial reward differences that supposedly drive this weight toward 1. These explanations are incompatible under the stated initialization. The capacity analysis has a separate confound: increasing the number of languages also reduces training examples per language from 2,000 to 166 and changes the language composition. This does not isolate interference from reduced data exposure, and does not establish the claimed fundamental representation limits.
Questions for authors
  1. Evaluation clarification: How do you reconcile the 20/25 positive changes under Aya with only 9/25 under Gemma in Table 10? What independent evidence establishes which measured changes correspond to better native-language preferences?
  2. Optimization clarification: Were the policy and reference identical at initialization in the cross-architecture experiments? If so, how can the initial sigmoid saturation described in Section 6 occur?
  3. Evidence for the capacity claim: Were CPP margins independently measured at every rank in Table 9? Its caption says the margins are measured at a canonical rank, whereas Section 4 claims that negative transfer persists across all ranks. Which results directly support that conclusion?
1 comment
AuthorsOpenReview ↗

Thank you for your actionable suggestions

We thank Reviewer xyYb for sharp technical questions—every anomaly you flagged led directly to bug fixes and new diagnostic experiments.

1. Mixture Table Duplicate CPP Values & Context Fix (Q3 — §4 & App. Table 12)

Auditing calibrated_r1_ratio.sh revealed GEN_FILE omitted ${RANK}, so r=16, 32, 64 reused r=8's generations in cpp_eval.py while capability_eval.py loaded each rank's true adapter. We also fixed 1,024-token Belebele truncation (-> 3,072 tokens) and unmerged Stage-1 bases in two-stage evaluations. Re-generating rank-isolated outputs on decontaminated aya_human_annotated prompts across all 20 (r, \gamma) cells (App. Table 12), γ=20%\gamma=20\% clean adherence (r=8..64r=8..64: 84.00%,87.33%,89.17%,88.67%84.00\%, 87.33\%, 89.17\%, 88.67\%) and Aya-32B WRcal\text{WR}_{\text{cal}} (55.00%,55.83%,54.00%,57.67%55.00\%, 55.83\%, 54.00\%, 57.67\%) vary cleanly by rank.

2. Reconciling Aya (20/2520/25) vs. Gemma (9/259/25), Near-Length-Matched Negatives & Coherent Subsets (W1 & Q1 — §4, §6, Tables 1, 6, 11, 14)

  • Zero-Thresholding Under Negative Priors: Cross-architecture cpp_eval.py scored I(ΔCPP>0)\mathbb{I}(\Delta_{\text{CPP}}>0) (WRraw\text{WR}_{\text{raw}}) uncentered. Under negative Gemma-4-31B-it priors ([−1.43,−0.04][-1.43, -0.04]), upward shifts staying <0\lt 0 count as losses (e.g., Granite-4.1-8B TR in App. Table 14 reaches CPP=0.553>0.500\text{CPP}=0.553>0.500 while WRraw\text{WR}_{\text{raw}} dips 7.1%→6.2%7.1\%\to 6.2\%; 7/257/25 mean-centered CPP>0.500\text{CPP}>0.500 cells in Table 14 and 14/2514/25 positive margin shifts in our N=100N=100 diagnostic log, e.g., Qwen2.5-7B TE −0.863→−0.157-0.863\to -0.157).
  • English Persona Coupling in Gemma: Even after median centering (Base=50%), Gemma-2-9B-it gives its highest scores (Base: 50.00%50.00\% CalWR / 61.33%61.33\% UncalWR at 95.0%/89.2%95.0\%\text{/}89.2\% ZH/AR drift; Fluent Self-SFT Neg: 44.83%/55.67%44.83\%\text{/}55.67\% at 85.0%/80.8%85.0\%\text{/}80.8\% drift) to English-drifting models and systematically penalizes zero-drift Staged SFT -> DPO down to 37.17%37.17\% CalWR (19.17%19.17\% ZH) as human Regional BLEnD peaks at 62.75%62.75\% (Table 6).
  • Near-Length-Matched Fluent Negatives & Coherent Subsets: Fluent, near-length-matched negatives (Self-SFT 1.5–2.6×1.5\text{–}2.6\times and Llama-3-Instruct 0.97–3.5×0.97\text{–}3.5\times char-length ratio vs. 9.7–78.7×9.7\text{–}78.7\times Base Greedy, App. Table 10) yield 58.58%–58.83%58.58\%\text{–}58.83\% BLEnD yet leave 58.67%58.67\% loops or 81%–85%81\%\text{–}85\% AR/ZH drift. Composed Staged SFT -> DPO reaches 56.11% Belebele, 69.21% CB-Tgt5, 62.75% BLEnD, and 85.17% CleanGen (5.50%5.50\% loops); on the n=26n=26 coherent-base Telugu prompts where Base already produced loop-free Telugu (App. Table 11), self-negative DPO reaches 80.77%80.77\% Aya-32B WRcal\text{WR}_{\text{cal}} vs. 57.69%57.69\% for Matched SFT (73.08% vs. 48.08% in TRL).

3. Initialization Identity & Post-Step Trajectory (W2 & Q2 — §2.1, §4.3, §6, Table 2)

Yes—in all cross-architecture runs (dpo.py, zero-init LoRA B0=0B_0=0, ref_model=None), πθ0≡πref\pi_{\theta_0} \equiv \pi_{\text{ref}} at t=0t=0, so r^θ0(x,yw)−r^θ0(x,yl)=0\hat{r}_{\theta_0}(x,y_w)-\hat{r}_{\theta_0}(x,y_l)=0 and σ(0)=0.5000\sigma(0)=0.5000. Profiling t=0→50t=0 \to 50 across 6 languages (Table 2) separates:

  • At t=0t=0: Length-normalized log-prob gap Δℓˉ0\Delta\bar{\ell}_0 (+2.354 YO, +2.133 TR, +2.030 AR, +1.527 ZH, +0.467 byte-diluted TE, -0.055 EN) and t=0t=0 DPO loss-gradient norm ∥∇θ0ℓDPO∥2\|\nabla_{\theta_0}\ell_{\text{DPO}}\|_2 (51.58 YO vs. 16.45 ZH, βσ(0)=0.05\beta\sigma(0)=0.05).
  • At t>0t > 0: Fragmented TE (297.8297.8 tok, NSRL=18.91\text{NSR}_L=18.91, unb. 50.0550.05) and YO (436.1436.1 tok, 21.5721.57, unb. 73.1473.14) have +50.7%–71.9%+50.7\%\text{–}71.9\% higher gradient noise than TR (98.898.8 tok, 12.5512.55, unb. 21.7721.77), slowing TE's t=50t=50 margin to +7.47+7.47 (σ=0.0551\sigma=0.0551) vs. +18.48+18.48 (σ=0.0134\sigma=0.0134) on TR.

4. Deconfounding RQ2 NN, Budget (MlangM_{\text{lang}}) & Cohorts (W2 — §5, Tables 3 & 4)

  • Fixed Cohorts (Table 3): On fixed {TE, YO} (N=6→12N=6 \to 12), r=32r=32 holds Acc_Bel (32.28%→32.33%32.28\% \to 32.33\%, +0.0005+0.0005) while r=64r=64 drops (33.22%→30.89%33.22\% \to 30.89\%, −0.0022-0.0022); r=32>r=64r=32 > r=64 at N=12N=12 also holds on fixed ZH (73.00%→67.57%73.00\% \to 67.57\%) and {ZH, AR, ES} (64.71%→63.08%64.71\% \to 63.08\%).
  • Fixed-Mlang=400M_{\text{lang}}=400, N=1N=1 vs. N=5N=5 (Table 4): Comparing 1-ep N=1N=1 (400400 samples; r=8→32r=8\to 32 lifts CleanGen 36.67%→56.67%36.67\%\to 56.67\% ZH, 30.00%→34.17%30.00\%\to 34.17\% TE) against 1-ep N=5N=5 (5×4005\times 400 at identical Mlang=400M_{\text{lang}}=400) shows positive cross-lingual stabilization transfer (CleanGen_ZH: 56.67%→84.17%56.67\% \to \mathbf{84.17\%}; CleanGen_TE: 34.17%→89.17%34.17\% \to \mathbf{89.17\%}; Belebele_TE: 37.78%→39.33%37.78\% \to \mathbf{39.33\%}), isolating Table 3's N=6→12N=6\to 12 drop to sample dilution (333→166333 \to 166) and 2.58×2.58\times ∥ΔW∥F\|\Delta W\|_F growth at r=64r=64.
Reviewer_SgKyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper combines general-purpose data with multilingual instruction-tuning data for SFT and DPO. The rejected completions used in DPO are outputs generated by the pretrained model for inputs in the Aya dataset. An LLM-as-judge metric (CPP) is also introduced, which compares conditional likelihood of model responses under different personas. The paper reports improvements in multilingual task performance and CPP for some languages and configs. Weaker evaluation results are attributed to multiple factors like tokenizer fragmentation, representation limitations, and adapter capacity.

Critical strengths

None

Critical weaknesses
  1. The paper introduces CPP as a measurement for localized cultural alignment. However, CPP measures the difference in a frozen evaluator's conditional likelihood of the same generated response under two contrasting persona instructions. This does not establish that the response actually expresses the intended cultural preference, nor that it is culturally appropriate. Table 5 shows per language personas in English with varying wording and cultural attributes. The resulting scores conditioning on the personas can therefore reflect other factors like evaluator sensitivity to lexical framing and predefined cultural stereotypes instead of the evaluated model's cultural performance. The paper also reports substantial disagreements between evaluator models and between English and translated conditioning prompts. These differences directly challenge the reliability of the proposed diagnostic. The other evaluation pillars do not solve this issue either. Accuracy on Belebele measures multiple-choice reading comprehension, not cultural alignment, and the Gemini-based evaluation uses a separate LLM judge without really establishing the cultural validity of its judgments. Furthermore, the evaluation instances used to report the CPP results are unclear. As a result, none of the metrics support the central claim of the paper on cultural alignment.
  2. The proposed training procedure is a mixture of existing components in literature including mixing general-purpose and multilingual training data, generating preference pairs using the base-model, and standard training with SFT followed by DPO. The paper characterizes the data mix as cross-lingual preference routing and describes its effects in terms of latent subspace steering and negative-gradient repulsion. However, no new routing mechanism, preference objective, or architectural modification is introduced. Much of the mathematical discussion describes existing DPO behavior rather than a new mechanism. More importantly, the experiments don’t properly isolate the contribution of the proposed rejected completion generation strategy. The paper assumes greedy base-model completions are appropriate rejected responses, but does not establish how frequently these completions are actually undesirable. There is no controlled evaluation of negative response quality, preference validity, or alternative negative-selection strategies. Comparisons with other established methods for multilingual or cultural preference alignment are also missing. Another important note is that the paper repeatedly makes causal claims about representation collapse, tokenizer fragmentation, gradient interference, and latent subspace capacity by relying on down stream task performance while making these claims requires measuring the relevant representations or optimization dynamics. Throughout the paper, unfavorable CPP results are explained by references to evaluator bias, generation artifacts, or tokenizer limitations. But the reported experiments can not distinguish failures of the proposed method or metric from these explanations. As a result, the experimental evidence is insufficient to support the claimed mechanisms and conclusions.
Questions for authors

My main questions and concerns are in the weaknesses above. They include the validity of the proposed cultural alignment metric, the contribution of the training procedure, and the lack of experimental evidence supporting the claimed mechanisms. These are fundamental concerns that would require substantial additional evidence to resolve.

1 comment
AuthorsOpenReview ↗

Thank you for your actionable suggestions

We thank Reviewer SgKy for a rigorous critique regarding metric validity, negative-sample characterization, controlled baselines, and direct optimization/representation telemetry. We ran targeted experiments for each point and recalibrated our claims accordingly.

1. Cultural Evaluation Beyond CPP: Human Judge-Free Benchmarks & Cross-Judge Diagnosis (W1 — §3, §6, Tables 1 & 6)

We accept your critique that persona-conditioned likelihood (CPP) is sensitive to lexical framing, persona stereotypes, and evaluator skew.

  • Human-Verified Judge-Free Cultural Benchmarks: We evaluated all models on deduplicated Regional BLEnD (N=1,200N=1,200 unique MCQs across CN, DZ, and nearest regional proxies AZ, AS-IN, NG) and CulturalBench-Easy (N=1,227N=1,227, 45 countries), plus stride-1 13-gram-decontaminated aya_human_annotated prompts (N=600N=600, 120120/lang, §3 & App. C). Without any LLM judge, Staged SFT -> DPO (62.75% BLEnD, 69.21% CB-Tgt5), On-Policy SFT -> DPO (62.58%, 65.70%), and RPO (61.42%, 65.09%) beat Base (58.67%, 63.99%), while CP-SMS and Staged SFT -> DPO lift open-ended target adherence from 7.17%7.17\% (Base) and 60.17%60.17\% (Matched SFT) to 87.33%87.33\% / 85.17%85.17\%.
  • Empirical Validation of Your CPP Critique: Comparing Gemma-2-9B-it vs. Aya-Expanse-32B under pluralistic persona ensembles confirmed that English persona prompts in Gemma reward English-drifting outputs (Base at 95.0%/89.2%95.0\%\text{/}89.2\% ZH/AR drift gets 50.00%50.00\% CalWR / 61.33%61.33\% UncalWR) while systematically penalizing zero-drift Staged SFT -> DPO down to 37.17%37.17\% CalWR (19.17%19.17\% ZH) despite peaking at 62.75%62.75\% on human BLEnD (Table 6). We therefore demote CPP to a bounded auxiliary probe and anchor primary claims on human-verified cultural QA (BLEnD, CulturalBench) and degeneration suppression.

2. Audit of Greedy Negatives, Controlled Negative Pools & Preference Objectives (W2 — §2.1, §4, Table 1, App. Table 10)

  • How Often Are Greedy Base Completions Undesirable? Across all 2,0002,000 target prompts in mix_20_percent.jsonl (App. Table 10), greedy base completions are undesirable in 92.8%–100.0%92.8\%\text{–}100.0\% of cases: clean fluent in-language rate is only 0.5% (ZH), 0.0% (AR), 4.2% (TR), 7.2% (TE), and 5.2% (YO), dominated by 4-gram loops (56.8%–88.8%56.8\%\text{–}88.8\%), script drift (94.8%94.8\% ZH, 94.0%94.0\% AR), and prompt echoing.
  • Controlled Negative-Selection & Objective Baselines: Fixing data, ywy_w, hyperparameters, and step budgets (Meta-Llama-3-8B, Table 1), we compared Matched SFT, DPO across four negative pools (Base Greedy τ=0.0\tau=0.0, Base Sampled τ=0.7\tau=0.7, Fluent Self-SFT τ=0.7\tau=0.7, Fluent Llama-3-8B-Instruct τ=0.0\tau=0.0), and RPO, SimPO, Staged SFT -> DPO, and On-Policy SFT -> DPO. Off-policy fluent negatives leave 58.67%58.67\% loops (Llama-3-Instruct) or 80.8%–85.0%80.8\%\text{–}85.0\% AR/ZH drift (Self-SFT), whereas on-manifold base greedy negatives extinguish base degenerate modes (74.67%–87.33%74.67\%\text{–}87.33\% clean), RPO and Staged SFT -> DPO reconcile Belebele (52.93%–56.11%52.93\%\text{–}56.11\%) and BLEnD (61.42%–62.75%61.42\%\text{–}62.75\%) with low loops (5.50%–11.00%5.50\%\text{–}11.00\%), and reference-free SimPO collapses (9.67%9.67\% clean, 43.5%43.5\% empty).

3. Direct Optimization, Gradient Alignment & Spectral Telemetry (W2 — §4, §5, §7, Tables 2, 4, 7, App. Table 15)

  • Cross-Lingual Gradient Cosines (t=10t=10, Table 4): Target languages share positive float64 gradient alignment (AR–TR +0.354+0.354, ZH–AR +0.346+0.346, ZH–TR +0.313+0.313, TR–TE +0.230+0.230), whereas EN aligns weakly negatively with ZH (−0.031-0.031), AR (−0.029-0.029), YO (−0.016-0.016), and TR (−0.011-0.011).
  • Subword Fragmentation, Zipfian Distortion & Variance (§4.3, §7, Tables 2, 7, 15): Mean chosen length expands 3.0–5.7×3.0\text{–}5.7\times from ZH/TR (76.0/98.876.0\text{/}98.8 tok) to TE/YO (297.8/436.1297.8\text{/}436.1 tok), raising plug-in NSRL\text{NSR}_L from 12.5512.55 (TR) and 17.1317.13 (ZH/AR) to 18.9118.91 (TE) and 21.5721.57 (YO; unbiased: 21.77→50.05–73.1421.77 \to 50.05\text{–}73.14). Across 12 aya_evaluation_suite reference pools, fertility flf_l linearly drives Tokenizer Zipfian DistortionΔαtok\Delta\alpha_{\text{tok}} (N=12N=12: r=0.9843,p=7.31×10−9r=0.9843, p=7.31\times 10^{-9}; no-TE N=11N=11: r=0.9358,p=2.30×10−5r=0.9358, p=2.30\times 10^{-5}), folding Telugu's flat word tail (C10word=7.5%C_{10}^{\text{word}}=7.5\%) into a 71.1%71.1\% byte head (C10BPE=0.711C_{10}^{\text{BPE}}=0.711).
  • 224-Matrix SVD Spectrum (Table 2): At N=12N=12 (1 ep), reff(ΔW)r_{\text{eff}}(\Delta W) scales linearly (7.27→14.50→29.14→58.207.27 \to 14.50 \to 29.14 \to 58.20) while ∥ΔW∥F\|\Delta W\|_F grows 2.58×2.58\times (0.103→0.2650.103 \to 0.265), and causal overclaims ("routing") are removed.
Reviewer_vozJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CP-SMS, which uses base model generations as negative samples, pairs them with existing native-language reference responses, and performs DPO training to improve multilingual responses while reducing the need for new preference annotations. The experiments compare training sequences, data mixture ratios, LoRA ranks, and multiple model families, and evaluate using reading comprehension, CPP diagnostic probes, and model judges. The authors argue that CP-SMS mainly reduces repetition and English drift in low-resource languages, while in high-resource languages it primarily steers pragmatic stance.

Critical strengths
  1. The proposed method uses the base model's own outputs paired with existing native-language reference responses, requiring no additional human preference annotations, making it convenient to train on existing instruction data.
  2. The experiments cover comparisons of multiple training sequences, languages, and model families, and report both reading comprehension and preference metrics, which helps observe training gains and capability trade-offs.
Critical weaknesses
  1. Overall, the practical value of this paper or the core insights it aims to convey are unclear; it seems more like a report of a limited methodological attempt. The positioning of LoRA in Contribution 3 is rather vague. The paper treats the analysis of LoRA capacity as one of its contributions, yet whether LoRA is used is itself not directly related to the task of low-resource languages. This was originally a choice of implementation setting by the authors, but the existing experiments mainly show performance changes under different ranks and different numbers of languages, without establishing a relationship between capacity and stance adjustment, so it is a stretch to count it as an independent mechanistic contribution.
  2. The writing of this paper is not good, with many puzzling overclaims. For example, the first sentence of the Abstract jumps through four layers at once: shared representations → English preference data → Western cultural bias → language confusion. However, these are not entirely causal relationships. Another example is the introduction of many terms to describe things, such as "generation-stabilization operator" and "capacity saturation," along with many strong verbs such as uncover, establish, and prove, which make many of the paper's findings appear to be universal laws, when in fact there are logical gaps underlying them.
  3. Problems on the experimental side: Table 1 compares SFT and DPO while simultaneously changing the training objective and whether negative samples are used, so it cannot explain why the gains come from CP-SMS. Positive samples, data ratios, and training budgets should be fixed, and negative samples from base greedy generation, sampling generation, and instruction model generation should be compared. The appendix discusses the latter but does not provide the experiment.
Questions for authors
  1. How does CPP distinguish between "more inclined toward collectivism" and "more consistent with local culture"? Why can the former represent an improvement in cultural alignment?
  2. Cross-model validation is something that should be done anyway to verify a pattern; why do the authors treat it as one of their independent contributions?
3 comments

We thank Reviewer vozJ for highlighting the need for controlled negative-source ablations, clearer LoRA rank positioning, grounded cultural evaluation beyond collectivism templates, and calibrated writing.

1. Controlled SFT vs. DPO & Negative-Source Ablation (W3 — Revised §4, Table 1 & App. Table 10)

You rightly noted that our original table changed both the objective and negative usage simultaneously. Using a single unified bfloat16 trainer (Meta-Llama-3-8B, fixed 8,000 EN + 2,000 Target prompts and positive references ywy_w, fixed r=16r=16, lr=5e-5, batch=64, seed=42), we compared Matched SFT (positive-only NLL), Base Greedy (τ=0.0\tau=0.0), Base Sampled (τ=0.7,p=0.9\tau=0.7, p=0.9), Fluent Self-SFT (τ=0.7\tau=0.7), Fluent Llama-3-8B-Instruct (τ=0.0\tau=0.0), RPO (LDPO+λNLLLNLL\mathcal{L}_{\text{DPO}}+\lambda_{\text{NLL}}\mathcal{L}_{\text{NLL}}), and two-stage sequential pipelines:

  • Why Greedy Base Negatives Work: Auditing the 2,0002,000 target prompts (Appendix Table 10) shows base greedy completions contain 56.8%–88.8%56.8\%\text{–}88.8\% repetition loops and up to 94.8%94.8\% script drift (only 3.4% macro clean in-language responses vs. 72.3%72.3\% for Llama-3-8B-Instruct and 80.6%80.6\% for Self-SFT).
  • Empirical Trade-Off Across Negative Sources: At 1 epoch (157 steps), positive-only Matched SFT grounds reading comprehension (Belebele 56.51%56.51\%) and cultural MCQA (Regional BLEnD 61.83%61.83\%, CulturalBench 69.19%69.19\%), while on-manifold Base Greedy (58.00%58.00\%) and Base Sampled (58.83%58.83\%) suppress degeneration. Off-policy fluent negatives (Llama-3-Instruct and Self-SFT) fail to penalize active base degeneracies—leaving 58.67%58.67\% repetition loops (Llama-3-Instruct, 33.16%33.16\% clean) or 80.8%–85.0%80.8\%\text{–}85.0\% English drift on AR/ZH (Self-SFT, 25.50%25.50\% clean). Our 979-slice Zipf–Mandelbrot and Dirichlet log-odds audit (§7, Table 7, App. Table 16) pinpoints why: Fluent Instruct negatives are already Zipf-matched to human text (DJSword=0.0579D_{\text{JS}}^{\text{word}}=0.0579 bits), so sequence-level DPO penalizes shared target-language declarative connectives (z=+7.32 to +12.75z=+7.32\text{ to }+12.75) and drives outputs into interrogative prompt-echoing loops (α^ZMBPE=3.370\hat{\alpha}_{\text{ZM}}^{\text{BPE}}=3.370). Combining SFT + DPO—via Staged SFT -> DPO (Belebele 56.11%56.11\%, CB-Tgt5 69.21%69.21\%, BLEnD 62.75%62.75\%, CleanGen 85.17%85.17\%, Rep 5.50%5.50\%) or RPO (Belebele 52.93%52.93\%, BLEnD 61.42%61.42\%, CleanGen 66.33%66.33\%, V1/V=0.694V_1/V=0.694)—reconciles comprehension, local cultural accuracy, and open-ended fluency.

2. Distinguishing Local Cultural Knowledge from Collectivist Stereotypes (Q1 — Revised §3 & Table 1)

We agree completely that single-axis collectivism prompts cannot establish authentic local cultural alignment. We therefore added two human-authored, stereotype-free cultural benchmarks: deduplicated Regional BLEnD (N=1,200N=1,200 unique stems across China CN, Algeria DZ, and nearest regional proxy tracks Azerbaijan AZ, Assam AS-IN, and Northern Nigeria NG, stripped of JSON formatting instructions) and CulturalBench-Easy (N=1,227N=1,227 across 45 countries). Staged SFT -> DPO and On-Policy SFT -> DPO improve Regional BLEnD from 58.67%58.67\% (Base) →62.75%\to 62.75\% / 62.58%62.58\% and CulturalBench Target-5 from 63.99%→69.21%63.99\% \to 69.21\% / 65.70%65.70\% (+3.33 pp+3.33\text{ pp} on Northern Nigerian cultural QA: 52.92%→56.25%52.92\% \to 56.25\%). We also replaced single-persona CPP with a pluralistic civic/solidarity/heritage ensemble and demoted CPP to an auxiliary diagnostic.

3. Repositioning LoRA Rank & Cross-Model Validation (W1 & Q2 — Revised §1, §5, Table 2)

We agree that LoRA rank scaling and cross-model validation are empirical characterization studies of practical PEFT deployment, not standalone algorithmic contributions. In the revised manuscript:

  • We consolidate Contributions 2 and 3 into empirical characterizations of multilingual PEFT trade-offs and cross-family generalizability rather than independent mechanistic laws.
  • Direct SVD spectrum measurements across all 224 LoRA matrices of ΔW=αLoRArBA\Delta W = \frac{\alpha_{\text{LoRA}}}{r}BA (Table 2) show that at N=12N=12 (166166 samples/lang, 1 ep), Shannon effective rank reffr_{\text{eff}} scales linearly (7.27→14.50→29.14→58.207.27 \to 14.50 \to 29.14 \to 58.20, ≈91%\approx 91\% utilization) while ∥ΔW∥F\|\Delta W\|_F grows 2.58×2.58\times (0.103→0.2650.103 \to 0.265; 2-ep: 0.178→0.447,2.50×0.178 \to 0.447, 2.50\times), explaining variance overfitting at r=64r=64 vs. r=32r=32 on fixed cohorts.

4. Eliminating Overclaims & Causal Jumps (W2 — Revised Title, Abstract, §1, §2)

We rewrote the Title and Abstract opening to separate empirical observations from hypotheses, replaced terms like "generation-stabilization operator", "routing", and "universal laws" with precise empirical descriptions, and scoped all conclusions strictly to tested 4B–8B models and budgets.

Thank you for the response. The new ablations address my concern about CP-SMS’s distinct benefits, while the revised contribution framing and wording improve the presentation. However, the connections among negative-sample construction, cultural knowledge evaluation, and adapter capacity analysis remain unclear. These components do not yet support a coherent central conclusion. Improved cultural knowledge accuracy also does not necessarily establish alignment with local cultural preferences. I hope the revised manuscript clarifies how these analyses jointly support its main claims and distinguishes empirical observations from mechanistic explanations.

We revised the paper (Abstract, §1, §§4–8, Limitations) to foreground our motivating constraint, make explicit how each stage motivates the next toward one conclusion, clarify cultural knowledge vs. preference scope, and separate empirical observations from mechanistic explanations.

1. Core Motivation & How the Findings Tie Together (Abstract, §1, §§4–8)

Our study is motivated by a practical post-training constraint: adapting an unaligned multilingual base model across diverse scripts without human preference labels or teacher distillation, using only native instruction-response pairs (aya_dataset) alongside general capability data inside one shared LoRA adapter. Under this constraint, negative construction, cultural knowledge grounding, and shared adapter capacity form one causal chain:

  1. Supervision Division (§4.1–§4.2 →\to §7.2): Regional factual knowledge cannot surface in user interactions if free-form generation collapses into loops or drifts into English. Positive fitting (LNLL(yw)\mathcal{L}_{\text{NLL}}(y_w)) grounds comprehension (Belebele 56.51%56.51\%) and factual regional QA (BLEnD 61.83%61.83\%, CulturalBench 69.19%69.19\%) yet leaves generation prone to loops (Matched SFT: 60.17%60.17\% clean). Off-policy fluent negatives (Llama-3-Instruct, Self-SFT) fail to clear base attractors (25.5%–33.2%25.5\%\text{–}33.2\% clean) because Zipf-matched fluent negatives (DJSword=0.0579D_{\text{JS}}^{\text{word}}=0.0579) cause sequence-level negative gradients to suppress shared target declarative connectives (§7.2). Deterministic base self-negatives (CP-SMS) target active greedy loops and drift directly, so combining positive anchoring with self-negative repulsion (Staged SFT -> DPO or RPO) expresses grounded comprehension and regional knowledge (Belebele 56.11%56.11\%, BLEnD 62.75%62.75\%) in fluent native-script text (85.17%85.17\% clean, DJSword=0.1081D_{\text{JS}}^{\text{word}}=0.1081).
  2. From Sequence Updates to Shared Adapter Capacity (§4.3 →\to §5 →\to §7.3): Because CP-SMS applies sequence-level updates across scripts inside one shared LoRA subspace, subword segmentation governs multi-language capacity scaling. Byte-fallback tokenization (TE 13.4513.45 subwords/word, YO diacritics) folds rare inflected words into shared UTF-8 byte heads (§7.3, r12=0.984r_{12}=0.984) and inflates gradient variance (NSRL\text{NSR}_L, Table 2). Scaling across NN languages (§5) tests whether shared updates interfere or transfer: at fixed per-language budgets (Mlang=400M_{\text{lang}}=400, Table 4), self-negative updates yield constructive cross-lingual stabilization transfer (CleanGen: +27.50 pp+27.50\text{ pp} ZH, +55.00 pp+55.00\text{ pp} TE; target cosines up to +0.35+0.35) by jointly suppressing English fallback and loops; under diluted wide mixtures (N=12,Mlang=166N=12, M_{\text{lang}}=166, Table 3), unconstrained r=64r=64 updates (2.58×2.58\times ∥ΔW∥F\|\Delta W\|_F at ≈91%\approx 91\% rank utilization, Table 2) absorb byte-fragmentation noise, explaining why intermediate rank (r=32r=32) regularizes wide mixtures.
  3. Central Conclusion (§1, §8): Zero-annotation multilingual adaptation succeeds when positive likelihood fitting (grounding comprehension and factual regional knowledge) is paired with on-manifold self-negative repulsion (clearing active degeneracies without depleting shared target syntax) inside a variance-regularized shared adapter subspace.

2. Factual Regional Knowledge vs. Local Cultural Preferences (§1, §6.2, §8, Limitations)

In §6.2, §8, and Limitations, we tie cultural claims directly to supervision source. Pairing general native instruction completions (ywy_w) with base degeneracies (yly_l)—rather than community preference rankings over competing norms—means gains on Regional BLEnD and CulturalBench-Easy establish propositional regional knowledge (everyday local facts and customs) grounded by LNLL\mathcal{L}_{\text{NLL}}, whereas aligning with subjective or pluralistic local cultural preferences requires community preference supervision and native human evaluation. Cross-evaluator auditing (Table 6) further shows that English persona prefixes in CPP lexically couple to English drift in English-centric evaluators, restricting CPP to an auxiliary diagnostic on verified in-script text.

3. Empirical Observations vs. Mechanistic Explanations (§§4–7)

Across §4.1–§4.3, §5.1–§5.2, §6.1–§6.2, and §7.2–§7.3, each subsection separates Empirical observation. (benchmark trade-offs, fixed-budget N=1→5N=1\to 5 transfer, cross-family shifts, evaluator inversions, and 12-language Zipfian curves) from Mechanistic explanation. (πbase\pi_{\text{base}} manifold density, LoRA gradient NSRL\text{NSR}_L and float64 cosines, 224-matrix SVD ∥ΔW∥F\|\Delta W\|_F scaling, English persona coupling, and Dirichlet log-odds connective suppression).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the Area Chair and reviewers (vozJ, SgKy, xyYb) for their rigorous critiques. Every concern fell into testable questions, addressed in our revised 10-page paper via 16 newly trained bfloat16 checkpoints, two human-authored, judge-free cultural benchmarks (Regional BLEnD & CulturalBench-Easy), full-context (3,072-token) and rank-isolated re-evaluations, 6-language optimization, 224-matrix SVD, and LoRA gradient telemetry, and a 979-slice dual-granularity Zipf–Mandelbrot and Dirichlet log-odds audit across 12 languages for Cross-Lingual Preference with Self-Negative Mode Suppression (CP-SMS):

1. Controlled Negative-Source & Objective Isolation (vozJ W3, SgKy W2, xyYb W1 — Revised §4, Table 1, Fig. 1a)

Holding Meta-Llama-3-8B (bfloat16), prompts xx, native chosen references ywy_w, mixture ratio (γ=0.20\gamma=0.20: 8,000 EN+2,000 Target8,000\text{ EN} + 2,000\text{ Target}), optimizer, LoRA rank (r=16r=16), batch size (6464), seed=42, and step budget strictly fixed in our unified trainer (512/384 prompt/response caps), we compared Matched SFT, DPO across four negative pools (Base Greedy τ=0.0\tau=0.0, Base Sampled τ=0.7\tau=0.7, Fluent Self-SFT τ=0.7\tau=0.7, Fluent Llama-3-8B-Instruct τ=0.0\tau=0.0), RPO (LDPO+λNLLLNLL\mathcal{L}_{\text{DPO}}+\lambda_{\text{NLL}}\mathcal{L}_{\text{NLL}}), SimPO, and On-Policy SFT -> DPO, alongside TRL (max_len=2048) CP-SMS Direct DPO and composed two-stage pipelines (Staged SFT -> DPO and Reverse DPO -> SFT, merging 2-ep Stage-1 + 2-ep Stage-2 at lr=2e-5):

Controlled Regime (r=16,γ=0.20r=16, \gamma=0.20)Belebele MacroCulturalBench (All-45 / Tgt-5)Human Reg. BLEnD (Macro / CN / NG)Open-Ended Clean Adherence4-gram RepLoop Rate
Unaligned Base45.47%45.47\%64.14%64.14\% / 63.99%63.99\%58.67%58.67\% (70.470.4 / 52.952.9)7.17%7.17\%66.83%66.83\%
Matched SFT (1 ep / 2 ep)56.51%56.51\% / 55.80%55.80\%69.19%69.19\%/64.61%64.61\% ; 69.19%69.19\%/65.61%65.61\%61.83%61.83\% / 61.58%61.58\%57.50%57.50\% / 60.17%60.17\%10.83%10.83\% / 15.50%15.50\%
Unified Greedy DPO (1 ep / 2 ep)48.96%48.96\% / 48.91%48.91\%66.42%66.42\% / 65.03%65.03\% ; 66.91%66.91\% / 64.99%64.99\%59.75%59.75\% / 59.25%59.25\%58.00%58.00\% / 74.67%74.67\%33.17%33.17\% / 20.00%20.00\%
CP-SMS Direct DPO (TRL, 2 ep)47.87%47.87\%65.04%65.04\% / 64.74%64.74\%59.50%59.50\% (71.371.3 / 51.351.3)87.33%87.33\%7.50%7.50\%
DPO: Base Sampled τ=0.7\tau=0.7 (1 / 2 ep)48.07%48.07\% / 47.87%47.87\%66.26%66.26\% / 63.72%63.72\% ; 66.83%66.83\% / 64.33%64.33\%59.83%59.83\% / 60.25%60.25\%58.83%58.83\% / 76.00%76.00\%33.83%33.83\% / 17.33%17.33\%
DPO: Fluent Self-SFT τ=0.7\tau=0.7 (1 ep)45.87%45.87\%65.69%65.69\% / 62.91%62.91\%58.83%58.83\% (71.371.3 / 52.152.1)25.50%25.50\%35.00%35.00\%
DPO: Fluent Llama-3-Instruct (1 ep)46.11%46.11\%65.61%65.61\% / 63.43%63.43\%58.58%58.58\% (69.269.2 / 53.853.8)33.16%33.16\%58.67%58.67\%
RPO (LDPO+LNLL\mathcal{L}_{\text{DPO}}+\mathcal{L}_{\text{NLL}}, 1 / 2 ep)52.51%52.51\% / 52.93%52.93\%66.59%66.59\% / 65.04%65.04\% ; 67.32%67.32\% / 65.09%65.09\%61.17%61.17\% / 61.42%61.42\%62.00%62.00\% / 66.33%66.33\%6.67%6.67\% / 11.00%11.00\%
SimPO (Ref-Free, Greedy, 2 ep)52.33%52.33\%66.67%66.67\% / 64.25%64.25\%59.58%59.58\% (75.075.0 / 52.152.1)9.67%9.67\% (43.5%43.5\% empty)10.17%10.17\%
Staged SFT -> DPO (Merged + 2-ep DPO)56.11%56.11\%69.19%69.19\% / 69.21%69.21\%62.75%62.75\% (72.172.1 / 56.356.3)85.17%85.17\%5.50%5.50\%
On-Policy SFT -> DPO (πref=πSFT\pi_{\text{ref}}=\pi_{\text{SFT}}, 1 ep)55.09%55.09\%68.54%68.54\% / 65.70%65.70\%62.58%62.58\% (72.172.1 / 54.254.2)63.83%63.83\%21.83%21.83\%
Reverse DPO -> SFT (Merged + 2-ep SFT)52.64%52.64\%67.16%67.16\% / 65.55%65.55\%61.50%61.50\% (70.870.8 / 53.353.3)65.83%65.83\%17.33%17.33\%
  • Core Takeaway: Positive NLL fitting (Matched SFT) grounds reading comprehension (56.51%56.51\%) and factual cultural MCQA (Regional BLEnD 61.83%61.83\%, CulturalBench 69.19%69.19\%) yet leaves open-ended generation prone to loops (57.5%–60.2%57.5\%\text{–}60.2\% clean adherence, 15.8%–25.0%15.8\%\text{–}25.0\% on Yoruba; Legacy Pure SFT 30.0%30.0\%). Off-policy fluent negatives (Llama-3-Instruct / Self-SFT) occupy low-density regions under πbase\pi_{\text{base}} and fail to penalize arg⁡max⁡yπbase(y∣x)\arg\max_y \pi_{\text{base}}(y\mid x), leaving 58.67%58.67\% repetition loops or 80.8%–85.0%80.8\%\text{–}85.0\% English drift on AR/ZH (25.50%25.50\% clean). Combining positive anchoring with on-manifold negative repulsion—via Staged SFT -> DPO (Belebele 56.11%56.11\%, CB-Tgt5 69.21%69.21\%, BLEnD 62.75%62.75\%, CleanGen 85.17%85.17\%, Rep 5.50%5.50\%), On-Policy SFT -> DPO, or RPO—reconciles comprehension, local cultural knowledge, and open-ended fluency, whereas inverting stage order (Reverse DPO -> SFT) overwrites contrastive boundaries (CleanGen −19.34 pp-19.34\text{ pp}, 3.15×3.15\times more loops).

2. Human-Verified Cultural Benchmarks & Reframing CPP (vozJ Q1, SgKy W1, xyYb W1/Q1 — §3, §6, Table 6, App. Fig. 5)

We evaluated all models on human-authored, judge-free cultural benchmarks: deduplicated Regional BLEnD (N=1,200N=1,200, 240240 unique stems/region across CN, DZ, and nearest regional proxies AZ, AS-IN, NG, stripped of JSON suffixes) and CulturalBench-Easy (N=1,227N=1,227, 45 countries). Anchored and two-stage models beat Base by +4.08 pp on Regional BLEnD (58.67%→62.75%58.67\% \to 62.75\%) and +5.22 pp on CulturalBench Target-5 (63.99%→69.21%63.99\% \to 69.21\%). Cross-judge auditing uncovered both uncalibrated zero-thresholding compression under negative Gemma priors and English persona-coupling bias in Gemma-2-9B-it, which gives its highest scores (50.00%50.00\% CalWR / 61.33%61.33\% UncalWR and 44.83%44.83\% / 55.67%55.67\%) to high-English-drift models (81%–95%81\%\text{–}95\% drift) while systematically penalizing zero-drift Staged SFT -> DPO down to 37.17%37.17\% CalWR (19.17%19.17\% ZH) as human BLEnD peaks at 62.75%62.75\%. We demote CPP to a bounded auxiliary probe and anchor primary claims on human-verified BLEnD, CulturalBench, and target adherence.

3. Mixture Surface Bug Fix (xyYb Q3) & Fixed-Budget N=1N=1 Controls (xyYb W2, vozJ W1 — §5, Tables 3–4, App. Table 12)

We traced the duplicated CPP columns in the original Appendix mixture table to a shell variable omission (GEN_FILE lacked ${RANK}), re-generated rank-isolated outputs on stride-1 decontaminated aya_human_annotated prompts across all 20 (r, \gamma) cells (App. Table 12), and trained 1-ep fixed-Mlang=400M_{\text{lang}}=400 single-language controls (N1_{ZH,TE}_400_{r8,r32}). Comparing 1-ep N=1N=1 (400400 samples) vs. 1-ep N=5N=5 (5×4005\times 400 samples at identical Mlang=400M_{\text{lang}}=400, Table 4) reveals positive cross-lingual stabilization transfer (CleanGen_ZH: 56.67%→84.17%56.67\% \to 84.17\%, +27.50 pp; CleanGen_TE: 34.17%→89.17%34.17\% \to 89.17\%, +55.00 pp; Belebele_TE: 37.78%→39.33%37.78\% \to 39.33\%), isolating Table 3's N=6→12N=6\to 12 drop to sample dilution (Mlang:333→166M_{\text{lang}}: 333 \to 166) and 2.58×2.58\times ∥ΔW∥F\|\Delta W\|_F growth at r=64r=64.

4. Direct Optimization, 224-Matrix SVD & Gradient Telemetry (xyYb Q2, SgKy W2, vozJ W1/W2 — §4–§6, Tables 2 & 4)

We measured chat-template-aligned t=0→50t=0 \to 50 mixed-stream DPO trajectories across all 6 languages (EN, ZH, AR, TR, TE, YO), 224-matrix 1-ep/2-ep SVD Shannon effective rank reff(ΔW)r_{\text{eff}}(\Delta W), float64 cross-lingual gradient cosines, and plug-in/unbiased NSRL\text{NSR}_L, replacing overstated terms ("universal laws", "generation-stabilization operator", "routing") with calibrated empirical statements.

5. Corpus-Scale Zipfian Scaling Dynamics & Lexical Tail Mining Across 979 Slices (SgKy W1/W2, vozJ W3, xyYb W1 — §7, Fig. 4, Table 7, App. Tables 15–16)

We mined 979 generation and reference slices across 12 languages via fixed-budget Zipf–Mandelbrot MLE (P(k)∝(k+b)−αP(k) \propto (k+b)^{-\alpha}, Kref=max⁡(4,096,V)K_{\text{ref}}=\max(4{,}096,V)), Heaps' growth (V(n)=KnβHeapsV(n)=Kn^{\beta_{\text{Heaps}}}), and Monroe Dirichlet log-odds:

  • Why Fluent Negatives Trigger Loops: Fluent Instruct negatives already match human Zipf curves (DJSword=0.0579D_{\text{JS}}^{\text{word}} = 0.0579 bits, α^word=0.812\hat{\alpha}_{\text{word}}=0.812), so sequence-level DPO penalizes shared target declarative connectives (ZH he/de z=+8.22/+7.32z=+8.22/+7.32, TR ve/bu z=+12.75/+7.76z=+12.75/+7.76, YO ni/pe z=+8.38/+8.05z=+8.38/+8.05) and drives interrogative prompt-echoing loops (α^ZMBPE=3.370\hat{\alpha}_{\text{ZM}}^{\text{BPE}}=3.370, DJSword=0.2490D_{\text{JS}}^{\text{word}}=0.2490). Staged SFT -> DPO (DJSBPE=0.0286D_{\text{JS}}^{\text{BPE}}=\mathbf{0.0286}, DJSword=0.1081D_{\text{JS}}^{\text{word}}=\mathbf{0.1081}) and CP-SMS (0.0328/0.13090.0328 / 0.1309, βHeaps=0.872\beta_{\text{Heaps}}=\mathbf{0.872}) best match human rank distributions, RPO preserves the richest non-collapsed hapax tail (V1/V=0.694V_1/V=\mathbf{0.694}, 96.5%96.5\% tail native purity), and SimPO suffers head collapse (α^word=1.628\hat{\alpha}_{\text{word}}=1.628, C10BPE=0.757C_{10}^{\text{BPE}}=0.757, DJSword=0.6513D_{\text{JS}}^{\text{word}}=0.6513, 43.5%43.5\% empty).
  • 12-Language Tokenizer Zipfian Distortion: Across 12 aya_evaluation_suite reference pools, subword fertility flf_l drives Δαtok=α^BPE−α^word\Delta\alpha_{\text{tok}} = \hat{\alpha}_{\text{BPE}} - \hat{\alpha}_{\text{word}} (N=12N=12: r=0.9843,p=7.31×10−9r = 0.9843, p = 7.31\times 10^{-9}, ρ=0.9510\rho = 0.9510; no-TE N=11N=11: r=0.9358,p=2.30×10−5,ρ=0.9364r = 0.9358, p = 2.30\times 10^{-5}, \rho = 0.9364): Telugu byte-fallback (fTE=13.45f_{\text{TE}}=13.45) folds a flat word tail (α^word=0.649,C10word=7.5%\hat{\alpha}_{\text{word}}=0.649, C_{10}^{\text{word}}=7.5\%) into a 71.1%71.1\% UTF-8 byte head (C10BPE=0.711C_{10}^{\text{BPE}}=0.711).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.