- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies multilingual preference tuning by pairing native Aya responses with greedy completions from the unaligned base model as rejected answers. It compares SFT, DPO, and sequential training, and examines data mixtures, language count, and LoRA rank. Evaluation combines Belebele, a persona-conditioned likelihood probe called CPP, and a generative judge. The authors interpret the results as evidence of generation stabilization, cultural preference steering, and capacity contention.
Critical strengths
The comparisons across training objectives, model families, and resource levels provide a useful empirical investigation of multilingual post-training. Evaluating reading comprehension separately from preference scores is particularly valuable: the results show that improvements in one do not necessarily imply improvements in the other. Using existing native responses and self-generated negatives is also a practical recipe worth examining.
Critical weaknesses
- The cultural-alignment conclusions are not sufficiently validated. CPP measures compatibility with selected collectivist versus individualist personas, rather than preferences established by the relevant language communities. More importantly, the claimed improvements are highly evaluator-dependent. Counting trained-versus-base changes in Table 10 gives positive probe win-rate changes in 20/25 cases under Aya, but only 9/25 under Gemma. This disagreement affects the direction of the measured gains. The coherent-subset analysis is useful, but without comparisons using fluent, length-matched rejected responses, it remains unclear how much the method learns cultural preferences beyond suppressing poor base-model continuations.
- The proposed mechanisms are stronger than the evidence supports. Section 2.1 correctly states that, when policy and reference coincide at initialization, both implicit rewards are zero and the DPO sigmoid weight is 0.5. Section 6 then explains larger gains through initial reward differences that supposedly drive this weight toward 1. These explanations are incompatible under the stated initialization. The capacity analysis has a separate confound: increasing the number of languages also reduces training examples per language from 2,000 to 166 and changes the language composition. This does not isolate interference from reduced data exposure, and does not establish the claimed fundamental representation limits.
Questions for authors
- Evaluation clarification: How do you reconcile the 20/25 positive changes under Aya with only 9/25 under Gemma in Table 10? What independent evidence establishes which measured changes correspond to better native-language preferences?
- Optimization clarification: Were the policy and reference identical at initialization in the cross-architecture experiments? If so, how can the initial sigmoid saturation described in Section 6 occur?
- Evidence for the capacity claim: Were CPP margins independently measured at every rank in Table 9? Its caption says the margins are measured at a canonical rank, whereas Section 4 claims that negative transfer persists across all ranks. Which results directly support that conclusion?
Thank you for your actionable suggestions
We thank Reviewer
xyYbfor sharp technical questions—every anomaly you flagged led directly to bug fixes and new diagnostic experiments.1. Mixture Table Duplicate CPP Values & Context Fix (Q3 — §4 & App. Table 12)
Auditing
calibrated_r1_ratio.shrevealedGEN_FILEomitted${RANK}, sor=16, 32, 64reusedr=8's generations incpp_eval.pywhilecapability_eval.pyloaded each rank's true adapter. We also fixed1,024-tokenBelebeletruncation (-> 3,072tokens) and unmerged Stage-1 bases in two-stage evaluations. Re-generating rank-isolated outputs on decontaminatedaya_human_annotatedprompts across all 20(r, \gamma)cells (App. Table 12), γ=20% clean adherence (r=8..64: 84.00%,87.33%,89.17%,88.67%) andAya-32BWRcal (55.00%,55.83%,54.00%,57.67%) vary cleanly by rank.2. Reconciling
Aya(20/25) vs.Gemma(9/25), Near-Length-Matched Negatives & Coherent Subsets (W1 & Q1 — §4, §6, Tables 1, 6, 11, 14)cpp_eval.pyscored I(ΔCPP>0) (WRraw) uncentered. Under negativeGemma-4-31B-itpriors ([−1.43,−0.04]), upward shifts staying <0 count as losses (e.g.,Granite-4.1-8B TRin App. Table 14 reaches CPP=0.553>0.500 while WRraw dips 7.1%→6.2%; 7/25 mean-centered CPP>0.500 cells in Table 14 and 14/25 positive margin shifts in our N=100 diagnostic log, e.g.,Qwen2.5-7B TE−0.863→−0.157).Gemma: Even after median centering (Base=50%),Gemma-2-9B-itgives its highest scores (Base: 50.00% CalWR / 61.33% UncalWR at 95.0%/89.2%ZH/ARdrift;Fluent Self-SFT Neg: 44.83%/55.67% at 85.0%/80.8% drift) to English-drifting models and systematically penalizes zero-driftStaged SFT -> DPOdown to 37.17% CalWR (19.17%ZH) as humanRegional BLEnDpeaks at 62.75% (Table 6).Self-SFT1.5–2.6× andLlama-3-Instruct0.97–3.5× char-length ratio vs. 9.7–78.7×Base Greedy, App. Table 10) yield 58.58%–58.83%BLEnDyet leave 58.67% loops or 81%–85%AR/ZHdrift. ComposedStaged SFT -> DPOreaches56.11%Belebele,69.21%CB-Tgt5,62.75%BLEnD, and85.17%CleanGen(5.50% loops); on the n=26 coherent-base Telugu prompts whereBasealready produced loop-free Telugu (App. Table 11), self-negative DPO reaches 80.77%Aya-32BWRcal vs. 57.69% forMatched SFT(73.08%vs.48.08%inTRL).3. Initialization Identity & Post-Step Trajectory (W2 & Q2 — §2.1, §4.3, §6, Table 2)
Yes—in all cross-architecture runs (
dpo.py, zero-init LoRA B0=0,ref_model=None), πθ0≡πref at t=0, so r^θ0(x,yw)−r^θ0(x,yl)=0 and σ(0)=0.5000. Profiling t=0→50 across 6 languages (Table 2) separates:+2.354YO,+2.133TR,+2.030AR,+1.527ZH,+0.467byte-dilutedTE,-0.055EN) and t=0 DPO loss-gradient norm ∥∇θ0ℓDPO∥2 (51.58YOvs.16.45ZH, βσ(0)=0.05).TE(297.8 tok, NSRL=18.91, unb. 50.05) andYO(436.1 tok, 21.57, unb. 73.14) have +50.7%–71.9% higher gradient noise thanTR(98.8 tok, 12.55, unb. 21.77), slowingTE's t=50 margin to +7.47 (σ=0.0551) vs. +18.48 (σ=0.0134) onTR.4. Deconfounding
RQ2N, Budget (Mlang) & Cohorts (W2 — §5, Tables 3 & 4){TE, YO}(N=6→12), r=32 holdsAcc_Bel(32.28%→32.33%, +0.0005) while r=64 drops (33.22%→30.89%, −0.0022); r=32>r=64 at N=12 also holds on fixedZH(73.00%→67.57%) and{ZH, AR, ES}(64.71%→63.08%).CleanGen36.67%→56.67%ZH, 30.00%→34.17%TE) against 1-ep N=5 (5×400 at identical Mlang=400) shows positive cross-lingual stabilization transfer (CleanGen_ZH: 56.67%→84.17%;CleanGen_TE: 34.17%→89.17%;Belebele_TE: 37.78%→39.33%), isolating Table 3's N=6→12 drop to sample dilution (333→166) and 2.58× ∥ΔW∥F growth at r=64.