acceptodds
Under review as a conference paper at ICLR 2027

LARA: Language-Adaptive Reward-Anchored Preference Optimization

44 views

Abstract

Preference optimization is typically applied to multilingual language models with one global margin scale, implicitly assuming that chosen-rejected log-probability gaps are comparable across languages. This assumption is fragile since tokenizer fertility, script, response length, and language entropy make the same semantic preference induce different numerical margins. Existing remedies reshape the loss or reweight examples, yet they still apply one global margin scale to every language. We therefore introduce Language-Adaptive Reward-Anchored Preference Optimization (LARA), which instead changes the unit of optimization while preserving the implicit-reward view of direct preference optimization (DPO). Concretely, LARA scores completions with tokenizer-independent effective-length normalization, calibrates margins with per-language scales estimated from reference gaps, and optionally adds cross-lingual anchoring and language-level group distributionally robust optimization (group-DRO). Across three backbones (Qwen3-4B, Aya Expanse 8B, and Gemma-4-12B-it) and seven languages, LARA improves average and worst-language held-out accuracy over every baseline while also balancing languages. On Gemma, where DPO barely moves the base model (51.6%/47.0%), LARA reaches 68.0%/67.3% against 60.7%/56.5% for odds-ratio preference optimization (ORPO), with no length bias and the smallest language gap of any method. Under a Friedman-Nemenyi rank test on Gemma, LARA outranks every baseline at the 5% level. Ablations and theory then isolate the mechanism, identifying effective-length normalization as the dominant driver and per-language calibration as a smaller lower-tail and balance gain, while diagnostics rule out verbosity. The margin unit therefore decides how much accuracy a multilingual objective recovers, and the per-language scale decides how evenly that accuracy spreads across languages.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multilingual Language Model, Preference Optimizatoin, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NsqNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Language-Adaptive Reward-Anchored Preference Optimization (LARA), a variant of DPO, to address language-dependent differences in preference optimization. It applies grapheme-based length normalization and language-specific margin calibration, along with optional cross-lingual anchoring and language-level group-DRO. Experiments show that most gains come from length normalization.

Critical strengths

1, The paper is well-structured and presents an extensive set of experiments, including comparisons across multiple models and languages, ablation studies, statistical analyses, and robustness checks. Figure 1 is particularly helpful for understanding this paper.

2, LARA is easy to implement, requiring only modifications to the DPO loss without changes to the model architecture or additional training data.

Critical weaknesses

1, The methodological contribution appears somewhat incremental, as the experiments suggest that most of the performance gains come from length normalization, which has been extensively studied in prior work. The main difference seems to be replacing token-length normalization with grapheme-based normalization, while the additional benefits of language-specific calibration are relatively modest.

2, The evaluation metric may favor the proposed method, as preference accuracy is measured using grapheme-normalized log-probabilities, which closely align with the proposed training objective. My understanding is that preference optimization should encourage the fine-tuned model to assign higher probabilities to preferred responses, rather than merely higher length-normalized scores. It would be helpful to report results using unnormalized sequence probabilities as well, to better understand whether the improvements reflect stronger preference learning or the choice of evaluation metric.

Questions for authors

1, Since length normalization has been extensively studied in preference optimization, could the authors clarify the advantages of grapheme-based normalization over existing token-based normalization? In particular, it would be helpful to include token-normalized DPO and SimPO as baselines.

2, Could the authors explain why normalized log probabilities aθa_{\theta} are more appropriate for evaluation? It would be helpful to report accuracy based on unnormalized log probabilities as well.

3, Proposition 3 establishes scale invariance under the assumption that language-specific effects multiply both the policy-reference margins and the reference-gap distribution by a common positive factor. The authors should provide empirical evidence or theoretical justification for this assumption, since the practical relevance of the result is limited if it fails to hold.

1 comment

Thank you for the review and the kind words on presentation. Your questions prompted several new experiments that we believe strengthen the paper, and we hope they support a more favourable score.

W1 / Q1. Novelty over token normalization and SimPO: Avg / worst / gap, %:

AyaGemmaQwen
DPO (summed)59.9 / 52.1 / −3.751.6 / 47.0 / −2.459.3 / 53.3 / −3.4
Per-token DPO (ref-based, no calib)66.7 / 65.3 / −1.5–67.6 / 65.3 / −0.9
SimPO (tokens, ref-free)66.1 / 64.4 / −1.368.7 / 67.3 / −0.666.4 / 63.8 / −1.5
LARA w/o calib65.7 / 63.5 / −2.866.6 / 64.1 / −0.267.6 / 66.3 / −1.6
LARA, token denominator66.9 / 64.0 / −0.168.2 / 66.2 / 0.1–
LARA (graphemes)67.0 / 65.4 / −1.768.0 / 67.3 / −0.067.2 / 64.9 / −1.1

(– = not run due to time limitations.)

  • LARA is on par with SimPO, a strong recent baseline: LARA − SimPO avg = +0.8 [−0.3, +1.9] Qwen, +1.0 [+0.0, +1.9] Aya, −0.7 [−1.5, +0.2] Gemma. At SimPO's recommended β=2.5 on Aya, LARA leads by +2.7 [+1.5, +3.8].
  • Per-token DPO reaches similar accuracy (Qwen: −0.4 [−1.2, +0.4]), confirming your point that normalization is the key ingredient.
  • Graphemes vs. tokens: Once calibrated, the unit is absorbed (grapheme − token LARA: +0.2 [−0.4, +0.7] Aya, −0.2 [−0.7, +0.3] Gemma), because tokens per grapheme vary little within a language (CV 0.05–0.16). Without calibration graphemes clearly matter: on Gemma, dividing by tokens shrinks Chinese margins about 3× relative to English.
  • Calibration helps on two of three backbones: +1.3 [+0.5, +2.1] Aya, +1.4 [+0.6, +2.3] Gemma, −0.3 Qwen, and on Aya +1.4 ± 0.3 over 3 seeds. A shared-scale control shows this comes mostly from setting the temperature.

In the final version, we will sharpen the contribution to a reference-anchored, length-normalized margin whose temperature is set from reference statistics, rather than treating graphemes and per-language scales as separate sources of accuracy. We believe this resolves the novelty concern.

W2 / Q2. Metric bias and unnormalized results: Same checkpoints and pairs, avg / worst:

Grapheme (paper)Unnormalized sumToken-norm.
Qwen DPO59.3 / 53.346.6 / 37.761.0 / 56.7
Qwen SimPO66.4 / 63.860.1 / 53.466.4 / 64.5
Qwen LARA67.2 / 64.963.0 / 61.567.8 / 66.1
Aya DPO59.9 / 52.147.1 / 36.763.2 / 58.5
Aya SimPO66.1 / 64.462.8 / 61.766.4 / 64.5
Aya LARA67.0 / 65.462.9 / 61.767.2 / 65.0
Gemma DPO51.6 / 47.043.8 / 34.058.8 / 55.5
Gemma SimPO68.7 / 67.360.5 / 55.668.7 / 67.7
Gemma LARA68.0 / 67.361.8 / 59.768.0 / 66.7

Even unnormalized, LARA leads DPO by 16–18 points on average and 24–26 on the worst language. We keep a normalized primary metric because summed scoring is itself length-biased: the preferred answer is longer in 57% of pairs, so base and DPO fall below chance (34–47%). We will include all three columns in the final version, along with DPO's implicit reward, where tuned or matched DPO scores higher, as expected for its own training objective.

Absolute likelihood: LARA raises the preferred response relative to the rejected one. Chosen log-prob per grapheme falls from −1.52 to −5.01 on Gemma (Aya −0.81 → −3.33, Qwen −0.93 → −5.91), similar to SimPO (−5.07). This is the well-known likelihood displacement shared by DPO-family losses, and notably calibration roughly halves it (w/o calib: −9.90). In a 1,024-token win-rate study against matched DPO, LARA scores 0.37 (Aya) and 0.395 (Gemma), on par with or above SimPO (0.38 and 0.372). Since our focus is preference ranking, we will discuss both points openly in the limitations. We hope this fully addresses the metric-bias concern.

Q3. Support for Proposition 3: Correlation across the 7 languages between s_ℓ and the per-language SD of uncalibrated grapheme-normalized margins:

CheckpointQwenAyaGemma
DPO (near reference)0.901.000.97
LARA w/o calib (trained)0.090.580.96

The assumption holds strongly near the reference, where s_ℓ is estimated. After training it continues to hold on Gemma and partly on Aya, less so on Qwen, which matches where calibration helps most. Given the shared-scale result, we will revise App. E to present the proposition with appropriate qualifications and offer the correlation as a practical check to run before training, which we hope answers your question.

Thank you again for your constructive feedback. We hope the additional evidence resolves your concerns and would be grateful if you would consider revisiting your score. We are happy to clarify anything further during the discussion.

Reviewer_rSEWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The problem this paper address: preference margins can have different numerical scales across languages, so applying the same DPO temperature may produce uneven training signals. Its core method makes three changes to the calculation:

  1. Divide each response’s log probability by its number of Unicode graphemes—roughly, user-perceived characters.
  2. Subtract the frozen reference model’s normalized preference gap.
  3. Divide that margin by a language-specific scale estimated from reference-model gaps.

The training inputs are prompts with preferred and rejected responses, plus language labels. The output is a model adapted through LoRA; ordinary generation requires no new decoding procedure. Optional components encourage agreement across translated examples and increase the weight of high-loss languages.

The experiments use 31,000 training pairs and 8,514 held-out pairs across seven languages. Arabic and Chinese are machine-translated extensions. Crucially, the headline accuracy measures whether the model assigns a higher normalized likelihood to the supplied preferred answer. It does not measure whether newly generated answers are better.

Critical strengths

First, estimating one scale per language is an inexpensive way to address uneven preference learning. It could be useful without changing the model architecture.

Second, the reported improvement reaches weak languages rather than merely improving the pooled average. On Gemma, LARA exceeds ORPO by 7.3% on average and 10.8% in worst-language accuracy. The improvement also survives changing the evaluation denominator from graphemes to tokens, although its margin over DPO becomes smaller.

Critical weaknesses
  1. The evidence does not isolate the benefit of graphemes over ordinary token normalization. Most of the gain appears before language calibration is added. Token-normalized preference objectives already exist, and SimPO explicitly studies reference-based DPO with token-length normalization in Appendix I [1]. The clean comparison would retain LARA’s reference subtraction and calibration while changing only the length denominator. Figure 13 instead changes how existing checkpoints are scored; it cannot establish which normalization is better during training.

    There is also a useful theoretical observation: if token count is a constant multiple of grapheme count within a language, that factor cancels between the normalized margin and its calibration standard deviation. Under that assumption, calibrated token and grapheme objectives coincide. Their practical difference therefore depends on variation within a language, which the reported average tokenization ratios do not establish.

  2. The calibration ablation changes overall temperature as well as its variation across languages. On Gemma, the reported scales range from 0.136 to 0.528. With the fixed base temperature of 0.1, calibration produces effective temperatures of approximately 0.19–0.74, compared with 0.1 without calibration. A suitably tuned shared temperature could therefore explain some of the improvement. The paper does not provide that control.

    This matters because the incremental calibration benefit is modest and inconsistent: it improves Aya and Gemma but reduces Qwen’s average and worst-language accuracy by 0.4 and 1.4 points. Those results support a potentially useful heuristic; they do not yet establish that language-specific scaling is the decisive mechanism.

  3. The comparisons do not cleanly support the authors’ causal explanation. Appendix I reports different learning rates and seeds for LARA and the baselines, with one training run per setting. Consequently, the statement that normalization alone accounts for the DPO-to-LARA improvement is stronger than the experiment permits. Bootstrap intervals quantify uncertainty over evaluation examples, not variability from training.

    There is a concrete baseline-description problem too: Sections 3.1 and D.3 classify ORPO as using summed, unnormalized margins. The original ORPO formulation uses token-average log likelihood [3]. This establishes a discrepancy in the manuscript. I am not sure how they implement the ORPO.

[1] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng et al. NeurIPS 2024. https://arxiv.org/abs/2405.14734

[2] ORPO: Monolithic Preference Optimization without Reference Model. Jiwoo Hong et al. EMNLP 2024. https://aclanthology.org/2024.emnlp-main.626/

Questions for authors

see above

2 comments

Thank you for a precise and constructive review. Your points on W1 and W2 were insightful and helped us state the contribution more clearly, and we hope the new controls below merit a higher score.

W1. Graphemes vs. token normalization: To test this directly, we kept LARA unchanged and switched only the denominator to model tokens (with scales re-estimated in tokens). Avg / worst / gap, %:

AyaGemmaQwen
LARA, graphemes67.0 / 65.4 / −1.768.0 / 67.3 / −0.067.2 / 64.9 / −1.1
LARA, model tokens66.9 / 64.0 / −0.168.2 / 66.2 / 0.1–
Per-token DPO (no calib)66.7 / 65.3 / −1.5–67.6 / 65.3 / −0.9
LARA w/o calib65.7 / 63.5 / −2.866.6 / 64.1 / −0.267.6 / 66.3 / −1.6

(– = not run due to time limitations.)

Your argument holds nicely: a constant within-language ratio cancels against s_ℓ. Tokens per grapheme vary little within a language (CV 0.05–0.16), so the two calibrated objectives nearly coincide (grapheme − token LARA: +0.2 [−0.4, +0.7] Aya, −0.2 [−0.7, +0.3] Gemma). Per-token DPO reaches similar accuracy (Qwen: −0.4 [−1.2, +0.4]). The final version will credit your observation, present graphemes as what makes the uncalibrated margin tokenizer-invariant (Gemma fertility 0.20 vs. 0.63 tokens/grapheme), and centre the contribution on the reference-anchored normalized margin.

We also added SimPO (β=0.1, γ=0.5, same data), a strong recent baseline, and LARA performs on par with it: LARA − SimPO avg = +0.8 [−0.3, +1.9] Qwen, +1.0 [+0.0, +1.9] Aya, −0.7 [−1.5, +0.2] Gemma. With the recommended β=2.5, γ/β=0.5 on Aya, SimPO drops to 64.4 / 59.5 and LARA leads by +2.7 [+1.5, +3.8]. We will adjust our comparison claims accordingly in the final version, which we believe addresses your W1 concern.

W2. Calibration changes the overall temperature: Agreed, the mean β/s_ℓ is 4–6× the base β=0.1. As a control, we trained with one shared scale s̄ at the same training-weighted temperature (avg / worst):

AyaGemmaQwen
w/o calib (β=0.1)65.7 / 63.566.6 / 64.167.6 / 66.3
Shared s̄66.6 / 64.368.1 / 67.067.4 / 64.5
Per-language s_ℓ67.0 / 65.468.0 / 67.367.2 / 64.9

Going from no calibration to s̄ gives +0.9 [+0.1, +1.7] Aya, +1.6 [+0.7, +2.4] Gemma and −0.1 [−1.0, +0.7] Qwen. Going from s̄ to s_ℓ adds +0.4 [−0.1, +0.9], −0.1 [−0.6, +0.4] and −0.2 [−0.9, +0.5]. Most of the gain therefore comes from setting the temperature from reference statistics, a simple and robust recipe, while the per-language refinement is within noise.

Qwen: Chinese has by far the widest reference dispersion (s = 0.818 vs. 0.136 for Italian), so it receives a 6× smaller temperature (66.3→64.9). The correlation between s_ℓ and trained margin spread is 0.96 on Gemma, 0.58 on Aya and 0.09 on Qwen, which gives practitioners a simple diagnostic. In the final version, we will replace "decisive mechanism" with more measured wording, make s̄ the default, and present per-language scales as an optional refinement guided by this correlation. Together, these results resolve the temperature confound you raised.

W3. Causal story: lr, seed, β: We re-implemented DPO inside LARA's script (same lr, seed, code and 4-bit loading, with the Gemma reference re-scored using the step-0 4-bit policy). Avg / worst:

AyaGemma
DPO, original59.9 / 52.151.6 / 47.0
DPO, matched60.4 / 53.152.2 / 47.9
DPO, matched, β=0.0264.3 / 62.555.7 / 53.1
LARA w/o calib65.7 / 63.566.6 / 64.1
LARA, 3 seeds67.1 ± 0.1 / 65.4 ± 0.168.0 / 67.3 (1 seed)
LARA w/o calib, 3 seeds65.7 ± 0.2 / 63.8 ± 0.3–

(– = not run due to time limitations.)

Matching changes DPO by only +0.5 (Aya) and +0.6 (Gemma), so the original comparison holds. On Gemma, DPO barely moves the policy (step-1 grad norm 1,444). β=0.02 helps (+3.9 Aya, +3.5 Gemma), yet LARA still leads (+2.7 Aya, +12.3 [+11.0, +13.6] Gemma). Normalization carries most of the gain (w/o calib − matched DPO: +5.3 Aya, +14.4 Gemma), and calibration adds a further 1.3–1.4. For completeness, on DPO's own implicit-reward accuracy, matched DPO (Aya) and β=0.02 DPO (Gemma) score higher, as one would expect for DPO's training objective. Due to time limitations, matched DPO on Qwen, more seeds on Gemma and Qwen, and a wider β/lr sweep will follow in the final version. With fully matched settings, we believe the causal concern is now addressed.

ORPO: We appreciate you pointing out this discrepancy. Like the original, TRL's ORPO uses token-averaged log-likelihoods. We will correct Sec. 3.1 and App. D.3 in the final version to describe ORPO as normalized and reference-free, which resolves this point.

We are grateful for your detailed feedback. If the new controls resolve your soundness concerns, we would be glad to see this reflected in your rating, and we welcome any further suggestions.

Thank you for the additional experiments and clarifications. Some of my concerns are not resolved well.

Specifically, the new results narrow the paper’s contribution. Grapheme and token normalization achieve similar average performance, while a shared scale recovers most of the calibration gain. The added SimPO comparison likewise shows no consistent average advantage for LARA. The worst-language results suggest some benefit from the proposed components, but the uncertainty of those differences remains unclear.

I therefore still lean toward weak reject since I remain unconvinced that the results establish a sufficiently distinct or useful advance over existing length-normalized preference optimization methods.

Reviewer_xLEiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that multilingual preference optimization with a single global margin scale is miscalibrated. Tokenizer fertility, script, response length, and language entropy change the numerical scale of chosen–rejected log-probability gaps, independently of the preference itself. LARA modifies DPO in two ways. First, it averages log-probabilities over Unicode grapheme clusters instead of summing them, keeping the reference subtraction. Second, it divides each language’s margin by sℓs_ℓ, the standard deviation of the reference model’s normalized gap on a held-out calibration split (≤500 pairs per language), which amounts to a per-language temperature β/sℓβ/s_ℓ. Optional extensions add a cross-lingual anchor penalty on translated clusters and language-level group-DRO.

Critical strengths
  1. The paper gives direct evidence that preference margins live on language-dependent scales. The estimated calibration scales span a factor of 3.9 on Gemma, from 0.136 for English to 0.528 for Chinese (Sec. 3.3), and they track empirical margin dispersion (r = 0.95, Fig. 15). The method needs only seven scalars, estimated once from reference-scored pairs. It requires no architecture change and no extra pass over the training data, so it would be easy to adopt in existing DPO pipelines.

  2. Thorough statistical reporting and candid limitations. Results are reported per language on three backbones, with grouped-bootstrap intervals over conversation groups, Friedman–Nemenyi and McNemar tests, length-decile diagnostics, and an error analysis.

Critical weaknesses
  1. The test metric matches LARA’s own training loss. All methods are scored by per-character likelihood ranking, which is exactly what LARA optimizes. DPO does not optimize this, so it looks weak: on Gemma, DPO equals the base model (51.6%). Switching the scoring to per-token alone cuts LARA’s lead over DPO from 16.4 to 9.3 points (Sec. 3.4). There is also no test of generation quality. App. G only measures length, and most outputs hit the 256-token cap. Please add DPO’s own metric and a multilingual win-rate evaluation.

  2. Missing baselines and weak ablations. Most of the gain comes from length normalization, a known idea, but SimPO and length-normalized DPO are not compared. Several methods the paper cites are not compared either: R-DPO, IPO, KTO, and Dr. DPO. Group-robust DPO (Ramesh et al., 2024) is not discussed. The new part, per-language calibration, adds only +1.3, +1.4, and −0.4 points. That is within the 0.1–1.4-point run-to-run noise the authors report (App. K.1). Each setup runs once, and LARA and the baselines use different learning rates and seeds (Table 9). The two optional parts do not help.

Questions for authors
  1. How does LARA compare with SimPO and with per-token-averaged DPO? Once margins are divided by a per-language scale, does using characters instead of tokens still matter?

  2. All Gemma scales are below 1, so calibration also makes the overall β about 2–7× larger. Could you add a run with one shared scale, or tune β for LARA without calibration, to see which effect matters?

  3. On Qwen, calibration lowers Hindi and Chinese, and Chinese becomes the worst language. Why would calibration help the worst language? Please also report the scales for Qwen and Aya.

  4. Were β and the learning rate tuned for DPO, especially on Gemma? Please report DPO’s implicit-reward accuracy, and rerun with the same learning rate and seed, using at least three seeds.

  5. DPO’s derivation needs the partition function Z(x) to cancel. After dividing by length, it no longer cancels when the two responses differ in length. In what sense is LARA still “in the same implicit-reward family as DPO”?

I'd be happy to raise the score if these questions have been justified.

2 comments

Thank you for a careful and thoughtful review. Your comments led to many new experiments, which we hope will encourage you to raise your score.

W1. The metric matches LARA's loss: We re-scored all models with rules that don't favour LARA (avg / worst, %):

Unnormalized sumDPO implicit reward
Qwen DPO / SimPO / LARA46.6/37.7 · 60.1/53.4 · 63.0/61.565.4/63.3 · 62.0/58.1 · 63.6/62.3
Aya DPO / SimPO / LARA47.1/36.7 · 62.8/61.7 · 62.9/61.763.3/62.5 · 63.4/62.6 · 63.6/62.5
Gemma DPO / SimPO / LARA43.8/34.0 · 60.5/55.6 · 61.8/59.755.4/52.8 · 63.6/61.4 · 64.0/62.5

With summed scores, LARA leads DPO by 16–18 points on average and 24–26 on the worst language. (Summing is length-biased: the preferred answer is longer in 57% of pairs.) On DPO's implicit reward, matched DPO (Aya 65.4/64.3) and β=0.02 DPO (Gemma 66.5/65.2) score higher, as expected for its own objective. We will frame LARA's claim around pair ranking and report the Gemma gap under token scoring (16.4→9.2).

Win rates vs. matched DPO (40 prompts × 7 languages, 1,024 tokens, Qwen2.5-14B judge, both orders):

AyaGemma
Base0.54 [0.50, 0.58]0.49 [0.45, 0.53]
LARA0.37 [0.33, 0.41]0.395 [0.36, 0.43]
SimPO (β=2.5)0.38 [0.34, 0.42]0.372 [0.362, 0.381]
LARA w/o calib0.07 [0.05, 0.09]0.0135 [0.012, 0.015]

We make no generation-quality claim, though LARA edges out SimPO on Gemma (0.395 vs. 0.372). Calibrated LARA is the most concise (Aya: 1,778 vs. 2,212 graphemes for DPO), and calibration is key for generation (w/o calib: 0.07 Aya, 0.0135 Gemma). We believe this addresses the metric concern.

W2. Baselines and seeds: We added SimPO (β=0.1, γ=0.5, same data), and LARA is on par: LARA − SimPO avg = +0.8 [−0.3, +1.9] Qwen, +1.0 [+0.0, +1.9] Aya, −0.7 [−1.5, +0.2] Gemma. At SimPO's recommended β=2.5 on Aya, LARA leads by +2.7 [+1.5, +3.8]. Per-token DPO is similar (Aya 66.7/65.3, Qwen 67.6/65.3 vs. 67.0/65.4, 67.2/64.9), so normalization is the key ingredient. Due to time limitations, IPO, R-DPO, KTO and Dr. DPO will follow in the final version, along with group-robust DPO (LARA rescales, not reweights, groups).

Aya, 3 seedsAvgWorst
LARA w/o calib65.7 ± 0.263.8 ± 0.3
LARA67.1 ± 0.165.4 ± 0.1
Δ, paired by seed+1.4 ± 0.3+1.6 ± 0.3

Calibration improves Aya at every seed (Aya only, due to time limitations). Qwen is the one exception, within noise (−0.3 [−1.1, +0.4]). The baseline and variance concerns are thus resolved.

Q1. Graphemes vs. tokens after calibration: Calibrated token-LARA gets 66.9/64.0 (Aya) and 68.2/66.2 (Gemma), so grapheme − token = +0.2 [−0.4, +0.7] and −0.2 [−0.7, +0.3]. Tokens per grapheme vary little within a language (CV 0.05–0.16), so the scale absorbs the unit. Graphemes matter for the uncalibrated margin, and we will clarify this, addressing your question.

Q2. Temperature confound: We trained with one shared scale s̄ at the same mean temperature:

avg / worstAyaGemmaQwen
w/o calib65.7 / 63.566.6 / 64.167.6 / 66.3
Shared s̄66.6 / 64.368.1 / 67.067.4 / 64.5
Per-language s_ℓ67.0 / 65.468.0 / 67.367.2 / 64.9

s̄ alone adds +0.9 [+0.1, +1.7] Aya and +1.6 [+0.7, +2.4] Gemma, and s_ℓ adds +0.4 [−0.1, +0.9] and −0.1 [−0.6, +0.4] on top. Most of the gain is thus from a reference-set temperature, which we will make clear. This resolves the confound.

Q3. Qwen: Scales (EN/FR/IT/PT/HI/AR/ZH) are .414/.178/.136/.179/.312/.387/.818 for Qwen and .155/.133/.132/.145/.290/.230/.452 for Aya. On Qwen, Chinese gets a 6× smaller temperature, hence weaker updates (66.3→64.9). After training, scale and margin spread correlate at 0.96 (Gemma), 0.58 (Aya) and 0.09 (Qwen), a useful diagnostic. We will make s̄ the default and note this as a limitation, which we hope answers your question.

Q4. DPO tuning: We re-ran DPO with matched settings:

avg / worstAyaGemma
DPO, original59.9 / 52.151.6 / 47.0
DPO, matched lr/seed/code60.4 / 53.152.2 / 47.9
DPO, matched, β=0.0264.3 / 62.555.7 / 53.1
LARA67.0 / 65.468.0 / 67.3

On Gemma, DPO barely moves the policy (step-1 grad norm 1,444). β=0.02 helps (+3.9 Aya, +3.5 Gemma), yet LARA still leads (+2.7 Aya, +12.3 [+11.0, +13.6] Gemma). This matched comparison addresses the tuning concern. Due to time limitations, multi-seed DPO and a wider sweep will follow in the final version.

Q5. Z(x): Correct. Normalization leaves a β log Z(x)(1/e(y₋) − 1/e(y₊)) term, so LARA is not the closed-form KL solution. Like SimPO and R-DPO, it keeps DPO's Bradley–Terry form on a reference-anchored implicit reward. Revising Sec. 2.1 accordingly resolves this point.

Thank you again for helping us improve the paper. We would sincerely appreciate an updated assessment in light of these additions.

Thank you for the detailed rebuttal and the substantial new experiments. I appreciate the transparent reporting and am raising my score by one. My remaining concern is that the new results narrow the contribution: tuned DPO leads on its own metric and in the win-rate evaluation, LARA is on par with SimPO and per-token DPO, and most of the calibration gain comes from the shared scale, not the per-language one.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.