- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Language-Adaptive Reward-Anchored Preference Optimization (LARA), a variant of DPO, to address language-dependent differences in preference optimization. It applies grapheme-based length normalization and language-specific margin calibration, along with optional cross-lingual anchoring and language-level group-DRO. Experiments show that most gains come from length normalization.
Critical strengths
1, The paper is well-structured and presents an extensive set of experiments, including comparisons across multiple models and languages, ablation studies, statistical analyses, and robustness checks. Figure 1 is particularly helpful for understanding this paper.
2, LARA is easy to implement, requiring only modifications to the DPO loss without changes to the model architecture or additional training data.
Critical weaknesses
1, The methodological contribution appears somewhat incremental, as the experiments suggest that most of the performance gains come from length normalization, which has been extensively studied in prior work. The main difference seems to be replacing token-length normalization with grapheme-based normalization, while the additional benefits of language-specific calibration are relatively modest.
2, The evaluation metric may favor the proposed method, as preference accuracy is measured using grapheme-normalized log-probabilities, which closely align with the proposed training objective. My understanding is that preference optimization should encourage the fine-tuned model to assign higher probabilities to preferred responses, rather than merely higher length-normalized scores. It would be helpful to report results using unnormalized sequence probabilities as well, to better understand whether the improvements reflect stronger preference learning or the choice of evaluation metric.
Questions for authors
1, Since length normalization has been extensively studied in preference optimization, could the authors clarify the advantages of grapheme-based normalization over existing token-based normalization? In particular, it would be helpful to include token-normalized DPO and SimPO as baselines.
2, Could the authors explain why normalized log probabilities are more appropriate for evaluation? It would be helpful to report accuracy based on unnormalized log probabilities as well.
3, Proposition 3 establishes scale invariance under the assumption that language-specific effects multiply both the policy-reference margins and the reference-gap distribution by a common positive factor. The authors should provide empirical evidence or theoretical justification for this assumption, since the practical relevance of the result is limited if it fails to hold.
Thank you for the review and the kind words on presentation. Your questions prompted several new experiments that we believe strengthen the paper, and we hope they support a more favourable score.
W1 / Q1. Novelty over token normalization and SimPO: Avg / worst / gap, %:
(– = not run due to time limitations.)
In the final version, we will sharpen the contribution to a reference-anchored, length-normalized margin whose temperature is set from reference statistics, rather than treating graphemes and per-language scales as separate sources of accuracy. We believe this resolves the novelty concern.
W2 / Q2. Metric bias and unnormalized results: Same checkpoints and pairs, avg / worst:
Even unnormalized, LARA leads DPO by 16–18 points on average and 24–26 on the worst language. We keep a normalized primary metric because summed scoring is itself length-biased: the preferred answer is longer in 57% of pairs, so base and DPO fall below chance (34–47%). We will include all three columns in the final version, along with DPO's implicit reward, where tuned or matched DPO scores higher, as expected for its own training objective.
Absolute likelihood: LARA raises the preferred response relative to the rejected one. Chosen log-prob per grapheme falls from −1.52 to −5.01 on Gemma (Aya −0.81 → −3.33, Qwen −0.93 → −5.91), similar to SimPO (−5.07). This is the well-known likelihood displacement shared by DPO-family losses, and notably calibration roughly halves it (w/o calib: −9.90). In a 1,024-token win-rate study against matched DPO, LARA scores 0.37 (Aya) and 0.395 (Gemma), on par with or above SimPO (0.38 and 0.372). Since our focus is preference ranking, we will discuss both points openly in the limitations. We hope this fully addresses the metric-bias concern.
Q3. Support for Proposition 3: Correlation across the 7 languages between s_ℓ and the per-language SD of uncalibrated grapheme-normalized margins:
The assumption holds strongly near the reference, where s_ℓ is estimated. After training it continues to hold on Gemma and partly on Aya, less so on Qwen, which matches where calibration helps most. Given the shared-scale result, we will revise App. E to present the proposition with appropriate qualifications and offer the correlation as a practical check to run before training, which we hope answers your question.
Thank you again for your constructive feedback. We hope the additional evidence resolves your concerns and would be grateful if you would consider revisiting your score. We are happy to clarify anything further during the discussion.