acceptodds
Desk rejected at ICLR 2027

Tail-Aware Direct Preference Optimization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

15 views

Abstract

The geometry of policy regularization is a central design choice in LLM alignment. KL-regularized RLHF and DPO are attractive because they yield exponential tilting and a simple log-ratio preference loss, but this geometry can amplify low-coverage reward-model errors and cause severe overoptimization. We propose Tail-Aware Direct Preference Optimization (TDPO), a direct preference method derived from R\'enyi-regularized RLHF. TDPO derives an exact direct-preference loss on the active constraint, whose Bradley–Terry score difference is a difference of density-ratio powers rather than the KL-induced log-ratio. The order controls statistical coverage, tail sensitivity, and anti-hacking strength as a hyperparameter of the preference law. We prove finite-sample bounds for the unregularized optimality gap and regularized sub-optimality gap under a Bradley–Terry preference model. A two-environment lower bound matches the latter's sample-size exponent for arbitrary policy-valued estimators. We further characterize the induced geometry: locally, R\'enyi regularization rescales the Fisher geometry of KL, while globally it provides polynomial rare-set control. A TL;DR study evaluates density-ratio tails and low-reference-coverage mass across DPO, ChiPO, and clipped/capped baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
direct Preference Optimization, Rényi divergence, reward hacking, RLHF, information geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cYKPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is a special case of f-DPO (ICLR 24) and proposes Tail-Aware Direct Preference Optimization (TDPO) by changing the regularization term in RLHF from KL to Rényi divergence. As proven in Lemma 3.1, unlike the conventional method, where the optimal solution is the reference distribution multiplied by an exponential function of the reward, the proposed method results in a power form of the reward. Thus, the paper claims that the probability does not differ significantly from the reference distribution.

Although the main text contains excessive theoretical explanations, I think the key point in the context of this paper is that the gap between the reference and policy distributions can also bound pointwise probability assignments. Theorem 5.1 and Theorem 5.2, which explains that KL and other divergences cannot provide such bounds, indirectly support this claim.

In very limited experiments, the method shows a slight reduction in probability assignments to regions far from the reference distribution, at the cost of some loss in win rate.

Critical strengths
  1. Explaining how the reward values are reflected in the form of the optimal solution is a good theoretical approach. Explain the formulation of optimal solution through Lemm3.1 looks clear.

  2. Although the main points are somewhat scattered due to excessive theoretical explanations, the approach is reasonable from the perspective of assigning probability to responses where the reference distribution has meaningful probability values (Theorem 5.1).

  3. Although the experiments are very limited, the Outside-ref-99% results in Table 2 support the main claim of the paper.

Critical weaknesses
  1. It is unclear whether the exponential reward term in the original DPO is problematic. Also, since the optimal reward model is in log-likelihood units, multiplying by its exponential is an appropriate scale for representing probability values.

  2. Since this paper is a subset of f-DPO, it only selects Rényi divergence as a specific choice of f. A broader comparison with other instances of f-DPO is needed to examine whether the reward is reflected in the optimal solution in exponential or power-law forms.

  3. According to f-PO (Han et al., AISTATS 2025) and BPO (Kim et al., NeurIPS 2025), even when the optimal solution is fixed, there is still room for discussion on how to match it. How are the training dynamics compared to the original DPO? For example, it might be useful to analyze the gradient update equation. Can the proposed method also be applied orthogonally to f-PO, BPO, and other methods?

  4. The experiments are very limited, and proposing only metrics such as "Outside-ref-99%" is problematic. It would be better to design experiments based on empirical cases where deviating from the reference distribution causes problems, and find test cases where the proposed method actually improves language quality.

Questions for authors

Could you conduc more pratical experiments that TDPO is actually have effects in better language quality? My intuition is that when reference model is good and preference dataset is not good enough, then TDPO can be better than DPO in win rate.

2 comments

Thank you for the constructive comments. We clarify the motivation, the relationship to prior work, and the scope of our experiments.

Exponential reweighting (W1). In the Bradley–Terry model, the reward difference determines the pairwise preference probability through the logistic link. The exponential reweighting of generation probabilities comes from a separate problem: maximizing expected reward under a KL constraint. It arises even when the reward is given directly and no preference model is used. Our concern is that this update can assign substantial probability to a response that is rare under the reference when its reward is overestimated.

Relation to ff-DPO and statistical contribution (W2). We agree that TDPO uses the f′f'-score form of ff-DPO. Other choices can yield different updates; for example, forward KL gives reciprocal reweighting. The exponentiated Rényi penalty is an ff-divergence up to an additive constant, but its generator has a finite right derivative at zero, so the reparametrization theorem of Wang et al. (2024) does not directly cover the thresholded solution. The generator is globally strongly convex only at α=2\alpha=2, so the general strongly convex guarantees of Zhao et al. (2026) and the formal analysis of ChiPO’s strongly convex mixed penalty (Huang et al., 2025) do not directly apply at other orders.

Our Bregman analysis is of independent technical interest: it reveals a curvature-driven phase transition in the minimax rate for the regularized gap at α=2\alpha=2. Policy normalization cancels the reward offsets that pairwise comparisons cannot identify, allowing us to bound the gap from logit estimation error via the penalty’s Bregman divergence. For α>2\alpha>2, quadratic curvature vanishes at zero density ratios, and the matching lower bound shows that the resulting slowdown is unavoidable in our setting.

Gradient updates and compatibility (W3). In the margin gradient, DPO gives the chosen and rejected log-probability gradients coefficients of equal magnitude and opposite sign. TDPO additionally scales each coefficient by (α−1)uα−1(\alpha-1)u^{\alpha-1}, where uu is that response’s policy-to-reference density ratio. TDPO can be combined with BPO (Kim et al., 2025) by using e−ΔTDPOe^{-\Delta_{\rm TDPO}} as the model ratio, and the Rényi optimizer can serve as the target in an ff-PO-style matching objective (Han et al., 2025).

Language quality (W4). We assess summary quality on TL;DR through pairwise judgments. Table 2 compares each method separately against SFT and does not establish a quality cost for TDPO. In the direct comparison in Table 10, TDPO’s win rate against DPO exceeds 50% in all three seeds, averaging 52.57%; these results are descriptive. If unreliable preferences favor poor summaries that a strong reference model rarely generates, TDPO could help preserve quality by limiting the probability assigned to those summaries. Appendix H illustrates this mechanism in a controlled example. Our TL;DR experiments did not vary preference reliability, so they do not directly test this hypothesis.

References

  • Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
  • Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, 2024.
  • Qingyue Zhao, Kaixuan Ji, Heyang Zhao, Tong Zhang, and Quanquan Gu. Towards a sharp analysis of offline policy learning for ff-divergence-regularized contextual bandits. In The Fourteenth International Conference on Learning Representations, 2026.
  • Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J. Foster. Correcting the mythos of KL-regularization: Direct alignment without overoptimization via chi-squared preference optimization. In The Thirteenth International Conference on Learning Representations, 2025.
  • Jiaqi Han, Mingjian Jiang, Yuxuan Song, Stefano Ermon, and Minkai Xu. ff-PO: Generalizing Preference Optimization with ff-divergence Minimization. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:1144–1152, 2025.
  • Yeongmin Kim, HeeSun Bae, Byeonghu Na, and Il-chul Moon. Preference Optimization by Estimating the Ratio of the Data Distribution. In Advances in Neural Information Processing Systems 38, 2025.

Thank you for the clarifications. W1 and W3 are largely resolved. Regarding W2, my main concern was whether other f-DPO variants also avoid the exponential reweighting issue identified in DPO, and if so, what makes TDPO particularly advantageous. While the additional theoretical contributions are interesting, this comparison remains unclear.

W4 also remains unresolved, as the experiments do not directly validate the practical benefits of TDPO under unreliable preference data.

Overall, I appreciate the theoretical contributions but believe stronger empirical evidence is needed. I therefore maintain my original rating.

Reviewer_hWbrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Tail-Aware Direct Preference Optimization (TDPO), which uses Rényi regularization to limit probability shifts toward responses that are rare under the reference policy. The paper derives the TDPO objective, establishes statistical guarantees under a Bradley–Terry model, and analyzes the probability assigned to rare sets. Experiments on TL;DR compare TDPO with several baselines and measure response quality and changes in policy probabilities.

Critical strengths

Strengths

  1. The paper addresses an important question: can regularization limit reward exploitation on rare responses? This strength carries limited weight in my score, because the paper does not show that rewards are inaccurate on rare responses or that shifting probability toward them causes harm.

  2. The mathematical derivations are interesting despite the poor presentation. The derivation of TDPO, its statistical guarantees, and the analysis of rare sets give the paper theoretical substance. This strength also carries limited weight in my score, because the paper does not explain how these results address reward exploitation on rare responses, and the presentation makes the derivations hard to follow.

Critical weaknesses

Weaknesses

  1. The paper does not clearly state or answer its central scientific question. The reader must reconstruct whether the goal is to reduce reward exploitation or to establish statistical optimality. Unexplained terminology, abrupt transitions, and poorly motivated derivations make this reconstruction difficult. Sections 3–5 present results without explaining how they combine to reach that goal. This unclear goal is the main reason for my score.

  2. The paper does not show that controlling probability on rare responses reduces reward exploitation. The authors do not show that rewards are inaccurate on rare responses or that increasing their probability causes harm. The main experiments do not close this gap: Table 2 and Figure 3 establish no advantage in response quality over the baselines, and the favorable diagnostic differences lack uncertainty estimates or evidence that they reduce harm. The manuscript also provides no code link, limiting independent verification of the experiments.

Questions for authors

Questions

  1. Central question (Introduction and Sections 3–5). What precise question does the paper answer, and how do the statistical guarantees and the bounds on the probability of rare sets combine to answer it?

  2. Low coverage and reward exploitation (lines 321–323). What evidence shows that rewards are inaccurate on rare responses and that increasing their probability causes harm? Is this mechanism an assumption that defines the paper’s setting, or a general claim about reward exploitation?

  3. Why Rényi regularization? (Introduction and Section 3). What limitation of existing methods motivates Rényi regularization, and what advantage does TDPO show over ChiPO or appropriately tuned KL regularization? Neither answer is clear from the current manuscript.

  4. Motivation and interpretation (Sections 3–5). For each mathematical derivation, could the authors explain why it is needed and how its result helps answer that question? As written, the paper reads as a collection of theoretical results without a clear connection to one another or to the central question.

  5. Discussion and limitations. Could the Discussion section address the limitations of the claimed connection between rare responses and reward exploitation? Under what conditions do rare responses receive inaccurate rewards, and when does increasing their probability cause harm? Please state which conditions the analysis assumes and which guarantees no longer apply when these conditions fail.

  6. Exact parametrization (line 212). Does the claimed parametrization remain exact when the optimizer assigns zero probability to some responses?

  7. Experimental evidence (Table 2, Figure 3, Appendix Tables 9–10). Can the authors report uncertainty estimates for the small differences in diagnostic metrics, and what evidence connects these differences to reduced reward exploitation? Given that the main comparison does not establish an advantage in response quality, how do Appendix Tables 9–10 establish an improvement over the baselines, and what uncertainty supports those comparisons?

2 comments

Thank you for pointing out that the paper did not clearly explain how Sections 3–5 fit together. We will revise the introduction to state the motivating question: Can offline preference optimization limit how much probability a policy assigns to mistakenly rewarded responses that are rare under the reference?

Tail-aware motivation (Q1, Q3, Q4). In offline preference learning, a response that the reference policy rarely generates may receive an mistakenly high reward. Optimizing that reward can then give the response much more probability than it had under the reference. We use Rényi regularization to limit this shift. Section 3 turns that idea into a preference objective, and Section 5 explains what it guarantees.

Evidence and scope (Q2). The Introduction (lines 42–45) motivates this as a possible failure mode: when offline data contain few informative comparisons involving a response, its estimated reward can be inaccurate. Figure 2 in the main text illustrates how a spurious reward spike on a reference-rare response leads to a larger probability shift under KL than under Rényi regularization. Appendix H goes further by specifying the true reward and showing how moving probability from a better response to a falsely rewarded rare response can reduce true expected reward. These examples illustrate when the mechanism can occur. Our TL;DR experiments do not test whether rare responses are systematically overrewarded or whether increasing their probability causes lower quality. We will clarify this scope in lines 321–323.

Discussion and limitations (Q5). Upweighting a rare response reduces true expected reward when it shifts probability away from responses with higher true reward. Since rarity alone tells us nothing about true reward, our rare-set bound limits the mass assigned to rare responses but does not by itself imply better response quality. We will make this distinction explicit in the Discussion.

Statistical efficiency as a separate contribution (Q1, Q4). Section 4 asks how efficiently the policy induced by the Rényi penalty can be learned from finite preference data. The penalty is an ff-divergence up to a constant, but its generator is globally strongly convex only at α=2\alpha=2. Thus the general strongly convex ff-divergence results of Zhao et al. and the formal analysis of ChiPO’s mixed penalty do not cover the other orders. The Bregman argument is of independent technical interest: it uses policy normalization to eliminate the reward offsets that pairwise comparisons cannot identify, then uses the penalty’s power curvature to convert logit estimation error into a bound on the regularized policy gap even when quadratic curvature vanishes. The matching lower bound shows that the slower rate for α>2\alpha>2 is unavoidable in our setting.

Exact parametrization (Q6). Equation (7) gives the exact penalized optimizer for a given reward, including when some responses receive zero probability. Conversely, a policy’s induced reward (line 206) yields that same policy as the optimizer. However, recovering the true preference logits requires the induced reward to agree with the true reward up to a prompt-dependent constant. Assumption 4.2 and Appendix F give sufficient conditions for this by excluding strict truncation.

Empirical evidence and reproducibility (Q7). Table 2 and Figure 3 show that TDPO assigns slightly less mass to reference-rare tokens than DPO at the measured checkpoints. Table 9 examines quality at matched KL, and Table 10 favors TDPO over DPO in all three direct comparisons, with a mean win rate of 52.57% and an across-seed standard deviation of 0.93 percentage points. These are descriptive results. We do not report uncertainty intervals for the tail diagnostic or the matched-KL comparisons, and the direct comparison does not establish statistical significance. Appendix A documents the experimental settings and per-seed results.

Reviewer_hWbrOpenReview ↗

Response to the Authors

Thank you for your response. I appreciate your clarifications. My main concern is that the paper has not been revised. It is still hard to follow. Without such a revision (e.g., a revision that connects Sections 3-5 to the research question), W1 remains partly addressed.

The authors' response clarifies W2. However, as the authors acknowledge in Evidence and scope (Q2), the experiments do not test whether rare responses are systematically overrewarded or whether increasing their probability causes lower quality. Thus, my main concern remains.

Questions Q1, Q2, Q5, and Q6 have been clarified. Questions Q3, Q4, and Q7 remain partly addressed. Given all these concerns, I intend to keep my original score.

Reviewer_rZW4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Tail-Aware Direct Preference Optimization (TDPO), a modification of DPO intended to reduce reward overoptimization in low-reference-coverage regions. It replaces DPO's log-density-ratio score difference, β[log⁡ν(y+)−log⁡ν(y−)]\beta[\log \nu(y^+) - \log \nu(y^-)], with a power transformation of the density ratio, β[ν(y+)α−1−ν(y−)α−1]\beta[\nu(y^+)^{\alpha-1} - \nu(y^-)^{\alpha-1}], where ν(y)=π(y∣x)/π0(y∣x)\nu(y)=\pi(y|x)/\pi_0(y|x) is the density ratio of the policy π\pi to the reference π0\pi_0, and α>1\alpha>1 is the Rényi order. The main contributions are the following.

  1. A direct preference optimization objective derived from Rényi-regularized RLHF, replacing DPO's log-density-ratio difference with a power-transformed density-ratio difference.
  2. Upper bounds on the regularized and unregularized optimality gaps, together with a minimax lower bound matching the sample-complexity exponent for the regularized gap.
  3. Polynomial rare-set probability bounds under Rényi regularization, a contrast with KL-based control, and a characterization of the local relationship between Rényi and KL divergences.
  4. An evaluation of TDPO against DPO, ChiPO, and clipping/capping baselines on Reddit's TL;DR dataset, measuring density-ratio tails and the probability mass assigned to low-reference-coverage regions.
Critical strengths
  1. The paper gives a formulation that derives TDPO from Rényi-regularized RLHF, replacing DPO's log-density-ratio score with a power-based score.
  2. It provides finite-sample upper bounds and a minimax lower bound for the regularized suboptimality, and establishes polynomial bounds on the probability mass assigned to low-reference-coverage regions.
  3. It offers a geometric insight that characterizes the local relationship between Rényi and KL divergences and explains their different tail-control properties.
  4. It reports experiments comparing TDPO with DPO, ChiPO, and clipping/capping methods.
Critical weaknesses

The main TL;DR experiment does not establish a statistically significant quality advantage for TDPO over DPO. For seed 42, TDPO achieves a 49.20% win rate against 49.92% for DPO, and the reported confidence intervals overlap. Although TDPO has a higher mean across three seeds, its standard deviation is also considerably larger, which makes the result less conclusive.

The paper's ChiPO discussion uses the mixed score ν+log⁡ν\nu+\log\nu, whereas the experimental implementation uses να−1+γlog⁡ν\nu^{\alpha-1}+\gamma\log\nu with α=1.5\alpha=1.5 in the selected configuration. This yields a different score, ν+log⁡ν\sqrt{\nu}+\log\nu. Unless this is explicitly justified as a generalized ChiPO variant, the comparison with the standard method is potentially misleading.

Finally, the related work is incomplete. The most relevant omissions are Gopalan et al. (2026) on DPO misspecification, and Nguyen et al. (2025) and Im et al. (2026) on reward overoptimization and distribution-shift control.

References

  • Aditya Gopalan, Sayak Ray Chowdhury, and Debangshu Banerjee. Why DPO is a misspecified estimator and how to fix it. ICLR, 2026.
  • Shawn Im, Federico Danieli, Skyler Seto, Barry-John Theobald, and Katherine Metcalf. Normalized rewards for preference optimization. ICML, PMLR 306:49627–49646, 2026.
  • Phuc Minh Nguyen, Ngoc-Hieu Nguyen, Duy H. M. Nguyen, Anji Liu, An Mai, Binh T. Nguyen, Daniel Sonntag, and Khoa D. Doan. Mitigating reward over-optimization in direct alignment algorithms with importance sampling. NeurIPS, 2025.
Questions for authors

See the weakness section. Comparison with AuxDPO (both theoretical and practical) is necessary (Gopalan et al., 2026).

2 comments

Thank you for the careful review. We clarify three points.

TL;DR results. Table 2 compares each method separately against the SFT reference, so overlapping confidence intervals do not test the difference between TDPO and DPO. In the direct comparison in Table 10, TDPO’s win rates against DPO are 51.5%, 53.0%, and 53.2% across the three seeds, averaging 52.57%. These point estimates consistently favor TDPO, although the reported results do not establish statistical significance.

ChiPO baseline. Our selected link, u+log⁡u\sqrt{u}+\log u, differs from the canonical u+log⁡uu+\log u in Huang et al.’s Algorithm 1. Huang et al. themselves tune the power exponent and the log-term weight in their own TL;DR experiments (Appendix E.2). We use the same generalized link family, uα−1+γlog⁡uu^{\alpha-1}+\gamma\log u, with α=1.5\alpha=1.5 and γ=1\gamma=1 selected on our validation set. We will cite their experimental setup explicitly and label our baseline ChiPO (tuned link).

Theory and related work. Our matching upper and lower bounds show that the regularized optimality gap scales as n−1n^{-1} for 1<α≤21<\alpha\le2 and as n−α/[2(α−1)]n^{-\alpha/[2(\alpha-1)]} for α>2\alpha>2. This identifies a statistical cost of increasing the Rényi order beyond two. AuxDPO addresses reward misspecification within a parametrized policy class, while our analysis characterizes the effect of the regularizer under score realizability. A direct comparison under misspecification would be informative. We will clarify this distinction and discuss Nguyen et al.’s importance sampling and Im et al.’s reward normalization as complementary approaches to overoptimization.

Thanks for the rebuttal. My main criticism: the comparison with AuxDPO is still unresolved. I believe this is very much needed for the paper. Hence, I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.