- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper is a special case of f-DPO (ICLR 24) and proposes Tail-Aware Direct Preference Optimization (TDPO) by changing the regularization term in RLHF from KL to Rényi divergence. As proven in Lemma 3.1, unlike the conventional method, where the optimal solution is the reference distribution multiplied by an exponential function of the reward, the proposed method results in a power form of the reward. Thus, the paper claims that the probability does not differ significantly from the reference distribution.
Although the main text contains excessive theoretical explanations, I think the key point in the context of this paper is that the gap between the reference and policy distributions can also bound pointwise probability assignments. Theorem 5.1 and Theorem 5.2, which explains that KL and other divergences cannot provide such bounds, indirectly support this claim.
In very limited experiments, the method shows a slight reduction in probability assignments to regions far from the reference distribution, at the cost of some loss in win rate.
Critical strengths
-
Explaining how the reward values are reflected in the form of the optimal solution is a good theoretical approach. Explain the formulation of optimal solution through Lemm3.1 looks clear.
-
Although the main points are somewhat scattered due to excessive theoretical explanations, the approach is reasonable from the perspective of assigning probability to responses where the reference distribution has meaningful probability values (Theorem 5.1).
-
Although the experiments are very limited, the Outside-ref-99% results in Table 2 support the main claim of the paper.
Critical weaknesses
-
It is unclear whether the exponential reward term in the original DPO is problematic. Also, since the optimal reward model is in log-likelihood units, multiplying by its exponential is an appropriate scale for representing probability values.
-
Since this paper is a subset of f-DPO, it only selects Rényi divergence as a specific choice of f. A broader comparison with other instances of f-DPO is needed to examine whether the reward is reflected in the optimal solution in exponential or power-law forms.
-
According to f-PO (Han et al., AISTATS 2025) and BPO (Kim et al., NeurIPS 2025), even when the optimal solution is fixed, there is still room for discussion on how to match it. How are the training dynamics compared to the original DPO? For example, it might be useful to analyze the gradient update equation. Can the proposed method also be applied orthogonally to f-PO, BPO, and other methods?
-
The experiments are very limited, and proposing only metrics such as "Outside-ref-99%" is problematic. It would be better to design experiments based on empirical cases where deviating from the reference distribution causes problems, and find test cases where the proposed method actually improves language quality.
Questions for authors
Could you conduc more pratical experiments that TDPO is actually have effects in better language quality? My intuition is that when reference model is good and preference dataset is not good enough, then TDPO can be better than DPO in win rate.
2 comments
Thank you for the clarifications. W1 and W3 are largely resolved. Regarding W2, my main concern was whether other f-DPO variants also avoid the exponential reweighting issue identified in DPO, and if so, what makes TDPO particularly advantageous. While the additional theoretical contributions are interesting, this comparison remains unclear.
W4 also remains unresolved, as the experiments do not directly validate the practical benefits of TDPO under unreliable preference data.
Overall, I appreciate the theoretical contributions but believe stronger empirical evidence is needed. I therefore maintain my original rating.
Thank you for the constructive comments. We clarify the motivation, the relationship to prior work, and the scope of our experiments.
Exponential reweighting (W1). In the Bradley–Terry model, the reward difference determines the pairwise preference probability through the logistic link. The exponential reweighting of generation probabilities comes from a separate problem: maximizing expected reward under a KL constraint. It arises even when the reward is given directly and no preference model is used. Our concern is that this update can assign substantial probability to a response that is rare under the reference when its reward is overestimated.
Relation to f-DPO and statistical contribution (W2). We agree that TDPO uses the f′-score form of f-DPO. Other choices can yield different updates; for example, forward KL gives reciprocal reweighting. The exponentiated Rényi penalty is an f-divergence up to an additive constant, but its generator has a finite right derivative at zero, so the reparametrization theorem of Wang et al. (2024) does not directly cover the thresholded solution. The generator is globally strongly convex only at α=2, so the general strongly convex guarantees of Zhao et al. (2026) and the formal analysis of ChiPO’s strongly convex mixed penalty (Huang et al., 2025) do not directly apply at other orders.
Our Bregman analysis is of independent technical interest: it reveals a curvature-driven phase transition in the minimax rate for the regularized gap at α=2. Policy normalization cancels the reward offsets that pairwise comparisons cannot identify, allowing us to bound the gap from logit estimation error via the penalty’s Bregman divergence. For α>2, quadratic curvature vanishes at zero density ratios, and the matching lower bound shows that the resulting slowdown is unavoidable in our setting.
Gradient updates and compatibility (W3). In the margin gradient, DPO gives the chosen and rejected log-probability gradients coefficients of equal magnitude and opposite sign. TDPO additionally scales each coefficient by (α−1)uα−1, where u is that response’s policy-to-reference density ratio. TDPO can be combined with BPO (Kim et al., 2025) by using e−ΔTDPO as the model ratio, and the Rényi optimizer can serve as the target in an f-PO-style matching objective (Han et al., 2025).
Language quality (W4). We assess summary quality on TL;DR through pairwise judgments. Table 2 compares each method separately against SFT and does not establish a quality cost for TDPO. In the direct comparison in Table 10, TDPO’s win rate against DPO exceeds 50% in all three seeds, averaging 52.57%; these results are descriptive. If unreliable preferences favor poor summaries that a strong reference model rarely generates, TDPO could help preserve quality by limiting the probability assigned to those summaries. Appendix H illustrates this mechanism in a controlled example. Our TL;DR experiments did not vary preference reliability, so they do not directly test this hypothesis.
References