- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Entropy-Guided DPO (EG-DPO), which modifies the DPO objective based on the token-wise entropy of the policy model. The motivation in Figure 1a is that a small subset of high-entropy tokens accounts for almost all reward contributions, and this paper assumes that many low-entropy tokens have small contributions, but the sum of their contributions is non-negligible, so it introduces some non-useful noise. Thus, it assigns more weight to high-entropy tokens. It reformulates the policy-generated implicit reward as a weighted one, so that it can still use BT models to derive the actual loss. Performance on instruction-following tasks looks better than that of some previous works, and it shows scalability up to 14B.
Critical strengths
- Simple but effective method, especially, explanation using the new definition of reward model looks clear.
- Strong results across multiple backbones, as shown in Table 1.
- Empirical scalability as shwon in Figure 5.
- Ablation about weighting strength in Figure 6 clearly shows the progressive improvements in scale.
Critical weaknesses
-
In general, sample entropy (diversity) and sample fidelity (e.g., LLM-judge-based win rate or MT-Bench score) are negatively correlated, and improving both is important. For example, Figure 3 of f-DPO [1] and Figure 1 of BPO [2] show the entropy-fidelity trade-off. In LLMs, we also often observe that reducing entropy improves fidelity. Therefore, it is unclear whether high-entropy tokens are truly important or whether your method simply improves fidelity by reducing the entropy of the policy model. In fact, Figure 3 even suggests that your method reduces sample diversity. What I would like to see is a trade-off plot with entropy on the x-axis and benchmark score on the y-axis. For all methods in Table 1, please vary the top-p decoding parameter (or temperature) and plot the trade-off curves to show that your method lies on the Pareto frontier. I would like to see this for all benchmarks, with multiple diversity metrics such as entropy, Self-BLEU, and Distinct-n.
-
The theoretical results suggest that equal weighting may not be optimal (Proposition 1) and show an indirect relationship between pairwise preference divergence and entropy. However, they do not explain the connection to the objective function in Eq. 8, which is derived from the entropy-weighted implicit reward model.
-
DPO can cause an alignment tax, but this paper lacks a discussion of how robust the proposed method is to this side effect. Please provide downstream task performance results similar to those in Table 9 of SimPO.
[1] (ICLR 24) BEYOND REVERSE KL: GENERALIZING DIRECTPREFERENCE OPTIMIZATION WITH DIVERSE DIVER-GENCE CONSTRAINTS
[2] (NeurIPS 25) Preference Optimization by Estimating the Ratio of the Data Distribution
Questions for authors
-
This paper improves the DPO loss by modifying the definition of the implicit reward model while keeping the BT model (in Eq.8.). Similarly, SimPO and f-DPO also modify the implicit reward model. Could your method be applied in a complementary way with these methods? Also, f-PO (AISTATS 2025) and BPO extend the loss from different perspectives. Could you explain whether your method is complementary to these loss extensions?
-
Can it be applicable to DPO for diffusion models? How can you define corresponding quantity of entropy for diffusion model's alignment?