acceptodds
Under review as a conference paper at ICLR 2027

Not All Tokens Are Equal: Entropy-Guided Credit Assignment for Direct Preference Optimization

205 views

Abstract

Direct Preference Optimization (DPO) aligns language models with a single sequence-level preference signal, leaving open how that signal should be attributed to individual tokens. We show empirically that this attribution is highly uneven: under the current policy, a small minority of high-entropy positions accounts for most of the implicit reward margin, whereas the low-entropy majority contributes almost nothing. Motivated by this observation, we propose Entropy-Guided DPO (EG-DPO), which reweights token-level policy–reference log-ratios by a detached, monotone function of the policy's own predictive entropy. The weights are normalized within each response, so the objective coincides with DPO when they are uniform, and they are recomputed at every step, so they track the evolving policy without adding a gradient path, an auxiliary model, or a forward pass. We justify the design with two complementary analyses: a signal-to-noise argument showing that uniform aggregation dilutes preference-relevant gradients, and a bound showing that the attainable pairwise preference-divergence at a position grows with the policy's entropy there. Across Llama-3-8B, Mistral-7B, and four Qwen2.5-Instruct scales, EG-DPO improves over DPO and recent token-level baselines on AlpacaEval 2, Arena-Hard v0.1, and MT-Bench, and yields the best average preference-classification accuracy on HH-RLHF, with its largest gain at the 14B scale.

Primary area:
reinforcement learning
Keywords:
Large Language Model Alignment, Direct Preference Optimization, Token-Level Credit Assignment, Entropy-Guided Weighting, Human Preference Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xxXkOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Entropy-Guided DPO (EG-DPO), which modifies the DPO objective based on the token-wise entropy of the policy model. The motivation in Figure 1a is that a small subset of high-entropy tokens accounts for almost all reward contributions, and this paper assumes that many low-entropy tokens have small contributions, but the sum of their contributions is non-negligible, so it introduces some non-useful noise. Thus, it assigns more weight to high-entropy tokens. It reformulates the policy-generated implicit reward as a weighted one, so that it can still use BT models to derive the actual loss. Performance on instruction-following tasks looks better than that of some previous works, and it shows scalability up to 14B.

Critical strengths
  1. Simple but effective method, especially, explanation using the new definition of reward model looks clear.
  2. Strong results across multiple backbones, as shown in Table 1.
  3. Empirical scalability as shwon in Figure 5.
  4. Ablation about weighting strength γ\gamma in Figure 6 clearly shows the progressive improvements in γ\gamma scale.
Critical weaknesses
  1. In general, sample entropy (diversity) and sample fidelity (e.g., LLM-judge-based win rate or MT-Bench score) are negatively correlated, and improving both is important. For example, Figure 3 of f-DPO [1] and Figure 1 of BPO [2] show the entropy-fidelity trade-off. In LLMs, we also often observe that reducing entropy improves fidelity. Therefore, it is unclear whether high-entropy tokens are truly important or whether your method simply improves fidelity by reducing the entropy of the policy model. In fact, Figure 3 even suggests that your method reduces sample diversity. What I would like to see is a trade-off plot with entropy on the x-axis and benchmark score on the y-axis. For all methods in Table 1, please vary the top-p decoding parameter (or temperature) and plot the trade-off curves to show that your method lies on the Pareto frontier. I would like to see this for all benchmarks, with multiple diversity metrics such as entropy, Self-BLEU, and Distinct-n.

  2. The theoretical results suggest that equal weighting may not be optimal (Proposition 1) and show an indirect relationship between pairwise preference divergence and entropy. However, they do not explain the connection to the objective function LEG−DPO\mathcal{L}_{\mathrm{EG-DPO}} in Eq. 8, which is derived from the entropy-weighted implicit reward model.

  3. DPO can cause an alignment tax, but this paper lacks a discussion of how robust the proposed method is to this side effect. Please provide downstream task performance results similar to those in Table 9 of SimPO.

[1] (ICLR 24) BEYOND REVERSE KL: GENERALIZING DIRECTPREFERENCE OPTIMIZATION WITH DIVERSE DIVER-GENCE CONSTRAINTS

[2] (NeurIPS 25) Preference Optimization by Estimating the Ratio of the Data Distribution

Questions for authors
  1. This paper improves the DPO loss by modifying the definition of the implicit reward model while keeping the BT model (in Eq.8.). Similarly, SimPO and f-DPO also modify the implicit reward model. Could your method be applied in a complementary way with these methods? Also, f-PO (AISTATS 2025) and BPO extend the loss from different perspectives. Could you explain whether your method is complementary to these loss extensions?

  2. Can it be applicable to DPO for diffusion models? How can you define corresponding quantity of entropy for diffusion model's alignment?

Reviewer_3f7vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Entropy-Guided DPO to assign larger weights to high-entropy tokens when computing the DPO loss function.

Critical strengths

Simple method, comprehensive empirical evaluation.

Critical weaknesses

The paper somewhat outdated. Several relevant papers on preference optimization are missing:

  1. MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training.
  2. Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization.
  3. Mitigating Mismatch within Reference-based Preference Optimization.
  4. Entropy Controllable Direct Preference Optimization.

MuSC seems particularly relevant, it already uses full-vocabulary predictive entropy to weight token-level log-ratios in DPO. MuSC is neither cited nor discussed. (my major concern)

Proposition 1 suggests weighting tokens according to their signal-to-noise ratio, whereas Proposition 2 bounds the expected squared preference divergence. However, in my opinion, a larger potential divergence does not necessarily imply a stronger preference signal or a higher gradient signal-to-noise ratio?

The bound in Proposition 2 does not establish that Shannon entropy consistently ranks positions by preference-divergence capacity?

Questions for authors

I like the human intelligence in this paper, and it should be a good paper in 2024to2025, however, I found it scooped. As encourage human intelligence, I still consider to provide a positive review, though I don’t think the likelihood to be high.

Reviewer_VzDtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how to allocate learning emphasis across tokens in DPO. The authors observe that a small subset of high-entropy positions contributes disproportionately to the implicit reward and propose EG-DPO, which uses detached current-policy entropy to reweight token-level log-ratios, placing greater emphasis on high-entropy positions. However, the observation does not establish that emphasizing these positions improves preference learning, and the theoretical analysis does not fully bridge this gap. Experiments across several model families nevertheless show improvements over DPO.

Critical strengths

Reweighting is an important and widely studied topic in preference optimization, which plays a central role in LLM alignment. Moreover, the proposed method is simple and easy to understand.

Critical weaknesses

I think the observed uneven attribution depends on both the data and the training strategy, which might not be that general. For example, when the model is trained on particularly difficult examples, more tokens are expected to have high entropy. Conversely, low entropy at certain positions suggests that the model has already learned the relevant information, so it is unclear why further learning from those positions would be beneficial. Further experiments across benchmarks of varying difficulty and a broader range of models are needed to establish the generality of the conclusions. A clearer explanation is also needed to better motivate this work.

To my knowledge, GRPO and PPO are also widely used for preference optimization. By focusing on DPO, the authors may implicitly assume that it offers advantages over these alternatives, particularly in performance. At a minimum, the experiments should include GRPO and PPO as baselines to substantiate this choice.

Previous work has also explored token-level variants of DPO. A natural question is whether these methods already address similar issues and what distinguishes the proposed approach from these existing variants.

From an entropy perspective, the reference model’s token probabilities may already influence token contributions through the implicit reward, potentially producing effects similar to those sought by the proposed method. This raises the question of whether standard DPO, with a carefully tuned β, could achieve comparable performance. Although adjusting β is not equivalent to introducing token-specific weights, a systematic comparison across β values is needed to establish the additional benefit of entropy-based weighting. I consider this a substantial concern, as the current results may not clearly distinguish the benefit of the proposed weighting scheme from that of tuning the existing DPO objective.

There is a substantial gap in the motivation for further emphasizing high-entropy tokens. If the uneven attribution described in the introduction is itself the problem, why should making it even more concentrated help? The authors should clarify why upweighting high-entropy tokens is preferable to giving greater weight to the low-entropy majority.

The authors do attempt a theoretical analysis, but Proposition 1 does not establish a relationship between entropy and the proposed weights, so it does not directly justify or motivate the design. The independence assumption in Proposition 2 is also unrealistic for DPO, and its connection to the proposed method remains unclear.

The hyperparameter sensitivity analysis is insufficient. It remains unclear whether the comparison with DPO is fair, whether both methods receive the same tuning budget, and whether DPO has been tuned as thoroughly as the proposed method.

The evaluation focuses on general preference alignment, without more recent and challenging benchmarks for math, coding, or reasoning. Further evaluation on these tasks would help establish the method’s broader applicability. Moreover, the HH-RLHF evaluation uses a policy–reference implicit reward for all methods, raising concerns about fairness when comparing methods with different objectives. Complementary LLM-as-a-judge evaluations on this dataset would be helpful.

Questions for authors

Please see the Weaknesses above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.