acceptodds
Under review as a conference paper at ICLR 2027

Less Is More in DPO: Learning from the Lowest-Entropy 20% of Tokens

Abstract

Direct Preference Optimization (DPO) is widely used to align large language models with human preferences. However, its sequence-level objective does not explicitly distinguish the contributions of individual tokens. We find that optimizing only a small subset of low-entropy tokens can outperform using all response tokens. Based on this observation, we propose **Low-Entropy Support DPO (LES-DPO)**. LES-DPO ranks tokens by the policy model's predictive entropy. It selects a low-entropy subset separately from each chosen and rejected response. The preference loss uses only these selected positions. LES-DPO requires no auxiliary models, additional policy forward passes, or extra trainable parameters. Experiments on six Qwen2.5, Llama-3.1, and Mistral models show consistent improvements on AlpacaEval 2, Arena-Hard, and MT-Bench. Gradient analysis shows that high-entropy tokens tend to produce larger gradients. Their directions are less consistent and conflict more often. These findings suggest that focusing on low-entropy tokens can provide more stable signals for preference learning.

Primary area:
reinforcement learning
Keywords:
Direct Preference Optimization, Large Language Model Alignment, Token-Level Credit Assignment, Predictive Entropy, Preference Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4L6ZOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Here the authors come up with an Entropy based method to choose which tokens to optmize for, in DPO, as opposed to standard version where every token is weighted equally. The paper takes into account the informational aspects of each tokens to calculate the importance based on Entropy. They rank the tokens in both chosen and rejected responses, by the current policies predictive entropy. Then they keep the lowest 20% entropy weight of the tokens and computes the weights only on those. This is recomputed at every training step. The selection process does not flow back in gradients, but only that the loss is calculated on the selected tokens. They train this setup using six models, from Qwen, Llama and Mistral families. Training uses UltraFeedback pairs. And evaluation is done on AlpacaEval 2, AreaHard and MTBecnh. Baselines include a suite of other techniques also. The results seem to suggest that LES-DPO beats DPO in 22 out of 24 model/metric comparisons. It shows very promising results on AlpacaEval 2 and ArenaHard. Some wins go to other methods. Other ablations show that, selecting by policy entropy is the best, compared to random selection or reference model entropy based selection. The highest entropy selection seems quite harmful. 30% selection does worse.

Critical strengths
  • This is a fairly straightforward idea with no need for any external methods or computation, but using the uncertainity information that we already have.
  • The low entropy usage is somewhat counterintuitive at first, but the paper covers the impact of this with several ablation studies.
  • Comprensive evaluation on 6 models, 3 model families, and on 3 benchmarks.
  • It beats DPO in most of the comparisons. And the gains are noticeable on the base models.
  • It does a strong comparison to other similar methods and comes on top on most cases.
  • Ablations cover all other interesting cases such as random selection, high entropy selection etc.
  • Investigation into Gradient based analysis gives you some explanation on what might be happening.
Critical weaknesses
  • The empirical results are convincing but not enough explanation on how the low entropy signals might be a good proxy for preference signals overall?
  • Whether this might work in other training settings, say full fine tuning, is a bit puzzling.
  • Somewhat low performance on MT-Bench.
Questions for authors

Overall I feel, this technique is pretty promising. Some open questions from my side,

  • What do you think is a principled way to come up with the threshold of 20%? any insights you gained from your sweep analysis?
  • Could this method have any length impacts in the responses? Can ArenaHard performance be explained by length or style changes?
  • The wins seems broad and consistent. But any idea on the statistical intervals for these?
Reviewer_M8xROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Low-Entropy Support DPO (LES-DPO), which uses the current policy’s predictive entropy to select the lowest-entropy 20% of tokens independently from each chosen and rejected response. The preference loss is constructed from the mean policy–reference log-ratios over the selected positions. LES-DPO requires no auxiliary models, additional policy forward passes, or new trainable parameters. The authors evaluate the method on six models from the Qwen2.5, Llama-3.1, and Mistral families and investigate selective optimization through ablation studies and gradient analysis. The results show that LES-DPO outperforms DPO on multiple preference evaluation metrics.

Critical strengths
  1. The method is simple and easy to implement. It uses the current policy’s predictive entropy to select tokens, requiring no auxiliary models, additional policy forward passes, or new trainable parameters.
  2. The experimental analysis is relatively systematic. The method is evaluated on six models from three model families. Ablation studies on entropy sources, retained token fractions, and token-weighting strategies, together with gradient-preserving training controls, examine the contributions of different tokens to preference optimization.
Critical weaknesses
  1. The paper discusses several token-level preference optimization methods, but its relationship to existing selective training and sparse token masking methods is not sufficiently clarified. The methodological contribution primarily lies in low-entropy support selection, and the distinct contribution of this selection criterion relative to existing methods requires further clarification. Its effectiveness is primarily supported by empirical results. The theoretical analysis does not establish a direct connection between predictive entropy and useful preference signals, leaving the conditions under which this selection rule succeeds or fails unclear. Gradient diagnostics and the gradient-preserving control provide some support for the gradient interference explanation. However, the theoretical analysis mainly characterizes local descent of the Low20 loss with fixed selection masks, which does not fully explain differences in final evaluation performance or establish generalization across settings.
  2. On Qwen2.5-3B-Instruct, DPO and the original checkpoint have identical reported scores on all four metrics, while the main ablation studies also use this model. The authors should explain this observation and provide training and generation checks to verify the validity of this baseline. The retained token fraction is selected based on the reported evaluation results, but the use of an independent validation set is not clearly described, raising concerns about configuration selection bias. Moreover, multi-seed validation remains incomplete, and uncertainty estimates are not reported, leaving the robustness of some small performance gains unclear. The paper includes random selection, full-token uniform weighting, and soft token-weighting controls, but these key comparisons primarily focus on a single model. Further comparisons of selection criteria under the same objective, normalization, and retained token fraction are needed. Controls matched for token categories and positions would also help distinguish the effect of predictive entropy from that of these associated attributes.
Questions for authors
  1. How does LES-DPO differ from existing selective training and sparse token masking methods, such as SePO, SparsePO, and ConfPO? Beyond adopting predictive entropy as the selection criterion, what is its distinct methodological contribution?
  2. Since low predictive entropy does not necessarily indicate correctness or preference relevance, under what conditions does low-entropy support selection provide useful preference signals? Can the authors identify settings in which this selection rule fails?
  3. How do the gradient diagnostics and gradient-preserving control connect local descent of the Low20 loss to final evaluation performance? What additional evidence can distinguish harmful gradient interference from potentially useful updates that differ from the Low20 direction?
  4. Why do DPO and the original Qwen2.5-3B-Instruct checkpoint have identical reported scores on all four metrics? Can the authors provide training logs and comparisons of generated responses to verify that the DPO baseline was correctly trained, loaded, and evaluated?
  5. Was the retained token fraction selected using an independent validation set, or the same evaluation benchmarks used to report the final results? Can the authors provide multi-seed results and confidence intervals to assess the robustness of the reported improvements?
  6. Can the authors extend the controlled comparisons to additional models and representative token-selection methods while matching the objective, normalization, and retained token fraction? Can controls matched for token categories and positions help determine whether the gains arise from predictive entropy itself or these associated attributes?
Reviewer_36g2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Low-Entropy Support DPO, which selects the lowest entropy 20% of tokens from chosen and rejected responses and optimizes their mean log probability ratios between the policy and reference model. The method uses predictive entropy from the current policy without additional models or forward passes. Experiments across six models from the Qwen2.5, Llama-3.1, and Mistral families report improvements over DPO in 22 of 24 comparisons on AlpacaEval 2, Arena-Hard, and MT-Bench. Additional analyses examine token selection, gradient alignment, and interference between different entropy regions.

Critical strengths

The paper proposes a simple and efficient token-selection strategy that improves preference optimization across multiple models, with useful gradient analyses supporting the empirical observations.

Critical weaknesses

The comparison with vanilla DPO is confounded by different log-ratio normalization schemes, making it unclear whether the gains arise from low-entropy token selection.

Questions for authors
  1. Does LES-DPO outperform mean-normalized or β-tuned all-token DPO?

  2. Why are the Qwen2.5-3B Base and DPO results identical?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.