- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Here the authors come up with an Entropy based method to choose which tokens to optmize for, in DPO, as opposed to standard version where every token is weighted equally. The paper takes into account the informational aspects of each tokens to calculate the importance based on Entropy. They rank the tokens in both chosen and rejected responses, by the current policies predictive entropy. Then they keep the lowest 20% entropy weight of the tokens and computes the weights only on those. This is recomputed at every training step. The selection process does not flow back in gradients, but only that the loss is calculated on the selected tokens. They train this setup using six models, from Qwen, Llama and Mistral families. Training uses UltraFeedback pairs. And evaluation is done on AlpacaEval 2, AreaHard and MTBecnh. Baselines include a suite of other techniques also. The results seem to suggest that LES-DPO beats DPO in 22 out of 24 model/metric comparisons. It shows very promising results on AlpacaEval 2 and ArenaHard. Some wins go to other methods. Other ablations show that, selecting by policy entropy is the best, compared to random selection or reference model entropy based selection. The highest entropy selection seems quite harmful. 30% selection does worse.
Critical strengths
- This is a fairly straightforward idea with no need for any external methods or computation, but using the uncertainity information that we already have.
- The low entropy usage is somewhat counterintuitive at first, but the paper covers the impact of this with several ablation studies.
- Comprensive evaluation on 6 models, 3 model families, and on 3 benchmarks.
- It beats DPO in most of the comparisons. And the gains are noticeable on the base models.
- It does a strong comparison to other similar methods and comes on top on most cases.
- Ablations cover all other interesting cases such as random selection, high entropy selection etc.
- Investigation into Gradient based analysis gives you some explanation on what might be happening.
Critical weaknesses
- The empirical results are convincing but not enough explanation on how the low entropy signals might be a good proxy for preference signals overall?
- Whether this might work in other training settings, say full fine tuning, is a bit puzzling.
- Somewhat low performance on MT-Bench.
Questions for authors
Overall I feel, this technique is pretty promising. Some open questions from my side,
- What do you think is a principled way to come up with the threshold of 20%? any insights you gained from your sweep analysis?
- Could this method have any length impacts in the responses? Can ArenaHard performance be explained by length or style changes?
- The wins seems broad and consistent. But any idea on the statistical intervals for these?