acceptodds
Withdrawn from ICLR 2027

Less Drift, Better Alignment: Selecting the Fittest Preference Data for Offline DPO

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Offline Direct Preference Optimization (DPO) relies on pre-collected preference data, but mismatches between such data and the current policy can cause large local perturbations, ultimately leading to excessive policy drift. To address this issue, we seek to identify preference pairs that induce smaller local perturbations to the current policy when trained on. We study computable proxies for this local update cost and identify a generation-level asymmetry between chosen and rejected responses. Controlled generation and gradient analyses show that chosen-side updates affect decoding more directly and reliably, whereas rejected-side effects depend on their overlap with the current policy's high-probability generation region. The chosen-gradient norm therefore provides a better proxy for local update cost. We further show that this cost is only weakly coupled with preference quality and semantic coverage, so low-cost subsets need not sacrifice either. Motivated by these results, we propose Low-, which ranks complete preference pairs by a length-normalized chosen-gradient score and selects the lowest-scoring subset under a fixed data budget before standard DPO training. Under local smoothness, lower cost tightens the upper bound on log-probability perturbation and raises the lower bound on retained probability mass for high-quality responses in the generation region. Across four models, three datasets, and five metrics, Low- ranks in the top two in 55 of 60 settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Direct Preference Optimization, Preference Data Selection, Gradient-based Data Selection, Policy Drift, Large Language Model Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YAjQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies data selection for offline DPO about mismatched preference data. The authors propose Low-g+g^+ to select preference pairs with small chosen-gradient norms.

Critical strengths

The method is simple and effective. The improvements are substantial across different models and datasets.

The analysis of chosen/rejected asymmetry is interesting.

Critical weaknesses

The theoretical analysis only bounds the chosen-side update, whereas actual DPO training uses both chosen and rejected gradients. While the authors argue that chosen-side updates have a more direct effect on generation, this does not mean that rejected-side gradients are irrelevant. In particular, Low-g+g^+ may select pairs with small chosen gradients but very large rejected gradients, potentially causing substantial policy drift. I guess, for example, directly measurement of policy KL during training can help.

The authors should consider citing Mitigating Mismatch within Reference-based Preference Optimization (ICLR 2026), which also studies mismatch in preference optimization.

The paper is largely relied on AI, but I found human intelligence. The author should consider improving the flow of paper and the writing, especially since there are many isolated lines, and the images also show excessive AI. But I am happy to found the human intelligenc, so the writing issuse is encouraged to fix (still a major part for junior researcher), not the major concern for the paper. If the author makes improvements in response to the Critical Weaknesses, I would be happy to raise my rating accordingly^_^

Questions for authors

N/A

My major question is g−g^- side.

Reviewer_H2p8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Low-(g+)(g^+), an offline DPO data-selection method that selects preference pairs with small length-normalized chosen-response gradient norms. The approach is motivated by an observed asymmetry between chosen- and rejected-side updates and supported by a local stability analysis. Experiments across four models and three preference datasets show consistent alignment improvements, with ablations examining alternative selection signals, data budgets, and computational efficiency.

Critical strengths
  1. The evaluation spans four models and three datasets, with comparisons against several selection methods. Alternative-gradient ablations, equal sample-presentation controls, and an approximately compute-matched Random baseline strengthen the evidence that the proposed selection signal is useful. This breadth is the main reason for my positive recommendation.

  2. Low-(g+)(g^+) preserves standard DPO training and requires no additional preference annotations. The LoRA-proxy validation and analyses of response length, preference quality, and semantic diversity address several plausible confounds, supporting the method’s practical applicability.

Critical weaknesses

Equations (8)–(10) analyze an isolated chosen-side update, whereas full DPO also includes rejected-side gradients. Appendix D acknowledges this limitation, but no direct drift measurements during full DPO training are reported. Moreover, the one-step asymmetry experiments do not match update norms, leaving update magnitude as a possible confound.

Questions for authors
  1. Can the authors provide evidence that Low-(g+)(g^+) reduces KL divergence from the SFT reference, or another direct measure of policy drift, during full DPO training compared with Random and full-gradient selection?

  2. Does the chosen/rejected asymmetry persist when parameter-update norms are matched and examples are stratified by their initial response likelihood?

Reviewer_XXu7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper selects DPO training pairs using the length-normalized gradient norm of the chosen response. It keeps low-scoring pairs and reports better alignment performance using less training data.

Critical strengths
  1. The method is simple and reports promising gains across four models and three datasets.
  2. Comparisons with other gradient scores are useful.
Critical weaknesses
  1. What does ‘less drift’ in the title mean? I am still confused after reading this paper. The authors should define terms like "drift" and "policy drift" well in the manuscript, given that "Less drift, better alignment" is the core claim of this paper. Equations 2,8,9 in this paper give different definitions of drift.

  2. Could the author give a deep analysis of how lower gradients lead to better alignment? A large gradient may indicate an error the model needs to correct, instead of a harmful change. How does the proposed selection rule distinguish these cases and avoid discarding useful training examples?

Questions for authors

See weaknesses.

  1. Does Low-g+g^+ still outperform Random on samples where the initial model gives poor answers? Please report results separately for samples where the initial model performs well and poorly, so we can see whether the method mainly strengthens what the model already does well.

  2. Can Random selection with a lower learning rate match Low-g+g^+ when both produce similar policy drift? This would help show whether the gain comes from selecting different data or simply making smaller updates.

Comments on the paper

1 comment

We thank the reviewers for their feedback. Additional experiments revealed a problem with the central claim of the paper. The paper argues that training on preference pairs whose chosen responses have small gradients keeps the model closer to its initial policy. However, the experiments showed the opposite. We computed the chosen-response gradient of each pair with LoRA, selected the pairs with small gradients, and trained on them with full fine-tuning. At the end of training, the policy KL between the trained model and the initial model was larger, not smaller. This may be because the gradients were computed with LoRA while training used full fine-tuning, or it may come from the training process itself; we do not yet know which. We have therefore decided to withdraw the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.