- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies data selection for offline DPO about mismatched preference data. The authors propose Low- to select preference pairs with small chosen-gradient norms.
Critical strengths
The method is simple and effective. The improvements are substantial across different models and datasets.
The analysis of chosen/rejected asymmetry is interesting.
Critical weaknesses
The theoretical analysis only bounds the chosen-side update, whereas actual DPO training uses both chosen and rejected gradients. While the authors argue that chosen-side updates have a more direct effect on generation, this does not mean that rejected-side gradients are irrelevant. In particular, Low- may select pairs with small chosen gradients but very large rejected gradients, potentially causing substantial policy drift. I guess, for example, directly measurement of policy KL during training can help.
The authors should consider citing Mitigating Mismatch within Reference-based Preference Optimization (ICLR 2026), which also studies mismatch in preference optimization.
The paper is largely relied on AI, but I found human intelligence. The author should consider improving the flow of paper and the writing, especially since there are many isolated lines, and the images also show excessive AI. But I am happy to found the human intelligenc, so the writing issuse is encouraged to fix (still a major part for junior researcher), not the major concern for the paper. If the author makes improvements in response to the Critical Weaknesses, I would be happy to raise my rating accordingly^_^
Questions for authors
N/A
My major question is side.
We thank the reviewers for their feedback. Additional experiments revealed a problem with the central claim of the paper. The paper argues that training on preference pairs whose chosen responses have small gradients keeps the model closer to its initial policy. However, the experiments showed the opposite. We computed the chosen-response gradient of each pair with LoRA, selected the pairs with small gradients, and trained on them with full fine-tuning. At the end of training, the policy KL between the trained model and the initial model was larger, not smaller. This may be because the gradients were computed with LoRA while training used full fine-tuning, or it may come from the training process itself; we do not yet know which. We have therefore decided to withdraw the submission.