- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Summary
They consider a specific type of adversarial attack in offline preference optimization, where a subset of the preference labels is flipped. They observe that, under a log-linear assumption, the magnitude of the effect of a flip on the DPO gradient can be measured by the difference between the feature vectors of the responses. Based on this observation, the authors propose Trim-K Robust DPO, which filters out the top-(K) samples with the largest feature differences as the most affected samples. They prove that, under several assumptions, DPO trained on the filtered dataset produces parameters whose deviations from those obtained by clean DPO are bounded. In the experiments, they compare their method against several baselines, including noise-robust methods, in a log-linear policy setting and show that their proposed method remains closer to the clean-DPO oracle under several label-flip attacks.
Critical strengths
The motivation is clear. In particular, the claim that preference optimization needs to handle noise beyond random label flipping is convincing. I also find Lemma 2.2 interesting, and the simplicity of the resulting algorithm is a strength.
Critical weaknesses
The weaknesses are mainly related to the empirical validation and the coverage of the related work.
- The main metric, candidate-set KL to clean DPO, is well suited for validating the theoretical bounds, and I value these results as a necessary empirical validation. However, since the authors present their method as a robust DPO method, it is also necessary to report standard performance metrics such as win rate, because closeness to DPO in an approximate KL metric does not necessarily guarantee good downstream performance.
- Furthermore, the model has only 384 trainable parameters under the log-linear assumption introduced by the authors. A comparison with standard full-parameter DPO variants is needed to better position the proposed method within the existing literature.
- Regarding related work, the paper misses several relevant filtering-based DPO methods for noise robustness, such as [1, 2]. Comparisons with these methods would provide stronger evidence for the effectiveness of the proposed method.
[1] Liang, Xize, et al. “ROPO: Robust Preference Optimization for Large Language Models.” ICML 2025. [2] Fujisawa, Masahiro, Masaki Adachi, and Michael A. Osborne. “Scalable Valuation of Human Feedback through Provably Robust Model Alignment.” NeurIPS 2025.
Questions for authors
Could the author addresses the weaknesses?
2 comments
Thank you for the detailed rebuttal and clarifications.
W1: Evaluation metrics
As stated in my original review, I appreciate the experiments evaluating the proximity to clean DPO, as they directly validate the theoretical results. However, I still believe that evaluating downstream performance, such as win rate, is essential to demonstrate the practical value of the proposed method as a robust DPO algorithm. Therefore, my original concern remains.
W2: Full-parameter DPO
I understand the theoretical motivation for using log-linear policies and appreciate the preliminary experiments with nonlinear models. However, the effectiveness of the proposed method in standard full-parameter DPO settings remains unverified. I believe such experiments are important for establishing its practical applicability.
W3: Related work and baselines
While ROPO assumes symmetric label noise in its theoretical analysis, it also considers other noise settings experimentally. Therefore, I believe a direct comparison under the same attack settings is still important. I appreciate the authors' commitment to adding comparisons with ROPO and Hölder-DPO, but these results have not yet been provided.
Overall, my main concerns regarding its practical effectiveness remain unresolved. The authors have promised additional experiments, but I cannot assess their implications without seeing the results. Therefore, I would like to maintain my original rating at this stage.
Thank you for the detailed review.
W1. Candidate-set KL and downstream performance
Our objective is to preserve the policy learned by DPO on the full clean dataset when preference labels are attacked. This is explicitly defined in Section 2.2 (p. 4, lines 166–190, Eq. (6)). Candidate-set KL directly measures this policy-preservation objective in our experimental setting.
In the main SHP experiment, we compute the exact categorical KL from clean DPO to the defended policy on a fixed list of 100 candidate entries per prompt, then average over 5,000 held-out prompts (Appendix G, p. 28, lines 1472–1490).
Win rate evaluates downstream response quality. Our question is how closely training on attacked data preserves clean-data DPO, whose absolute response quality is a separate issue. Appendix G (p. 28, lines 1491–1493) already states this distinction. The simulations evaluate the robustness target established by the theory; we will make that purpose clearer in the main text.
W2. Policy class and full-parameter DPO
The log-linear family is an established theoretical setting. Section 2.1 (p. 3, lines 117–120) explicitly follows Nika et al. (2025), Chowdhury et al. (2024), and Xu et al. (2025). Our analysis applies to general dimension d (Assumption 2.1, p. 3, lines 120–125). The 384 parameters come from the MiniLM features chosen for the experiments (Section 4, p. 8, lines 422–424).
This setting makes each flip's gradient shift independent of the current parameters (Lemma 2.2, p. 3, lines 140–146). The impact score ranks pairs by how much each would change the gradient if its label were flipped. A clean pair can therefore rank among the top K and be trimmed (Section 3.2, p. 5, lines 218–222). The main experiments use the same features and policy class across methods to compare the effects of the defenses under the same attacks.
The paper also explores extensions beyond log-linear policies. Appendix O.1 (p. 40, lines 2145–2151, Lemma O.1) gives the flip-gradient identity for differentiable policies, and Appendix O.3–O.4 (p. 41, lines 2170–2209, Table 7) evaluates a nonlinear head trained on frozen features. At the 30% attack budget, TRIM_REG and SOFT_TRIM_REG reduce mean candidate-set KL by 18.7% and 9.6% relative to matched DPO_REG.
Thank you for suggesting a comparison using full-parameter language-model fine-tuning. We will address this in the revision.
W3. ROPO and Hölder-DPO
Thank you for suggesting both methods. ROPO’s analysis shares rDPO’s assumption that each preference label is flipped with the same probability in either direction. Hölder-DPO studies a broader model in which a fraction of the data comes from a contaminating distribution.
Our guarantees cover every fixed set of at most K adversarial flips (Section 2.2, p. 4, lines 162–190). We already evaluate rDPO (listed as NR_DPO) as a baseline designed for random flips. Table 1 reports its performance under random, top-impact, and targeted BMP-A flips (Appendix J.1, p. 33, lines 1737–1744, and Table 1, p. 36, lines 1890–1906). In the revision, we will discuss ROPO and Hölder-DPO in the Introduction and related work and compare both methods directly under the same attacks.