acceptodds
Withdrawn from ICLR 2027

Robust Direct Preference Optimization under Preference Label-Flip Attacks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Direct Preference Optimization (DPO) is widely used for offline Reinforcement Learning from Human Feedback. However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy. To address this issue, we design robust DPO methods that can defend against such attacks. Building on the property that lable flips induce parameter-independent gradient shifts in log-linear DPO, we propose Trim- Robust-DPO, which removes the highest-impact pairs, and Soft-Trim- Robust-DPO, which smoothly downweights them. We establish deterministic gradient perturbation bounds uniformly over all attack sets containing at most flips, explicitly separating defense-induced bias from surviving attack effects. Under bounded-region assumptions, these bounds yield time-uniform parameter and policy deviation guarantees. We further quantify how quadratic regularization improves curvature while introducing additional bias. Experiments on Stanford Human Preferences, HH-RLHF, and UltraFeedback-Binarized validate our analysis.

Primary area:
learning theory
Keywords:
Reinforcement learning from human feedback, direct preference optimization, adversarial robustness, preference label-flip attacks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_h3jdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Summary

They consider a specific type of adversarial attack in offline preference optimization, where a subset of the preference labels is flipped. They observe that, under a log-linear assumption, the magnitude of the effect of a flip on the DPO gradient can be measured by the difference between the feature vectors of the responses. Based on this observation, the authors propose Trim-K Robust DPO, which filters out the top-(K) samples with the largest feature differences as the most affected samples. They prove that, under several assumptions, DPO trained on the filtered dataset produces parameters whose deviations from those obtained by clean DPO are bounded. In the experiments, they compare their method against several baselines, including noise-robust methods, in a log-linear policy setting and show that their proposed method remains closer to the clean-DPO oracle under several label-flip attacks.

Critical strengths

The motivation is clear. In particular, the claim that preference optimization needs to handle noise beyond random label flipping is convincing. I also find Lemma 2.2 interesting, and the simplicity of the resulting algorithm is a strength.

Critical weaknesses

The weaknesses are mainly related to the empirical validation and the coverage of the related work.

  • The main metric, candidate-set KL to clean DPO, is well suited for validating the theoretical bounds, and I value these results as a necessary empirical validation. However, since the authors present their method as a robust DPO method, it is also necessary to report standard performance metrics such as win rate, because closeness to DPO in an approximate KL metric does not necessarily guarantee good downstream performance.
  • Furthermore, the model has only 384 trainable parameters under the log-linear assumption introduced by the authors. A comparison with standard full-parameter DPO variants is needed to better position the proposed method within the existing literature.
  • Regarding related work, the paper misses several relevant filtering-based DPO methods for noise robustness, such as [1, 2]. Comparisons with these methods would provide stronger evidence for the effectiveness of the proposed method.

[1] Liang, Xize, et al. “ROPO: Robust Preference Optimization for Large Language Models.” ICML 2025. [2] Fujisawa, Masahiro, Masaki Adachi, and Michael A. Osborne. “Scalable Valuation of Human Feedback through Provably Robust Model Alignment.” NeurIPS 2025.

Questions for authors

Could the author addresses the weaknesses?

2 comments

Thank you for the detailed review.

W1. Candidate-set KL and downstream performance

Our objective is to preserve the policy learned by DPO on the full clean dataset when preference labels are attacked. This is explicitly defined in Section 2.2 (p. 4, lines 166–190, Eq. (6)). Candidate-set KL directly measures this policy-preservation objective in our experimental setting.

In the main SHP experiment, we compute the exact categorical KL from clean DPO to the defended policy on a fixed list of 100 candidate entries per prompt, then average over 5,000 held-out prompts (Appendix G, p. 28, lines 1472–1490).

Win rate evaluates downstream response quality. Our question is how closely training on attacked data preserves clean-data DPO, whose absolute response quality is a separate issue. Appendix G (p. 28, lines 1491–1493) already states this distinction. The simulations evaluate the robustness target established by the theory; we will make that purpose clearer in the main text.

W2. Policy class and full-parameter DPO

The log-linear family is an established theoretical setting. Section 2.1 (p. 3, lines 117–120) explicitly follows Nika et al. (2025), Chowdhury et al. (2024), and Xu et al. (2025). Our analysis applies to general dimension dd (Assumption 2.1, p. 3, lines 120–125). The 384 parameters come from the MiniLM features chosen for the experiments (Section 4, p. 8, lines 422–424).

This setting makes each flip's gradient shift independent of the current parameters (Lemma 2.2, p. 3, lines 140–146). The impact score ranks pairs by how much each would change the gradient if its label were flipped. A clean pair can therefore rank among the top KK and be trimmed (Section 3.2, p. 5, lines 218–222). The main experiments use the same features and policy class across methods to compare the effects of the defenses under the same attacks.

The paper also explores extensions beyond log-linear policies. Appendix O.1 (p. 40, lines 2145–2151, Lemma O.1) gives the flip-gradient identity for differentiable policies, and Appendix O.3–O.4 (p. 41, lines 2170–2209, Table 7) evaluates a nonlinear head trained on frozen features. At the 30% attack budget, TRIM_REG and SOFT_TRIM_REG reduce mean candidate-set KL by 18.7% and 9.6% relative to matched DPO_REG.

Thank you for suggesting a comparison using full-parameter language-model fine-tuning. We will address this in the revision.

W3. ROPO and Hölder-DPO

Thank you for suggesting both methods. ROPO’s analysis shares rDPO’s assumption that each preference label is flipped with the same probability in either direction. Hölder-DPO studies a broader model in which a fraction of the data comes from a contaminating distribution.

Our guarantees cover every fixed set of at most KK adversarial flips (Section 2.2, p. 4, lines 162–190). We already evaluate rDPO (listed as NR_DPO) as a baseline designed for random flips. Table 1 reports its performance under random, top-impact, and targeted BMP-A flips (Appendix J.1, p. 33, lines 1737–1744, and Table 1, p. 36, lines 1890–1906). In the revision, we will discuss ROPO and Hölder-DPO in the Introduction and related work and compare both methods directly under the same attacks.

Thank you for the detailed rebuttal and clarifications.

W1: Evaluation metrics

As stated in my original review, I appreciate the experiments evaluating the proximity to clean DPO, as they directly validate the theoretical results. However, I still believe that evaluating downstream performance, such as win rate, is essential to demonstrate the practical value of the proposed method as a robust DPO algorithm. Therefore, my original concern remains.

W2: Full-parameter DPO

I understand the theoretical motivation for using log-linear policies and appreciate the preliminary experiments with nonlinear models. However, the effectiveness of the proposed method in standard full-parameter DPO settings remains unverified. I believe such experiments are important for establishing its practical applicability.

W3: Related work and baselines

While ROPO assumes symmetric label noise in its theoretical analysis, it also considers other noise settings experimentally. Therefore, I believe a direct comparison under the same attack settings is still important. I appreciate the authors' commitment to adding comparisons with ROPO and Hölder-DPO, but these results have not yet been provided.

Overall, my main concerns regarding its practical effectiveness remain unresolved. The authors have promised additional experiments, but I cannot assess their implications without seeing the results. Therefore, I would like to maintain my original rating at this stage.

Reviewer_L1TPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes defenses for offline DPO against adversarial preference label flips. Building on the observation that a label flip shifts each pair’s gradient by a fixed amount in log-linear policies, the authors score every pair by its potential flip impact and propose two defence methods, Trim-K, which removes the K highest-scoring pairs, and Soft-Trim-K, which downweights them. Each method also has a regularized variant that keeps the parameters close to the reference policy. The authors also proves the worst-case bounds on how far the defended model can drift from clean DPO and evaluate the defenses by training a linear head over frozen sentence embeddings, measuring how closely each one tracks clean DPO under random, high-impact and targeted flips on the Stanford Human Preferences, HH-RLHF and UltraFeedback-Binarized datasets.

Critical strengths
  1. The paper is well written and easy to follow. Robustness of offline preference optimization to targeted label flips is a relevant problem

  2. The paper splits the deviation from clean DPO into two parts, the error the defense introduces even on clean data and the effect of the flips that survive the defense. This split is clean and transparent, and it makes it easy to see what any defense that removes or downweights training pairs gains and what it costs.

Critical weaknesses
  1. The theory does not show that the defense improves over standard DPO without any defense. The worst-case bounds proved for the proposed methods are never shown to be tighter than the bound that already holds for undefended DPO
  2. The empirical results show a strong advantage only in a narrow setting, mainly the high-impact attack, which flips exactly the pairs the defense removes, so the gains hold by construction.
Questions for authors
  1. Can the authors show a setting where the worst-case bound for the proposed methods is tighter than the bound for standard DPO without any defense?
  2. How do the methods perform against an adaptive attacker that knows the defense and restricts its flips to pairs outside the highest-impact set?
1 comment

Thank you for the detailed review.

W1 / Q1. Comparing worst-case bounds with undefended DPO

Thank you for pointing out this gap. Theorem 3.2 bounds the parameter distance between Trim-KK trained on attacked data and standard DPO trained on the full clean dataset (Section 3.2, p. 6, lines 270–277). In a future revision, we will prove a lower bound on the parameter distance between standard DPO trained on attacked data and standard DPO trained on clean data.

W2. Evidence beyond the top-impact attack

Under the top-impact attack, hard trimming removes every flipped pair. The comparison is with DPO trained on the full clean dataset, so discarding those pairs still has a cost (Appendix J.2, p. 34, lines 1823–1829).

The targeted BMP-A experiment also shows a benefit, even though only 42.15% of the flipped pairs are in the top-impact set (Appendix L, p. 37, lines 1968–1969). At the same regularization strength, λ=10−3\lambda=10^{-3}, candidate-set KL to clean DPO is 0.1296 for DPO_REG, 0.1176 for TRIM_REG, and 0.1232 for SOFT_TRIM_REG. These are reductions of 9.3% and 4.9% relative to DPO_REG in this experiment (Appendix K, p. 36, lines 1920–1935, Table 2).

Q2. An attacker that flips only retained pairs

Our bounds cover this attacker. If the attacker knows the defense and selects up to KK pairs outside the top-impact set, those flips remain in the training data and contribute to the remaining attack term. Theorems 3.1 and 3.4 bound the gradient deviation for every fixed set of up to KK flips, including one chosen this way (Section 3.2, p. 5, lines 235–254 and Section 3.3, p. 7, lines 338–344). For each different attack method used as an example, the experiments use the same attacked dataset across defense methods for comparison.

Reviewer_DHcPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies DPO under budgeted adversarial preference-label flips. Building on an existing gradient-shift identity for log-linear policies, it proposes removing the highest-impact preference pairs or downweighting them through a soft weighting rule. The analysis separates defense-induced bias from residual attack effects and derives parameter and policy stability bounds under bounded-region and curvature assumptions. Experiments cover three preference datasets and a small nonlinear-policy extension.

Critical strengths

Strategically selected preference corruption is a meaningful threat, and the distinction from random label noise is well motivated. The proposed rules are simple, and their scores can be computed without knowing which labels were flipped. The analysis explicitly accounts for the clean-data bias introduced by trimming and regularization. The experiments also include useful controls: matched regularization, randomized score assignments, clean-data retention costs, and misspecified attack budgets. These provide evidence of an additional contribution from impact-based selection while making its trade-offs visible.

Critical weaknesses

The analysis provides a conditional stability characterization, but offers limited guidance on when the proposed intervention is beneficial. The score measures the magnitude of an individual flip’s gradient perturbation, while targeted poisoning also depends on the alignment of multiple perturbations and the objective’s curvature. Conditions under which suppressing high-impact pairs compensates for the loss of clean information would make the theoretical contribution more informative. The numerical usefulness of the resulting bounds is also not assessed.

The empirical advantage of trimming is uneven. In the main BMP-A experiment, unregularized TRIM and SOFT TRIM have higher candKL than DPO: 0.335 and 0.320 versus 0.289. Regularization substantially improves the results, but full-data DPO REG already reaches 0.1296, compared with 0.1176 for TRIM REG. The matched controls support an incremental benefit, yet WDPO and KLDPO perform better in this instance. The strongest trimming result occurs under the top-impact attack, where the attacked and removed subsets coincide. Together, these results leave the favorable regime for impact-based selection insufficiently characterized.

The experimental scope is consistent with the log-linear focus, but applicability to language-model fine-tuning remains largely untested; the nonlinear extension uses a small head over frozen representations. Candidate-set KL measures proximity to clean DPO, without directly establishing preservation of response quality or safety. The budget-selection study also relies on a clean-policy reference, so it does not establish how to select the budget under the stated information constraints.

Questions for authors

Can the authors identify conditions on the feature geometry and attack set under which trimming improves the total deviation, accounting for both clean-data bias and curvature? A numerical comparison of the bounds with observed deviations would also help assess their informativeness.

Does the incremental benefit persist after comparable regularization tuning across methods? Although the theory covers arbitrary fixed attack supports, an experiment that optimizes the attack with knowledge of the retained set or weighting rule would provide a stronger empirical test.

How should (K) be selected without access to the clean-policy reference used in Appendix N, and how sensitive is the resulting robustness–retention trade-off to that choice?

2 comments

Thank you for the detailed review.

W1 / Q1. When trimming helps

Thank you for suggesting this comparison. In a future revision, we will examine when trimming reduces the effect of flipped labels enough to outweigh the bias from removing clean pairs. We will account for gradient alignment and curvature and compare the resulting bounds with observed deviations.

W2 / Q2: Empirical benefits, regularization, and defense-aware attacks

W2 / Q2, first part: Benefits beyond regularization

Both regularized defenses improve over matched DPO_REG under top-impact and BMP-A attacks. Appendix K compares all three methods on the same attacked data with the same quadratic penalty and λ=10−3\lambda=10^{-3}:

AttackDPO_REG candKLTRIM_REG candKLSOFT_TRIM_REG candKL
Top-impact0.10890.0803 (26.3% lower)0.0934 (14.2% lower)
BMP-A0.12960.1176 (9.3% lower)0.1232 (4.9% lower)

These results show an additional benefit from impact-based trimming and weighting beyond the shared regularizer in both settings (Appendix K, p. 36, lines 1920–1935, Table 2).

The score-assignment controls provide further evidence: under BMP-A, impact-based selection improves over random trimming and permuted weights by 0.0090±0.00040.0090\pm0.0004 and 0.0048±0.00020.0048\pm0.0002 candKL across ten paired trials (Appendix L, p. 37, lines 1953–1975, Table 3). Only 42.15% of BMP-A's flipped pairs lie in the removed top-impact region (lines 1968–1969), so these gains extend beyond attacks whose flipped and removed sets coincide.

Regarding the tuning question, these comparisons use a shared regularization strength. Whether the gains persist after separately tuning λ\lambda for each method is an additional comparison that was not performed in the submitted experiments.

Our claim is robustness over all fixed admissible attack sets under the stated assumptions, rather than empirical superiority under every attack and parameter choice (Section 3.2, p. 6, lines 279–284).

Q2, second part: Defense-aware attacks

Our guarantees cover an attacker that chooses its flips with full knowledge of the defense methods, including our kept set or soft weights. The impact scores, kept set, and weights are invariant to label flips. Theorem 3.1 (p. 5, lines 250–254) and Theorem 3.4 (p. 7, lines 338–344) hold for every fixed support AA with ∣A∣≤K|A|\le K, including attack supports optimized against the defense and attacks restricted to retained pairs. Appendix J.2 (p. 36, lines 1911–1915) explicitly states this coverage.

The submitted experiments use common attack supports across methods (Appendix H, p. 28, lines 1497–1502). A separately optimized defense-aware attack would provide an additional empirical test within the scope of these guarantees.

W3 / Q3: Evaluation scope and budget selection

W3: Policy class and potential extensions

Log-linear policies are an established setting for theoretical analysis of DPO. Our primary contribution is a robustness analysis in this setting. Section 2.1 (p. 3, lines 117–120) follows Nika et al. (2025), Chowdhury et al. (2024), and Xu et al. (2025). The log-linear structure makes each flip's gradient shift independent of the current parameters (Lemma 2.2, p. 3, lines 140–146), enabling the impact-score construction and subsequent stability bounds.

We also investigate the potential to extend the approach to other policy classes. Appendix O.1 (p. 40, lines 2145–2151, Lemma O.1) gives the flip-gradient identity for differentiable policies, and Appendix O.3–O.4 (p. 41, lines 2170–2209, Table 7) evaluates a nonlinear head over frozen features. At the 30% attack budget, TRIM_REG and SOFT_TRIM_REG reduce mean candidate-set KL by 18.7% and 9.6% relative to matched DPO_REG. These experiments provide preliminary evidence of the approach's potential beyond the log-linear setting.

W3: Policy preservation and response quality

Our goal is to keep the defended policy trained on attacked data close to standard DPO trained on the full clean dataset. This target is defined in Section 2.2 (p. 4, lines 175–190).

The absolute quality and safety of clean DPO depend on the starting model, preference data, and training settings. Improving that baseline is a separate objective from controlling the deviation caused by corrupted labels. By fixing the clean-DPO baseline, our study measures how closely the defended policy preserves it under attack.

Candidate-set KL directly measures this policy deviation on the evaluated responses (Appendix G, p. 28, lines 1472–1493). The simulations therefore evaluate the robustness target established by the theory. They measure policy closeness rather than directly testing response quality or safety. We will make this distinction clearer in the main text.

W3 / Q3: The role of KK and the clean-policy reference

Our setting assumes that a known or estimated upper bound KK on the number of label flips is supplied to the method. This supplied value is kept fixed during each training run (Section 2.2, p. 4, lines 166–168). Section 3.3 explains what happens when the estimate is inaccurate: underestimating the true number of flips falls outside the guarantee, while a larger upper bound can remove or downweight more clean data (p. 7, lines 376–377). The upper bound need not equal the true flip count.

To examine this sensitivity empirically, Appendix N holds the actual attack fixed and varies the supplied KK. The true number of flips is 1,200 out of 6,000 pairs (20%), and the same flipped pairs are reused within each dataset and seed. We will clarify the experimental procedure and the role of the clean-policy reference as follows:

  1. Prepare the clean comparison policy and attacked data. Since the experiment starts with clean preferences, we first train standard DPO on the original 6,000 training pairs. This is the clean policy against which we measure deviation. Its parameters also define the reflected target used to construct the BMP-A attack (Appendix M, p. 38, lines 2017–2022). This attack construction is inherited by Appendix N.
  2. Vary the assumed upper bound while keeping the true flips fixed. We train separate runs with K/N∈{0,1,2,4,8,16,32}%K/N\in\{0,1,2,4,8,16,32\}\% on the same attacked dataset. This tests how underestimating or overestimating the actual flip count affects the results. Each run uses its supplied KK throughout training (Appendix N, p. 39, lines 2072–2078).
  3. Compare the results with clean DPO. Table 5 reports held-out policy KL for every tested budget (p. 39, lines 2080–2105). Table 6 reports which tested budget gives the lowest KL to clean DPO on 500 validation pairs. We then evaluate the selected run on another 1,000 held-out pairs (p. 39, lines 2074–2076; p. 40, lines 2106–2112). This selection compares policies already trained at fixed budgets.

We use clean DPO in this experiment to compare different choices of KK and understand their effects. The Table 6 comparison uses the clean policy because it is available in the experiment; we are not proposing clean policy as a way to choose KK when only corrupted data are available to the defender. The selected value measures which tested budget best preserves clean DPO, and is not an estimate of the number of flips (p. 40, lines 2115–2120).

Table 5 shows that the REG variants remain closer to clean DPO than attacked, unregularized DPO across the tested grid on all three datasets, while unregularized trimming generally increases deviation. We will revise Appendix N to make clear that this experiment studies sensitivity to the supplied upper bound, with clean DPO used to compare the resulting policies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.