acceptodds
Under review as a conference paper at ICLR 2027

BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment

Abstract

Preference-based alignment methods such as Direct Preference Optimization (DPO) use pairwise preferences labeled by human annotators to fine-tune language models. However, annotators carry systematic biases toward some attributes: a name that signals a gender or an ethnicity, a persona, a language variety, a formatting convention, or length. If not properly addressed, these systematic biases can be absorbed and amplified during alignment. Existing methods address length bias or annotator disagreement, but fail to eliminate biases toward arbitrary attributes. To address this limitation, we propose _Bias-Adjusted DPO_ (BA-DPO), a generalization of DPO that adds one bias parameter per annotator toward responses carrying a declared attribute. We prove that the objective is convex in the bias parameters and that the votes identify each annotator's bias up to a shared constant. The remaining constant is what fixes the aligned model's attribute rate: by default the rate of the reference model, or a target rate, which we use to bring a biased policy to statistical parity. On a corpus with planted biases, DPO drives the attribute from a balanced start to probability and BA-DPO removes to of that shift; on MultiPref with real annotators it removes about half of DPO's lengthening. Both hold at 0.5B with full fine-tuning and at 8B with LoRA, at no higher KL than DPO and no loss in judged quality.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Direct preference optimization, RLHF, Fairness, Annotator bias, Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1JNuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents BA-DPO, a DPO post-training method that models annotator biases per preference. Each annotator bias is modeled as θˉ+ϵκ\bar{\theta} + \epsilon_\kappa, with shared preferences and deviations. The method is theoretically characterized and evaluated against other DPO variants.

Critical strengths
  1. Removing annotator bias from DPO preferences is an important and understated problem.
  2. The method is thoroughly tested across 2 datasets, 2 model sizes, and 5-6 baselines.
Critical weaknesses
  1. The rationale for providing annotator annotations to DPO preferences is not present, and it's unclear whether other methods were tested in the same setting. What if the preferences had no annotator annotations, thus anonymous?
  2. It's unclear what the relative improvements are, and some parts of it are inconsistent. Is it high "Removed"? Then why is the method BA-DPO lower than R-DPO for MultiPref? Also, "Removed" is not measured in Table 3. Other metrics are provided without an explanation of whether the model is good or bad. The tables should be self-sufficient with enough explanations.
Questions for authors

The benefit of the theoretical convexity should be moved to the main content and be sufficiently explained.

4 comments

W1. […] it's unclear whether other methods were tested in the same setting. What if the preferences had no annotator annotations, thus anonymous?

We first want to clarify that BA-DPO treats the annotators as anonymous. All it needs is, for each comparison, which response was preferred and the declared attribute of the two responses (with no attribute declared, BA-DPO is plain DPO), so it applies to any preference dataset, including those that keep only the majority preference per comparison. Without annotator IDs it learns one bias per attribute, shared by all annotators (the pooled variant, Section 3), which is in every table.

Annotator IDs are therefore optional, and they are only identifiers: they say which judgments come from the same annotator, nothing about who the annotator is, and the annotator can be a person or a model used as a judge. When they are available, the same training run also estimates each annotator's own bias, which makes the annotators auditable (Section 5.2).

All methods are trained on the same judgments, and each baseline gets the information it can use (App E.3): R-DPO and SamPO use no IDs, while Group-DRO and EM-DPO use the annotator IDs or classes. Even without IDs, BA-DPO removes the bias, and these two do not (Tables 1-2). We will state in each table which information each method uses, and say in Section 3 that annotator IDs are optional.


W2. It's unclear what the relative improvements are […]. Is it high "Removed"? Then why is the method BA-DPO lower than R-DPO for MultiPref? Also, "Removed" is not measured in Table 3. Other metrics are provided without an explanation of whether the model is good or bad. […]

Thank you, we agree that the tables should stand on their own. The metrics are defined in Section 5.1, but the captions only point there. We describe each one below, with the direction that is better, and we will make every caption self-contained:

  • Attribute rate (p(woman)p(\text{woman}), p(black)p(\text{black}), Tokens, Rate): how strongly the policy produces the attribute. The reference shows where training started, and DPO how far the bias pushed it.
  • Removed: the share of DPO's shift that a method undoes, from 0% (as biased as DPO) to 100% (back to the reference). Higher is better on Signed-UltraFeedback, where the reference is balanced by construction, so the unbiased point is known.
  • Held-out gap: how much better the policy predicts the annotators' votes on pairs that differ in the attribute than on pairs that do not. A policy that learned the bias has a clearly positive gap, and the gap falls as the bias leaves the policy.
  • RM and RM−\mathrm{RM}^{-}: judge scores of the answers (RM−\mathrm{RM}^{-} with the attribute stripped), higher is better, compared within a model only. Under length the raw score also rewards longer answers.
  • KL: distance from the reference. It shows that a method does not remove the bias simply by staying at the reference.

On MultiPref no length is the correct one, so removing more is not automatically better, and the gap is the main readout. This is why R-DPO's higher "Removed" is not a better result: its length penalty is set by hand, the larger its weight the shorter the answers, and at its published weight they collapse to 138 tokens (Table 9). BA-DPO instead estimates from the labels how much of the preference for longer answers is bias, and removes that part. Table 3 has no "Removed" column because DPO barely changes the markdown rate there, so a share of that change would be noise; the gap is the readout instead.


Q1. The benefit of the theoretical convexity should be moved to the main content and be sufficiently explained.

Thank you, we agree. The result is stated in the main text (Theorem 3), but its use is not explained, and we will add a paragraph after the theorem. Convexity gives two things.

First, the bias parameters are easy to fit. For a fixed policy, fitting θ\theta is a logistic regression, which has a single optimum as long as the annotator does not vote for the attribute every time, or against it every time. The bias parameters therefore have no bad local optimum to get stuck in. Convexity also gives a sample-complexity result for the bias parameters, which we will add to the paper (stated with its proof in our reply to Reviewer Gpkz): with nn comparisons that differ in the attribute, an annotator's bias is estimated to within about 1/κn1/\sqrt{\kappa n}, where κ≤1/4\kappa \le 1/4 is the average variance of a vote.

Second, it lets Theorem 3 describe all solutions, not only those near one solution. Because the loss is convex in the reward values and the bias parameters, every solution gives the same probability to every vote. This is what allows us to state exactly what the labels determine (how the annotators' biases differ from each other) and what they leave open (their shared level, which sets how strongly the policy favours the attribute and is fixed by the anchor).

Thank you for the answer. It's good that consistent results are shown for BA-DPO regarding the "Removed" metric (even considering R-DPO), but I'm worried that this metric may be prone to cherry-picking.

Regarding RM and KL, what is the correct way to interpret the results? Seems like RM is negative for all Qwen2.5-0.5B-Instruct models and positive for Llama-3.1-8B-Instruct models. Why is that? If those are only sanity checks, then only the Signed-UltraFeedback result is clear, which is an interesting but limited result.

Thanks for your answer!

Removed is natural on Signed-UltraFeedback because the unbiased level is known: the reference is balanced by construction, so the target is its rate, and Removed measures how close a method gets to it. Since this is not a standard metric, we also report a standard test-set one, the held-out gap: the policy's accuracy on held-out annotator votes, on pairs that differ in the attribute minus pairs that do not. On MultiPref, where no length is the correct one, the gap is the main readout.

The judge's score is defined only up to a constant, so its sign carries no meaning: the 0.5B models write weaker answers than the 8B ones and score lower, both before DPO (the reference) and after it. RM is read within a model, and it checks that BA-DPO improves the answers over the reference about as much as DPO does. KL checks that BA-DPO moves the policy by the same order of magnitude as DPO. Both are checks: the MultiPref result rests on the held-out gap, which DPO opens on both attributes and every model, and which BA-DPO reduces by about half or more (Tables 2 and 3).

Dear Reviewer, we hope our answer on Removed, RM and KL clarifies how to read the tables. We would be glad to know whether you have any further concerns we have not addressed yet.

Reviewer_YxxKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper applies the existing BARP model on DPO to mitigate systematic annotator biases toward specific response attributes. The proposed BA-DPO introduces learnable bias parameters into the DPO objective. Experiments across multiple datasets and model families demonstrate that BA-DPO substantially reduces bias amplification to standard DPO.

Critical strengths

The paper is well-written and clearly structured, with clear motivation and a comprehensive presentation. It includes both theoretical analysis and extensive experiments to support the proposed method.

Critical weaknesses

1, The methodological contribution appears somewhat incremental, as BA-DPO mainly incorporates the existing BARP model into the standard DPO framework.

2, While the method shows substantial improvements on synthetic data (SIGNED-ULTRAFEEDBACK), the gains on real-world data (MULTIPREF) are less pronounced, which weakens the practical significance of the proposed approach.

3, Some theoretical results, particularly Theorem 3, mainly characterize properties of the underlying BARP model rather than DPO or language model alignment specifically. This limits their contribution to the central research problem of the paper.

Questions for authors

1, Theorem 3 appears to characterize the identifiability properties of the underlying BARP model rather than any specific property of DPO or language models. It would be helpful to clarify what new theoretical insights Theorem 3 provides beyond the existing model and how they inform the training of language models.

2, The improvements on real-world data (MULTIPREF) appear less conclusive than those on synthetic data (SIGNED-ULTRAFEEDBACK). In particular, reducing response length or the held-out accuracy gap does not necessarily indicate successful bias removal. Could the authors provide more direct evidence that BA-DPO separates annotator bias from genuine response quality?

3 comments

W1. The methodological contribution appears somewhat incremental, as BA-DPO mainly incorporates the existing BARP model into the standard DPO framework.

We agree that the likelihood is BARP's, and putting it inside DPO is a short step (Prop 2). The step is short on purpose: because the bias is additive, it passes through the partition function, so modelling annotator bias costs one scalar per annotator rather than one policy per annotator type, as in mixture methods. What happens once BARP is inside DPO, however, does not follow from BARP.

Two things in particular do not follow, and both come with new theory. First, BARP's own form does not work inside DPO as it is: with one free bias per annotator, the method removes little more bias than with shuffled annotator IDs (Table 9). We show why: only the shared level of the biases reaches the policy (Prop 5), and with free biases this level is learned too slowly to get there before the policy absorbs the bias (Prop 6); writing each bias as a shared mean plus a deviation fixes this. Second, BARP has no identifiability result. Theorem 3 gives one, and shows that in DPO the part the labels leave free is how strongly the aligned policy favours the attribute, so debiasing amounts to choosing an anchor (see our answer to Q1 below).

Finally, the problem matters for how we steer a policy. Without a bias term, DPO reads every label as evidence of quality, so whatever the annotators favour for other reasons, such as a name or the length of an answer, becomes a preference of the policy (Tables 1-3). This is exactly what we want to avoid: the policy should follow the quality of the answer, not signals that should have no bearing on it. Existing methods do not prevent it: those built for annotator disagreement leave a shared bias in place, and those built for length apply to no other attribute.


W2. While the method shows substantial improvements on synthetic data (SIGNED-ULTRAFEEDBACK), the gains on real-world data (MULTIPREF) are less pronounced, which weakens the practical significance of the proposed approach.

Q2. The improvements on real-world data (MULTIPREF) appear less conclusive than those on synthetic data (SIGNED-ULTRAFEEDBACK). In particular, reducing response length or the held-out accuracy gap does not necessarily indicate successful bias removal. Could the authors provide more direct evidence that BA-DPO separates annotator bias from genuine response quality?

We agree that this is the key question on real data: a longer answer can also be a better one, so length and quality have to be separated, not length removed.

BA-DPO separates them by asking how much more often the longer answer wins than its content explains. The policy's reward scores the whole answer, so when a longer answer is better because of what it says, the reward gets the credit; the bias term takes only the advantage the longer side gets on top of its content. A longer answer that is genuinely better therefore still wins under BA-DPO, and what disappears is the push that length gives regardless of content. Some reference point for this split is unavoidable: by Theorem 3, no method can separate a bias that all annotators share from quality using the labels alone, and BA-DPO takes the reference policy as that point (see our answer to Q1).

This is also why the gains on MultiPref look smaller than on Signed-UltraFeedback without being weaker. On Signed-UltraFeedback the names carry no quality, so all of DPO's shift is bias and the target is to remove all of it. On MultiPref part of the preference for length is genuine, so the right amount to remove is less than all: BA-DPO removes about half, its answers stay longer than the reference's, and removing more by hand, as R-DPO does, lowers the judge's scores (Table 9).

As direct evidence we use the held-out gap, which isolates the part of the policy's agreement with the annotators that comes from length. Both kinds of held-out pairs differ in content, and only one differs clearly in length. Under DPO the policy agrees with the annotators much more often on pairs that differ in length, because it has learned their preference for length; under BA-DPO this extra agreement largely disappears, while agreement on the other pairs is unchanged (Tables 2 and 9). On Signed-UltraFeedback, where we know which part is bias, the gap falls steadily as the bias is removed (Table 4). We will add this validation to Section 5.1.

W3. Some theoretical results, particularly Theorem 3, mainly characterize properties of the underlying BARP model rather than DPO or language model alignment specifically. This limits their contribution to the central research problem of the paper.

Q1. Theorem 3 appears to characterize the identifiability properties of the underlying BARP model rather than any specific property of DPO or language models. It would be helpful to clarify what new theoretical insights Theorem 3 provides beyond the existing model and how they inform the training of language models.

Thank you for the question. We first want to clarify that Theorem 3 is a result about the BA-DPO loss: the DPO loss with the bias term inside its logit, where the reward is the policy itself, r=βlog⁡π/πrefr = \beta\log\pi/\pi_{\mathrm{ref}}. The proof uses only the form of this logit, so the same statement also holds when the reward is a free score per item, which is BARP's model. The theorem is therefore new for both: the BARP paper gives no identifiability result, and in DPO the result has consequences that BARP cannot have, which we describe below.

For BARP itself, Bradley-Terry scores are known to be identified up to one constant. Theorem 3 shows that BARP has a second flat direction: raising the score of every item carrying the attribute by cc and lowering every evaluator's bias by cc changes no comparison. What stays identified is how the annotators' biases differ from each other. This also corrects our own Section 2, which says that cross-group comparisons identify each evaluator's bias: they identify only the differences between the biases.

In DPO this flat direction has a concrete meaning. Because the reward is the policy itself, moving along the flat direction changes how much the trained policy favours the attribute without changing the fit to any vote. In other words, the labels alone cannot set how strongly the aligned model favours the attribute; the reference, or a target rate we choose, sets it. This holds even for pairs that differ only in the attribute: the votes on such a pair can be explained either by the annotators' bias or by a policy that favours the attribute itself. Together with Prop 5, the theorem also splits the roles of the parameters: the shared level of the biases is the only part a policy can absorb, so it is the part that debiases; the deviations are fixed by the labels but invisible to the policy, so they describe the annotators.

These results guide training in four ways. The attribute must be declared, since otherwise no method can separate a shared bias from quality. The shared level must be learned fast, which is why we use the shared-mean or pooled form (Prop 6; BARP's form, with one free bias per annotator, removes little bias, Table 9). The anchor must be chosen: starting from πref\pi_{\mathrm{ref}} with θ=0\theta = 0 stays near the reference's rate, and a fixed offset reaches a target rate (App C.5). And every annotator needs comparisons that differ in the attribute for their own bias to be identified, which our sample-complexity result quantifies (see our reply to Reviewer Gpkz, W1). We will add a remark after Theorem 3 on what is specific to DPO.

Dear Reviewer, we clarified that Theorem 3 and our new theory are results about the BA-DPO loss, not only BARP (W1, W3), and that on MultiPref the held-out gap measures the bias directly (W2). We would be glad to know whether you have any further concerns we have not addressed yet.

Reviewer_GpkzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines a limitation of preference-based optimization: preference labels may reflect not only response quality but also systematic annotator preferences for attributes such as demographic-coded names or response length. To address this issue, the authors propose Bias-Adjusted DPO (BA-DPO), which introduces an additive bias parameter for each annotator and declared attribute. The theoretical analysis characterizes which components of annotator bias can be identified from preference judgments and explains the need for an additional anchoring choice to separate shared attribute preferences from reward. Experiments on Signed-UltraFeedback with planted biases and MultiPref with real annotators, across three language models, show that BA-DPO reduces attribute shifts induced by standard DPO.

Critical strengths
  1. This paper addresses an important problem in preference-based alignment: distinguishing preferences for declared nuisance attributes from useful information about response quality.

  2. The authors evaluate the proposed method on both synthetic and real-annotator preference data across three language models, providing empirical evidence of its effectiveness in reducing the evaluated attribute shifts.

Critical weaknesses
  1. The reliability of the annotator-specific bias parameters requires further clarification. Although the paper explains how these parameters are jointly optimized with the policy, it is less clear how reliably they can be estimated when an annotator provides only a limited number of relevant comparisons. More evidence on their stability under sparse annotation would help establish the reliability of the individual-bias estimates.
  2. The construction of Signed-UltraFeedback raises a question about the scope of the empirical validation. As described in Appendix D.2, its preference labels are synthetically generated using the assumed additive bias model, rather than collected from human annotators. This provides a useful controlled test, but it remains unclear how well the findings generalize to real annotator biases that do not follow this model.
  3. The evaluation focuses on reducing declared attribute shifts, but it remains unclear whether these reductions translate into better alignment with the target users’ intended preferences. Removing an attribute preference is not an end in itself: attributes such as length and formatting may also reflect legitimate user needs. While the reported reward-model scores provide some indirect evidence, a direct comparison of user preferences for the resulting responses would better establish the method’s practical benefit, rather than attribute-shift reduction alone.
  4. The method’s practical applicability is limited by its reliance on declared, computable bias attributes. In many settings, the relevant attributes may be implicit, difficult to measure, or context-dependent. Although the paper acknowledges this limitation, the current results mainly establish effectiveness for known and explicitly specified attributes, leaving its applicability to less structured real-world biases uncertain.
Questions for authors

See Critical Weaknesses.

6 comments

W1. The reliability of the annotator-specific bias parameters requires further clarification. […] it is less clear how reliably they can be estimated when an annotator provides only a limited number of relevant comparisons. […]

Thank you for the question! This question inspired a sample-complexity result, which answers in a principled way how many judgments an annotator's bias needs.

Only cross-group judgments inform θk\theta_k (Theorem 3). Orient each one so that the response carrying the attribute comes first; with quality margin mjm_j, annotator kk picks it with probability σ(mj+θk)\sigma(m_j+\theta_k). For given margins, estimating θk\theta_k is a one-parameter logistic regression. Let κ=1n∑jσ′(mj+θk)≤1/4\kappa = \frac1n\sum_j \sigma'(m_j+\theta_k) \le 1/4 be the average variance of a vote. We prove matching upper and lower bounds (in a separate comment below): for bounded ε\varepsilon, estimating θk\theta_k to within ε\varepsilon with probability 1−δ1-\delta takes n=Θ(log⁡(1/δ)/(κε2))n = \Theta(\log(1/\delta)/(\kappa\varepsilon^2)) cross-group judgments, with δ/m\delta/m in place of δ\delta for all mm annotators at once. In practice the standard error is about 1/κn1/\sqrt{\kappa n}, which matches the observed error in simulation from n≈25n \approx 25 on.

In BA-DPO the margins are those of the current policy, and the bounds treat them as given; they do not include the margins' own estimation error. Shifting the margins along the flat direction of Theorem 3 moves all the θk\theta_k together, so it changes the shared level of the biases but not how they differ from each other.

Detecting a bias of size Δ\Delta (5% level, 80% power) takes about 7.8/(κΔ2)7.8/(\kappa\Delta^2) judgments. κ\kappa is largest when quality does not already decide the vote, so close pairs, such as those that differ only in the attribute, are the most informative.

We can check this on our results: on Signed-UltraFeedback, with about 155 cross-group judgments per annotator, this standard error predicts the recovery we measure (r=0.95r = 0.95 and 0.980.98, Section 5.2).

A natural follow-up to your question is what to do when annotators give too few judgments for their own bias to be estimated (below about 50 for a bias of 1 in log-odds). The answer is to estimate it from a group of annotators rather than from each one alone:

  1. Pooling. One bias for all annotators, estimated from all their judgments together. Since only the shared level of the bias reaches the policy (Prop 5), this removes as much bias as θˉ+εk\bar\theta+\varepsilon_k (Tables 1-2), including on MultiPref, where many annotators have few judgments.
  2. Partial pooling. A prior on the deviations εk\varepsilon_k shrinks each annotator's estimate toward the shared level, more strongly the fewer judgments they have.
  3. Known groups. When annotators come with metadata, such as crowd workers and experts on MultiPref, the class form θˉ+δc+εk\bar\theta + \delta_c + \varepsilon_k (Section 3) pools within each group.

Debiasing therefore never needs many judgments from every annotator; only auditing an individual annotator does. We will add the result and its proof to the final version, together with the prior on the deviations as an option.


W4. The method’s practical applicability is limited by its reliance on declared, computable bias attributes. In many settings, the relevant attributes may be implicit, difficult to measure, or context-dependent. Although the paper acknowledges this limitation, the current results mainly establish effectiveness for known and explicitly specified attributes, leaving its applicability to less structured real-world biases uncertain.

We agree that BA-DPO needs a declared, measurable attribute, but the requirement is not specific to our method. By Theorem 3, a bias that all annotators share looks exactly like genuine quality in the labels, so no method can remove it without being told which attribute it concerns. Methods for annotator heterogeneity, such as Group-DRO and EM-DPO, need no declaration, but they capture how annotators differ from each other; a bias that all annotators share is not such a difference, so they do not remove it (Tables 1-2).

Within this limit the attribute can be quite general: anything computed from the prompt and the responses, continuous or produced by a classifier or a language-model labeller, and several attributes at once. A bias that appears only in some contexts, for instance on some topics, can be declared as the attribute multiplied by a topic indicator.

Declaring an attribute that turns out to carry no bias is harmless: BA-DPO then learns a small bias and trains as DPO, as we checked on MultiPref with a hedging opener and a score from a language-model labeller. An attribute only suspected of bias can therefore be declared safely. The real limit is the opposite case: a biased attribute that is not declared stays in the policy (App F.1).

W2. […] As described in Appendix D.2, its preference labels are synthetically generated using the assumed additive bias model, rather than collected from human annotators. This provides a useful controlled test, but it remains unclear how well the findings generalize to real annotator biases that do not follow this model.

Thank you for pointing this out. The two corpora are meant to play complementary roles. On Signed-UltraFeedback we plant the biases, so we know exactly how much bias there is to remove and can verify that BA-DPO removes it. MultiPref is the test on real biases: no model generated its labels, and BA-DPO still brings the held-out gap close to zero at 0.5B and halves it at 8B, for length (Table 2) and for formatting (Table 3).

Your question made us check directly whether the Signed-UltraFeedback results depend on the additive vote model. We rebuilt the corpus with everything fixed (pairs, names, annotators, planted biases, reference) and changed only how the annotators vote. With qq the quality gap between the two answers and bb the annotator's bias toward one of the names, our model votes σ(q+b)\sigma(q + b). We tried two rules that break this form in non-linear directions:

  • Fading bias, σ(q+e−∣q∣ b)\sigma(q + e^{-|q|}\,b): the name only decides close calls, and the bias fades when one answer is clearly better.
  • Heuristic voter: with probability tanh⁡(∣b∣/2)\tanh(|b|/2) the annotator picks the favoured name without reading the answers; otherwise they vote on quality alone.
Vote ruleDPO, p(woman)p(\text{woman}) / p(black)p(\text{black})Removed, pooledRemoved, θˉ+εk\bar\theta+\varepsilon_krr with planted θk\theta_k
Additive (paper, Table 1)0.96 / 0.9989% / 81%91% / 83%0.95 / 0.98
Fading0.93 / 0.9793% / 84%93% / 86%0.91 / 0.97
Heuristic voter0.97 / 1.0090% / 82%91% / 85%0.96 / 0.98

(0.5B, mean of two seeds; each cell is gender-coded / race-coded; the reference is balanced at 0.50 / 0.52, and Removed is the share of DPO's shift that BA-DPO undoes.)

The results barely move. BA-DPO removes 82 to 93% of DPO's shift under both rules, as under our model, and still recovers each annotator's bias (r≥0.91r \ge 0.91). The intuition is that a policy can take up the name's effect only as one overall tilt toward the name (Prop 5), and the bias term absorbs that tilt whatever the exact rule. Judge scores stay within 0.1 of DPO's or above, and KL is about two thirds of DPO's, as in Table 1. We will add both builds to the paper.


W3. […] it remains unclear whether these reductions translate into better alignment with the target users’ intended preferences. Removing an attribute preference is not an end in itself: attributes such as length and formatting may also reflect legitimate user needs. […] a direct comparison of user preferences for the resulting responses would better establish the method’s practical benefit […]

Thank you for the good point, we should have stated the objective more clearly. BA-DPO does not try to remove length or formatting. It removes only the part of the annotators' preference that the attribute explains by itself, with the content held fixed. A longer answer can still win, but because of what it says, not because it is longer: the policy should learn that the content is better, not that longer answers are always better.

This matters most for attributes that must never act as a signal of quality. DPO cannot tell an attribute from quality: it learns whatever the annotators favour as evidence of a better answer. BA-DPO strips the attribute from the quality signal, so that quality alone drives the policy's choices, also when quality and the attribute go together. Consider a model that chooses among job candidates of different races, trained on judgments from biased annotators. BA-DPO at inference may still prefer candidates of one race, but the annotators' preference for the race itself no longer drives that choice: training kept it in the bias term and out of the policy, which learned the quality of the applications with that preference decoupled from it.

Which attributes count as bias is the user's choice: if an attribute may carry quality for a given use, it should not be declared, and BA-DPO then trains as DPO. Some such input cannot be avoided: by Theorem 3, the labels alone cannot tell a bias that all annotators share from genuine quality, whatever the method. It is also why we measure the effect with held-out votes rather than with a new user study. Raters who share the bias would prefer the biased answers again. With held-out votes we can compare pairs that differ in the attribute with pairs that do not, and since the bias acts only on the first kind, the difference between the two shows how much of it the policy has kept.

We will state these points more clearly.

The sample-complexity result of our reply to W1: statements and proofs

Setting. Fix annotator kk and their nn cross-group judgments. Orient each judgment so that the response carrying the attribute comes first, let zj∈{0,1}z_j \in \{0,1\} say whether it won, and let mjm_j be the quality margin (the reward of that response minus the other's), taken as given. Under the bias-adjusted model the zjz_j are independent with P(zj=1)=σ(mj+θ⋆)P(z_j = 1) = \sigma(m_j + \theta^\star). The maximum-likelihood estimate θ^\hat\theta is the root of the score S(θ)=1n∑j=1n(zj−σ(mj+θ)),S(\theta) = \frac1n \sum_{j=1}^n \bigl(z_j - \sigma(m_j + \theta)\bigr), which is continuous and strictly decreasing; a root exists unless all nn votes went the same way. Let κ=1n∑jσ′(mj+θ⋆)∈(0,1/4]\kappa = \frac1n \sum_j \sigma'(m_j + \theta^\star) \in (0, 1/4], where σ′=σ(1−σ)\sigma' = \sigma(1-\sigma) is the variance of a vote.

Lemma 1. σ′(x+h)≥e−∣h∣ σ′(x)\sigma'(x+h) \ge e^{-|h|}\,\sigma'(x) for all x,hx, h.

Proof. ddxlog⁡σ′(x)=1−2σ(x)∈(−1,1)\frac{d}{dx}\log\sigma'(x) = 1 - 2\sigma(x) \in (-1, 1). □\square

Proposition A (upper bound). If n≥2e2ε(1+ε/3)log⁡(2/δ)/(κε2)n \ge 2e^{2\varepsilon}(1+\varepsilon/3)\log(2/\delta)/(\kappa\varepsilon^2), then with probability at least 1−δ1-\delta the estimate exists and ∣θ^−θ⋆∣<ε|\hat\theta - \theta^\star| < \varepsilon.

Proof. By Lemma 1, −S′(θ)≥e−εκ-S'(\theta) \ge e^{-\varepsilon}\kappa for ∣θ−θ⋆∣≤ε|\theta - \theta^\star| \le \varepsilon, so with t=e−εκεt = e^{-\varepsilon}\kappa\varepsilon we have S(θ⋆+ε)≤S(θ⋆)−tS(\theta^\star+\varepsilon) \le S(\theta^\star) - t and S(θ∗−ε)≥S(θ⋆)+tS(\theta^*-\varepsilon) \ge S(\theta^\star) + t. If ∣S(θ⋆)∣<t|S(\theta^\star)| < t, then SS changes sign on (θ⋆−ε,θ⋆+ε)(\theta^\star-\varepsilon, \theta^\star+\varepsilon) and its root lies there. Now nS(θ⋆)=∑jXjnS(\theta^\star) = \sum_j X_j with Xj=zj−σ(mj+θ⋆)X_j = z_j - \sigma(m_j+\theta^\star) independent, of mean zero, ∣Xj∣≤1|X_j| \le 1 and ∑jVar(Xj)=nκ\sum_j \mathrm{Var}(X_j) = n\kappa. Bernstein's inequality gives P(∣S(θ⋆)∣≥t)≤2exp⁡(−nt22(κ+t/3))≤2exp⁡(−nκε22e2ε(1+ε/3))≤δ.□P\bigl(|S(\theta^\star)| \ge t\bigr) \le 2\exp\Bigl(-\frac{n t^2}{2(\kappa + t/3)}\Bigr) \le 2\exp\Bigl(-\frac{n\kappa\varepsilon^2}{2e^{2\varepsilon}(1+\varepsilon/3)}\Bigr) \le \delta. \quad \square

Proposition B (lower bound). Fix the margins, θ0\theta_0, ε>0\varepsilon > 0 and δ∈(0,1/4)\delta \in (0, 1/4), with κ\kappa evaluated at θ0\theta_0. If n<e−4εlog⁡(1/(4δ))/(4κε2)n < e^{-4\varepsilon}\log(1/(4\delta))/(4\kappa\varepsilon^2), then every estimator θ~\tilde\theta has Pθ(∣θ~−θ∣≥ε)>δP_\theta(|\tilde\theta - \theta| \ge \varepsilon) > \delta for θ=θ0\theta = \theta_0 or for θ=θ0+2ε\theta = \theta_0 + 2\varepsilon.

Proof. Let P0,P1P_0, P_1 be the laws of the votes under θ0\theta_0 and θ1=θ0+2ε\theta_1 = \theta_0 + 2\varepsilon, and ψ=1[θ~≥θ0+ε]\psi = \mathbf 1[\tilde\theta \ge \theta_0 + \varepsilon]. An error below ε\varepsilon forces ψ=0\psi = 0 under P0P_0 and ψ=1\psi = 1 under P1P_1, so by the Bretagnolle-Huber inequality the two error probabilities sum to at least P0(ψ=1)+P1(ψ=0)≥12e−KL(P1∥P0)P_0(\psi=1) + P_1(\psi=0) \ge \frac12 e^{-\mathrm{KL}(P_1\|P_0)}, and the larger is at least 14e−KL(P1∥P0)\frac14 e^{-\mathrm{KL}(P_1\|P_0)}. With xj=mj+θ0x_j = m_j + \theta_0, pj=σ(xj)p_j = \sigma(x_j) and qj=σ(xj+2ε)q_j = \sigma(x_j + 2\varepsilon), we bound the KL divergence by the chi-square divergence and use Lemma 1, which gives qj−pj≤2εe2εσ′(xj)q_j - p_j \le 2\varepsilon e^{2\varepsilon}\sigma'(x_j): KL(P1∥P0)≤∑j(qj−pj)2σ′(xj)≤4ε2e4ε nκ<log⁡14δ.□\mathrm{KL}(P_1\|P_0) \le \sum_j \frac{(q_j - p_j)^2}{\sigma'(x_j)} \le 4\varepsilon^2 e^{4\varepsilon}\, n\kappa < \log\frac{1}{4\delta}. \quad \square

Remarks.

  1. Together, A and B give n=Θ(log⁡(1/δ)/(κε2))n = \Theta(\log(1/\delta)/(\kappa\varepsilon^2)) for bounded ε\varepsilon; a union bound replaces δ\delta by δ/m\delta/m for all mm annotators at once. The explicit constants are conservative: for fixed margins the estimate is asymptotically normal with standard error 1/κn1/\sqrt{\kappa n}, which matches simulation from n≈25n \approx 25 on.
  2. The margins are taken as given. In BA-DPO they are those of the current policy, and the result does not include their own estimation error. Shifting all margins along the flat direction of Theorem 3 moves every θk\theta_k by the same amount, so it does not change how the annotators' biases differ.
  3. For a vector attribute, the argument applies coordinate by coordinate to the judgments that differ in that coordinate only.

Comments on the paper

1 comment

Dear Reviewers,

We are writing to inform you that we have incorporated all the changes emerged during the rebuttal into the revised paper. Thank you again for your valuable feedback that helped improving our work.

As the discussion period is coming to an end, we would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns, or if there are any remaining points we can clarify.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.