- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents BA-DPO, a DPO post-training method that models annotator biases per preference. Each annotator bias is modeled as , with shared preferences and deviations. The method is theoretically characterized and evaluated against other DPO variants.
Critical strengths
- Removing annotator bias from DPO preferences is an important and understated problem.
- The method is thoroughly tested across 2 datasets, 2 model sizes, and 5-6 baselines.
Critical weaknesses
- The rationale for providing annotator annotations to DPO preferences is not present, and it's unclear whether other methods were tested in the same setting. What if the preferences had no annotator annotations, thus anonymous?
- It's unclear what the relative improvements are, and some parts of it are inconsistent. Is it high "Removed"? Then why is the method BA-DPO lower than R-DPO for MultiPref? Also, "Removed" is not measured in Table 3. Other metrics are provided without an explanation of whether the model is good or bad. The tables should be self-sufficient with enough explanations.
Questions for authors
The benefit of the theoretical convexity should be moved to the main content and be sufficiently explained.
4 comments
Thank you for the answer. It's good that consistent results are shown for BA-DPO regarding the "Removed" metric (even considering R-DPO), but I'm worried that this metric may be prone to cherry-picking.
Regarding RM and KL, what is the correct way to interpret the results? Seems like RM is negative for all Qwen2.5-0.5B-Instruct models and positive for Llama-3.1-8B-Instruct models. Why is that? If those are only sanity checks, then only the Signed-UltraFeedback result is clear, which is an interesting but limited result.
Thanks for your answer!
Removed is natural on Signed-UltraFeedback because the unbiased level is known: the reference is balanced by construction, so the target is its rate, and Removed measures how close a method gets to it. Since this is not a standard metric, we also report a standard test-set one, the held-out gap: the policy's accuracy on held-out annotator votes, on pairs that differ in the attribute minus pairs that do not. On MultiPref, where no length is the correct one, the gap is the main readout.
The judge's score is defined only up to a constant, so its sign carries no meaning: the 0.5B models write weaker answers than the 8B ones and score lower, both before DPO (the reference) and after it. RM is read within a model, and it checks that BA-DPO improves the answers over the reference about as much as DPO does. KL checks that BA-DPO moves the policy by the same order of magnitude as DPO. Both are checks: the MultiPref result rests on the held-out gap, which DPO opens on both attributes and every model, and which BA-DPO reduces by about half or more (Tables 2 and 3).
Dear Reviewer, we hope our answer on Removed, RM and KL clarifies how to read the tables. We would be glad to know whether you have any further concerns we have not addressed yet.
We first want to clarify that BA-DPO treats the annotators as anonymous. All it needs is, for each comparison, which response was preferred and the declared attribute of the two responses (with no attribute declared, BA-DPO is plain DPO), so it applies to any preference dataset, including those that keep only the majority preference per comparison. Without annotator IDs it learns one bias per attribute, shared by all annotators (the pooled variant, Section 3), which is in every table.
Annotator IDs are therefore optional, and they are only identifiers: they say which judgments come from the same annotator, nothing about who the annotator is, and the annotator can be a person or a model used as a judge. When they are available, the same training run also estimates each annotator's own bias, which makes the annotators auditable (Section 5.2).
All methods are trained on the same judgments, and each baseline gets the information it can use (App E.3): R-DPO and SamPO use no IDs, while Group-DRO and EM-DPO use the annotator IDs or classes. Even without IDs, BA-DPO removes the bias, and these two do not (Tables 1-2). We will state in each table which information each method uses, and say in Section 3 that annotator IDs are optional.
Thank you, we agree that the tables should stand on their own. The metrics are defined in Section 5.1, but the captions only point there. We describe each one below, with the direction that is better, and we will make every caption self-contained:
On MultiPref no length is the correct one, so removing more is not automatically better, and the gap is the main readout. This is why R-DPO's higher "Removed" is not a better result: its length penalty is set by hand, the larger its weight the shorter the answers, and at its published weight they collapse to 138 tokens (Table 9). BA-DPO instead estimates from the labels how much of the preference for longer answers is bias, and removes that part. Table 3 has no "Removed" column because DPO barely changes the markdown rate there, so a share of that change would be noise; the gap is the readout instead.
Thank you, we agree. The result is stated in the main text (Theorem 3), but its use is not explained, and we will add a paragraph after the theorem. Convexity gives two things.
First, the bias parameters are easy to fit. For a fixed policy, fitting θ is a logistic regression, which has a single optimum as long as the annotator does not vote for the attribute every time, or against it every time. The bias parameters therefore have no bad local optimum to get stuck in. Convexity also gives a sample-complexity result for the bias parameters, which we will add to the paper (stated with its proof in our reply to Reviewer Gpkz): with n comparisons that differ in the attribute, an annotator's bias is estimated to within about 1/κn, where κ≤1/4 is the average variance of a vote.
Second, it lets Theorem 3 describe all solutions, not only those near one solution. Because the loss is convex in the reward values and the bias parameters, every solution gives the same probability to every vote. This is what allows us to state exactly what the labels determine (how the annotators' biases differ from each other) and what they leave open (their shared level, which sets how strongly the policy favours the attribute and is fixed by the anchor).