acceptodds
Under review as a conference paper at ICLR 2027

Auditing Length Confounds in DPO Data Selection

Abstract

Preference-data selection methods for Direct Preference Optimization (DPO) are often credited with improving alignment when they outperform standard or hard-negative baselines. Such comparisons can be confounded by response length: a selector may change the length relation between chosen and rejected responses rather than isolate better preference evidence. We present a shared-surface audit for length confounds in DPO data-selection studies. It requires a length-controlled baseline, common evaluation pairs, length-matched and slate-level metrics, threshold sweeps, and clustered uncertainty. Across controlled DPO runs over UltraFeedback TruthfulQA slates, soft length weighting, graph negatives, and exposure-aware selection initially improve over a hard-negative baseline, but their apparent wins become unstable under length control: one run reverses across all primary metrics, while another produces metric-dependent outcomes rather than a stable dominance relation. On a frozen RewardBench evaluation, standard DPO, length control, and exposure-aware selection obtain 50.8%, 58.3%, and 61.7% accuracy, respectively, but the latter's +3.3 point delta over length control has a 95% paired-bootstrap interval spanning zero. Additional held-out and generated-task evaluations show both supporting and countervailing cases. The contribution is not a universal selector; it is a falsification standard for deciding when DPO data-selection gains survive a shared, confound-aware audit.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
direct preference optimization, data selection, length bias, alignment evaluation, confound auditing

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_vydcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a method to evaluate whether a method used to select the training set used for DPO is effective beyond mitigating DPO's length bias. Several

Critical strengths
  • Understanding the value of a data selection method after controlling for known confounds is an important problem to tackle.
  • There is a human evaluation and the section describing the human evaluation protocol is easy to follow.
Critical weaknesses
  • The writing of the paper leaves me largely unable to understand what has been done or why it has been done. Several aspects of the writing contribute to my challenges in following what the authors are saying. This is not due to unfamiliarity with DPO nor data selection techniques having written papers on both. (1) The authors appear to pull precise and technical language from the literature, but do not provide definitions or explanations for what the words mean. They are then used to modify each other or in a list (e.g. "length-controlled, shared, multi-metrics audit", "the paired bootstrap resamples aligned policy contrasts", "UltraFeedback TruthfulQA prompt slates"). Individually the words make sense, but put together I cannot always understand what exactly they are saying about the work done in the paper. (2) The paper does not dedicate space to making sure the reader is on the correct page about how data selection is done, what standard practices are, and what domain-specific words mean. (3) Several metrics and pieces of information are listed, but the information they provide about the proposed method is not made clear. For example, a git commit ID is provided, but no repo is linked and it is not clear why that git commit detail is needed.
  • The abstract ends with this text, "The contribution is not a universal selector; it is a falsification standard for deciding when DPO data-selection gains survive a shared, confound-aware audit." This makes it sound like the proposed method should apply to arbitrary confounds, but only length appears to be explored.
Questions for authors

I generally struggled to understand what the paper is doing, so am not able to articulate specific questions to ask. To improve overall clarity, it would be helpful to provide extra context. For example, start with an overview of standard data selection approach in Section 3 to help the reader more quickly pick up what the maybe technical terms mean.

Reviewer_d5m7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose an audit protocol for response-length confounds in DPO data selection, combining shared evaluation pairs, a length-controlled selector, five primary metrics, threshold sweeps and clustered uncertainty. It reports small-model reversals and a larger confirmation study, where exposure-aware selection passes its gate at three of four scales, with further external, human, and retention evaluations.

Critical strengths

None

Critical weaknesses
  1. The presentation is difficult to follow and several of the key design decisions are poorly justified. The paper suggests length as a potential shortcut, but it’s not clear how often this confound alters conclusions about existing data-selection methods, or why the proposed audit is the correct benchmark. The experimental scope is also limited compared to the framing, as the selectors primarily select rejected responses while keeping the prompt and selected response constant. It is an open question how the protocol generalizes to the pool-based setting of complete preference triplets, where prompt coverage and the quality of selected responses are also affected. Figure 1 provides only a numbered description of the steps. Finally, the candidate baselines are internally built variants, not published selection methods. The practical relevance of auditing representative published methods would be better established, distinguishing the protocol’s contribution from the behavior of these particular selectors. In conclusion, I am not convinced about the need to follow the auditing pipeline proposed by the authors.

  2. The length-controlled baseline does not isolate the effect of length. Choosing a rejection with a smaller length gap also changes its content, correctness, difficulty, and other quality attributes, all of which can affect the five evaluation metrics. So, a candidate losing advantage versus this baseline does not prove its initial advantage was due to length; the baseline may just be picking more useful preference pairs for other reasons. While the authors acknowledge that the audit does not establish causal effects, its interpretation of reversals as proof of length confounding requires more caution. The authors should characterize the quality and difficulty distributions induced by each selector, and add controlled experiments that vary length while preserving preference-relevant content as closely as possible.

Questions for authors

Please refer to the weakness section.

Reviewer_Vp3mOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether improvements attributed to DPO data-selection methods may partly reflect differences in response length, or the use of different evaluation pairs across selectors. The authors propose an audit protocol that combines shared evaluation pairs, a length-controlled baseline, five primary metrics, length-threshold sweeps, and clustered confidence intervals. In retrospective experiments, they find that gains over standard or hard-negative DPO can become smaller, reverse, or depend on the metric when compared against length control. The paper further reports 78 confirmatory training units, along with external benchmarks, placebo controls, capability-retention tests, and human evaluations. Exposure-aware selection passes the proposed gate at three of the four Qwen2.5 scales, and the authors report an overall gain of 2.6 points over length control across scales.

Critical strengths

The paper addresses a practically relevant question. Response length can influence both how preference pairs are constructed and how they are scored, so treating it as an explicit control is a useful contribution to evaluation practice.

The argument for shared evaluation is well motivated. Comparing all policies on the same preference pairs is a simple and sensible way to avoid differences that come from selector-specific test sets.

The retrospective examples are clear and informative. They illustrate how a method can outperform a hard-negative baseline while still trailing a length-controlled one, and how conclusions can depend on the matching threshold.

The evaluation is broad, covering external preferences, capability retention, placebo controls, and human judgments. I also appreciate that the authors report unresolved and unfavorable results, which makes the scope of the claims easier to understand.

Critical weaknesses

The graph/exposure scores, negative-type construction, and tie-breaking rules are currently described mainly in prose. Since these choices shape the training data, formal definitions or pseudocode would make it easier for readers to understand where the reported gains come from.

The confirmatory experiments could be documented in more detail. The retrospective runs are described thoroughly, but the larger confirmatory study includes fewer training and evaluation details, and the supplement mainly provides an aggregate ledger. Per-prompt and per-seed records would allow readers to reproduce the clustered intervals and check pair alignment. Since the paper positions the audit as an executable standard, this additional material would strengthen the contribution considerably.

The paper describes blinding and inter-annotator agreement clearly. It would also be useful to know the prompt sources, how models and seeds were assigned, the decoding settings, how factuality was assessed, and how meaning was preserved during rewriting. These details would help readers interpret the 25.6-point preference gap and how much of it remains after length matching.

Questions for authors

Could you provide pseudocode or formal definitions for each selector, including the graph/exposure scores, negative-type labels, and tie-breaking, as well as the full training configurations for the confirmatory runs?

Does the anonymous artifact include per-prompt scores for each confirmatory policy and seed? If so, which scripts reproduce Tables 7–12 and their confidence intervals?

At what stage were the gate, matching threshold, scale-advancement rule, and noninferiority margins determined? Would the conclusions hold under a leave-one-seed-out analysis, or under a gate that also requires interval-based noninferiority on the remaining metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.