acceptodds
Under review as a conference paper at ICLR 2027

Conformal Preference Weighting for Label-Noise-Robust Direct Preference Optimization

Abstract

Direct Preference Optimization (DPO) and its uncertainty-aware variants assume the preference label attached to each response pair is correct. A flipped label does not merely add noise: it inverts the training signal, so the policy is trained toward the response it should reject. TUR-DPO addresses this by reweighting pairs with a model-internal uncertainty heuristic. We show that heuristic is non-discriminative in practice — across every run we measure it emits a single constant (, zero variance) for every pair, clean and corrupted alike — and, on a complete grid of four arms, four noise levels and three matched seeds, that it is worse than not reweighting at all, falling below plain DPO in of paired runs under noise. We replace it with Conformal Preference Weighting (CPW), which derives the weight from a held-out calibration split via split conformal prediction and conformal risk control, and with CPW+, which fuses a local and a relational view of each pair before a single selective-risk calibration. CPW+ exceeds DPO under label noise by paired accuracy over the runs that certified a threshold ( of positive), and on accuracy retention — each arm against its own clean baseline — exceeds TUR-DPO in all nine matched runs at every noise level. Calibration uses no ground-truth corruption labels at any point. We are equally explicit about what does not work: the method costs accuracy on clean data; a third, optimization-aware view of our own design scores flipped pairs higher than clean ones and is given zero weight; the calibration certified no threshold in of runs; the structural risk term was degenerate under the extractor used for the accuracy runs, so those numbers are a lower bound rather than an estimate; and on MATH the weights fail to separate corrupted pairs (AUROC ) and CPW is the weakest arm.

Primary area:
optimization
Keywords:
Direct Preference Optimization (DPO), Conformal Prediction, Conformal Preference Alignment, Heuristic Uncertainty Weighting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nyySOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CPW and CPW+, which down-weight unreliable preference pairs during DPO using conformal calibration and reasoning-graph information. CPW+ additionally considers consistency between comparisons. The authors report improved robustness to noisy preferences across reasoning and dialogue tasks.

Critical strengths
  1. Addresses a practical problem: incorrect preference labels can actively harm alignment.
  2. Evaluates multiple model sizes and families, including real human- and AI-annotated preference datasets.
  3. Matched-seed comparisons and weight-shuffle controls provide useful evidence that selecting the right pairs matters.
Critical weaknesses
  1. The main experiments do not validate the final guarantee. They use an earlier calibration rule without the stated statistical protection. Moreover, the guarantee controls graph structure, rather than whether preference labels are correct.
  2. The GSM8K setting provides strong additional supervision. The method checks responses against ground-truth answers, which already reveal the correct preference. A simple verifier-based filtering or relabeling baseline is needed to establish the benefit of conformal calibration.
  3. The practical advantage over standard DPO remains unclear. DPO with β = 0.1 achieves slightly higher GSM8K solve rates than CPW+ at every tested noise level (Table 3).
  4. The evaluation needs cleaner separation. Prompts can overlap across training, validation, and calibration splits, while preference accuracy is measured against noisy labels. Prompt-disjoint splits and direct evaluation against clean labels would strengthen the conclusions.
Questions for authors

See Weaknesses Above.

1 comment
AuthorsOpenReview ↗

To Reviewer nyyS, 1/3: the guarantee

We thank the reviewer. Three of the four weaknesses are shared with ifGA and fWD7, and we have accepted them; this reply states our position and adds what is specific to this review.

The main experiments do not validate the final guarantee. Correct. The main grid used an earlier gate with a pointwise bound and no correction for selecting the threshold on the same data, so it is not valid after selection. Those runs show that the fused weight helps empirically; they are not evidence for Eq. 11, and the revision says so in Appendix B.6 and in the main-grid table captions. Under the rule-based extractor structural risk is binary, so the corrected pair-level procedure returns no threshold; under a model-based extractor and the pair-level split it returns λ∗=0.46\lambda^* = 0.46 on clean data, 0.470.47 at 10% noise and 0.480.48 at 20% and 30% noise, under the stated assumptions (Table 10 of the revision; our reply to ifGA 1/4 gives the details). These pair-level values are separate from the prompt-disjoint result, λ∗=0.478±0.004\lambda^* = 0.478 \pm 0.004 over seeds 42-44 (Appendix E.7; our reply to fWD7 4/4). Because the implemented scores use calibration ranks and shared-prompt relations, this guarantee is approximate for the implementation.

The guarantee controls graph structure, not whether labels are correct. Also correct. Appendix B.5 (Remark 4) implements an oracle loss (the indicator that a pair is corrupted), which would make the certificate a statement about labels but presumes a verified subset, and the structural loss, which needs no corruption labels. Every result uses the structural loss.

The link between low weight and a flipped label is therefore empirical: on D1 the low-weight detector gives AUROC 0.911 / 0.840 / 0.833 at 10 / 20 / 30% noise, and on MATH 0.605. In the revision, Eq. 11 is named a structural-risk certificate, the abstract states that it does not certify label correctness, and the detector AUROC is the only claimed link to label noise.

Reviewer_fWD7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is written in the context of direct preference optimization under preference label noise. Existing works like TUR-DPO do uncertainty-aware reweighting of preference pairs or correct the loss using a known noise rate but fail in providing any guarantee on which pairs are unreliable. Further, the heuristic weights can collapse to a constant that does no selection. This work proposes Conformal Preference Weighting (CPW) and its multi-view extension CPW+ and shows theoretical results on finite-sample, distribution-free control of structural risk on the accepted pairs. The paper also shows empirical results on 4 settings (GSM8K, MATH, UltraFeedback, Anthropic-HH) showcasing the proposed approach outperforms baselines in the noisy-label setting.

Critical strengths
  1. The method proposed in the paper doesn’t require oracle noise rates or ground-truth corruption labels, which is often a requirement in existing works. Instead, it proposes a novel approach that fuses local confidence with relational graph consistency to selectively down-weight unreliable pairs.
  2. The experiments are carefully designed and insights such as the weight-shuffle control are also helpful in showing that the gains come from which pairs are down-weighted and not just from a smaller step size.
Critical weaknesses
  1. The work only compares to limited baselines like TUR-DPO. There are multiple classes of work on being robust to preference noise, for instance those based on distributionally robust preference optimization (like Dr. DPO, DPO-PRO), and loss correction with a known noise rate (like rDPO).
  2. The theoretical guarantee bounds structural risk on accepted pairs and not label correctness, which is the actual problem the paper is addressing.
Questions for authors
  1. The experiment results are shown on older models like Qwen2.5-0.5B, Llama-3.2-3B and Gemma-2-2B. Can you show at least one set of results that extend to more modern LLMs like Qwen3.5 class or Gemma 4 models?
  2. The splits are drawn over pairs and not prompts, so calibration and training pairs can share a prompt. Can you show whether the results hold with prompt-disjoint splits?
5 comments
AuthorsOpenReview ↗

To Reviewer fWD7, 1/4: baselines beyond TUR-DPO

We thank the reviewer for naming the specific classes of related work.

Loss correction with a known noise rate (rDPO, cDPO). The submitted appendix already compared against rDPO (Chowdhury et al., 2024) and cDPO on D2 at all three noise levels, seeds 42-44, same beta and budget. rDPO is run with the true flip rate (an oracle CPW never receives) and with a misspecified rate. Given the true rate, rDPO is the strongest arm at every noise level, ahead of CPW+. Among arms not given the rate, CPW+ is strongest at 20% and 30% (0.613 vs 0.609 and 0.544 vs 0.531 for misspecified rDPO); at 10% noise, where the fixed misspecification parameter (\hat{\rho}=0.10) coincidentally matches the injected noise rate, rDPO leads (0.718 vs 0.715). The reviewer's point about placement stands: this comparison moves into the main results (Table 2 of the revision), and the contributions state that CPW+ competes with methods not given the noise rate.

Distributionally robust preference optimization (Dr. DPO, DPO-PRO). This class was missing. The revision cites Dr. DPO (Wu et al., ICLR 2025) and DPO-PRO (Kim et al., 2025) and evaluates both under the main-grid D2 protocol (beta = 2.0, seeds 42-44, same budget; Appendix E.1). At 30% noise Dr. DPO reaches 0.529 +/- 0.013 and DPO-PRO 0.525 +/- 0.013, against 0.544 +/- 0.015 for CPW+. Both were run at the main-grid beta only, with default robustness hyperparameters (beta' = 1.0 for Dr. DPO) and without a beta sweep, which we state. The distinction is conceptual as well: Dr. DPO is a distributionally robust objective that reweights pairwise DPO losses through a log-sum-exp (KL-based robust) objective, with the additional parameter beta' controlling robustness, so each pair's weight depends on the current policy's loss (Appendix E.1); DPO-PRO hedges against uncertainty in the preference probability; neither produces a per-pair reliability score from external evidence, which is what our detector analysis (Table 6 of the revision) measures.

On TUR-DPO as the headline baseline. TUR-DPO in the main grid is the collapsed configuration; its non-degenerate version under the model-based extractor (Table 10) still falls below CPW and CPW+. Each table now states which configuration it uses.

AuthorsOpenReview ↗

To Reviewer fWD7, 2/4: structural risk, not label correctness

This is correct, and it is the most important scope condition of the method. Eq. 11 bounds the selective structural risk of the accepted pairs, E[R | s >= lambda*] <= alpha, at confidence 1 - delta (Theorem 1 of the revision, which now states this estimand exactly as the code computes it). Label correctness cannot be certified without clean labels, which the method deliberately does not use.

Appendix B.5 (Remark 4) implements two losses for the bound. An oracle loss, the indicator that a pair is corrupted, would make the certificate a statement about labels, but presumes a verified subset of the calibration split. The structural loss uses the pair's topology risk and requires no corruption labels. Every result in the paper uses the structural loss.

The link between low weight and a flipped label is therefore empirical, and we measure it. On D1, low CPW weight as a flip detector gives AUROC 0.911 / 0.840 / 0.833 at 10 / 20 / 30% noise (Table 6 of the revision); on MATH the same detector gives 0.605, so the association does not transfer automatically. Under the model-based extractor, the pair-level LTT procedure returns a non-trivial threshold under its stated assumptions; the guarantee is approximate for the implemented scores because calibration ranks and shared-prompt relations induce dependence. The label contamination of the accepted set falls from 30.0% to 11.8% +/- 0.5%; this is a measurement, not a consequence of the risk bound.

Changes in the revision.

  1. Eq. 11 is called a structural-risk certificate throughout.
  2. The abstract states that it does not certify label correctness.
  3. The detector AUROC, with the MATH result beside it, is the only claimed link to label noise.
  4. The contributions separate the proved claim (structural-risk control on the accepted set) from the measured one (reduced degradation under label noise).

A reader who takes Eq. 11 as a guarantee about labels has been misled by our framing, and the fix is framing, not a stronger theorem.

AuthorsOpenReview ↗

To Reviewer fWD7, 3/4: more recent models

The reviewer asked for at least one set of results on a Qwen3.5- or Gemma-4-class model. The revision reports two, under the same protocol as the other backbones, beside the base and instruction-tuned models of the earlier generation.

Results (Table 14 of the revision; 30% noise, seeds 42-44):

Backbonez^rel coverageDPOTUR-DPO (collapsed)CPW+
Qwen2.5-3B68.2%0.568 +/- 0.0110.561 +/- 0.0150.597 +/- 0.012
Llama-3.2-3B65.8%0.554 +/- 0.0130.548 +/- 0.0140.583 +/- 0.012
Gemma-2-2B67.4%0.546 +/- 0.0150.539 +/- 0.0160.574 +/- 0.014
Qwen2.5-3B-Instruct72.4%0.582 +/- 0.0120.575 +/- 0.0140.614 +/- 0.010
Gemma-2-2B-IT70.8%0.574 +/- 0.0130.566 +/- 0.0150.605 +/- 0.011
Qwen3.5-9B75.1%0.613 +/- 0.0110.605 +/- 0.0120.648 +/- 0.009
Gemma-4-E4B-it73.2%0.598 +/- 0.0120.590 +/- 0.0130.631 +/- 0.010

How to read the table.

  • z^rel coverage is the fraction of pairs whose relational view z^rel is defined, that is, whose prompt carries at least two preference edges (Appendix B.3). The 94.6% figure in our other replies is a different quantity: availability on the calibration split under prompt-disjoint splitting (Table 17). It is not the fraction of pairs accepted by the gate.
  • TUR-DPO applies the topology- and uncertainty-aware heuristic weight of Eq. 2 instead of a conformally calibrated weight, and shares the shaped loss with CPW and CPW+; it is a heuristic reweighting baseline, not a conformal method. Here it is the collapsed rule-based configuration (constant weight 0.8912), as in the main grid (Section 5.1); its non-degenerate configuration is reported separately in Table 10.
  • Mean +/- s.d. is over three seeds (42, 43, 44), and every arm is run with the same three seeds.

On the two current-generation backbones the CPW+ margin over DPO is +3.5 points (Qwen3.5-9B) and +3.3 points (Gemma-4-E4B-it), against +2.8 to +3.2 points on the earlier-generation backbones. The stronger models raise the absolute level (DPO moves from 0.582 on Qwen2.5-3B-Instruct to 0.613 on Qwen3.5-9B), and the relative benefit of calibrated weighting is preserved rather than absorbed by the stronger backbone. Together with the Qwen2.5 scale ladder from 0.5B to 7B (Table 13), the evaluation now covers two model generations, sizes from 0.5B to 9B, and both base and instruction-tuned variants.

Reviewer_ifGAOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses preference-label noise in DPO through Conformal Preference Weighting (CPW) and its multi-view extension, CPW+. CPW combines conformal p-values from an offline reward margin with a structural-risk penalty. CPW+ adds relational consistency and a threshold-selection procedure intended to control structural risk.

The evaluation covers synthetic label flips, human- and AI-annotated preference datasets, and several model scales and families. The paper reports improvements over β-matched DPO and includes controls supporting the usefulness of pair-specific weighting.

Critical strengths

The weight-shuffle and constant-weight controls provide useful evidence that the gains depend on which pairs are down-weighted, rather than simply reducing the overall update magnitude.

The paper identifies a concrete failure mode of uncertainty weighting under deterministic extraction and appropriately limits this finding to that configuration.

The evaluation extends beyond preference accuracy to downstream generation, real preference corpora, and multiple model scales and families.

Critical weaknesses

Appendix B.5 explains that the main grid used an earlier gate without correction for threshold selection. The corrected procedure certifies no threshold with the main rule-based extractor, so the headline results cannot be treated as evidence for the guarantee in Eq. 11.

In Table 3, DPO with β = 0.1 has numerically higher downstream solve rates than CPW+ at every noise level. Without comparable tuning for both methods, the gains over β = 2.0 DPO do not establish an advantage over a well-tuned standard baseline.

Questions for authors

With the same verifier and shaped loss, how does CPW+ compare with direct verifier-based filtering or continuous weighting without conformal calibration?

Does CPW+ retain an advantage when β is tuned for both methods under the same budget, particularly on downstream generation rather than preference accuracy alone?

For Table 11, please also specify the evaluation sample size, reference policy, scoring implementation, and whether the win-rate checkpoints were trained with additional injected noise.

4 comments
AuthorsOpenReview ↗

To Reviewer ifGA, 1/4: the main grid and Eq. 11

We thank the reviewer for identifying this, and we accept the criticism in full.

Main-grid provenance (Appendix B.6). The main-grid CPW+ runs (Tables 3-5 of the revision) predate the Learn-then-Test certificate of Eq. 11. They used an earlier gate that scanned the candidate thresholds and kept the first whose pointwise Clopper-Pearson bound on the accepted-set risk was at most α\alpha, with no correction for selecting the threshold on the same calibration data. A pointwise bound is not valid after selection, so those thresholds are selected, not certified, and the runs carry no guarantee. On D2 the gate returned a threshold in 10 of the 12 CPW+ runs; the other 2 trained with wi=1w_i = 1. The main-grid runs therefore measure whether the weighting helps empirically; they are not evidence for Eq. 11. The abstract, the contributions, Section 5.4, Appendix B.6 and the captions of Tables 3-5 now state this.

The corrected procedure under a model-based extractor (Table 10). Under the deterministic rule-based parser, structural risk is near-binary (0 for multi-sentence responses, 1 for single-sentence ones), so the corrected pair-level Learn-then-Test procedure returns no threshold. With a model-based judge (Qwen2.5-3B-Instruct; Appendix E.2, Table 10), structural risk takes 20 distinct values, and the procedure returns a non-trivial threshold under the stated pair-level assumptions: λmulti∗=0.46\lambda^*_{multi} = 0.46 on clean data, 0.470.47 at 10% noise and 0.480.48 at 20% and 30% noise. CPW+ preference accuracy under this extractor is 0.553 at 30% noise.

Splits and the relational view. Under the pair-level split used for Table 10, the 452 calibration pairs are drawn across D2 while zrelz^{rel} is computed over prompt comparison graphs across the dataset, so 68-80% of calibration pairs have a defined zrelz^{rel} partner (which may fall in calibration or training). Under the prompt-disjoint split (Table 17), prompts are partitioned intact so no prompt crosses splits; relational-view availability on the calibration split is 94.6% +/- 0.3%, and the procedure returns λmulti∗=0.478±0.004\lambda^*_{multi} = 0.478 \pm 0.004 with empirical test structural risk 0.081±0.003≤α=0.100.081 \pm 0.003 \le \alpha = 0.10 (Appendix E.7). Because the implemented scores use calibration ranks (zlocalz^{local}) and shared-prompt relations (zrelz^{rel}), and calibration pairs can share a prompt, the guarantee is exact for pair-wise scores and approximate for the implemented ones (Appendix B.6).

AuthorsOpenReview ↗

To Reviewer ifGA, 2/4: Table 3 and the beta = 0.1 baseline

We accept the criticism of the presentation. The submitted Table 3 had DPO at both beta = 0.1 and beta = 2.0, but CPW+ only at beta = 2.0. DPO at beta = 0.1 (0.258 at 30% noise) was above every beta = 2.0 arm, including CPW+ at beta = 2.0 (0.256). The bolding showed this, while the text implied an advantage for CPW+. That comparison mixed values of beta.

The table below adds the missing row, CPW+ at beta = 0.1, under the same protocol. It does not contradict the submitted table: once beta is matched, CPW+ is higher at both beta = 0.1 and beta = 2.0.

Changes in the revision. Table 3 (Table 4 in the revision) is bolded within each beta group, states that DPO at beta = 0.1 is not beta-matched to the beta = 2.0 arms, and adds CPW+ at beta = 0.1. The abstract states the downstream result as a matched-beta comparison.

Protocol (applies to everything below). All arms use the same D2 data and splits, the same budget (one epoch, effective batch 32, learning rate 5e-6, seeds 42-44), and the same evaluation: GSM8K test solve rate on 1,319 problems with greedy decoding. For each method and noise level, beta* was selected by maximizing the mean GSM8K solve rate across seeds on the GSM8K questions represented in D2's pair-level validation split. Hyperparameter selection was performed using validation questions only. The 1,319 GSM8K test problems were strictly held out and were not used for hyperparameter tuning (Appendix E.9); they only report the numbers below. As in the main grid, the CPW+ runs at beta = 0.1 use the earlier, uncertified gate (caption of Table 4 of the revision; provenance in Appendix B.6).

Matched beta. GSM8K solve rate, mean +/- s.d. over three seeds:

Armclean10%20%30%
DPO beta=0.1.298+/-.008.284+/-.009.271+/-.008.258+/-.009
CPW+ beta=0.1.301+/-.007.291+/-.008.280+/-.008.271+/-.009
DPO beta=2.0.286+/-.009.268+/-.010.249+/-.011.231+/-.012
CPW+ beta=2.0.294+/-.008.282+/-.008.267+/-.007.256+/-.009

Independent tuning under the same budget (Question 2). We swept beta in {0.05, 0.1, 0.5, 1.0, 2.0} for both methods (Table 20). Validation selects beta* = 0.1 for both. At that beta and 30% noise, CPW+ is ahead by +1.3 pp in mean test solve rate (0.271 vs. 0.258, mean over seeds 42-44; Table 20). We report these as descriptive mean differences across matched seeds (0.271 vs. 0.258, mean over seeds 42--44; Table 20); given the small sample size (N=3N=3), no inferential significance claim is made (Appendix E.11).

What we claim, and what we do not. CPW+ has the higher mean solve rate in every cell of the matched-beta table. With three seeds, the differences are of the order of the across-seed s.d., so the paper reports them as descriptive matched-seed differences and makes no significance claim (Appendix E.11). We therefore claim:

  • a higher mean solve rate in every reported matched-beta condition;
  • at the validation-selected beta and 30% noise, a +1.3 pp mean gain.

We do not claim that CPW+ is better than DPO in general, or at every beta and noise level.

AuthorsOpenReview ↗

To Reviewer ifGA, 3/4: verifier filtering and continuous weighting (Question 1)

The reviewer asked how CPW+ compares, with the same verifier and shaped loss, with direct verifier-based filtering and with continuous weighting without conformal calibration. Table 18 of the revision reports both, together with verifier relabeling (D2, beta = 2.0, seeds 42-44). At 30% noise:

ArmPreference accuracyGSM8K solve rate
DPO.519 +/- .013.231 +/- .012
Verifier filtering.535 +/- .014.245 +/- .012
Continuous sigmoid weighting.528 +/- .016.239 +/- .013
CPW.542 +/- .009.248 +/- .011
CPW+.544 +/- .015.256 +/- .009
Verifier relabeling (oracle)‡.814 +/- .015 true-label (.626† noisy).284 +/- .009

Losses: DPO, Verifier Filtering (corresponds to the same filtering strategy as the data-cleaning baseline "DPO on verifier-consistent pairs" in Table 2; verifier-contradicted pairs are dropped, and the remaining pairs are trained with plain DPO, wi=1w_i = 1, γ=0\gamma = 0), and Verifier Relabeling (contradicted pairs are corrected and trained with plain DPO, wi=1w_i = 1, γ=0\gamma = 0) use the plain DPO objective. Its reported preference accuracies coincide with the Table 2 values (0.708 / 0.609 / 0.535 at 10 / 20 / 30% noise).

Objective distinction and mechanistic analysis (plain DPO vs. shaped loss). As documented in the caption of Table 18 and in Appendix E.8, Verifier Filtering and Verifier Relabeling are trained with the standard unshaped DPO objective (wi=1w_i = 1, γ=0\gamma = 0), whereas Continuous Sigmoid Weighting (w=σ(Δrϕ)w = \sigma(\Delta r_\phi)) and CPW+ use the shaped objective (γ=1.0\gamma = 1.0). The comparison between Verifier Filtering and CPW+ therefore reflects both different training objectives and different treatments of preference pairs. We did not run a hard-filtering baseline with γ=1.0\gamma = 1.0, so this comparison does not isolate the effect of calibrated weighting; Table 18 and Appendix E.8 state this limitation.

Under the same shaped objective (γ=1.0\gamma = 1.0), Table 18 compares Continuous Sigmoid Weighting with CPW+. At 30% noise, CPW+ has higher preference accuracy (0.544 vs. 0.528) and downstream solve rate (0.256 vs. 0.239). This is a controlled comparison of calibrated and uncalibrated continuous weighting under the same shaped objective. Because CPW+ also uses the relational view zrelz^{rel} and the gate, this comparison does not isolate conformal calibration alone. The closer comparison is single-view CPW against the same sigmoid weights, since both use only the local view and the same shaped loss: 0.542 vs. 0.528 accuracy and 0.248 vs. 0.239 solve rate at 30% noise. That pair still differs in more than calibration, because CPW also applies the structural-risk term and the weight floor of Eq. 7, so no arm in Table 18 isolates conformal calibration on its own.

The two methods also differ in how they treat preference pairs:

  1. Pair retention. At 30% noise on D2, the verifier flags 6,780 contradicted D2 pairs (Appendix E.8), and Verifier Filtering discards the contradicted training pairs among them. CPW+ removes no pairs: pairs below the threshold selected by the earlier, uncertified gate remain in training with the floor weight wmin⁡=0.05w_{\min} = 0.05 (Eq. 12; Appendix B.6).
  2. Binary filtering vs. graded weighting. Verifier Filtering makes a binary selection from the final-answer check (qtaskq_{task}) and gives every retained pair the same weight (wi=1w_i = 1). CPW+ assigns graded pair weights from its fused score smultis^{multi}, which combines the split-conformal p-value of the local view (zlocalz^{local}) with relational graph consistency (zrelz^{rel}) (Section 3.5, Eq. 12).

(‡ The relabeling row reports accuracy against the true labels, not the noisy validation labels, so it is not directly comparable with the other accuracy rows. †\dagger Analytically derived oracle reference value, not a measured noisy-validation result: the measured true-label accuracy of .814 under uniform 30% flips of the validation labels gives 0.814 x 0.70 + 0.186 x 0.30 = 0.626. It holds only under this uniform-flip noise model and is not a general upper bound for other noise types.)

AuthorsOpenReview ↗

To Reviewer ifGA, 4/4: Table 11

The reviewer is right that the submitted paper did not document the win-rate evaluation. The revision adds a specification table (Table 11 of the revision), which answers the four questions:

  1. Sample size: 1,000 held-out test prompts per corpus, each judged in both orders (2,000 pairwise comparisons).
  2. Reference policy: Qwen2.5-3B-Instruct used as an unaligned SFT checkpoint (Table 11), i.e. the policy before our preference training. Win-rate is measured against this checkpoint.
  3. Scoring: GPT-4o (gpt-4o-2024-08-06, temperature 0) as a pairwise judge, with full position swap; ties count 0.5.
  4. Injected noise: the win-rate checkpoints were trained under natural annotator noise only, without additional injected flips. Preference accuracy is reported both under natural noise and with +20% injected flips.

On UltraFeedback, CPW+ reaches 55.4% against 51.2% for DPO (+4.2 points). On Anthropic-HH, no prompt carries a second comparison, so the relational view is unavailable. There CPW+ falls back to the local view and is on par with single-view CPW in preference accuracy. The +4.2-point figure appears in the abstract, Appendix E.3 and Table 12, with the same protocol everywhere.

What this review changes in the paper.

  1. The main-grid result is presented as an empirical gain under an uncertified gate; the provenance is stated in Appendix B.6.
  2. Eq. 11 is a structural-risk certificate for the selective risk of accepted pairs. Theorem 1, Algorithm 1 and the code use the same estimand and unit.
  3. Table 3 is re-bolded within beta groups and extended with CPW+ at beta = 0.1. A matched-beta sweep and a validation-tuned beta sweep are added.
  4. GSM8K/MATH are framed as a controlled testbed with an oracle verifier. Verifier relabeling, verifier filtering and uncalibrated weighting are reported.
  5. The win-rate protocol is documented.

These were defects in what we claimed and how we presented it, and the first one is one we should have caught ourselves. We would be glad to answer anything further during the discussion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To Reviewer nyyS, 2/3: the verifier, and the advantage over DPO

GSM8K provides strong additional supervision. The reviewer is right. On D2 the final-answer check flags exactly the injected flips (2,263 / 4,623 / 6,780 at 10 / 20 / 30% noise), so verifier relabeling recovers the clean labels. We ran the requested baseline (Table 18 of the revision): relabeling reaches 0.814 true-label accuracy and 0.284 solve rate at 30% noise, above CPW+ (0.256). CPW+ does not beat it, and we do not claim it does; it uses the benchmark's reference answers, which open-ended preference data lack. With the same verifier signal, CPW+ outperforms hard filtering (0.544 vs. 0.535 preference accuracy, and 0.256 vs. 0.245 solve rate at 30% noise); hard filtering here corresponds to the same filtering strategy as the data-cleaning baseline reported in Table 2 as "DPO on verifier-consistent pairs". CPW+ also outperforms uncalibrated sigmoid weighting (0.544 vs. 0.528 and 0.256 vs. 0.239). Hard filtering uses plain DPO (γ=0\gamma = 0) while CPW+ uses the shaped loss (γ=1.0\gamma = 1.0), so that comparison differs in the objective as well as the weighting and does not isolate the effect of conformal calibration. The sigmoid-weighting arm uses the same shaped loss (γ=1.0\gamma = 1.0) as CPW+; this is a controlled comparison under the same shaped objective, but because CPW+ also uses the relational view zrelz^{rel} and the gate, this comparison does not isolate conformal calibration alone. The closer comparison is single-view CPW against the same sigmoid weights, since both use only the local view and the same shaped loss: 0.542 vs. 0.528 accuracy and 0.248 vs. 0.239 solve rate at 30% noise. That pair still differs in more than calibration, because CPW also applies the structural-risk term and the weight floor of Eq. 7, so no arm in Table 18 isolates conformal calibration on its own. The revision introduces GSM8K and MATH as a controlled testbed for weighting quality, not as evidence that conformal calibration improves on direct use of a verifier.

The practical advantage over standard DPO. Accepted for the submitted presentation: in Table 3, DPO at beta = 0.1 was ahead of every beta = 2.0 arm. The revision re-bolds the table within beta groups and adds CPW+ at beta = 0.1. When beta is matched, CPW+ has a higher mean solve rate than DPO at every noise level at both beta = 0.1 and 2.0. With beta selected for both methods by solve rate on the GSM8K questions represented in D2's pair-level validation split, never on the 1,319 test problems (Appendix E.9, Table 20), both select beta* = 0.1, and CPW+ keeps a +1.3 pp mean advantage at 30% noise (mean over seeds 42-44). We report these as descriptive mean differences across matched seeds (0.271 vs. 0.258, mean over seeds 42--44; Table 20); given the small sample size (N=3N=3), no inferential significance claim is made (Appendix E.11). The full table is in our reply to ifGA (2/4).

AuthorsOpenReview ↗

To Reviewer nyyS, 3/3: cleaner separation of the evaluation

Prompt overlap across splits. On D2 a prompt contributes several comparisons, and the submitted splits were drawn over pairs. We re-ran with prompt-disjoint partitioning (DPO and CPW+, all four noise levels, seeds 42-44, --split prompt). The cost is about one point for both methods, and the CPW+ advantage is preserved at every noise level. The full table is in our reply to fWD7 (4/4) and in Table 17 of the revision.

Evaluation against clean labels. The submitted Table 15 reported a closed-form debiased estimate, which recovers true accuracy only in expectation. The revision reports accuracy measured directly against the recorded is_flipped mask (Table 16; D2, mean +/- s.d., three seeds):

Arm10% true20% true30% true
DPO.738+/-.014.674+/-.011.542+/-.018
TUR-DPO.732+/-.012.651+/-.019.528+/-.022
CPW.752+/-.016.668+/-.015.601+/-.015
CPW+.771+/-.021.687+/-.006.607+/-.021

CPW+ has the highest true-label accuracy at every noise level; at 30% noise its margin over DPO is 6.5 points (0.607 vs 0.542).

We thank the reviewer for pressing on the evaluation; both points improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.