- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CPW and CPW+, which down-weight unreliable preference pairs during DPO using conformal calibration and reasoning-graph information. CPW+ additionally considers consistency between comparisons. The authors report improved robustness to noisy preferences across reasoning and dialogue tasks.
Critical strengths
- Addresses a practical problem: incorrect preference labels can actively harm alignment.
- Evaluates multiple model sizes and families, including real human- and AI-annotated preference datasets.
- Matched-seed comparisons and weight-shuffle controls provide useful evidence that selecting the right pairs matters.
Critical weaknesses
- The main experiments do not validate the final guarantee. They use an earlier calibration rule without the stated statistical protection. Moreover, the guarantee controls graph structure, rather than whether preference labels are correct.
- The GSM8K setting provides strong additional supervision. The method checks responses against ground-truth answers, which already reveal the correct preference. A simple verifier-based filtering or relabeling baseline is needed to establish the benefit of conformal calibration.
- The practical advantage over standard DPO remains unclear. DPO with β = 0.1 achieves slightly higher GSM8K solve rates than CPW+ at every tested noise level (Table 3).
- The evaluation needs cleaner separation. Prompts can overlap across training, validation, and calibration splits, while preference accuracy is measured against noisy labels. Prompt-disjoint splits and direct evaluation against clean labels would strengthen the conclusions.
Questions for authors
See Weaknesses Above.
To Reviewer nyyS, 1/3: the guarantee
We thank the reviewer. Three of the four weaknesses are shared with ifGA and fWD7, and we have accepted them; this reply states our position and adds what is specific to this review.
The main experiments do not validate the final guarantee. Correct. The main grid used an earlier gate with a pointwise bound and no correction for selecting the threshold on the same data, so it is not valid after selection. Those runs show that the fused weight helps empirically; they are not evidence for Eq. 11, and the revision says so in Appendix B.6 and in the main-grid table captions. Under the rule-based extractor structural risk is binary, so the corrected pair-level procedure returns no threshold; under a model-based extractor and the pair-level split it returns λ∗=0.46 on clean data, 0.47 at 10% noise and 0.48 at 20% and 30% noise, under the stated assumptions (Table 10 of the revision; our reply to ifGA 1/4 gives the details). These pair-level values are separate from the prompt-disjoint result, λ∗=0.478±0.004 over seeds 42-44 (Appendix E.7; our reply to fWD7 4/4). Because the implemented scores use calibration ranks and shared-prompt relations, this guarantee is approximate for the implementation.
The guarantee controls graph structure, not whether labels are correct. Also correct. Appendix B.5 (Remark 4) implements an oracle loss (the indicator that a pair is corrupted), which would make the certificate a statement about labels but presumes a verified subset, and the structural loss, which needs no corruption labels. Every result uses the structural loss.
The link between low weight and a flipped label is therefore empirical: on D1 the low-weight detector gives AUROC 0.911 / 0.840 / 0.833 at 10 / 20 / 30% noise, and on MATH 0.605. In the revision, Eq. 11 is named a structural-risk certificate, the abstract states that it does not certify label correctness, and the detector AUROC is the only claimed link to label noise.