- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This submission proposes Hard Preference Calibration (HPC), which augments OPD with verifier-guided preference optimization. HPC constructs trajectory pairs using verifier outcome and introduces a weighted contrastive loss, so student prefer correct trajectories. Experiments on math and coding datasets demonstrate improvements over OPD and SCOPE across student sizes and teacher configurations.
Critical strengths
-
A relatively simple approach to address a meaningful limitation of OPD. Using verifier-based preferences is an intuitive mechanism,
-
Promising empirical improvements across several configurations (math and coding, two student sizes, two teacher settings). The ablation experiments also suggest that the reliability gate provides additional benefits.
Critical weaknesses
-
Limited methodological novelty and insufficient comparison with alternatives. The method can essentially be understood as combining OPD with an additional verifier-guided pairwise preference objective, where the teacher's relative sequence likelihood determines the weight assigned to each pair. While this is reasonable, the paper does not sufficiently establish why the particular calibration mechanism represents a substantial contribution over combinations of existing objectives. E.g., OPD augmented with a standard pairwise preference loss, and other verifier-guided OPD approaches such as reward-gated OPD.
-
Insufficient empirical validation. The experiments do not include rejection sampling followed by supervised fine-tuning, stronger verifier-guided OPD variants, or more direct combinations of OPD and standard preference optimization. For metrics, some individual results are comparable to or worse than simpler baselines. Without results across multiple random seeds or uncertainty estimates, it is difficult to establish the reliability of the smaller gains. No executable code or supplementary implementation is provided.
Questions for authors
See comments above around comparison with standard preference optimization and other verifier-guided OPD methods, reliability of empirical results
Response to Reviewer ydtg
Thank you for this helpful comment. In our method, verifier correctness determines which response should be preferred, while the teacher’s relative sequence scores determine how strongly each pair is corrected. Unlike uniform pairwise correction, HPC focuses on pairs where the teacher’s preference is weak or incorrect. In our ablation, uniform pairwise correction improves the average from 38.36% for OPD to 39.41%, while HPC reaches 40.54%, roughly doubling the improvement over OPD.
To compare HPC directly with a standard preference objective that weights all verifier-labeled response pairs equally, we have added a theoretical result in Appendix B.3. At the OPD reference point, let δi,p,n be the teacher margin deficit for a correct and incorrect response pair, and wi,p,n its HPC reliability weight. Both increase as the teacher’s relative score for the correct response decreases. Their covariance over pairs is therefore nonnegative, which gives
δˉical=∑(p,n)wi,p,n∑(p,n)wi,p,nδi,p,n≥∣Ei∣1∑(p,n)δi,p,n=δˉiunif.
Each deficit measures how much correction is needed for that pair at the OPD reference point. Therefore, under the same normalized correction budget, HPC assigns more correction to pairs with larger teacher deficits than uniform pairwise preference optimization. These reliability-weighted pairs contribute to the correction certificate in our existing analysis, which gives a lower generation error bound than OPD and implies an improvement in Pass@k.
Thank you again for this insightful comment. We have incorporated the discussion and theoretical analysis into Appendix B.3.
Thank you for this detailed feedback. We have expanded the comparison to include an outcome-guided margin calibration method like Uni-OPD and trajectory-level reward gating like OPDVR. The table reports results on four mathematics benchmarks. Scores are percentages (%).
As shown in the table, HPC achieves the strongest average performance among the compared methods. The main experiments show similarly consistent average gains across student and teacher settings. For individual Pass@k results, HPC exceeds OPD in 93.8% of the mathematics results and 95.8% of the code results, and outperforms the other methods on most benchmark-level results. Post-training requires substantial computational resources, and we have included means and standard deviations across multiple runs for some experiments in Appendix C.2. We also provide detailed configurations in Appendix C and core implementation in the supplementary material. Thank you again for this valuable feedback and for encouraging a broader evaluation.