- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose the LatentCP-DPO method to address the problem that DPO is prone to overfitting when preference data are scarce and noisy. Conventional DPO trains language models using human preference data, but preference labels only indicate which response is better; they cannot directly indicate how reliable the preference judgment is, nor can they reflect the statistical uncertainty involved. Therefore, the authors mainly investigate two questions: first, how to measure uncertainty in true preference strength; and second, how to exploit this uncertainty in preference optimization to make model training more stable. To this end, the authors propose LatentCP-DPO. The method first builds on conformal prediction to convert prediction sets into uncertainty sets of latent relative rewards that cannot be directly observed. It then uses pessimistic preference optimization to optimize under the worst case over the preference distributions corresponding to these uncertainty sets. In addition, the authors prove that the latent reward uncertainty sets have a marginal coverage guarantee under finite samples, and provide an upper bound on policy suboptimality and a worst-case lower bound. Finally, experiments validate the effectiveness of the method.
Critical strengths
1、To address the problem that DPO is prone to overfitting on limited and noisy preference data, the authors propose an uncertainty aware preference optimization method, which is innovative to a certain extent. The method combines conformal prediction with pessimistic preference optimization and offers a statistically grounded solution. In addition, the method incorporates uncertainty sets directly into the optimization objective, which gives the paper strong research value.
2、The theoretical analysis in the paper is fairly comprehensive. Through three theorems, namely the finite sample marginal coverage guarantee, the finite sample suboptimality guarantee, and the worst case lower bound for LatentCP-DPO, the paper provides a theoretical basis for the method and also reveals the potential limitations of pessimistic optimization. These results not only explain the conditions under which the method obtains performance guarantees but also point out that pessimistic optimization may be overly conservative, which deepens the theoretical analysis of the paper.
Critical weaknesses
1、The authors investigate the problem that DPO is prone to overfitting on limited and noisy preference data. In the experiments, however, the authors do not seem to sufficiently demonstrate that LatentCP-DPO can indeed better mitigate overfitting than standard DPO on limited and noisy preference data.
2、Ablation experiments on the two core components, namely conformal prediction and pessimistic preference optimization, are missing from the experimental section of the paper, so the independent contribution of each component to model performance cannot be clearly established.
Questions for authors
1、The authors did not conduct independent ablation experiments on the two core components, conformal prediction and pessimistic preference optimization. How can it be clarified whether the performance improvement comes from one of these mechanisms or from their combination?
2、In the theoretical section, the authors discuss the effects of the conformal parameters α and γ on the policy suboptimality upper bound for the latent reward uncertainty set. In the experiments, however, they fix α = 0.3 and γ = 0.2. How were these values determined? Also, how can model performance under different combinations of parameter values be clarified?
2 comments
To authors
Thank you for the detailed response and additional experiments. The new controlled experiments show that LatentCP-DPO mitigates late-stage overfitting under the tested noise conditions. The conformal constraint ablation and parameter sensitivity analysis also address my main concerns. I therefore maintain my original rating (Weak Accept).
Response to Reviewer sAcW
Thank you for your positive assessment and constructive comments. We agree that directly evaluating overfitting and conducting ablation studies would strengthen the empirical evaluation. In the revised manuscript, we add a controlled experiment on late-stage overfitting and ablations of the conformal compatibility constraint.
We present our responses to your concerns below.
Overfitting. To evaluate overfitting directly, we compared LatentCP-DPO and Pure DPO on HelpSteer2 using Qwen2.5-0.5B-Instruct with BT feedback (Appendix F.7). We independently flipped each preference label in the non-test splits with probability 0.20 and evaluated both methods on the original test labels. From epoch 4 to epoch 8, Pure DPO’s NLL on noisy optimization labels decreased, while its test NLL rose from 0.68205 to 0.73442. Over the same interval, LatentCP-DPO’s test NLL stayed nearly constant (0.67898 to 0.67874), indicating less late-stage overfitting under the tested training schedule (Fig. 8).
Ablation of conformal compatibility. We retain the worst-case objective and replace the conformally induced ambiguity set in Eq. (11) with the full simplex of ranking distributions. In the synthetic PL experiment, full-simplex pessimism yields higher policy suboptimality at every tested nopt and approaches a higher plateau (Fig. 2(c)). On semi-synthetic Nectar data, LatentCP-DPO reduces mean policy suboptimality by 90.79% relative to full-simplex pessimism and achieves 81.18% pairwise accuracy versus 50.00% (Table 4, Appendix F.4). These comparisons show how conformal compatibility can reduce the cost of worst-case optimization. Conformal prediction and pessimistic optimization are two stages of the same training procedure. The former constructs calibrated uncertainty sets, and the latter uses those sets to define the worst-case policy objective. Isolating the contribution of pessimism would require comparing this objective with an alternative that uses the same calibrated sets. It remains a future work.
Sensitivity to α and γ. The choice of α sets the target marginal coverage 1−α for the latent reward uncertainty sets. Choosing γ involves a trade-off because a smaller γ can enlarge the conformal prediction sets while tightening the compatibility constraint. We selected γ=0.20 on Qwen3-8B validation data and used the same value for the Llama models. In Appendix F.8, we vary α and γ one at a time on HelpSteer2 with BT feedback. Over the tested ranges, performance on held-out preference data varies modestly with α and shows larger, nonmonotonic changes with γ (Fig. 9).
We appreciate the time and effort you devoted to reviewing our work. Your comments helped us make the manuscript more complete. We hope our responses and revisions address your concerns.