acceptodds
Under review as a conference paper at ICLR 2027

LatentCP-DPO: Conformal Pessimism for DPO with Finite-Sample Guarantees

Abstract

Direct Preference Optimization (DPO) learns a response policy from human preference data, but with a limited dataset it can overfit to noisy or weakly informative preference labels. We introduce LatentCP-DPO, a framework that combines conformal prediction with pessimistic preference optimization for Bradley–Terry (BT) pairwise feedback and Plackett–Luce (PL) ranking feedback. Building on LatentCP’s construction of uncertainty sets for unobserved model parameters, we address how to incorporate this uncertainty into preference optimization and characterize its effect on policy performance. Under exchangeability and a correctly specified preference model, we construct uncertainty sets for latent relative rewards with finite-sample marginal coverage from conformal prediction sets for observable preferences. We then learn a response policy by maximizing the worst-case expected preference log-likelihood over these uncertainty sets, using an independent collection of prompts and candidate responses without preference labels. Under suitable assumptions, we establish finite-sample upper bounds on regularized policy suboptimality that separate policy-class approximation error, conformal uncertainty, and statistical estimation error. For BT feedback, we further establish a worst-case lower bound for LatentCP-DPO that matches the upper bound’s reciprocal dependence on the number of unlabeled optimization examples. The lower bound also shows that the uncertainty sets can induce a bias that persists as the optimization sample grows. Together, these results characterize both the statistical accuracy and potential limitations of preference optimization using conformal uncertainty sets. Experiments support the benefits of the proposed framework in finite preference datasets.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, uncertainty quantification, direct preference optimization (DPO), pessimistic optimization, latent rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sAcWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose the LatentCP-DPO method to address the problem that DPO is prone to overfitting when preference data are scarce and noisy. Conventional DPO trains language models using human preference data, but preference labels only indicate which response is better; they cannot directly indicate how reliable the preference judgment is, nor can they reflect the statistical uncertainty involved. Therefore, the authors mainly investigate two questions: first, how to measure uncertainty in true preference strength; and second, how to exploit this uncertainty in preference optimization to make model training more stable. To this end, the authors propose LatentCP-DPO. The method first builds on conformal prediction to convert prediction sets into uncertainty sets of latent relative rewards that cannot be directly observed. It then uses pessimistic preference optimization to optimize under the worst case over the preference distributions corresponding to these uncertainty sets. In addition, the authors prove that the latent reward uncertainty sets have a marginal coverage guarantee under finite samples, and provide an upper bound on policy suboptimality and a worst-case lower bound. Finally, experiments validate the effectiveness of the method.

Critical strengths

1、To address the problem that DPO is prone to overfitting on limited and noisy preference data, the authors propose an uncertainty aware preference optimization method, which is innovative to a certain extent. The method combines conformal prediction with pessimistic preference optimization and offers a statistically grounded solution. In addition, the method incorporates uncertainty sets directly into the optimization objective, which gives the paper strong research value.

2、The theoretical analysis in the paper is fairly comprehensive. Through three theorems, namely the finite sample marginal coverage guarantee, the finite sample suboptimality guarantee, and the worst case lower bound for LatentCP-DPO, the paper provides a theoretical basis for the method and also reveals the potential limitations of pessimistic optimization. These results not only explain the conditions under which the method obtains performance guarantees but also point out that pessimistic optimization may be overly conservative, which deepens the theoretical analysis of the paper.

Critical weaknesses

1、The authors investigate the problem that DPO is prone to overfitting on limited and noisy preference data. In the experiments, however, the authors do not seem to sufficiently demonstrate that LatentCP-DPO can indeed better mitigate overfitting than standard DPO on limited and noisy preference data.

2、Ablation experiments on the two core components, namely conformal prediction and pessimistic preference optimization, are missing from the experimental section of the paper, so the independent contribution of each component to model performance cannot be clearly established.

Questions for authors

1、The authors did not conduct independent ablation experiments on the two core components, conformal prediction and pessimistic preference optimization. How can it be clarified whether the performance improvement comes from one of these mechanisms or from their combination?

2、In the theoretical section, the authors discuss the effects of the conformal parameters α and γ on the policy suboptimality upper bound for the latent reward uncertainty set. In the experiments, however, they fix α = 0.3 and γ = 0.2. How were these values determined? Also, how can model performance under different combinations of parameter values be clarified?

2 comments
AuthorsOpenReview ↗

Response to Reviewer sAcW

Thank you for your positive assessment and constructive comments. We agree that directly evaluating overfitting and conducting ablation studies would strengthen the empirical evaluation. In the revised manuscript, we add a controlled experiment on late-stage overfitting and ablations of the conformal compatibility constraint.

We present our responses to your concerns below.

  • Overfitting. To evaluate overfitting directly, we compared LatentCP-DPO and Pure DPO on HelpSteer2 using Qwen2.5-0.5B-Instruct with BT feedback (Appendix F.7). We independently flipped each preference label in the non-test splits with probability 0.20 and evaluated both methods on the original test labels. From epoch 4 to epoch 8, Pure DPO’s NLL on noisy optimization labels decreased, while its test NLL rose from 0.68205 to 0.73442. Over the same interval, LatentCP-DPO’s test NLL stayed nearly constant (0.67898 to 0.67874), indicating less late-stage overfitting under the tested training schedule (Fig. 8).

  • Ablation of conformal compatibility. We retain the worst-case objective and replace the conformally induced ambiguity set in Eq. (11) with the full simplex of ranking distributions. In the synthetic PL experiment, full-simplex pessimism yields higher policy suboptimality at every tested noptn_{\mathrm{opt}} and approaches a higher plateau (Fig. 2(c)). On semi-synthetic Nectar data, LatentCP-DPO reduces mean policy suboptimality by 90.79% relative to full-simplex pessimism and achieves 81.18% pairwise accuracy versus 50.00% (Table 4, Appendix F.4). These comparisons show how conformal compatibility can reduce the cost of worst-case optimization. Conformal prediction and pessimistic optimization are two stages of the same training procedure. The former constructs calibrated uncertainty sets, and the latter uses those sets to define the worst-case policy objective. Isolating the contribution of pessimism would require comparing this objective with an alternative that uses the same calibrated sets. It remains a future work.

  • Sensitivity to α\alpha and γ\gamma. The choice of α\alpha sets the target marginal coverage 1−α1 - \alpha for the latent reward uncertainty sets. Choosing γ\gamma involves a trade-off because a smaller γ\gamma can enlarge the conformal prediction sets while tightening the compatibility constraint. We selected γ=0.20\gamma = 0.20 on Qwen3-8B validation data and used the same value for the Llama models. In Appendix F.8, we vary α\alpha and γ\gamma one at a time on HelpSteer2 with BT feedback. Over the tested ranges, performance on held-out preference data varies modestly with α\alpha and shows larger, nonmonotonic changes with γ\gamma (Fig. 9).

We appreciate the time and effort you devoted to reviewing our work. Your comments helped us make the manuscript more complete. We hope our responses and revisions address your concerns.

Reviewer_sAcWOpenReview ↗

To authors

Thank you for the detailed response and additional experiments. The new controlled experiments show that LatentCP-DPO mitigates late-stage overfitting under the tested noise conditions. The conformal constraint ablation and parameter sensitivity analysis also address my main concerns. I therefore maintain my original rating (Weak Accept).

Reviewer_LcMkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentCP-DPO, a method that combines conformal prediction with pessimistic DPO. The method first uses a pilot policy and a calibration set to build conformal prediction sets for preference outcomes, and then converts them into uncertainty sets over latent relative rewards under BT or PL preference models. The final policy is trained against the worst-case preference distribution within these uncertainty sets. The paper provides a closed-form objective for BT feedback and an MM-based optimization method for PL ranking feedback. It also gives finite-sample guarantees for latent reward coverage and policy suboptimality. Experiments on Nectar show that the method can produce smaller conformal prediction sets than several DPO-based baselines while keeping similar pairwise accuracy.

Critical strengths

S1: The paper studies an interesting problem: how to account for uncertainty in finite preference data instead of treating all preference labels as equally reliable.

S2: The connection between conformal prediction and pessimistic preference optimization is well motivated. The BT formulation is especially intuitive, as the uncertainty directly controls the preferred margin.

S3: The theoretical analysis is relatively complete. The paper provides both an upper bound on policy suboptimality and a lower bound showing that conformal uncertainty can introduce persistent bias.

S4: The method covers both pairwise BT feedback and listwise PL feedback, and the PL case includes a practical optimization procedure.

Critical weaknesses

W1: The theoretical results are mainly stated for the exact LatentCP-DPO optimizer, while the PL experiments rely on an approximate MM solver. It is not clear how the approximation error affects the final policy-level guarantee.

W2: The empirical improvement is mainly in conformal prediction-set size. The pairwise accuracy is almost the same as standard DPO, so the practical benefit for actual alignment quality is still unclear.

W3: The experimental evaluation is limited to one dataset and mainly the PL ranking setting with K=6. The BT case has substantial theoretical development but no direct empirical validation.

W4: The guarantees rely on a correctly specified BT or PL preference model. This assumption may be strong for real human preference data, where preferences can be noisy, inconsistent, or heterogeneous.

Questions for authors
  • The policy-level theory assumes the exact LatentCP-DPO optimizer, but the PL experiments use an approximate MM solution. Can the authors explain how the MM approximation affects the theoretical guarantee?

  • The main empirical gain is smaller conformal prediction sets, while pairwise accuracy is almost unchanged. Do the authors have results on downstream response quality, such as reward-model scores, LLM-based evaluation, or human preference?

  • Why are there no experiments for the BT setting, given that the paper provides a simple closed-form BT objective and substantial theoretical analysis for it?

  • How robust is LatentCP-DPO when the BT or PL preference model is misspecified?

  • How does the PL method scale when K becomes larger than 6? Is it possible to avoid explicit enumeration of all K! rankings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer LcMk (Part I)

Thank you for your thoughtful feedback. Our paper studies how pessimistic optimization over conformally calibrated sets affects DPO policies. Under the stated assumptions, these sets provide finite-sample marginal coverage for latent relative rewards, and optimizing pessimistically over them can introduce policy bias. In the revision, we quantify the effect of model misspecification on coverage, clarify how MM approximation enters the policy suboptimality bound, and add BT experiments and evaluations of generated responses.

We present our detailed responses to your concerns below:

  • MM approximation error. Corollary 1 in Appendix D.1 extends Theorem 2 to any approximate policy π~\widetilde{\pi}, including a policy returned by the MM-based PL procedure, by adding a term proportional to Δopt\Delta_{\mathrm{opt}}, defined as the empirical pessimistic loss at π~\widetilde{\pi} minus the infimum of that loss over Π\Pi. In the new synthetic PL experiments, we measure Δopt\Delta_{\mathrm{opt}} against an independently computed global reference for the same empirical objective. All 15 runs across K∈4,5,6K \in \\{4, 5, 6\\} achieved Δopt(50)<10−8\Delta_{\mathrm{opt}}^{(50)} < 10^{-8} at outer iteration 50 (Fig. 2(b)), demonstrating that the implemented MM and outer updates can achieve small empirical optimization error in this construction.

  • Empirical evidence. Our original Nectar experiments evaluated pairwise accuracy and ranking prediction-set size. At matched target coverage, LatentCP-DPO narrows the set of plausible rankings while maintaining comparable pairwise accuracy. In a new synthetic PL experiment with known rewards and fixed pilot and calibration data, mean KL-regularized policy suboptimality declines at a rate consistent with nopt−1n_{\mathrm{opt}}^{-1} over intermediate sample sizes, then approaches a nonzero plateau, illustrating persistent bias from conformal pessimism (Fig. 2(a)). On HelpSteer2, we independently flipped each label in the non-test splits with probability 0.20. From epoch 4 to epoch 8, Pure DPO’s NLL on noisy optimization labels decreased while its NLL on the original test labels rose from 0.68205 to 0.73442. Over the same interval, LatentCP-DPO’s test NLL stayed nearly constant (0.67898 to 0.67874), indicating less late-stage overfitting under this schedule (Fig. 8). On held-out Nectar prompts, responses generated by LatentCP-DPO also received a slightly higher mean ArmoRM score than those generated by Pure DPO (Table 6).

  • BT evaluation. In the new HelpSteer2 experiment with BT feedback, LatentCP-DPO achieves higher pairwise accuracy than Pure DPO (59.72% vs. 58.39%) and yields slightly smaller prediction sets at 90% nominal coverage (mean size 1.6647 vs. 1.6664, Table 5). With only two possible outcomes per pair, BT feedback leaves less room for reductions in prediction-set size.

  • Preference model misspecification. BT and PL are standard models in theoretical analyses of preference learning and policy optimization for pairwise and complete-ranking feedback, respectively [1, 2]. To quantify how departures from these working models affect latent-reward coverage, we extend Theorem 1 to obtain the marginal miscoverage bound Pun+1⋆∉U^α(Dn+1;γ)≤min⁡1,α(1+Δ/γ)\mathbb{P} \\{\boldsymbol{u}_{n+1}^\star \notin \widehat{\mathcal{U}}_\alpha(D_{n+1}; \gamma)\\} \leq \min \\{1, \alpha (1 + \Delta / \gamma)\\}, where Δ\Delta is the average total variation distance between the true conditional preference distribution and the working BT or PL distribution evaluated at the target latent rewards. When Δ=0\Delta = 0, this recovers the original 1−α1 - \alpha marginal coverage guarantee. For the policy suboptimality analysis in Theorem 2, we make the simplifying assumption that the working BT or PL preference model is correctly specified. In Appendix F.3, we use real Nectar contexts with assigned rewards and simulated rankings to test two mild departures from PL. In the experiment over a finite policy class, mean KL-regularized policy suboptimality increased by 7.53% relative to unperturbed PL at a 10% adjacent-swap probability. It increased by 5.38% when choices became progressively more random as the ranking was formed (with sampling temperature increasing by up to 10%; Fig. 5).

[1] Xingyu Zhou, Yulian Wu, and Francesco Orabona. A unified theoretical analysis of private and robust offline alignment: from RLHF to DPO. In International Conference on Machine Learning, vol. 267, pp. 79366–79388. PMLR, 2025.

[2] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, vol. 202, pp. 43037–43067. PMLR, 2023.

AuthorsOpenReview ↗

Response to Reviewer LcMk (Part II)

  • Scalability of the PL implementation. The model-dependent constant CPLC_{\mathrm{PL}} in Eq. (28) grows as O(K4)O(K^4) with the other parameters fixed. The current MM implementation explicitly enumerates all K!K! rankings, so it is best suited to small or moderate KK (Appendix C.4). One possible extension is to use top-kk feedback, which records only the ordered top kk of the KK candidates. This reduces the outcome space to K!/(K−k)!=O(Kk)K!/(K-k)!=O(K^k) for fixed kk. Extending the method to this setting would require further analysis of latent-reward coverage, constraint feasibility, and MM convergence and is beyond the scope of the current analysis.

We appreciate your careful review and helpful suggestions. We hope the clarifications and additional results addressed the concerns you raised.

Reviewer_9BHKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper fits an initial DPO policy, calibrates prediction sets on separate labeled data, and converts these into uncertainty sets for relative rewards. The method then trains a policy on independent prompts and candidate responses without preference labels by maximizing the worst-case expected preference log-likelihood over these sets. The authors provide bounds on KL-regularized policy suboptimality (Theorem 2) and construct an instance highlighting a persistent performance gap caused by the uncertainty sets, conditional on a particular initial training event (Theorem 3).

Critical strengths
  • Theorem 2 separates policy approximation error, conformal uncertainty, and estimation error from the unlabeled optimization sample. This provides a connection between calibrated reward sets and policy performance under the stated assumptions.

  • Theorem 3 constructs an instance of the Bradley–Terry model with π⋆∈Π\pi^\star\in\Pi and Ccomp(π⋆;Π)≤1C_{\mathrm{comp}}(\pi^\star;\Pi)\leq1. Conditional on HpilH_{\mathrm{pil}}, the lower bound holds with probability at least 1/161/16. Its uncertainty contribution remains positive as noptn_{\mathrm{opt}} grows, identifying a possible persistent bias even without policy approximation error.

Critical weaknesses
  • The latent-reward and policy guarantees require more than exchangeability. Equation 8 and Theorem 1 assume that preferences follow a correctly specified BT or PL model. Although this assumption is explicitly stated, the experiments do not assess sensitivity to its violation on Nectar. Conformal coverage of observed rankings does not by itself establish coverage of latent rewards under a misspecified preference model, which limits the interpretation of the guarantees for these experiments.

  • Theorem 3 conditions on a specific initial training event, HpilH_{\mathrm{pil}}. Its construction implies Pr⁡(Hpil)≤8−(npil−2dΠ)\Pr(H_{\mathrm{pil}})\leq8^{-(n_{\mathrm{pil}}-2d_\Pi)}. For fixed dΠd_\Pi, this event’s probability vanishes exponentially as npiln_{\mathrm{pil}} grows. Although the conditioning is explicitly acknowledged, the result does not establish a persistent gap with probability bounded below by a positive constant over all samples, independently of the initial training sample size.

  • Theorem 2 does not show how many labeled comparisons are needed to learn a policy close to optimal. In particular, the paper does not quantify how additional labeled data improve the prediction sets and reduce the uncertainty term in the bound. Moreover, at the experimental settings α=0.3\alpha=0.3 and γ=0.2\gamma=0.2, this term satisfies Ψα,γ(δ)≥5/9\Psi_{\alpha,\gamma}(\delta)\geq5/9, regardless of sample size. Thus, the bound cannot guarantee that the uncertainty contribution becomes small under the settings used in the experiments.

  • The experiments provide limited evidence about policy performance. The prediction-set improvements are useful, but measurements such as Qwen3-8B achieving 66.6% pairwise accuracy versus 66.5% for Pure DPO do not directly evaluate the KL-regularized policy objective in Theorem 2. The evaluation uses fixed candidate responses from Nectar and does not examine the dependence on noptn_{\mathrm{opt}} or illustrate the persistent bias identified in the BT analysis. The experiments also do not quantify the errors from approximately solving the inner and outer optimization problems, leaving the size of Δopt\Delta_{\mathrm{opt}} in Corollary 1 unknown.

Questions for authors
  • How sensitive are the inferred reward sets and learned policies to violations of the assumed PL preference model?

  • Can the persistent bias identified in Theorem 3 be shown to hold with constant probability over the initial training, calibration, and optimization samples without conditioning on HpilH_{\mathrm{pil}}?

  • What evidence suggests that the optimization error Δopt\Delta_{\mathrm{opt}} is small for the policies trained in the experiments? Does Corollary 1 provide an informative performance guarantee for these policies?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9BHK (Part I)

Thank you for your careful reading and thoughtful comments. LatentCP-DPO uses conformally calibrated uncertainty sets for latent relative rewards in pessimistic preference optimization. Our upper bound quantifies the potential cost of this uncertainty in terms of KL-regularized policy suboptimality. In the revision, we characterize how model misspecification affects latent-reward coverage and give a BT example in which pessimistic bias persists with probability one over all three samples.

We recap your valuable comments and present our detailed response below:

  • Preference model misspecification. BT and PL are standard models in theoretical analyses of preference learning and policy optimization for pairwise and complete-ranking feedback, respectively [1, 2]. To quantify how departures from these working models affect latent-reward coverage, we extend Theorem 1 to obtain the marginal miscoverage bound Pun+1⋆∉U^α(Dn+1;γ)≤min⁡1,α(1+Δ/γ)\mathbb{P} \\{\boldsymbol{u}_{n+1}^\star \notin \widehat{\mathcal{U}}_\alpha(D_{n+1}; \gamma)\\} \leq \min \\{1, \alpha (1 + \Delta / \gamma)\\}, where Δ\Delta is the average total variation distance between the true conditional preference distribution and the working BT or PL distribution evaluated at the target latent rewards. When Δ=0\Delta = 0, this recovers the original 1−α1 - \alpha marginal coverage guarantee. In Appendix F.3, we use real Nectar contexts with assigned rewards and simulated rankings to test two kinds of departures from PL. Across the tested perturbations, empirical coverage of the assigned rewards stayed within 2 percentage points of its value under the unperturbed PL model. In a separate experiment over a finite policy class, mean KL-regularized policy suboptimality increased relative to unperturbed PL by 7.53% at a 10% adjacent-swap probability and by 5.38% when choices became progressively more random as the ranking was formed (with sampling temperature increasing by up to 10%; Fig. 5).

  • Persistent conformal bias. Following the split-conformal setup, we fit the pilot policy on an independent sample and keep it fixed during calibration and optimization. Theorem 3 characterizes the dependence on noptn_{\mathrm{opt}} conditional on HpilH_{\mathrm{pil}}, whose probability stays constant as noptn_{\mathrm{opt}} grows with the pilot setup fixed. In the revised manuscript, we add Proposition 2 to show that the bias can persist unconditionally. Under its stated conditions, the proposition gives a fixed BT instance in which every exact LatentCP-DPO empirical optimizer has a strictly positive KL-regularized policy suboptimality gap bounded away from zero with probability one over the pilot, calibration, and optimization samples.

  • The effect of labeled comparisons. Theorem 2 conditions on the realized pilot sample and requires neither consistency nor a convergence rate for the pilot estimator (Remark 1, Appendix D.1). It separates the dΠ/noptd_\Pi/n_{\mathrm{opt}} estimation term at fixed confidence from the potential cost of conformal uncertainty. The pilot fit enters the bound through η~α,γ\widetilde{\eta}_{\alpha, \gamma}, which measures how much preference information the calibrated prediction sets leave unresolved. Existing BT and PL estimation results [2] can help characterize pilot accuracy under their respective assumptions. Translating those rates into a bound on η~α,γ\widetilde{\eta}_{\alpha, \gamma} would require further assumptions linking pilot accuracy to prediction-set informativeness.

[1] Xingyu Zhou, Yulian Wu, and Francesco Orabona. A unified theoretical analysis of private and robust offline alignment: from RLHF to DPO. In International Conference on Machine Learning, vol. 267, pp. 79366–79388. PMLR, 2025.

[2] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, vol. 202, pp. 43037–43067. PMLR, 2023.

AuthorsOpenReview ↗

Response to Reviewer 9BHK (Part II)

  • Conformal uncertainty at the experimental settings. Theorem 2 separates the estimation term, which scales as dΠ/noptd_\Pi/n_{\mathrm{opt}} at fixed confidence, from the potential cost of conformal pessimism. At the Nectar settings α=0.30\alpha = 0.30 and γ=0.20\gamma = 0.20, the conformal uncertainty term in Theorem 2 satisfies Ψα,γ(δ)≥5/9\Psi_{\alpha, \gamma}(\delta) \geq 5/9 for all sample sizes. The finite-sample bound controls suboptimality in the worst case under the stated assumptions, so its numerical value can be conservative. Even so, this reflects a non-vanishing conservative contribution to the upper bound under fixed calibration parameters.

  • Evaluation of policy suboptimality. To directly evaluate the KL-regularized policy suboptimality studied in Theorems 2 and 3, we added a synthetic PL experiment with known rewards. Figure 2(a) of the revised manuscript plots mean suboptimality against noptn_{\mathrm{opt}} on log–log axes, holding the pilot fit and calibration data fixed. Over intermediate sample sizes, the curves decline at a rate consistent with the nopt−1n_{\mathrm{opt}}^{-1} reference slope. They then approach a nonzero plateau, illustrating how the bias induced by conformal pessimism can persist as more unlabeled optimization contexts become available.

  • Empirical optimization error. To quantify the optimization gap Δopt\Delta_{\mathrm{opt}} in Corollary 1, we run MM in the full (K−1)(K-1)-dimensional latent PL space alongside the outer policy updates. In the synthetic experiment, at outer iteration 50, all 15 runs across K∈4,5,6K \in \\{4, 5, 6\\} achieved Δopt(50)<10−8\Delta_{\mathrm{opt}}^{(50)} < 10^{-8} (Fig. 2(b) in the revised manuscript), demonstrating that the implemented MM and outer updates can achieve small empirical optimization error in this construction. This measured gap further lets us use Corollary 1 to quantify how much approximate optimization adds to the policy suboptimality bound in this setting. For the Nectar LLM runs, a comparable global reference is unavailable, making the gap difficult to quantify.

Thank you again for your constructive feedback, which helped us strengthen the manuscript. We hope our responses and revisions address your concerns.

I thank the authors for their substantial effort. The additional results strengthen the paper and address many of my original concerns. My remaining reservation concerns the empirical evidence of policy improvement. The real-data evaluation primarily demonstrates smaller prediction sets over fixed candidate responses, with essentially unchanged pairwise accuracy compared with DPO. The synthetic experiments help illustrate the theoretical results, but do not establish improvements in generated-response quality. The revision introduces substantial new theoretical and experimental material that would benefit from another round of careful verification. I have not fully verified all these additions within the rebuttal period. I now consider the paper very borderline and am more positive than before. Nevertheless, the remaining limitations in demonstrating practical policy benefits leave the contribution marginally below the acceptance threshold in my assessment, and I maintain my weak reject.

Dear Reviewer 9BHK,

Thank you for your careful reassessment and for taking the time to examine the new results.

We agree that broader evaluation would strengthen the evidence on generated-response quality. To evaluate the practical motivation of mitigating overfitting to noisy preferences, we conducted an experiment on HelpSteer2, independently flipping each preference label in the non-test splits with probability 0.20. From epoch 4 to epoch 8, Pure DPO’s NLL on noisy optimization labels decreased while its NLL on the original test labels rose from 0.68205 to 0.73442. Over the same interval, LatentCP-DPO’s test NLL stayed nearly constant (0.67898 to 0.67874), providing evidence of reduced late-stage overfitting under the tested schedule (Fig. 8). We also evaluated generated responses on held-out Nectar prompts, where LatentCP-DPO received a slightly higher mean ArmoRM score than Pure DPO (Table 6).

We appreciate your feedback, which helped us strengthen the evaluation and clarify the scope of our empirical conclusions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank the reviewers for their constructive feedback. To mitigate overfitting in preference optimization, LatentCP-DPO optimizes policies pessimistically over conformally calibrated uncertainty sets. Under the stated assumptions, these sets provide finite-sample marginal coverage for latent relative rewards. Our upper and lower bounds jointly characterize the dΠ/noptd_{\Pi}/n_{\mathrm{opt}} scaling of the statistical error from unlabeled optimization contexts and quantify the potential cost of conformal pessimism (i.e., a bias that can persist as noptn_{\mathrm{opt}} grows).

The revised manuscript includes the following updates:

Theory. We extend the coverage result in Theorem 1 to quantify the effect of preference model misspecification. We also clarify how the empirical optimization gap of the MM-based procedure enters the policy suboptimality bound in Corollary 1.

Experiments. In a new synthetic PL experiment, mean policy suboptimality declines at a rate consistent with the predicted nopt−1n_{\mathrm{opt}}^{-1} scaling over intermediate sample sizes and then approaches a nonzero plateau. We also add a BT evaluation, a controlled experiment showing less late-stage overfitting under the tested schedule, ablations of conformal compatibility, and sensitivity analyses for α\alpha and γ\gamma.

We appreciate your time and would welcome any further thoughts on the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.