acceptodds
Under review as a conference paper at ICLR 2027

Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution

Abstract

We introduce PEPO (Pessimistic Ensemble based Preference Optimization}), a single-step Direct Preference Optimization (DPO)-like algorithm to mitigate the well-known over-optimization issue in preference learning without requiring the knowledge of the data-generating distribution or learning an explicit reward model. PEPO achieves pessimism via an ensemble of preference-optimized policies trained on disjoint data subsets and then aggregates them through a worst case construction that favors the agreement across models. In the tabular setting, PEPO achieves sample complexity guarantees depending only on a single-policy concentrability coefficient, thus avoiding the all-policy concentrability which affects the guarantees of algorithms prone to over-optimization, such as DPO. The theoretical findings are corroborated by a convincing practical performance, while retaining the simplicity and the practicality of DPO-style training.

Primary area:
reinforcement learning
Keywords:
direct preference optimization, LLMs, pessimism

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jDASOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PEPO, a DPO-like method for reducing over-optimization without access to the data-generating policy or an explicitly trained reward model. It splits the preference dataset into disjoint subsets, trains a separate DPO policy on each subset, and combines the policies through pessimistic aggregation at inference time.

In the tabular setting, the authors derive sample-complexity guarantees that depend on single-policy concentrability and provide guidance on the ensemble size. The experiments compare PEPO with DPO and pessimistic DPO variants on four backbones, using a token-level approximation for the main LLM evaluations. The reported results show better stability at later checkpoints and higher final-checkpoint scores in the tested configurations.

Critical strengths

S1. The paper addresses a useful theoretical question: whether pessimism can be introduced into a DPO-like method without knowing the policy that generated the preference data. The tabular analysis is developed in substantial detail, with supporting proofs and a discussion of how ensemble size trades off pessimism against estimation error. This provides a principled motivation for the aggregation rule. The analysis's scope and structure are a substantive strength.

S2. The experimental study is reasonably broad. It includes four backbones, synthetic experiments that control the relationship between the data and reference policies, comparisons with DPO, χ²PO, and DPO+SFT, and additional analyses of ensemble size and inference latency. Reporting training curves and both best-observed and final-checkpoint results is particularly useful. These results support the observation that PEPO is more stable during the later part of the tested training runs.

Critical weaknesses

W1. Connection between the theoretical guarantee and the implemented LLM method may not be clear enough.

Theorem 3.3 relies on a sample-complexity bound scaling with the massive sizes of |X||A|⁴, and the sufficient ensemble size L far exceeds the 2–4 members used in the experimental implementations. There may not be a sufficiently clear explanation of which part of the theoretical guarantee should still be expected to hold in the reported LLM runs, or how the practical choice of L relates to the condition needed to establish pessimism.

Furthermore, while the theory uses response-level aggregation, the experiments rely on token-level aggregation. Although Appendix D analyzes this, the error and normalization-mismatch terms are not quantified for practical settings. A clearer account is needed to show that these approximation terms do not undermine the theoretical connection.

W2. The comparisons may not sufficiently separate pessimistic aggregation from training exposure.

Each PEPO member (LoRA model) trains on only 1/3 to 1/4 of the full dataset, receiving fewer updates per epoch than a full-dataset DPO model. Consequently, the epoch-aligned comparisons in Figures 1, 7, and 8 conflate the benefits of pessimistic aggregation with the shorter training history of each member.

A single DPO member trained on the same data shard with matched updates, alongside comparisons of different aggregation rules using an identical set of trained members, would better isolate the source of the gains.

W3. The choice of final checkpoints needs more justification.

Table 1 uses epoch 16 for 7B/8B models and epoch 4 for Yi-34B, while Figure 8 and Table 3 use epoch 5 for Yi-34B. It remains unclear why these budgets were chosen, why multiple passes over the preference dataset are needed or beneficial, or how models perform beyond these points.

While using a uniform endpoint helps study late-training stability, Table 4 only shows proximity to the best score within a window, not that the endpoint is optimal.

Additionally, earlier checkpoints complicate the cost-benefit analysis. In Table 3, Mistral-7B DPO reaches 85.3 on AlpacaEval2 at epoch 8 (vs. PEPO's 84.0 at epoch 14), and Zephyr-7B DPO+SFT hits 82.5 at epoch 1 (vs. PEPO's 77.6 at epoch 12). These results suggest that early-stopped baselines can sometimes match or exceed PEPO, meaning ensemble gains may not be universal evidence of superior resource efficiency.

W4. The additional inference cost makes the practical advantage less clear.

PEPO requires multiple forward passes during token-level inference, and the exact response-level sampler is much slower with a decreasing acceptance rate during training (Figure 2 and Appendix C.3.9). Direct comparisons at matched training and inference budgets are necessary. Without these controls, the paper demonstrates late-training stability more effectively than a compelling overall deployment advantage.

Questions for authors

Q1. Clarification of the implemented pessimism terms. How are mℓm_\ell, the centering factors ζℓ\zeta_\ell, and the penalty CC implemented in the LLM experiments? For the token-level version, how are the prefix counts and penalties constructed? Providing the actual inference rule and specifying which assumptions of the theoretical analysis remain satisfied in the LLM experiments would be helpful.

Q2. Clarification of the training endpoint. What determined the budgets of 16 epochs for the 7B/8B models and 4 epochs for Yi-34B (5 in MT-Bench)? If tens of epochs are necessary for preference learning, is there evidence about stability beyond these endpoints?

Q3. Control for training exposure. A matched-update control for the individual members would clarify whether the gains actually stem from pessimistic aggregation.

Q4. Length control and evaluation robustness. The length-controlled win rate (LC-WR) of AlpacaEval2 is a useful metric for evaluating the model's ability beyond length bias. How do average response lengths compare across methods? Reporting these results would help assess whether response-length differences contribute to the observed gains. Additional metrics or evaluation with a different judge would also help establish the robustness of the comparison.

I would be very happy to update my assessment if the authors can clarify these points and provide additional analysis. Looking forward to further discussion.

7 comments
AuthorsOpenReview ↗

Response to Reviewer jDAS (1/4)

Dear reviewer,

Thank you for appreciating our theoretical analysis as "a substantive strength" and for defining our experimental study as "reasonably broad". We are happy to receive your thought-provoking questions which we address below.

W1. The tabular analysis sets sufficient conditions for a provably efficient aggregation rule, and the LLM runs test that rule. Before going into details, we would like to clarify that our tabular analysis serves as a guideline to define a reasonable pessimistic aggregation rule for an ensemble of models. Our analysis serves to establish sufficient conditions for the rule to be provably efficient in the tabular setting. The goal is not to provide sample-complexity guarantees for the LLM version used in practice, which would clearly be overambitious. Having clarified this, we can go on answering your specific points.

The ensemble size LL. The LL of the theory is a worst-case sufficient condition, and LL = 2–4 works in practice. The value of LL prescribed by our theory grows logarithmically in the number of states and actions, which is clearly larger than the values of 2–4 which we found effective in practice. The cause of this mismatch is that our tabular analysis is for the worst-case instance which is unlikely to be the one faced in practice. In the controlled bandit of App. C.1.1 (Fig. 3), PEPO finds the optimal action for every LL we tried, and L=3L=3 performs best.

Response- vs. token-level aggregation. The guarantee for the token-level rule (App. D) depends on C⋆C^\star and the token-level estimation error, not on CallC^{\mathrm{all}}. The token-level aggregation is preferred in practice as it can be implemented in closed form by summing over the vocabulary instead of resorting to rejection sampling. Its analysis in Appendix D should be thought of as an error propagation analysis as it allows us to conclude that the squared suboptimality of the learned policy is at most the token-level generalization error ⟨πdata,Errtoken2⟩\langle \pi_{\mathrm{data}},\mathrm{Err}_{\mathrm{token}}^2 \rangle times the inflation factor C⋆C^\star. Therefore, thanks to the pessimistic aggregation at token level, CallC^\mathrm{all} does not enter the bound. While we do not prove that ⟨πdata,Errtoken2⟩\langle \pi_{\mathrm{data}},\mathrm{Err}_{\mathrm{token}}^2 \rangle decreases with the dataset size NN, our analysis says that the suboptimality is small whenever ⟨πdata,Errtoken2⟩\langle \pi_{\mathrm{data}},\mathrm{Err}_{\mathrm{token}}^2 \rangle is. This is, in spirit, an error propagation analysis which, even if it does not lead to sample complexity guarantees, is an informative kind of analysis in the RL literature. Among several potential examples, we mention some in imitation learning [1, 2, 3] and in policy optimization [4, 5]. We, therefore, think that error propagation analyses like ours for token-level PEPO are important to motivate methods also in learning from preferences.

[1] Efficient Imitation under Misspecification, Espinosa-Dice et al.

[2] Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap, Swamy et al.

[3] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, Ross et al.

[4] Leveraging the Average: an Analysis of KL Regularization in RL, Vieillard et al.

[5] On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift, Agarwal et al.

AuthorsOpenReview ↗

Response to Reviewer jDAS (2/4)

Q1. In the LLM runs, the ensemble aggregation is the only active pessimistic ingredient. In the practical LLM implementation, we consider the centering shift ζℓ(x)\zeta^\ell(x) and the penalty C(x,a)C(x,a) to be constants. We consider the constant approximation to work well because in the LLM experiments a state action pair is visited either once or never, i.e. Nℓ(x,a)∈{0,1}N^\ell(x,a) \in \lbrace 0, 1 \rbrace. Inspecting the expression for C(x,a)C(x,a) in Theorem 3.3, and omitting the numerator, we see that C(x,a)C(x,a) decays as (Nℓ(x,a)+∣A∣)−1/2(N^\ell(x,a) + |\mathcal{A}|)^{-1/2}. Therefore, when the action space is very large as in LLMs and the counts Nℓ(x,a)N^\ell(x,a) are binary we have that (1+∣A∣)−1/2≊∣A∣−1/2(1 + |\mathcal{A}|)^{-1/2} \approxeq |\mathcal{A}|^{-1/2} and the constant approximation for C(x,a)C(x,a) is justified. Finally, we set ζℓ\zeta^\ell to zero because when each prompt appears only once we have that ζℓ=0/0\zeta^\ell = 0/0 (because each prompt in Dℓ\mathcal{D}^\ell never appears again in D∖Dℓ\mathcal{D}\setminus\mathcal{D}^\ell). By our convention, we have 0/0=00/0=0 and therefore we set ζℓ=0\zeta^\ell = 0 and as a consequence mℓm^\ell to be constant in the experiment.

Our main takeaway is that when the data are scarce, that is, when no state-action pair is revisited, the ensemble aggregation is the only active pessimistic ingredient of our algorithm. In other words, the stability of PEPO in the LLM experiments is due solely to the ensemble aggregation rule and it is not necessary to introduce additional pessimistic penalties. Following the same reasoning, we consider as constants also the prefix counts of the token-level PEPO implementation.

The exact inference (aggregation) rule. Token-level PEPO generates each token from the renormalized minimum of the members' next-token probabilities. With greedy decoding, as in our tables, it takes the most likely token under this distribution. The probability of generating token yty_t after prompt xx and previous tokens y1:t−1y_{1:t-1} is given by min⁡ℓ∈[L]πℓ(yt∣x,y1:t−1)∑y∈Vmin⁡ℓ∈[L]πℓ(y∣x,y1:t−1),\frac{\min_{\ell\in [L]} \pi^\ell(y_t| x, y_{1:t-1} )}{\sum_{y \in \mathcal{V}} \min_{\ell\in [L]} \pi^\ell(y| x, y_{1:t-1} )},

where V\mathcal{V} is the token vocabulary and it is usually small enough to compute the exact sum over it.

Validity of the theoretical analysis in practice. The tabular bounds are too loose to explain PEPO's results quantitatively, and they serve as a design principle. The LLM setup actually has finitely many states and actions, as our theory requires. However, our bounds would be extremely loose because they consider each prompt and response as completely independent from any other possible pairs. This leads to guarantees that scale with the number of states and actions which are larger than the available dataset size in language applications. The tabular guarantees therefore are not sufficient to explain the success of PEPO in practice even though they serve as a strong designing principle. What is missing from our analysis and what could lead to tighter theoretical guarantees in the future is the parameter sharing of LLMs across similar prompts and responses, which allows good performance with dataset and ensemble sizes much smaller than what our tabular analysis would predict.

AuthorsOpenReview ↗

Response to Reviewer jDAS (3/4)

W2 / Q3 (a single member on its own shard). Training on 1/L of the data does not prevent over-optimization.

Each ensemble member is DPO trained on its own 1/L of the data. The following table shows the AlpacaEval2 win rates (WR) of each member alone, of PEPO and of DPO, on Llama-3-8B with L=3 (Fig. 10 in App. C.3.11 of the revised paper shows the WRs at every epoch).

ModelBest WR (epoch)Final WR (epoch 16)Drop from best
DPO69.3 (5)60.88.5
Member 067.3 (10)66.11.2
Member 175.3 (5)70.74.6
Member 269.6 (14)67.71.9
PEPO75.0 (10)73.81.2

Member 1's WR peaks at epoch 5, the same epoch as DPO's, but after a third as many updates. It then drops by 4.6 pp. PEPO's WR drops by only 1.2 pp, even though member 1 is part of its ensemble. The WRs of members 0 and 2 drop less than member 1's but are below PEPO's at all 16 epochs, by 1.4–9.7 pp. PEPO's final WR is 3.1–7.7 pp above each member's.

W2 / Q3 (matched updates). PEPO is not DPO trained more slowly. Each member of the ensemble receives 1/L1/L of DPO's updates in one epoch. PEPO's results do not come from this smaller number of updates.

  • Higher win rate (WR) with fewer updates: On Llama-3-8B, PEPO's WR is above DPO's at every epoch (by 0.6–13.9 pp). On Yi-34B (L=4), it is 6.8 pp above at epoch 4.
  • Much higher WR with as many updates: At epoch LkLk each member has received as many updates as DPO at epoch kk. PEPO's WR at these points is 4.6–12.2 pp above DPO's on Llama-3-8B, and 11.4 pp above on Yi-34B (Fig. 12 in App. C.3.13 of the revised paper shows all checkpoints).
  • No drop past DPO's peak: On Zephyr-7B, PEPO's members are trained past the number of updates at which DPO's WR peaks and PEPO's WR does not drop while DPO's does. DPO's WR peaks at epoch 3, and with L=3 PEPO's members receive as many updates at epoch 9. Over the same numbers of updates (DPO's epochs 3 to 5.3 and PEPO's epochs 9 to 16), DPO's WR drops from 75.3 to 72.0 (epoch 5), while PEPO's rises from 75.0 to 77.0.
  • One ensemble member peaks at the same point as DPO with fewer updates: On Llama-3-8B, member 1's WR peaks at epoch 5, as DPO's does, even though it received only a third of the updates. By epoch 16, member 1's WR drops by 4.6 pp and DPO's by 8.5 pp, while PEPO's drops by only 1.2 pp from its best (Fig. 10 in App. C.3.11 of the revised paper).

W2 / Q3 (aggregation rules on identical members). Our guarantees need the minimum, and on Llama-3-8B the minimum has a higher WR than the mean of the same members at 15 of the 16 epochs. Our guarantees hold for the minimum (Lemma 3.4) and do not cover the mean. In the 3-armed bandit of Fig. 4b, the suboptimality under the mean is higher than under the minimum from N = 500 on, and higher than DPO's from N = 1000 on. On Llama-3-8B with L=3, we generated answers from the same ensemble with the mean of the members' next-token probabilities instead of their minimum. With greedy answers, the WR under the mean is lower at 15 of the 16 epochs, by 1.6±0.61.6\pm0.6 pp on average over epochs (paired over prompts). With sampled answers (T=1, top-p 0.9), it is lower at all five epochs that we sampled, by 1.4±0.61.4\pm0.6 pp on average. Under both rules, the greedy WR drops by at most 1.3 pp from the best to the final checkpoint, against 8.5 pp for DPO. On this backbone, both ensemble rules therefore avoid DPO's drop, and the minimum, which our guarantees cover, is the better of the two. App. C.3.14 (Fig. 13) of the revised paper shows both rules at every epoch.

W3. The early checkpoints you cite cannot be identified during training. As discussed in Appendix C.3.6 and Figure 9, identifying them requires generating answers and judging them at every checkpoint, which is costly. The validation loss is cheap (since it needs no generation) but its minimum is not at the checkpoint with the highest WR (App. C.3.6). For your two examples, it is lowest at epoch 4 for Mistral-7B DPO (WR 80.2, not the 85.3 of epoch 8) and at epoch 10 for Zephyr-7B DPO+SFT (70.1, not the 82.5 of epoch 1). Both are below PEPO's final WR on the same backbone (82.9 and 77.0). The same holds for DPO on the other three backbones, by 5.0–13.1 pp. We do not claim that the endpoint is optimal. Table 4 shows that PEPO's final checkpoint is close to its best, so no checkpoint has to be selected.

Reviewer_XjhwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PEPO, a DPO-style method for reducing over-optimization without requiring access to the data-generating policy. The method trains several DPO models on different subsets of the preference data and combines them using a pessimistic ensemble rule. The paper provides theoretical guarantees in the tabular setting and evaluates the method on several 7B–34B language models, showing more stable performance than standard DPO during longer training.

Critical strengths

S1:The paper studies an important and practical problem in preference optimization. In particular, the assumption that the data-generating policy is known is often unrealistic for preference datasets collected from multiple or proprietary models.

S2:The main idea is simple and intuitive. Using disagreement across independently trained DPO models to construct a pessimistic policy is easy to understand and does not require an explicit reward model.

S3:The theoretical result is interesting. The sample complexity depends on a single-policy concentrability coefficient rather than the stronger all-policy concentrability required by standard DPO.

S4:The experiments cover several model families and scales, and the training curves clearly show that PEPO is generally more stable than DPO at later training epochs.

Critical weaknesses

W1: There is a gap between the main theoretical result and the method used in the large-scale experiments. The main theory is developed for response-level aggregation, while most LLM experiments use the more practical token-level approximation. Although the appendix provides additional analysis, the connection between the theory and the practical implementation is still not completely clear.

W2: The theoretical guarantee is limited to the tabular setting. It is unclear how much of the analysis extends to neural-network function approximation, which is the setting used in all of the main LLM experiments.

W3: Most large-scale experiments appear to use only one training seed. Since PEPO depends on data partitioning and ensemble diversity, results across multiple seeds or different dataset partitions would make the empirical evidence more convincing.

W4: The exact response-level rejection sampling method is very expensive in practice. The reported acceptance rate becomes very low during later training, making the theoretically motivated version difficult to use at scale.

Questions for authors
  • Can the authors provide multi-seed results for at least one representative model? How sensitive is PEPO to the random partition of the preference dataset?

  • How large is the additional approximation term introduced by the token-level normalization in practice? Can the authors measure it empirically?

  • How should the main theoretical result be interpreted for the token-level PEPO implementation used in most experiments?

  • Can the authors provide a stronger ablation to separate the effect of pessimistic aggregation from the effect of training each ensemble member on only (1/L) of the dataset?

  • Is the response-level rejection sampler intended to be practically usable for large models, or should it mainly be viewed as a theoretical construction?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Xjhw (1/3)

We thank the reviewer for the time spent reviewing our submission and for the positive comments on our paper. In particular, you find our theoretical result interesting and our main idea intuitive, and you note that PEPO is more stable than DPO in our experiments. Reading the weaknesses that you list and your negative evaluation, we feel that your main concern is a mismatch between theory and practice.

Regarding this concern, we would like to summarize our learning process that led to the solution in the paper. This will highlight the importance of the theoretical contribution in designing the practical version for PEPO.

We started with the question of how we could use an ensemble of models to avoid over-optimization in DPO without knowing the data distribution. An aggregation rule for an ensemble of DPO models that achieves our goal was absent from the literature (previous ensemble methods, such as Coste et al. [1], first learn an ensemble of reward models); therefore, we came up with the one at the foundation of our Algorithm 1. In order to make sure that it is a meaningful one to use, we came up first with the tabular analysis, and then we also demonstrated the effectiveness of this update in LLM experiments. However, it would have been impossible for us to come up with the practical update without the theoretical analysis, although the latter does not directly apply to neural networks. This is also acknowledged by Reviewer jDAS, who says that the tabular analysis serves as "principled motivation for the aggregation rule".

Moreover, we want to make clear here that in reinforcement learning and in LLM post-training there are, to our knowledge, no algorithms whose theoretical guarantees cover the training of neural networks as done in practice. A theory-practice mismatch is therefore unavoidable, and whether the gap is too large is a subjective point.

[1] Reward Model Ensembles Help Mitigate Overoptimization, Coste et al., ICLR 2024.

W3 / Q1 (seeds and the partition). Averaged over three training seeds, PEPO's WR is above DPO's at every epoch. On Llama-3-8B (L=3), we trained PEPO and DPO with two more seeds (43 and 44). The seed determines the members' shards, the batch order, and the initial weights and dropout masks of the LoRA adapters. PEPO's final WR is above DPO's by 11.1±0.911.1\pm0.9 pp and its LC win rate by 12.0±0.612.0\pm0.6 pp (mean ±\pm standard error over seeds). PEPO's final WR varies across seeds by 1.2 pp (standard deviation), far less than its gap to DPO. From the best to the final checkpoint, PEPO's WR drops by 2.0 pp on average, against 4.7 pp for DPO. App. C.3.16 (Fig. 15) of the revised paper shows every epoch, with the spread over seeds.

Q2 The additional approximation term introduced by token-level PEPO is captured by the additional term β(DKL(πouttoken,πoutprod)−DKL(π⋆,πoutprod))\beta (D_{\mathrm{KL}}(\pi_{\mathrm{out}}^{\mathrm{token}},\pi_{\mathrm{out}}^{\mathrm{prod}}) - D_{\mathrm{KL}}(\pi^\star,\pi_{\mathrm{out}}^{\mathrm{prod}})) in equation (14) which is absent from the response-level guarantees in Lemma 3.5. While there is no efficient way to measure DKL(πouttoken,πoutprod)−DKL(π⋆,πoutprod)D_{\mathrm{KL}}(\pi_{\mathrm{out}}^{\mathrm{token}},\pi_{\mathrm{out}}^{\mathrm{prod}}) - D_{\mathrm{KL}}(\pi^\star,\pi_{\mathrm{out}}^{\mathrm{prod}}) in practice, it can be controlled via a function of the token-level normalization constants (Lemma D.2 of the revised paper), which can be computed exactly on any sampled response. The approximation cost can be controlled by setting low values of β\beta.

AuthorsOpenReview ↗

Response to Reviewer Xjhw (2/3)

W1 / Q3. Lemma D.1 gives an error propagation bound for the token-level implementation, and this bound depends on C⋆C^\star, not on CallC^{\mathrm{all}}. In fact, in Appendix D, we provide a complete error propagation analysis for token-level PEPO which is what we used for most of the experiments. This analysis style is common in the RL literature (see below) and we therefore think that it serves as a strong enough motivation for our practical version. In particular, Lemma D.1 guarantees that, up to the additional term of Q2, the policy suboptimality is small whenever C⋆C^\star and ⟨πdata,Errtoken2⟩\langle \pi_{\mathrm{data}},\mathrm{Err}_{\mathrm{token}}^2 \rangle are, no matter which is the value of CallC^{\mathrm{all}}. Therefore, also token-level PEPO avoids over-optimization in the sense that it circumvents a dependence on CallC^{\mathrm{all}}. The theoretical advantage of the response level version is that we provide explicit control of the generalization error in Lemma 3.6.

Examples of error propagation analysis in RL literature Error propagation analyses have long been a valuable tool in the RL community, for example, in imitation learning [2, 3, 4] and in policy optimization [5, 6]. We, therefore, think that error propagation analyses like ours are important to motivate methods also in learning from preferences as our analysis of token-level PEPO does.

[2] Efficient Imitation under Misspecification, Espinosa-Dice et al.

[3] Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap, Swamy et al.

[4] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, Ross et al.

[5] Leveraging the Average: an Analysis of KL Regularization in RL, Vieillard et al.

[6] On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift, Agarwal et al.

W2. The guarantee is tabular, and tabular analyses have often led to practical algorithms with neural networks. A tabular analysis does not apply verbatim to neural networks, but this kind of analysis has historically given clear guidelines for practice (see the references below). In our case, the guideline is to aggregate an ensemble of DPO models pessimistically, by their minimum, to fight over-optimization instead of trying to do the same with a single model.

Historically, the tabular analysis of value iteration with count-based exploration bonuses [7] is at the foundations of the practical approach in [8] which uses neural networks, even though the analysis in [7] is tabular only. Moreover, several analyses for Bayesian RL methods [9] are limited to the tabular setting, but they lead to scalable methods. More recently, [10] shows that an imitation learning method with provable guarantees in the tabular setting can lead to effective methods for robotics.

We hope that these examples show why tabular guarantees matter for deriving practical algorithms.

[7] Minimax Regret Bounds for Reinforcement Learning, Mohammad Gheshlaghi Azar, Ian Osband, and Remi Munos

[8] Unifying Count-Based Exploration and Intrinsic Motivation, Bellemare et al.

[9] Probabilistic Inference in Reinforcement Learning Done Right, Tarbouriech et al.

[10] Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning, Wagenmaker et al.

AuthorsOpenReview ↗

Response to Reviewer Xjhw (3/3)

Q4 (each member evaluated alone). Training on 1/L of the data does not prevent over-optimization.

Each ensemble member is DPO trained on its own 1/L of the data. The following table shows the AlpacaEval2 win rates (WR) of each member alone, of PEPO and of DPO, on Llama-3-8B with L=3 (Fig. 10 in App. C.3.11 of the revised paper shows the WRs at every epoch).

ModelBest WR (epoch)Final WR (epoch 16)Drop from best
DPO69.3 (5)60.88.5
Member 067.3 (10)66.11.2
Member 175.3 (5)70.74.6
Member 269.6 (14)67.71.9
PEPO75.0 (10)73.81.2

Member 1's WR peaks at epoch 5, the same epoch as DPO's, but after a third as many updates. It then drops by 4.6 pp. PEPO's WR drops by only 1.2 pp, even though member 1 is part of its ensemble. The WRs of members 0 and 2 drop less than member 1's but are below PEPO's at all 16 epochs, by 1.4–9.7 pp. PEPO's final WR is 3.1–7.7 pp above each member's.

Q4 (as many updates as DPO). PEPO is not DPO trained more slowly. Each member of the ensemble receives 1/L1/L of DPO's updates in one epoch. PEPO's results do not come from this smaller number of updates.

  • Higher win rate (WR) with fewer updates: On Llama-3-8B, PEPO's WR is above DPO's at every epoch (by 0.6–13.9 pp). On Yi-34B (L=4), it is 6.8 pp above at epoch 4.
  • Much higher WR with as many updates: At epoch LkLk each member has received as many updates as DPO at epoch kk. PEPO's WR at these points is 4.6–12.2 pp above DPO's on Llama-3-8B, and 11.4 pp above on Yi-34B (Fig. 12 in App. C.3.13 of the revised paper shows all checkpoints).
  • No drop past DPO's peak: On Zephyr-7B, PEPO's members are trained past the number of updates at which DPO's WR peaks and PEPO's WR does not drop while DPO's does. DPO's WR peaks at epoch 3, and with L=3 PEPO's members receive as many updates at epoch 9. Over the same numbers of updates (DPO's epochs 3 to 5.3 and PEPO's epochs 9 to 16), DPO's WR drops from 75.3 to 72.0 (epoch 5), while PEPO's rises from 75.0 to 77.0.
  • One ensemble member peaks at the same point as DPO with fewer updates: On Llama-3-8B, member 1's WR peaks at epoch 5, as DPO's does, even though it received only a third of the updates. By epoch 16, member 1's WR drops by 4.6 pp and DPO's by 8.5 pp, while PEPO's drops by only 1.2 pp from its best (Fig. 10 in App. C.3.11 of the revised paper).

Q4 (aggregation rule on the same members). Our guarantees need the minimum, and on Llama-3-8B the minimum has a higher WR than the mean of the same members at 15 of the 16 epochs. Our guarantees hold for the minimum (Lemma 3.4) and do not cover the mean. In the 3-armed bandit of Fig. 4b, the suboptimality under the mean is higher than under the minimum from N = 500 on, and higher than DPO's from N = 1000 on. On Llama-3-8B with L=3, we generated answers from the same ensemble with the mean of the members' next-token probabilities instead of their minimum. With greedy answers, the WR under the mean is lower at 15 of the 16 epochs, by 1.6±0.61.6\pm0.6 pp on average over epochs (paired over prompts). With sampled answers (T=1, top-p 0.9), it is lower at all five epochs that we sampled, by 1.4±0.61.4\pm0.6 pp on average. Under both rules, the greedy WR drops by at most 1.3 pp from the best to the final checkpoint, against 8.5 pp for DPO. On this backbone, both ensemble rules therefore avoid DPO's drop, and the minimum, which our guarantees cover, is the better of the two. App. C.3.14 (Fig. 13) of the revised paper shows both rules at every epoch.

W4 / Q5 (the exact sampler). The exact sampler is mainly a theoretical construction, and all LLM tables use the token-level rule. In Fig. 2, the sampler has the highest WR at epochs 1–3, but its cost grows during training. As the members diverge during training, the sampler (Algorithm 2) accepts fewer proposals. On Llama-3-8B with L=4 (805 prompts), the sampler draws 2.3 proposals per prompt at epoch 1 and 10.9 at epoch 3. By epoch 6, 97% of prompts reach the cap of 16 proposals, and for these prompts the sampler returns the best proposal seen. App. C.3.9 reports about 80 hours for one AlpacaEval evaluation on an A100 GPU with the sampler, against 1.25 hours with the token-level rule. The token-level rule runs one forward pass per member for each token. With L=4, it generates a token in about 24 ms in the run of Fig. 2, against about 6 ms for DPO. App. D gives an error propagation bound for this rule, with the additional term of Q2, which Lemma D.2 bounds.

We believe that these results address W1–W4, and we would be grateful if you would reconsider your score in light of them. We are happy to answer any further questions.

Dear reviewer Xjhw,

we are reaching out because the end of the discussion period is approaching and, unfortunately, we did not have a chance to discuss. We think that our responses clarify your concerns.

We would be grateful if you could look at our response and reach back to us before the end of the discussion period. We are happy to provide additional clarifications if needed.

Best, Authors

Reviewer_EBwcOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

PEPO trains LoRA adapters on disjoint shards of the preference data. It learns the policy that maximizes a pessimistic virtual objective based on the minimum over the ensemble's implicit rewards. The output can be sampled exactly by rejection sampling or approximated by token-level min aggregation.

Empirically, this paper shows performances over several models including Llama-3.1-Tulu-3-8B-SFT, Zephyr-7B, Mistral-7B, and Yi-34B. Datset are UltraFeedback Binarized. Win rates are measured on AlpacaEval2 and MT-Bench against each model's own SFT checkpoint, using a Llama-3-70B judge. DPO degrades late in training, while token-level PEPO stabilizes and attains the best final-checkpoint score.

Critical strengths
  • This paper studies a DPO-like method with no explicit reward model and a single-policy concentrability guarantee, without access to the data generated by the optimal policy π\pi.

  • The method is simple. Empirically, the final checkpoint stays close to the best one. It means the real problem, which is the checkpoint selection for DPO is unreliable.

Critical weaknesses
  • The deployed algorithm may lack the mechanism that the theory relies on. In the analysism it seams the pessimism comes from both the count-based penalty and the centering. Therefore, the practical method seems to reduce to a renormalized token-level minimum over the members' probabilities.

  • The empirical comparison is confounded, and its statistical support is limited. For example, all runs use a single seed, and the error bars are over evaluation prompts only. Win rates are measured against each model's own SFT checkpoint, without length control or response-length statistics.

Questions for authors

Please add the following controls: mean aggregation of the same adapters, DPO on a single shard, step-matched DPO, and DPO with larger β\beta or IPO.

Please report results over multiple seeds for at least one backbone, together with length-controlled AlpacaEval or response-length statistics.

What is the acceptance rate of the rejection sampler in practice?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EBwc (1/3)

Dear reviewer,

Thank you very much for the time spent reviewing our paper and for your positive evaluation. Please find below our responses to your interesting questions.

W1. The deployed algorithm keeps the main mechanism of the theory.

We thank the reviewer for the interesting question. The response is that in our experimental setup the count-based penalty is responsible for removing the all-policy concentrability coefficient from lower-order terms of our upper bound. In contrast, our pessimistic aggregation is the most important mechanism which moves the all-policy concentrability coefficient from the leading term of the bound to the lower order one. We sketch the proof of this result below.

Proof Sketch As a first observation, we notice that in UltraFeedback each prompt comes with a single compared pair of responses, so the pairs compared at each prompt are vertex-disjoint. Therefore, the observed preferences are realized exactly by a BT model, and DxℓD^\ell_x in equation (18) equals 00 for the UltraFeedback dataset (Lemma 3.4, Step 1). As a consequence, also the inconsistency slack, denoted as slackℓ(x,a,b)\mathrm{slack}^\ell(x,a,b) in Lemma 3.4 (Step 1), equals 00.

Given this observation and inspecting the equation at line 1924 of the revised paper (which is at the end of the first display in the proof of Lemma 3.4 (Step 3)), we notice that we have Δ^(x,a,b)≤Δr⋆(x,a,b)+γ(x,a,b)−C(x,a).\hat{\Delta}(x,a,b) \leq \Delta_{r^\star}(x,a,b) + \gamma(x,a,b) - C(x,a). Therefore, if there is no count-based penalty mechanism (i.e. C(x,a)=0C(x,a)=0), we have that pessimism holds up to a factor γ(x,a,b)\gamma(x,a,b): Δ^(x,a,b)−Δr⋆(x,a,b)≤γ(x,a,b).\hat{\Delta}(x,a,b) - \Delta_{r^\star}(x,a,b)\leq \gamma(x,a,b). As a consequence, the regret bound without count-based penalty would feature an additional term of order Call∑x,a,bν0(x)πdata(a∣x)πdata(b∣x)γ(x,a,b)≤O(∣X∣∣A∣2Calllog⁡NN)C^{\mathrm{all}} \sum_{x,a,b} \nu_0(x) \pi_{\mathrm{data}}(a|x) \pi_{\mathrm{data}}(b|x) \gamma(x,a,b) \leq \mathcal{O}\bigg( \frac{ |\mathcal{X}| |\mathcal{A}|^2 C^{\mathrm{all}} \log N}{N} \bigg), where the last inequality can be proven with similar steps to the ones that lead to Eq. (25) in our revised submission. This is a lower-order term because the main one, that depends only on C⋆C^\star and not on CallC^\mathrm{all}, decays at the slower rate of N−1/2N^{-1/2}. This concludes the proof sketch.

If the reviewer finds this discussion interesting, we can add it to our revision.

Regarding the centering, we find that it is not necessary in practice because each ensemble member is trained from the same initialization scheme with the same optimization algorithm and is therefore unlikely to converge to solutions with very different magnitude of the log policy ratio. In particular, at epoch 16, the average of log⁡(πℓ/πref)\log(\pi^\ell/\pi_{\mathrm{ref}}) over the held-out answers is −146.8-146.8, −165.6-165.6 and −154.3-154.3, which are all on the same order of magnitude.

Finally, we notice that the factor C(x,a)C(x,a) can be well approximated by a constant in the LLM experiments in which each pair x,ax,a is visited only once in the training set and for the same reason ζℓ\zeta^\ell can be set to zero. Under this approximation, neither C(x,a)C(x,a) nor ζℓ\zeta^\ell has an effect on the policy update. We provide this perspective in the response to Q1 of Reviewer jDAS.

AuthorsOpenReview ↗

Response to Reviewer EBwc (2/3)

W2 / Q2 (seeds). Averaged over three training seeds, PEPO's WR is above DPO's at every epoch. On Llama-3-8B (L=3), we trained PEPO and DPO with two more seeds (43 and 44). The seed determines the members' shards, the batch order, and the initial weights and dropout masks of the LoRA adapters. PEPO's final WR is above DPO's by 11.1±0.911.1\pm0.9 pp and its LC win rate by 12.0±0.612.0\pm0.6 pp (mean ±\pm standard error over seeds). PEPO's final WR varies across seeds by 1.2 pp (standard deviation), far less than its gap to DPO. From the best to the final checkpoint, PEPO's WR drops by 2.0 pp on average, against 4.7 pp for DPO. App. C.3.16 (Fig. 15) of the revised paper shows every epoch, with the spread over seeds.

W2 (SFT reference). The WR against a model's own SFT outputs measures what preference training changes. On each backbone, PEPO and DPO are judged against the same SFT outputs, so the comparison between them is direct. Liu et al. [1] report the same kind of comparison (pairwise win rates among their method, DPO and the reference model, which is their SFT model).

[1] Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer, Liu et al., NeurIPS 2024.

W2 / Q2 (length control). PEPO beats DPO on length-controlled (LC) win rate for every backbone and degrades least. PEPO's final LC win rate is above DPO's on all four backbones (68.3 vs. 56.6 on Llama, 70.7 vs. 65.9 on Zephyr, 75.9 vs. 66.8 on Mistral, 70.8 vs. 58.7 on Yi). On Llama, Zephyr and Mistral, the difference between PEPO's and DPO's final win rate (WR) changes by at most 1.3 pp when we control for answer length (from 13.0, 6.0 and 8.5 pp to 11.7, 4.8 and 9.1 pp). In addition, at the last checkpoint PEPO loses at most 2.2 pp of LC win rate from its best checkpoint, while the baselines lose 2.7–12.3 pp. PEPO's answers are shorter than DPO's on Zephyr and Yi, and longer on Llama and Mistral. They are shorter than χ2\chi^2PO's on all three backbones we ran it on. App. C.3.12 of the revised paper gives the LC win rates and mean answer lengths of all methods (Table 7), the drops (Table 8) and the LC curves (Fig. 11).

Q1 (mean aggregation of the same adapters). Our guarantees need the minimum, and on Llama-3-8B the minimum has a higher WR than the mean of the same members at 15 of the 16 epochs. Our guarantees hold for the minimum (Lemma 3.4) and do not cover the mean. In the 3-armed bandit of Fig. 4b, the suboptimality under the mean is higher than under the minimum from N = 500 on, and higher than DPO's from N = 1000 on. On Llama-3-8B with L=3, we generated answers from the same ensemble with the mean of the members' next-token probabilities instead of their minimum. With greedy answers, the WR under the mean is lower at 15 of the 16 epochs, by 1.6±0.61.6\pm0.6 pp on average over epochs (paired over prompts). With sampled answers (T=1, top-p 0.9), it is lower at all five epochs that we sampled, by 1.4±0.61.4\pm0.6 pp on average. Under both rules, the greedy WR drops by at most 1.3 pp from the best to the final checkpoint, against 8.5 pp for DPO. On this backbone, both ensemble rules therefore avoid DPO's drop, and the minimum, which our guarantees cover, is the better of the two. App. C.3.14 (Fig. 13) of the revised paper shows both rules at every epoch.

AuthorsOpenReview ↗

Response to Reviewer EBwc (3/3)

Q1 (DPO on a single shard). Training on 1/L of the data does not prevent over-optimization.

Each ensemble member is DPO trained on its own 1/L of the data. The following table shows the AlpacaEval2 win rates (WR) of each member alone, of PEPO and of DPO, on Llama-3-8B with L=3 (Fig. 10 in App. C.3.11 of the revised paper shows the WRs at every epoch).

ModelBest WR (epoch)Final WR (epoch 16)Drop from best
DPO69.3 (5)60.88.5
Member 067.3 (10)66.11.2
Member 175.3 (5)70.74.6
Member 269.6 (14)67.71.9
PEPO75.0 (10)73.81.2

Member 1's WR peaks at epoch 5, the same epoch as DPO's, but after a third as many updates. It then drops by 4.6 pp. PEPO's WR drops by only 1.2 pp, even though member 1 is part of its ensemble. The WRs of members 0 and 2 drop less than member 1's but are below PEPO's at all 16 epochs, by 1.4–9.7 pp. PEPO's final WR is 3.1–7.7 pp above each member's.

Q1 (step-matched DPO). PEPO is not DPO trained more slowly. Each member of the ensemble receives 1/L1/L of DPO's updates in one epoch. PEPO's results do not come from this smaller number of updates.

  • Higher win rate (WR) with fewer updates: On Llama-3-8B, PEPO's WR is above DPO's at every epoch (by 0.6–13.9 pp). On Yi-34B (L=4), it is 6.8 pp above at epoch 4.
  • Much higher WR with as many updates: At epoch LkLk each member has received as many updates as DPO at epoch kk. PEPO's WR at these points is 4.6–12.2 pp above DPO's on Llama-3-8B, and 11.4 pp above on Yi-34B (Fig. 12 in App. C.3.13 of the revised paper shows all checkpoints).
  • No drop past DPO's peak: On Zephyr-7B, PEPO's members are trained past the number of updates at which DPO's WR peaks and PEPO's WR does not drop while DPO's does. DPO's WR peaks at epoch 3, and with L=3 PEPO's members receive as many updates at epoch 9. Over the same numbers of updates (DPO's epochs 3 to 5.3 and PEPO's epochs 9 to 16), DPO's WR drops from 75.3 to 72.0 (epoch 5), while PEPO's rises from 75.0 to 77.0.
  • One ensemble member peaks at the same point as DPO with fewer updates: On Llama-3-8B, member 1's WR peaks at epoch 5, as DPO's does, even though it received only a third of the updates. By epoch 16, member 1's WR drops by 4.6 pp and DPO's by 8.5 pp, while PEPO's drops by only 1.2 pp from its best (Fig. 10 in App. C.3.11 of the revised paper).

Q1 (larger β\beta, IPO). Stronger regularization does not prevent the drop in the baselines that we ran, and prior work reports the same for larger β\beta and for IPO. We did not train DPO with a larger β\beta or IPO. The paper has two regularized baselines: χ2\chi^2PO, which adds a χ2\chi^2-divergence term to DPO's KL regularization, and DPO+SFT, which adds an SFT loss. The WR of both still drops from the best to the final checkpoint, by 5.1–7.9 pp for χ2\chi^2PO and by 4.8–12.8 pp for DPO+SFT, while PEPO's drops by at most 1.2 pp in the setting of Table 1 (Table 4). Rafailov et al. [2] trained DPO and IPO with seven values of β\beta each and report over-optimization for both.

[2] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms, Rafailov et al., NeurIPS 2024.

Q3 (acceptance rate). The acceptance rate falls as the members diverge during training. On Llama-3-8B with L=4 (805 prompts), the sampler draws 2.3 proposals per prompt at epoch 1 and 10.9 at epoch 3. By epoch 6, 97% of prompts reach the cap of 16 proposals, and for these prompts the sampler returns the best proposal seen (as in Fig. 2). Fig. 2 compares the generation time of the sampler with that of token-level PEPO and DPO, and App. C.3.9 reports about 80 hours for one AlpacaEval evaluation on an A100 GPU with the sampler, against 1.25 hours with the token-level rule. For this reason, we use the token-level rule in all LLM tables.

We believe that these results answer your questions, and we would be grateful if you would reconsider your score in light of them. We are happy to answer any further questions.

Dear reviewer EBwc,

as the end of the discussion phase is approaching, we are reaching out to ask if our response clarifies your questions. Moreover, we are happy to continue the discussion in case new questions arise.

Thank you for your positive evaluation of our work.

Best, Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

New Experiments and Revised Paper

We thank the reviewers for their careful reading. The reviewers agree that PEPO is more stable late in training. EBwc notes that "the final checkpoint stays close to the best one", Xjhw that PEPO is "generally more stable than DPO at later training epochs", and jDAS that it is "more stable during the later part of the tested training runs". They also credit the theory (Xjhw, jDAS), which jDAS calls "a principled motivation for the aggregation rule".

To address the reviewers' questions and concerns, we ran new experiments, extended the analysis of the token-level implementation and uploaded a revised paper (additions in red). The experiments test whether PEPO's gain over DPO comes from answer length, from members that train on less data or with fewer updates, from the judge or from the training seed. It does not.

  1. Length control (EBwc, jDAS). PEPO's final checkpoint has a higher length-controlled (LC) win rate than DPO's on all four backbones. Its LC win rate also drops less from the best to the final checkpoint than that of any baseline. See App. C.3.12 (Fig. 11, Tables 7 and 8).

  2. DPO on one shard (EBwc, Xjhw, jDAS). On Llama-3-8B with L=3, each of PEPO's members is DPO trained on a third of the data. We evaluated each member alone. From the best to the final checkpoint, one member's win rate (WR) drops by 4.6 pp while PEPO's drops by only 1.2 pp. The WRs of the other two members are below PEPO's at all 16 epochs. See App. C.3.11 (Fig. 10, Table 6). Therefore, individual ensemble members can still over-optimize and the PEPO aggregation rule mitigates this effect.

  3. Step-matched DPO (EBwc, Xjhw, jDAS). PEPO is not DPO trained more slowly. On Llama-3-8B, PEPO's WR is above DPO's at every epoch, although each member has received a third of DPO's updates. On Zephyr-7B, each member receives more updates than DPO has at its peak, and PEPO's WR still does not drop. See App. C.3.13 (Fig. 12).

  4. Mean aggregation (EBwc, Xjhw, jDAS). Our theoretical guarantees require the minimum. On Llama-3-8B, the mean of the same members' next-token probabilities has a lower WR than their minimum at 15 of the 16 epochs, by 1.6±0.61.6\pm0.6 pp with greedy answers and 1.4±0.61.4\pm0.6 pp with sampled answers. In the tabular setting, the mean aggregation rule is outperformed by PEPO from N = 500 on and by DPO from N = 1000 on. See App. C.3.14 (Fig. 13).

  5. Second judge (jDAS). We repeated the AlpacaEval2 evaluation with Claude Haiku 5.5 as the judge. PEPO's final checkpoint has a higher WR than every baseline's on every backbone. From the best to the final checkpoint, PEPO's WR drops by at most 2.1 pp, against 5.3–11.4 pp for DPO. See App. C.3.15 (Table 9, Fig. 14).

  6. Training seeds (EBwc, Xjhw). On Llama-3-8B, we trained PEPO (L=3) and DPO with two more seeds (43 and 44). Averaged over the seeds, PEPO's WR and LC win rate are above DPO's at every epoch. PEPO's final WR is higher by 11.1±0.911.1\pm0.9 pp and its LC win rate by 12.0±0.612.0\pm0.6 pp (mean ±\pm standard error over seeds). From the best to the final checkpoint, its WR drops by 2.0 pp on average, against 4.7 pp for DPO. See App. C.3.16 (Fig. 15).

  7. Theory-practice gap (Xjhw, jDAS). When working with large language models, a theory-practice gap is of course unavoidable, but we think that our theoretical result is of fundamental importance to motivate the pessimistic aggregation rule used in the experiments. Indeed, we came up first with the tabular analysis that shows that the PEPO aggregation rule is meaningful at least in the tabular setting and then we demonstrated also the effectiveness of this update in LLM experiments. However, it would have been impossible for us to come up with the practical update without the theoretical analysis, albeit this one does not directly apply to neural networks. This is also acknowledged by Reviewer jDAS that says that the tabular analysis serves as "principled motivation for the aggregation rule". The LLM experiments use the token-level implementation, whose guarantee (Lemma D.1) contains an additional term, and the new Lemma D.2 bounds this term by a quantity that does not depend on π⋆\pi^\star. See App. D.

We also unified and improved the style of Figures 1, 2, 7, 8 and 9. The values in them stayed untouched.

AuthorsOpenReview ↗

Second revision: Yi-34B MT-Bench correction and validation-loss checkpoints in Fig. 9

We uploaded a second revision.

  1. Yi-34B MT-Bench (jDAS). Table 1 reported PEPO's WR at epoch 5 (62.6), although its caption says epoch 4. We thank reviewer jDAS for noticing this error. Fig. 8 now stops at epoch 4, the last epoch of DPO's run and of the AlpacaEval2 results, and Tables 1 and 3 now report PEPO's epoch 4 instead of epoch 5 (as the final epoch in Table 1 and the best epoch in Table 3). At epoch 4, PEPO's WR is 59.7, still above DPO's 56.0, so no conclusion changes.
  2. Validation loss for checkpoint selection (jDAS). Fig. 9 (App. C.3.6) now shows the validation loss and the WR of DPO, DPO+SFT and χ2\chi^2PO on every backbone, and a table gives the WR at the checkpoint with the lowest validation loss in each run. As for DPO, early stopping on the validation loss does not select the checkpoint with the best WR for DPO+SFT and χ2\chi^2PO.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.