- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PEPO, a DPO-like method for reducing over-optimization without access to the data-generating policy or an explicitly trained reward model. It splits the preference dataset into disjoint subsets, trains a separate DPO policy on each subset, and combines the policies through pessimistic aggregation at inference time.
In the tabular setting, the authors derive sample-complexity guarantees that depend on single-policy concentrability and provide guidance on the ensemble size. The experiments compare PEPO with DPO and pessimistic DPO variants on four backbones, using a token-level approximation for the main LLM evaluations. The reported results show better stability at later checkpoints and higher final-checkpoint scores in the tested configurations.
Critical strengths
S1. The paper addresses a useful theoretical question: whether pessimism can be introduced into a DPO-like method without knowing the policy that generated the preference data. The tabular analysis is developed in substantial detail, with supporting proofs and a discussion of how ensemble size trades off pessimism against estimation error. This provides a principled motivation for the aggregation rule. The analysis's scope and structure are a substantive strength.
S2. The experimental study is reasonably broad. It includes four backbones, synthetic experiments that control the relationship between the data and reference policies, comparisons with DPO, χ²PO, and DPO+SFT, and additional analyses of ensemble size and inference latency. Reporting training curves and both best-observed and final-checkpoint results is particularly useful. These results support the observation that PEPO is more stable during the later part of the tested training runs.
Critical weaknesses
W1. Connection between the theoretical guarantee and the implemented LLM method may not be clear enough.
Theorem 3.3 relies on a sample-complexity bound scaling with the massive sizes of |X||A|⁴, and the sufficient ensemble size L far exceeds the 2–4 members used in the experimental implementations. There may not be a sufficiently clear explanation of which part of the theoretical guarantee should still be expected to hold in the reported LLM runs, or how the practical choice of L relates to the condition needed to establish pessimism.
Furthermore, while the theory uses response-level aggregation, the experiments rely on token-level aggregation. Although Appendix D analyzes this, the error and normalization-mismatch terms are not quantified for practical settings. A clearer account is needed to show that these approximation terms do not undermine the theoretical connection.
W2. The comparisons may not sufficiently separate pessimistic aggregation from training exposure.
Each PEPO member (LoRA model) trains on only 1/3 to 1/4 of the full dataset, receiving fewer updates per epoch than a full-dataset DPO model. Consequently, the epoch-aligned comparisons in Figures 1, 7, and 8 conflate the benefits of pessimistic aggregation with the shorter training history of each member.
A single DPO member trained on the same data shard with matched updates, alongside comparisons of different aggregation rules using an identical set of trained members, would better isolate the source of the gains.
W3. The choice of final checkpoints needs more justification.
Table 1 uses epoch 16 for 7B/8B models and epoch 4 for Yi-34B, while Figure 8 and Table 3 use epoch 5 for Yi-34B. It remains unclear why these budgets were chosen, why multiple passes over the preference dataset are needed or beneficial, or how models perform beyond these points.
While using a uniform endpoint helps study late-training stability, Table 4 only shows proximity to the best score within a window, not that the endpoint is optimal.
Additionally, earlier checkpoints complicate the cost-benefit analysis. In Table 3, Mistral-7B DPO reaches 85.3 on AlpacaEval2 at epoch 8 (vs. PEPO's 84.0 at epoch 14), and Zephyr-7B DPO+SFT hits 82.5 at epoch 1 (vs. PEPO's 77.6 at epoch 12). These results suggest that early-stopped baselines can sometimes match or exceed PEPO, meaning ensemble gains may not be universal evidence of superior resource efficiency.
W4. The additional inference cost makes the practical advantage less clear.
PEPO requires multiple forward passes during token-level inference, and the exact response-level sampler is much slower with a decreasing acceptance rate during training (Figure 2 and Appendix C.3.9). Direct comparisons at matched training and inference budgets are necessary. Without these controls, the paper demonstrates late-training stability more effectively than a compelling overall deployment advantage.
Questions for authors
Q1. Clarification of the implemented pessimism terms. How are , the centering factors , and the penalty implemented in the LLM experiments? For the token-level version, how are the prefix counts and penalties constructed? Providing the actual inference rule and specifying which assumptions of the theoretical analysis remain satisfied in the LLM experiments would be helpful.
Q2. Clarification of the training endpoint. What determined the budgets of 16 epochs for the 7B/8B models and 4 epochs for Yi-34B (5 in MT-Bench)? If tens of epochs are necessary for preference learning, is there evidence about stability beyond these endpoints?
Q3. Control for training exposure. A matched-update control for the individual members would clarify whether the gains actually stem from pessimistic aggregation.
Q4. Length control and evaluation robustness. The length-controlled win rate (LC-WR) of AlpacaEval2 is a useful metric for evaluating the model's ability beyond length bias. How do average response lengths compare across methods? Reporting these results would help assess whether response-length differences contribute to the observed gains. Additional metrics or evaluation with a different judge would also help establish the robustness of the comparison.
I would be very happy to update my assessment if the authors can clarify these points and provide additional analysis. Looking forward to further discussion.
7 comments
Response to Reviewer jDAS (2/4)
Q1. In the LLM runs, the ensemble aggregation is the only active pessimistic ingredient. In the practical LLM implementation, we consider the centering shift and the penalty to be constants. We consider the constant approximation to work well because in the LLM experiments a state action pair is visited either once or never, i.e. . Inspecting the expression for in Theorem 3.3, and omitting the numerator, we see that decays as . Therefore, when the action space is very large as in LLMs and the counts are binary we have that and the constant approximation for is justified. Finally, we set to zero because when each prompt appears only once we have that (because each prompt in never appears again in ). By our convention, we have and therefore we set and as a consequence to be constant in the experiment.
Our main takeaway is that when the data are scarce, that is, when no state-action pair is revisited, the ensemble aggregation is the only active pessimistic ingredient of our algorithm. In other words, the stability of PEPO in the LLM experiments is due solely to the ensemble aggregation rule and it is not necessary to introduce additional pessimistic penalties. Following the same reasoning, we consider as constants also the prefix counts of the token-level PEPO implementation.
The exact inference (aggregation) rule. Token-level PEPO generates each token from the renormalized minimum of the members' next-token probabilities. With greedy decoding, as in our tables, it takes the most likely token under this distribution. The probability of generating token after prompt and previous tokens is given by
where is the token vocabulary and it is usually small enough to compute the exact sum over it.
Validity of the theoretical analysis in practice. The tabular bounds are too loose to explain PEPO's results quantitatively, and they serve as a design principle. The LLM setup actually has finitely many states and actions, as our theory requires. However, our bounds would be extremely loose because they consider each prompt and response as completely independent from any other possible pairs. This leads to guarantees that scale with the number of states and actions which are larger than the available dataset size in language applications. The tabular guarantees therefore are not sufficient to explain the success of PEPO in practice even though they serve as a strong designing principle. What is missing from our analysis and what could lead to tighter theoretical guarantees in the future is the parameter sharing of LLMs across similar prompts and responses, which allows good performance with dataset and ensemble sizes much smaller than what our tabular analysis would predict.
Response to Reviewer jDAS (3/4)
W2 / Q3 (a single member on its own shard). Training on 1/L of the data does not prevent over-optimization.
Each ensemble member is DPO trained on its own 1/L of the data. The following table shows the AlpacaEval2 win rates (WR) of each member alone, of PEPO and of DPO, on Llama-3-8B with L=3 (Fig. 10 in App. C.3.11 of the revised paper shows the WRs at every epoch).
| Model | Best WR (epoch) | Final WR (epoch 16) | Drop from best |
|---|---|---|---|
| DPO | 69.3 (5) | 60.8 | 8.5 |
| Member 0 | 67.3 (10) | 66.1 | 1.2 |
| Member 1 | 75.3 (5) | 70.7 | 4.6 |
| Member 2 | 69.6 (14) | 67.7 | 1.9 |
| PEPO | 75.0 (10) | 73.8 | 1.2 |
Member 1's WR peaks at epoch 5, the same epoch as DPO's, but after a third as many updates. It then drops by 4.6 pp. PEPO's WR drops by only 1.2 pp, even though member 1 is part of its ensemble. The WRs of members 0 and 2 drop less than member 1's but are below PEPO's at all 16 epochs, by 1.4–9.7 pp. PEPO's final WR is 3.1–7.7 pp above each member's.
W2 / Q3 (matched updates). PEPO is not DPO trained more slowly. Each member of the ensemble receives of DPO's updates in one epoch. PEPO's results do not come from this smaller number of updates.
- Higher win rate (WR) with fewer updates: On Llama-3-8B, PEPO's WR is above DPO's at every epoch (by 0.6–13.9 pp). On Yi-34B (L=4), it is 6.8 pp above at epoch 4.
- Much higher WR with as many updates: At epoch each member has received as many updates as DPO at epoch . PEPO's WR at these points is 4.6–12.2 pp above DPO's on Llama-3-8B, and 11.4 pp above on Yi-34B (Fig. 12 in App. C.3.13 of the revised paper shows all checkpoints).
- No drop past DPO's peak: On Zephyr-7B, PEPO's members are trained past the number of updates at which DPO's WR peaks and PEPO's WR does not drop while DPO's does. DPO's WR peaks at epoch 3, and with L=3 PEPO's members receive as many updates at epoch 9. Over the same numbers of updates (DPO's epochs 3 to 5.3 and PEPO's epochs 9 to 16), DPO's WR drops from 75.3 to 72.0 (epoch 5), while PEPO's rises from 75.0 to 77.0.
- One ensemble member peaks at the same point as DPO with fewer updates: On Llama-3-8B, member 1's WR peaks at epoch 5, as DPO's does, even though it received only a third of the updates. By epoch 16, member 1's WR drops by 4.6 pp and DPO's by 8.5 pp, while PEPO's drops by only 1.2 pp from its best (Fig. 10 in App. C.3.11 of the revised paper).
W2 / Q3 (aggregation rules on identical members). Our guarantees need the minimum, and on Llama-3-8B the minimum has a higher WR than the mean of the same members at 15 of the 16 epochs. Our guarantees hold for the minimum (Lemma 3.4) and do not cover the mean. In the 3-armed bandit of Fig. 4b, the suboptimality under the mean is higher than under the minimum from N = 500 on, and higher than DPO's from N = 1000 on. On Llama-3-8B with L=3, we generated answers from the same ensemble with the mean of the members' next-token probabilities instead of their minimum. With greedy answers, the WR under the mean is lower at 15 of the 16 epochs, by pp on average over epochs (paired over prompts). With sampled answers (T=1, top-p 0.9), it is lower at all five epochs that we sampled, by pp on average. Under both rules, the greedy WR drops by at most 1.3 pp from the best to the final checkpoint, against 8.5 pp for DPO. On this backbone, both ensemble rules therefore avoid DPO's drop, and the minimum, which our guarantees cover, is the better of the two. App. C.3.14 (Fig. 13) of the revised paper shows both rules at every epoch.
W3. The early checkpoints you cite cannot be identified during training. As discussed in Appendix C.3.6 and Figure 9, identifying them requires generating answers and judging them at every checkpoint, which is costly. The validation loss is cheap (since it needs no generation) but its minimum is not at the checkpoint with the highest WR (App. C.3.6). For your two examples, it is lowest at epoch 4 for Mistral-7B DPO (WR 80.2, not the 85.3 of epoch 8) and at epoch 10 for Zephyr-7B DPO+SFT (70.1, not the 82.5 of epoch 1). Both are below PEPO's final WR on the same backbone (82.9 and 77.0). The same holds for DPO on the other three backbones, by 5.0–13.1 pp. We do not claim that the endpoint is optimal. Table 4 shows that PEPO's final checkpoint is close to its best, so no checkpoint has to be selected.
Response to Reviewer jDAS (1/4)
Dear reviewer,
Thank you for appreciating our theoretical analysis as "a substantive strength" and for defining our experimental study as "reasonably broad". We are happy to receive your thought-provoking questions which we address below.
W1. The tabular analysis sets sufficient conditions for a provably efficient aggregation rule, and the LLM runs test that rule. Before going into details, we would like to clarify that our tabular analysis serves as a guideline to define a reasonable pessimistic aggregation rule for an ensemble of models. Our analysis serves to establish sufficient conditions for the rule to be provably efficient in the tabular setting. The goal is not to provide sample-complexity guarantees for the LLM version used in practice, which would clearly be overambitious. Having clarified this, we can go on answering your specific points.
The ensemble size L. The L of the theory is a worst-case sufficient condition, and L = 2–4 works in practice. The value of L prescribed by our theory grows logarithmically in the number of states and actions, which is clearly larger than the values of 2–4 which we found effective in practice. The cause of this mismatch is that our tabular analysis is for the worst-case instance which is unlikely to be the one faced in practice. In the controlled bandit of App. C.1.1 (Fig. 3), PEPO finds the optimal action for every L we tried, and L=3 performs best.
Response- vs. token-level aggregation. The guarantee for the token-level rule (App. D) depends on C⋆ and the token-level estimation error, not on Call. The token-level aggregation is preferred in practice as it can be implemented in closed form by summing over the vocabulary instead of resorting to rejection sampling. Its analysis in Appendix D should be thought of as an error propagation analysis as it allows us to conclude that the squared suboptimality of the learned policy is at most the token-level generalization error ⟨πdata,Errtoken2⟩ times the inflation factor C⋆. Therefore, thanks to the pessimistic aggregation at token level, Call does not enter the bound. While we do not prove that ⟨πdata,Errtoken2⟩ decreases with the dataset size N, our analysis says that the suboptimality is small whenever ⟨πdata,Errtoken2⟩ is. This is, in spirit, an error propagation analysis which, even if it does not lead to sample complexity guarantees, is an informative kind of analysis in the RL literature. Among several potential examples, we mention some in imitation learning [1, 2, 3] and in policy optimization [4, 5]. We, therefore, think that error propagation analyses like ours for token-level PEPO are important to motivate methods also in learning from preferences.
[1] Efficient Imitation under Misspecification, Espinosa-Dice et al.
[2] Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap, Swamy et al.
[3] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, Ross et al.
[4] Leveraging the Average: an Analysis of KL Regularization in RL, Vieillard et al.
[5] On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift, Agarwal et al.