- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes about the tendency of hyperparameter of Direct preference optimization (DPO), especially in three aspects: (1) validation loss is poor and unreliable proxy for hyperparameter optimization (2) optimal hyperparameter configuration changes along dataset and scale (3) Partial runs can be a informative proxy scale-dependentably, and adaptive HPO outperforms random search. Based on those analysis, authors provide the practical recommendations for DPO hyperparameter selection, such as select configurations using downstream evaluations, retune for different setting, use partial target-model runs.
Critical strengths
- This paper proposes diverse analysis for DPO hyperparameter, such as the learning-rate trend shows the downward trend as the model size gets bigger, or similar model size (especially 0.6B-1.7B, 4B-14B) show some similar trend at hyperparameter optimization. Such analysis can be a good guide for DPO hyperparameter selection, selecting lower learning rate for bigger models, or trying to use similar model size's optimal hyperparameter as a starting point.
- This paper proposes nice practical recommendations for DPO hyperparameter optimization such as use diverse downstream evaluations rather than validation loss, retuning for target setting (scale, dataset), and use partial runs as proxy before whole evaluation. Such recommendations can be a good starting reference for DPO hyperparameter selection for the researchers who are conducting DPO training, and is worth for the research community.
Critical weaknesses
- Additional analysis of hyperparameter landscape at fixed model scale and benchmark. It seems analysis of hyperparameter landscapes at certain model scale and benchmark are needed, whether there exists some multiple optimal setting (multi-mode landscape), or shows high at certain settings. And additional analysis or interpretation about why certain hyperparameters may have been selected or shows good performance at certain model scale and benchmarks may validate the result of the experiments.
- More interpretation of validation loss. The authors claimed that validation loss is not informative for DPO hyperparameter optimization, but it seems like current 'validation loss' can be used only as proxy measure only within the same training, not across different training setting. Validation loss of DPO measures the relative probability between current and reference model, so the loss may differ when the reference model is different (e.g., model scale). It seems like some normalization for such reference model, or beta should be done to precisely analyze the correlation between validation loss and the model.
Questions for authors
- I wonder about the 'reason' of the validation loss was poor proxy in current setting. Validation loss should differ according to model and benchmark type so it seems like some 'normalization' of beta/ should be done to appropriately use the loss as proxy (also I have seen the beta can affect the loss in the appendix). Please give additional discussions and analysis for this.
- I am also curious whether beta shows any correlation with benchmark difficulty or other task-specific characteristics. (In Appendix D.1.12, the beta changes as the benchmark changes.) Additionally, are there any systematic scaling relationships for β, similar to those potentially observed for learning rates? It would be interesting to analyze whether such relationships exists for beta too.
3 comments
Thank you for the rebuttal and the revisions, including the validation loss, landscape, and beta scaling. I will maintain my score.
We thank the reviewer for considering our revisions and following-up.
We would appreciate clarification on which concerns remain and whether there is anything we could clarify or address before the discussion period ends.
Why validation loss fails, hyperparameter landscape at a fixed size, and correlations of benchmarks with beta
We thank the reviewer for their insightful questions and address them below:
W2 / Q1. Why validation loss fails, and normalising by β or the reference.
When varying only the learning rate, the β and the reference model are fixed, so normalisation does not change how the loss relates to quality within these runs. From 4B up, chat performance peaks before the final checkpoint in 48 of 50 runs while the validation loss keeps falling (Fig. 9). Across runs, the lowest-loss final checkpoint of a size and β has the lowest chat performance in 9 of 10 cases from 4B up. And ranking learning rates by validation loss within each β (App. D.1.2) gives inconsistent correlations with Z_All across sizes.
The same loss can also mean very different quality. We see that at 4B with β ≥ 0.02, larger learning rates can reach the same validation loss, but with 9 to 30 percentage points lower chat performance on Arena-Hard and MT-Bench. We now include these analyses in the paper in App. D.1.3 (Fig. 9, Table 11). We hypothesize that this is due to the model getting better at separating the chosen and rejected responses (DPO loss), while sacrificing chat quality, i.e., overoptimising on the DPO loss.
W1. Landscape at a fixed size.
We counted the local maxima of each final-grid surface [App. D.1.9, Table 15]. The chat landscape (Z_Dynamic) has a single maximum at every size from 1.7B up. The secondary maxima elsewhere exceed their neighbours by less than the 95% uncertainty of a difference between two scores, so every surface is consistent with a single optimum.
Q2. β, benchmark characteristics and scaling.
On Dolci, β = 0.01, the lowest tested value, wins Z_All at every size. It wins Z_Dynamic at every size except 1.7B (β = 0.02). With the winner at the edge of our grid, we cannot fit a β scaling law.
Most benchmarks also prefer β = 0.01 [App. D.1.6, Table 13]. The exceptions are IFEval (a larger β at four of five sizes) and PIQA and GPQA (β = 0.04 at three sizes each).
We did not measure benchmark difficulty directly. GPQA and PIQA vary across the grid by less than one interval width at every size, so their preferred β is within noise. IFEval's preference fits its training dynamics: IFEval declines as the policy drifts from the SFT model, and a larger β limits drift.
The β variation you noted in the appendix is between preference datasets: the Dolci winners use β = 0.01, and the UltraFeedback winners use β = 0.04 from 1.7B up [Sec. 4.2; App. D.1.16, Table 17].