acceptodds
Under review as a conference paper at ICLR 2027

The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

Abstract

Direct preference optimisation (DPO) is widely used for language model post-training, but its performance depends strongly on the choice of its hyperparameters, such as the learning rate and the reference-policy regularisation parameter . To better understand the DPO hyperparameter landscape, we conduct extensive experiments using a dense hyperparameter grid across different model scales from the same family. This allows us to analyse how the optimal DPO hyperparameters evolve with model size and the number of update steps, with intermediate checkpoints providing an inexpensive signal to accelerate the hyperparameter optimisation process. Furthermore, by evaluating our models on both static and dynamic benchmarks, we identify which benchmarks provide the strongest signal for hyperparameter selection. Finally, we investigate how different state-of-the-art hyperparameter optimisation strategies perform in this setting. Our experiments show that the optimal learning rate varies with model size and that hyperparameters are not transferable across training datasets. We also find that validation loss is a poor proxy for hyperparameter optimisation, as hyperparameters that minimize validation loss only weakly correlate with optimal hyperparameters for downstream benchmarks. Therefore, DPO hyperparameters need to be tuned specifically for the target model, dataset, and evaluation objective. We also show that short candidate training runs can substantially reduce the cost of this selection. We release all our training configurations and evaluation results to support reproducible analysis of selection quality and training cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Direct Preference Optimization, preference optimization, hyperparameter transfer, hyperparameter optimization, multi-fidelity optimization, language-model post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kz1fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes about the tendency of hyperparameter of Direct preference optimization (DPO), especially in three aspects: (1) validation loss is poor and unreliable proxy for hyperparameter optimization (2) optimal hyperparameter configuration changes along dataset and scale (3) Partial runs can be a informative proxy scale-dependentably, and adaptive HPO outperforms random search. Based on those analysis, authors provide the practical recommendations for DPO hyperparameter selection, such as select configurations using downstream evaluations, retune for different setting, use partial target-model runs.

Critical strengths
  • This paper proposes diverse analysis for DPO hyperparameter, such as the learning-rate trend shows the downward trend as the model size gets bigger, or similar model size (especially 0.6B-1.7B, 4B-14B) show some similar trend at hyperparameter optimization. Such analysis can be a good guide for DPO hyperparameter selection, selecting lower learning rate for bigger models, or trying to use similar model size's optimal hyperparameter as a starting point.
  • This paper proposes nice practical recommendations for DPO hyperparameter optimization such as use diverse downstream evaluations rather than validation loss, retuning for target setting (scale, dataset), and use partial runs as proxy before whole evaluation. Such recommendations can be a good starting reference for DPO hyperparameter selection for the researchers who are conducting DPO training, and is worth for the research community.
Critical weaknesses
  • Additional analysis of hyperparameter landscape at fixed model scale and benchmark. It seems analysis of hyperparameter landscapes at certain model scale and benchmark are needed, whether there exists some multiple optimal setting (multi-mode landscape), or shows high at certain settings. And additional analysis or interpretation about why certain hyperparameters may have been selected or shows good performance at certain model scale and benchmarks may validate the result of the experiments.
  • More interpretation of validation loss. The authors claimed that validation loss is not informative for DPO hyperparameter optimization, but it seems like current 'validation loss' can be used only as proxy measure only within the same training, not across different training setting. Validation loss of DPO measures the relative probability between current and reference model, so the loss may differ when the reference model is different (e.g., model scale). It seems like some normalization for such reference model, or beta should be done to precisely analyze the correlation between validation loss and the model.
Questions for authors
  • I wonder about the 'reason' of the validation loss was poor proxy in current setting. Validation loss should differ according to model and benchmark type so it seems like some 'normalization' of beta/ should be done to appropriately use the loss as proxy (also I have seen the beta can affect the loss in the appendix). Please give additional discussions and analysis for this.
  • I am also curious whether beta shows any correlation with benchmark difficulty or other task-specific characteristics. (In Appendix D.1.12, the beta changes as the benchmark changes.) Additionally, are there any systematic scaling relationships for β, similar to those potentially observed for learning rates? It would be interesting to analyze whether such relationships exists for beta too.
3 comments
AuthorsOpenReview ↗

Why validation loss fails, hyperparameter landscape at a fixed size, and correlations of benchmarks with beta

We thank the reviewer for their insightful questions and address them below:

W2 / Q1. Why validation loss fails, and normalising by β or the reference.

When varying only the learning rate, the β and the reference model are fixed, so normalisation does not change how the loss relates to quality within these runs. From 4B up, chat performance peaks before the final checkpoint in 48 of 50 runs while the validation loss keeps falling (Fig. 9). Across runs, the lowest-loss final checkpoint of a size and β has the lowest chat performance in 9 of 10 cases from 4B up. And ranking learning rates by validation loss within each β (App. D.1.2) gives inconsistent correlations with Z_All across sizes.

The same loss can also mean very different quality. We see that at 4B with β ≥ 0.02, larger learning rates can reach the same validation loss, but with 9 to 30 percentage points lower chat performance on Arena-Hard and MT-Bench. We now include these analyses in the paper in App. D.1.3 (Fig. 9, Table 11). We hypothesize that this is due to the model getting better at separating the chosen and rejected responses (DPO loss), while sacrificing chat quality, i.e., overoptimising on the DPO loss.


W1. Landscape at a fixed size.

We counted the local maxima of each final-grid surface [App. D.1.9, Table 15]. The chat landscape (Z_Dynamic) has a single maximum at every size from 1.7B up. The secondary maxima elsewhere exceed their neighbours by less than the 95% uncertainty of a difference between two scores, so every surface is consistent with a single optimum.


Q2. β, benchmark characteristics and scaling.

On Dolci, β = 0.01, the lowest tested value, wins Z_All at every size. It wins Z_Dynamic at every size except 1.7B (β = 0.02). With the winner at the edge of our grid, we cannot fit a β scaling law.

Most benchmarks also prefer β = 0.01 [App. D.1.6, Table 13]. The exceptions are IFEval (a larger β at four of five sizes) and PIQA and GPQA (β = 0.04 at three sizes each).

We did not measure benchmark difficulty directly. GPQA and PIQA vary across the grid by less than one interval width at every size, so their preferred β is within noise. IFEval's preference fits its training dynamics: IFEval declines as the policy drifts from the SFT model, and a larger β limits drift.

The β variation you noted in the appendix is between preference datasets: the Dolci winners use β = 0.01, and the UltraFeedback winners use β = 0.04 from 1.7B up [Sec. 4.2; App. D.1.16, Table 17].

Thank you for the rebuttal and the revisions, including the validation loss, landscape, and beta scaling. I will maintain my score.

We thank the reviewer for considering our revisions and following-up.

We would appreciate clarification on which concerns remain and whether there is anything we could clarify or address before the discussion period ends.

Reviewer_dBnAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an empirical study of hyperparameter selection for DPO, focusing on the learning rate and β.

Through experiments across model scales, preference datasets, and evaluation benchmarks, it investigates configuration-selection signals, hyperparameter transfer, and the usefulness of partial training runs.

It reports observations about evaluation-dependent configuration rankings, limitations of direct transfer, and opportunities for early selection.

Its main outputs are these empirical findings and a released collection of checkpoint-level training trajectories, evaluation scores, and training costs that supports an offline benchmark for studying DPO hyperparameter selection.

Critical strengths

1.The experiments cover multiple model sizes and preference datasets, and the released checkpoint-level trajectories provide a resource for offline research on DPO hyperparameter selection. 2.The finding that validation loss is a poor selection signal is clearly supported. 3.The authors are also open about several limits, such as preferred values sitting at grid boundaries.

Critical weaknesses

1.Limited incremental insight over prior work. Several central observations are consistent with existing literature. Rafailov et al. (2024) report little relationship between direct-alignment loss and downstream performance within a model size. β-DPO motivates adapting β to preference-data characteristics, and Tülu 3 reports data-dependent DPO learning-rate choices. The paper characterizes these effects more systematically, but the additional actionable insight remains limited. The fitted learning-rate trend is explicitly descriptive rather than predictive, and the recommendations largely reiterate downstream evaluation and target-specific tuning. The contribution would be clearer if the paper separated confirmations of prior findings from genuinely new guidance supported by its experiments.

2.The practical model-selection value remains unclear. For pipelines that continue from DPO to RL, the paper further leaves untested whether DPO-stage selection signals predict final model quality. These gaps limit the evidence that the reported observations translate into better practical model-selection decisions

Rafael Rafailov et al. (2024), Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms.

Questions for authors

See Weakness 2.

4 comments
AuthorsOpenReview ↗

Weakness 1: Insights over prior work and practical recommendations

We thank the reviewer for their valuable feedback. We address each of the reviewer's concerns below:

W1. Insights over prior work

Sec. 2 now separates confirmations from new findings. These are summarised below:

What we confirm from prior work, and how we extend it:

  • Validation loss is a poor selection signal (confirms Rafailov et al., 2024)
    • We extend this with larger and newer models, general chat preference data instead of text-summarisation data, a joint sweep of learning rate and beta instead of beta alone, and 11 benchmarks instead of one GPT-4 win rate. We also quantify the cost of selecting configurations by validation loss.
  • Preferred configurations depend on the training data (confirms Tülu 3 and Beta-DPO)
    • We additionally measure the transfer regret and address several limitations listed by Beta-DPO (see below)

New contributions:

  • Chat benchmarks rank configurations more consistently than static ones (tested with 2 judges)
  • Partial runs rank configurations well after about 30% of training (to our knowledge, this is new for DPO)
  • From 4B up, choosing for chat quality maintains static performance, while choosing for static benchmarks costs chat quality (between 11-20 percentage points on Arena-hard)
  • We release an offline benchmark of checkpoint-trajectories and evaluation scores.

More details on the comparison to Tülu 3:
Tülu3 lowered the DPO learning rate at 70B after a small check, kept β fixed, and selected on one suite average with one LLM-judged benchmark.

  • We vary learning rate and β together on models up to 14B, use 11 benchmarks (4 of which were tested with two LLM judges) and separately judge models on them based on chat-performance and static evaluation capability

More details on the comparison to β-DPO:
The paper adapts β at a fixed learning rate, whereas we jointly vary the learning rate and β. We also test β values below their smallest tested value of 0.1.

Additionally, our study covers some limitations listed in their paper:

  • “Enhanced Evaluation Standards….. to comprehensively gauge model performance.” - They only report GPT-4 win rates, whereas we evaluate our models on 11 benchmarks (4 judged by LLMs, and 7 static)
  • “Automated Parameter Tuning… Pursuing automation in parameter tuning… broaden accessibility.” - We systematically test and evaluate multiple HPO methods and their benefits.

Additionally, their finding that the best β depends on the data fits ours: β = 0.01 wins on Dolci at every size, and the UltraFeedback winners use β = 0.04 from 1.7B up [Sec. 4.2; App. D.1.16, Table 17].

For more details on comparison to Rafailov et al., please see our reply to reviewer Rw3K (W2) or refer to the point above on Validation Loss


W1. Practical Recommendations

Each recommendation in Sec. 5 now also states its empirical evidence:

  • 1. Select on chat benchmarks (Also partly addressing the practical model-selection value as mentioned in W2). Recipes commonly select DPO configurations on an average over largely static benchmarks (Tülu 3, Olmo 3, the HF Smol Training Playbook). From 4B up, choosing for chat keeps static performance, while choosing for static benchmarks loses 11 to 20 percentage points on Arena-Hard. Selecting the checkpoint with the lowest validation loss, as the Hugging Face Trainer used by TRL does with `load_best_model_at_end`, loses a median of 110 Elo from 4B up (if no other evaluation metric is manually set).
  • 2. Retune for the target setting. Copying across the size-group boundary chose a near-worst configuration (regret 0.91), and changing the preference dataset also gave high regret.
  • 3. Use partial runs to narrow the candidates. After about 30% of training, rankings correlate well with the final rankings. Additionally, under sufficient budget, automated HPO methods like CQR improve on random search under all four tested surrogates.

References
- Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. https://arxiv.org/abs/2411.15124
- Team Olmo. Olmo 3. https://arxiv.org/abs/2512.13961
- Ben Allal et al. The Smol Training Playbook: The Secrets to Building World-Class LLMs. Hugging Face, 2025. https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook

AuthorsOpenReview ↗

Weakness 2: DPO followed by RL

W2. DPO followed by RL

We ran a small-scale test [Sec. 6; App. D.1.17]. Five Qwen3-4B DPO models spread over the Z_Dynamic ranking each received the same short GRPO stage on GSM8K-train (about one epoch). The scores before and after RL are in the table of that section. The five models keep their order through RL on GSM8K-test, GSM-Plus, Arena-Hard and Elo, and the best configuration selected after the DPO stage stays the best on all four chat benchmarks (Arena-Hard, AlpacaEval, MT-Bench, and Elo score) after this short-RL stage.

Adequately studying the connection between DPO and RL would require testing different RL algorithms on top of each finished DPO run. This would increase the amount of training runs significantly and cost a lot of additional compute. We acknowledge this as a limitation of our paper and leave it for future work.

Thank you for the clarifications and additional results. I will keep my score.

We thank the reviewer for their follow-up and for acknowledging our revisions.

We would appreciate some clarification regarding any remaining concerns and how we could address them.

Reviewer_Rw3KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to select the DPO learning rate and β\beta. The authors start from base Qwen3 models, run SFT on Dolci-Instruct-SFT, and then DPO grids over learning rate and β\beta, with one seed each. The authors score intermediate and final checkpoints on four LLM-judged chat benchmarks, seven static benchmarks, and held-out DPO loss. From rank correlations and normalized regret, the authors conclude that:

  • validation loss is a poor selection signal;
  • chat benchmarks rank configurations more consistently than static ones, and are robust to the choice of judge above 0.6B;
  • configurations chosen at one model size or on one preference dataset often rank poorly at another;
  • rankings at about 30% of training already track the final ones.

The checkpoints are released.

Critical strengths
  1. A controlled, released record of DPO tuning.
    • One recipe is swept over five Qwen3 sizes, three Llama models, and two preference datasets.
    • Intermediate checkpoints (every ~10% of training) are scored on eleven benchmarks with two judges, and released. This is very valuable for the community.

These checkpoints are rich enough to test new selection or transfer rules offline.

  1. The message of the paper is important for the community: validation loss is a poor selection signal; configurations chosen at one model size or on one preference dataset often rank poorly at another; these are interesting conclusions that open the door to several research projects.
Critical weaknesses
  1. The transfer experiments show that copying a configuration fails, not that transfer fails.

    • Hyperparameter transfer goes beyond simple copying across scale; see e.g. Mu-P.
    • Here, only copying is tested. Transfer across sizes means copying the same absolute (LR,β)(\text{LR}, \beta) within the fixed 3×33\times3 grid.

    This matters because "when not to transfer" is the paper's headline: the title, contribution 2, and Sec. 7, which says the results "argue against a universal configuration or simple transfer rule". That copying fails is expected: Tülu 3 (cited; Tables 19–20) already lowered the DPO learning rate for its 70B model and found that the optimum depended on the preference mix. My understanding is that the paper currently does not show that there is no simple rule for transfers.

  2. Claim 1 (validation loss) is not new.

    • The finding was reported before. Rafailov et al. (2024, https://arxiv.org/abs/2406.02900, Sec. 3.4 and Fig. 4) found, across seven β\beta values and three Pythia sizes, that DPO loss and implicit-reward accuracy bear little relation to win rate, which aligns with this paper's findings. There is still a lot of value in the present paper thanks to the more modern models studied, but this work should be credited.
Questions for authors
  • Can you try more involved transfer rules than copying from one scale to the other?
  • Adding error bars to numbers like in fig.1 would greatly enhance the paper, and give us a sense of how much signal there is within these numbers.
3 comments
AuthorsOpenReview ↗

Transfer rules beyond copying, relation to Rafailov et al. (2024), and Error Bars

We thank the reviewer for their positive review and concrete suggestions. We address them below:

W1 / Q1. Transfer rules beyond copying

We agree with the reviewer and now make a refined claim in Contribution 2, Sec. 4.3 and the conclusion. μTransfer was developed and validated for pretraining and requires the model to be parametrised with μP from initialisation. To our knowledge, no μP-style transfer method has been validated for DPO. We therefore test empirical transfer rules that work with our checkpoints [Sec. 2].

We now compare three ways of choosing a configuration in a leave-one-size-out test over the five Qwen3 sizes on Dolci, against a random choice [App. D.5.3, Table 26].

  • Copying configuration from the nearest size
  • Copying configuration only from the nearest size within post-hoc identified groups (0.6B-1.7B; 4B-14B)
  • A power law fitted on the other sizes

Each picks one configuration from the other sizes and is scored by normalised regret on the held-out size's grid. For chat quality, copying within certain size groups works best. For static scores, none of the three is reliable.

Regarding the comparison with Tülu 3: they tuned at 8B and lowered the learning rate at 70B. We test five sizes from 0.6B–14B under four evaluation aggregates instead of one, and show that the preferred learning rate falls with size (Fig. 4, App. D.1.4).


W2. Rafailov et al. (2024).

We now cite it and compare our work with the paper by Rafailov et al. (2024) in Sec. 2. Our finding confirms theirs and extends it in the following ways:

  • Larger and newer models: Qwen3 at five sizes (0.6B to 14B) and Llama at 1B, 3B and 8B, against Pythia 1B to 6.9B.
  • General chat preference data (Dolci, UltraFeedback), against Reddit TL;DR summarisation data.
  • Joint exploration of learning rate and β, against varied β values at a fixed learning rate.
  • 11 benchmarks with two LLM judges, against one GPT-4 win rate.
  • We quantify the cost of selecting via validation loss: from 4B up, the checkpoint with the lowest validation loss loses a median of 110 Elo against the best checkpoint, and it is the worst of four rules in all nine held-out cases [App. D.1.3, Table 11].

Q2. Error bars

Figure 1 and every final score now reports 95% intervals [Fig. 1; App. D.1.13, Table 16].

The main effects are large against these intervals. For example: in the 4B subgrid of Figure 1, the best and worst configurations differ by 294.6 Elo points (95% interval 227.5 to 344.8).

The weak static agreement in Sec. 4.1 comes mainly from benchmarks disagreeing with one another because they prefer different configurations. IFEval, for example, prefers a larger β than most benchmarks [App. D.1.6, Table 13].

Dear authors,

Thanks for running the additional experiments. I maintain my initial score.

Thank you for considering our response and the additional experiments.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised PDF

We have made changes to the submitted PDF to address the reviewers’ comments. We have also added an overview at the start of the appendix to group its sections for better readability. All of these changes are highlighted in text colour blue in the PDF.

AuthorsOpenReview ↗

General Response

We would like to thank all the reviewers for their important feedback and thoughtful comments.

We appreciate that the reviewers find our released data “very valuable for the community” and "rich enough to test new selection or transfer rules offline”.

We are also pleased to see that they note that “the finding that validation loss is a poor selection signal is clearly supported”, and see the analysis as “a good guide” or “a good starting reference” for DPO hyperparameter selection due to the paper proposing "nice practical recommendations for DPO hyperparameter optimization", with “interesting conclusions that open the door to several research projects".


Summary of Changes

As mentioned previously, we have revised the PDF to address the reviewers’ comments and added an overview at the start of the appendix to group its sections for better readability. All of these changes are highlighted in blue in the PDF and we also summarise them below:

  • 1. Prior work. (as pointed out by reviewers Rw3K, dBnA)
    We now cite Rafailov et al. (2024) in our related work section (Sec. 2). We also include a comparison of our study with Rafailov et al. (2024), β-DPO and Tülu 3 to clearly distinguish our work from prior findings.

  • 2. Comparing transfer rules across model sizes. (Rw3K)
    We now additionally include a leave-one-size-out test comparing copying configurations, copying within a group of sizes, and a power law. We have also narrowed our claim on transferring configurations between models. It now states that copying a configuration from a nearby size works only for chat quality within post-hoc identified groups of sizes (0.6B-1.7B and 4B-14B). It does not work well across these groups, for different training datasets, or for static benchmarks. These are present in Sec. 4.3 and App. D.5.3 (Table 26).

  • 3. Practical Model-selection. (dBnA)
    Each practical recommendation made in Sec. 5 now reports a measured result taken from our experiments. We now explicitly state the cost for choosing configurations based on chat or static performance in App. D.1.7 (Table 14). From 4B up, choosing for chat keeps static performance but gains 11 to 20 percentage points in Arena-Hard over choosing for static benchmarks and choosing by validation loss is the worst of four selection rules [added to App. D.1.3; Fig. 9, Table 11]. We would like to emphasize that our paper explicitly studies the DPO hyperparameter landscape and only provides practical recommendations for DPO hyperparameter selection based on our findings.

  • 4. Validation loss as a selection signal. (kz1f, Rw3K)
    We show that chat quality peaks while validation loss keeps falling, which is one reason it picks poor configurations, and that selecting by lowest validation loss is the worst of four selection rules. The same loss can also lead to models of very different quality. We see that at 4B with β ≥ 0.02, larger learning rates can reach the same validation loss, but with 9 to 30 percentage points lower chat performance on Arena-Hard and MT-Bench. We now include these analyses in the paper in App. D.1.3 (Fig. 9, Table 11).

  • 5. Uncertainty in benchmark scores. (Rw3K)
    We now report the 95% intervals for Figure 1 in the main text and for every benchmark score in App. D.1.13 (Table 16).

  • 6. Local maxima on the tested grid. (kz1f).
    We now report the local optima of each surface (App. D.1.9, Table 15) and the preferred β of each benchmark in App. D.1.6, Table 13. In general, we found the landscape to be unimodal, which is also typical of other hyperparameter optimization tasks in the literature [Lourie et al. (2025)]

  • 7. DPO followed by RL. (dBnA)
    Connecting DPO with RL is an important research direction. In a small-scale test at 4B, the DPO-stage ranking held on the chat benchmarks after a short GRPO stage, and the configuration selected at the DPO stage stayed best on all four [added to Sec. 6; App. D.1.17, Table 19]. Properly evaluating the predictive relationship between DPO and post-RL performance is a distinct research direction concerning multi-stage post training pipelines that requires a significant amount of compute. We therefore leave an extensive evaluation of this for future work and note this as a limitation of our work.

  • 8. Released Dataset
    The reviewers noted the value of our released checkpoint trajectories and scores, and we used them to answer various reviewer questions (items 2 to 4 and 6) with no new training.


References:

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.