acceptodds
Withdrawn from ICLR 2027

When Does DPO Add Value over Continued SFT? Evidence from Correction-Preservation Frontiers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

After supervised fine-tuning (SFT), when does Direct Preference Optimization (DPO) add value over chosen-only continued SFT (CSFT)? Starting from shared SFT checkpoints, we track correction of inherited errors and preservation of initially correct decisions. Across two models and two ranking datasets, controlled increases in training disagreement widen the DPO-CSFT accuracy gap by 2.62–4.08 percentage points, mainly through CSFT decline rather than increased DPO gain over SFT. DPO incurs less corruption over overlapping correction ranges; the tested CSFT schedule and KL variants on MS MARCO do not recover its high-correction, low-corruption region. In naturally sampled ESCI and TL;DR, DPO instead gains mainly through additional correction despite greater corruption. These contrasting results show how DPO's relative value shifts between correction and preservation across training conditions. Exploratory analysis suggests that high training disagreement combined with a small SFT-error pool can signal stringent preservation requirements before further training; predicting realized gains remains open.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Direct Preference Optimization, preference optimization, supervised fine-tuning, error correction, decision-level evaluation, learning dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_vpCqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares continued chosen-only supervised finetuning against direct preference optimization for a number of open LLM models in the 4B-8B range. Through a curated selected of finetuning dataset with specific ratios of stage 1 SFT prediction alignment against a preference dataset, the effect on corrected, preserved and corrupted samples are observed. The results indicate that DPO shows more robust behavior in terms of less corruption of stage 1 correct results when the stage 2 dataset contains a higher ratio of samples that the initial SFT checkpoint gets wrong.

Critical strengths

The question of when and why CSFT can match DPO is relevant. The evaluation protocol of clearly separating along the three axes of corrected, preserved and corrupted while varying the stage 2 dataset mass is informative and gives insight into the mechanisms of DPO and CSFT.

The evaluation is robust and includes a number of different frontier architectures that shows that the conclusions are model agnostic.

Critical weaknesses

If i understand correctly, all stage 2 training is done using LoRA. This opens the question of whether these results has any bearing on full model fine tuning. This should be clearly stated in the claims in the manuscript that currently only has a mention in passing in section 3.1.

The manuscript, although easy to follow, fails to coherently tie together the main points through abstract, contributions, results and conclusions. It is hard to verify the exact statements of e.g. the contributions against the bulk material.

To the best of my reading of the supplied code, the CSFT stage 2 training is using the full context as training target, whereas DPO only trains on the answer part. I couldn’t find a clear mention of this in the manuscript. This seems to make it difficult to separate the effect of CSFT vs DPO from the change in training protocol.

Questions for authors

Can the authors clarify if CSFT is using unmasked training on the context, and if so, why the results should be comparable to DPO?

“Evidence that DPO’s additional correction in natural ranking settings is often partly offset by preservation loss, which aggregate accuracy conceals.” (“43185_When_Does_DPO_Add_Value_”, p. 2) I find it hard to trace this contribution statement to the results. It would be helpful for the reader if the manuscript clearly connects the results, discussion and conclusion to the chosen formulations in the contribution points.
In particular "... often partly offset by preservation loss" is hard to connect to any specific quantitative result. In the bulk it is often referred to that the CSFT shows preservation loss under an increased mass. Is this what the statement refers to?

“Unless varied explicitly, all stages use LoRA” (“43185_When_Does_DPO_Add_Value_”, p. 3) Do you expect any transfer of these results to full model SFT? And how do you view the relevance of these results as a general CSFT vs DPO comparison?

“mi(θ)” (“43185_When_Does_DPO_Add_Value_”, p. 4) Why has notation switched between subscript and argument compared to section 3.1?

“An exploratory location study found the clearest late frontier separation in the intermediate-margin stratum, which was then fixed as the focal population; the exploratory comparison is not pooled with the intervention.” (“43185_When_Does_DPO_Add_Value_”, p. 4) This is opaque.What is  "exploratory location study", "late frontier separation", "intermediate-margin stratum"?

“Quantitative claims draw on the experimental results” (“43185_When_Does_DPO_Add_Value_”, p. 10) "draw on"? Is there any reason to believe that the quantitative claims do not strictly match the experimental results?

“The 2, 5, 10, and 20% focal sets are nested prefixes of this master after stratification by margin and length tertiles;” (“43185_When_Does_DPO_Add_Value_”, p. 17) What margin and length tertiles?

“covariate-only pooled-centroid rule was declared and then applied” (“43185_When_Does_DPO_Add_Value_”, p. 17) what is the rule?

Reviewer_J3d4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper compares DPO with continued SFT by tracking which existing errors they fix and which correct decisions they break. It looks at datasets constructed with different levels of disagreement with the base SFT model (low levels of disagreement mean the CSFT model would not need to move much). They show that DPO has a growing advantage with higher levels of disagreement, but this advantage is mainly because CSFT damages more correct decisions.

Critical strengths

The disentanglement of correction and corruption gives a better idea of the behavior of DPO compared to CSFT. Figure 2 especially shows that the gap between the two largely comes from CSFT being unable preserve behavior on samples that were originally correct, rather than DPO learning new things.

Critical weaknesses
  • The writing needs a lot of work. It currently reads very LLM generated and is difficult to read. There's a lot of technical report-like language that is hard to follow. This is a major issue, and I think on its own is significant enough to not accept the paper.
  • The design of Figure 1b almost implies that DPO cannot do better than CSFT when preserving, which we later see is not the case. The two bars should not be directly opposed to each other like that.
  • In the shown experiments, it doesn't seem like there is a big change in overall accuracy for DPO compared to the SFT checkpoint for any setup. Even in the controlled mass experiments, while there is a lot of change in correction, the overall accuracy doesn't move. It would be useful to see the dynamics in a setting where there is a big change from SFT.
  • The correction-preservation framing has close precedents. Chen et al. already examine correct-to-incorrect and incorrect-to-correct preference transitions, and FlipGuard directly addresses regressions during preference alignment. The paper needs to explain its contribution relative to these more clearly.
Questions for authors
  • Do you have a potential explanation for why the frontier shape is task dependent? As in what properties of the tasks explain the different correction-corruption frontiers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.