acceptodds
Under review as a conference paper at ICLR 2027

DPO's Recovery Advantage Depends on Answer Presentation

Abstract

Direct preference optimization (DPO) can restore answers lost during supervised fine-tuning (SFT), but option order affects which failures are selected and how recovery is measured. We test whether DPO’s advantage over supervised continuation persists under a different option order. On MMLU-Pro, we select questions that Base answers correctly and SFT answers incorrectly, then evaluate unchanged SFT and both continuations on the same questions under both orders. Reordering alone lets unchanged SFT answer 29.6–34.1% of the selected failures correctly. On questions selected under the original order, DPO’s advantage falls from 8.72 to 2.58 percentage points for Qwen3-8B and from 9.53 to 1.46 for OLMo-2-7B. Gains in its relative advantage on the remaining questions conceal this decrease in the full-test comparison. Reversing the screening and evaluation orders gives the same pattern. Within Qwen3-8B’s recovery pools, DPO also changes more SFT answers, but a smaller fraction of those changes is correct. Recovery evaluation should therefore retest fixed questions across presentations and report how often each continuation changes an answer, alongside how often those changes succeed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Direct preference optimization, Post-training evaluation, Capability recovery, Multiple-choice evaluation, Option-order sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tEfhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how answer presentation affects the measured recovery of answers lost during supervised fine-tuning. For Qwen3-8B and OLMo-2-7B, the authors compare DPO with chosen-response supervised continuation from the same SFT checkpoint. They select MMLU-Pro questions answered correctly by Base and incorrectly by SFT under one option order, then evaluate the same questions and checkpoints under both orders. DPO retains an advantage, but that advantage decreases outside the screening presentation, while full-test comparisons can conceal the decrease. Further analyses distinguish answer-change frequency from successful changes, cross content positions with letter assignments on MMLU-Pro and HellaSwag, and examine shared readout corrections and final-token attention-parameter interventions.

Critical strengths
  1. A well-controlled and consequential evaluation finding. Crossing screening and evaluation order on fixed questions separates the definition of a recovery candidate from its subsequent measurement. In Table 1, DPO's advantage on original-screen candidates decreases from 8.72 to 2.58 percentage points for Qwen3-8B and from 9.53 to 1.46 for OLMo-2-7B. Reversing the screening order gives the same qualitative pattern. Figure 2 then demonstrates why full-test accuracy alone can miss this effect: changes outside the recovery pool offset the decrease within it. This identifies a specific issue in evaluating post-training recovery beyond the already known sensitivity of multiple-choice accuracy to option order.

  2. Several complementary controls strengthen the interpretation. Both continuations share their SFT initialization and use the same retained preference examples, with the supervised control using chosen responses. The matched-display comparison in Section 5 distinguishes content-to-letter mapping and position effects from the displayed letter sequence alone and extends to HellaSwag. Bidirectional parameter exchange and the fixed-aggregate replay in Section 6 provide evidence that the presentation response can change through attention output projections under the specified interventions. The explicit decomposition of corrections, losses, and answer changes also makes the recovery results more informative than a single accuracy score.

Critical weaknesses
  1. The central recovery comparison still has a limited range of readouts and presentations. The full screening-by-evaluation crossing uses MMLU-Pro with next-token letter selection and one alternative permutation per question. The HellaSwag paired-swap analysis is useful additional evidence, but does not repeat the complete crossing with reverse screening. Likewise, the label-free diagnostic tests content-choice changes under parameter exchange rather than the full Base/SFT/DPO/supervised recovery chain. It therefore remains unclear how strongly the measured attenuation persists under generated-answer or candidate-content scoring. This limits the breadth of the evaluation recommendation, although it does not undermine the reported letter-readout result.

  2. The connection between answer-change frequency and recovery advantage remains descriptive. Section 4's step-312 DPO checkpoint has nearly the same overall change count as supervised continuation and a much smaller recovery advantage. However, Appendix S.1 lacks the candidate-selection history, the supervised model's validation change count, and change counts within the recovery pools. The compared checkpoints also differ in training duration and which questions change. The score-scaling analyses provide useful corroborating diagnostics, but modify the readout rather than produce a matched training comparison. The paper acknowledges these distinctions; the remaining gap limits how much this analysis explains the origin of DPO's advantage.

Questions for authors
  1. Does the screening-by-evaluation attenuation persist when the full checkpoint chain is scored by candidate-content likelihood or generated answers, using the same readout consistently for screening and evaluation? A complete HellaSwag crossing with reverse screening would also help establish how broadly the effect holds.

  2. Can the step-312 selection record and its change counts within each fixed recovery pool be provided? In particular, are the DPO and supervised checkpoints still comparable in change frequency within those pools, rather than only over the full test set?

  3. Appendix P's earlier-correctness-state predictor approximates the pooled attenuation but poorly captures category-level variation. Which residual behavior do the parameter-exchange results explain beyond this state-mixture description, and can that connection be made explicit?

Reviewer_eFYDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors investigated the well-known problem of option order bias. They decided to examine the connection between this problem and recent works claiming that DPO models recall facts forgotten during the SFT stage. They conducted a series of experiments on Qwen3-8B and OLMo-2-7B models. By measuring the performance gap between DPO and supervised continuation on shuffled answers, they found that this gap shrinks significantly.

They performed a deep analysis using different shuffling variants (e.g., separating text movement from letter reassignment). They also conducted a parameter exchange experiment with swapping the last token's attention matrices. This was done to verify whether actual knowledge was added to the model, or if this effect is merely tied to the output format and letter-binding bias.

At the same time, the study was conducted using a single random seed and only on small-scale models (up to 8B parameters), and the main results heavily rely on this specific setup. The authors explicitly acknowledge this in the limitations section.

As a conclusion, the authors questioned the assumption that DPO restores knowledge compared to SFT, which forgets it. They provided theory and wide experimental results that contradict this assumption, showing that the recovery is largely driven by formatting artifacts and higher answer-changing frequency.

Critical strengths
  1. Closed the gap and connected two research areas: the dependence of results and models' sensitivity to option order in benchmarks, and the works dedicated to DPO's advantages in recovering and improving performance on these benchmarks.
  2. Extensive and broad experimental design with various comprehensive comparisons, including: comparison against a proper baseline with supervised continuation, separation of screening and evaluation orders, and checking the interpretability of the results with parameter exchange experiments on attention matrices.
  3. The mathematical derivations and the theory for decoupling position bias and letter assignment bias, along with the calibration mathematics, are consistent with the results.
Critical weaknesses

As stated in the Limitations:

  1. Only one seed was used. This is insufficient to draw firm conclusions, and the results may lack statistical robustness across training runs.
  2. Only families of small models were used. The paper does not compare larger models, for example, in range 14-32B.
  3. Taking an intermediate point in DPO (step 312) is mathematically not rigorous, because it acts as a post-hoc selection. To make this claim valid, the authors should either evaluate checkpoints at fixed intervals to show the trend, or select the checkpoint based on a strictly disjoint validation set.

Other weaknesses: 4. The core screening-by-evaluation cross-order experiment (Table 1) was done exclusively on MMLU-Pro. The paper uses too few benchmarks to support the analysis; you could add more (e.g., HellaSwag, which you are using later). 5. Calibration and its related experiments were conducted on very small sample sizes (42 or 70 samples), which could heavily influence the results, especially given the single seed. 6. The tests on the ARC-Challenge dataset in Appendix K were done on the old D8 checkpoints instead of the corrected D8R ones and should be redone. 7. The text is overloaded with metrics and notation, making it hard to follow.

Questions for authors
  1. Does this effect persist on larger models in range 14-32B?
  2. Is it possible to add more seeds to verify the statistical significance of the results?
  3. Is it possible to conduct testing on a broader random sample of permutations? It would be interesting to see how the results compare in this setting. Even in the current setting, you can swap different answers in various ways.
  4. Could you add more benchmarks, at least ARC-Challenge, which was run on a different (older) checkpoint in the appendix? Additionally, you can add, for example, CommonsenseQA and TruthfulQA.
  5. Using the provided results, can you formulate a guideline for properly testing DPO and other methods on such benchmarks?
  6. Discussed option order bias is known to be highly sensitive to the exact prompt template. Have you tested whether the reduction in the DPO-SFT gap persists under different prompt templates?
Reviewer_4LmuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates DPO's advantage over continued SFT in recovering previously correct answers ("recovering" means: base model answers correctly -> SFT answers wrongly -> DPO / continued SFT answers correctly). The authors select questions that base model answers correctly, but SFT fails, and evaluate 3 settings: (1) initial SFT, (2) initial SFT + DPO, (3) intial SFT + continued SFT trained on chosen responses in the preference dataset.

Results show that DPO retains a recovery advantage, but this advantage decreases significantly when the evaluation order is different from the original order. This reduction is not observed when using the full evaluation set (not just the ones that base model is correct but SFT answers wrongly). The paper further did anlysis on: answer-change frequency v.s. success rate, sensitivity to option positions and mappings, and whether parameter exchange at the final input token will affect model's behavior.

The paper's main contribution seems to be an empirical analysis of how DPO recovering capacity is dependant on presentation of the input. The findings might be beneficial for certain benchmark configures.

Critical strengths

None

Critical weaknesses

Scope of the work is much limited.

While the paper studies a useful question (DPO's recovering capacity under different settings), the work lacks sufficient width and depth to make a practically useful impact. For example, some results are presented, but it is unclear how we should change our understanding, development, and/or evaluation of DPO based on these results. The experiments are also only conducted on two datasets, two model families, and a single training seed, making its generalizability and scope much constrained.

Questions for authors

The authors recommend evaluating the same questions under different presentations. How exactly should this be done? Have you tested whether this leads to more reliable assessments of DPO's recovery capacity? How much improvements does it provide?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.