- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how answer presentation affects the measured recovery of answers lost during supervised fine-tuning. For Qwen3-8B and OLMo-2-7B, the authors compare DPO with chosen-response supervised continuation from the same SFT checkpoint. They select MMLU-Pro questions answered correctly by Base and incorrectly by SFT under one option order, then evaluate the same questions and checkpoints under both orders. DPO retains an advantage, but that advantage decreases outside the screening presentation, while full-test comparisons can conceal the decrease. Further analyses distinguish answer-change frequency from successful changes, cross content positions with letter assignments on MMLU-Pro and HellaSwag, and examine shared readout corrections and final-token attention-parameter interventions.
Critical strengths
-
A well-controlled and consequential evaluation finding. Crossing screening and evaluation order on fixed questions separates the definition of a recovery candidate from its subsequent measurement. In Table 1, DPO's advantage on original-screen candidates decreases from 8.72 to 2.58 percentage points for Qwen3-8B and from 9.53 to 1.46 for OLMo-2-7B. Reversing the screening order gives the same qualitative pattern. Figure 2 then demonstrates why full-test accuracy alone can miss this effect: changes outside the recovery pool offset the decrease within it. This identifies a specific issue in evaluating post-training recovery beyond the already known sensitivity of multiple-choice accuracy to option order.
-
Several complementary controls strengthen the interpretation. Both continuations share their SFT initialization and use the same retained preference examples, with the supervised control using chosen responses. The matched-display comparison in Section 5 distinguishes content-to-letter mapping and position effects from the displayed letter sequence alone and extends to HellaSwag. Bidirectional parameter exchange and the fixed-aggregate replay in Section 6 provide evidence that the presentation response can change through attention output projections under the specified interventions. The explicit decomposition of corrections, losses, and answer changes also makes the recovery results more informative than a single accuracy score.
Critical weaknesses
-
The central recovery comparison still has a limited range of readouts and presentations. The full screening-by-evaluation crossing uses MMLU-Pro with next-token letter selection and one alternative permutation per question. The HellaSwag paired-swap analysis is useful additional evidence, but does not repeat the complete crossing with reverse screening. Likewise, the label-free diagnostic tests content-choice changes under parameter exchange rather than the full Base/SFT/DPO/supervised recovery chain. It therefore remains unclear how strongly the measured attenuation persists under generated-answer or candidate-content scoring. This limits the breadth of the evaluation recommendation, although it does not undermine the reported letter-readout result.
-
The connection between answer-change frequency and recovery advantage remains descriptive. Section 4's step-312 DPO checkpoint has nearly the same overall change count as supervised continuation and a much smaller recovery advantage. However, Appendix S.1 lacks the candidate-selection history, the supervised model's validation change count, and change counts within the recovery pools. The compared checkpoints also differ in training duration and which questions change. The score-scaling analyses provide useful corroborating diagnostics, but modify the readout rather than produce a matched training comparison. The paper acknowledges these distinctions; the remaining gap limits how much this analysis explains the origin of DPO's advantage.
Questions for authors
-
Does the screening-by-evaluation attenuation persist when the full checkpoint chain is scored by candidate-content likelihood or generated answers, using the same readout consistently for screening and evaluation? A complete HellaSwag crossing with reverse screening would also help establish how broadly the effect holds.
-
Can the step-312 selection record and its change counts within each fixed recovery pool be provided? In particular, are the DPO and supervised checkpoints still comparable in change frequency within those pools, rather than only over the full test set?
-
Appendix P's earlier-correctness-state predictor approximates the pooled attenuation but poorly captures category-level variation. Which residual behavior do the parameter-exchange results explain beyond this state-mixture description, and can that connection be made explicit?