- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper compares continued chosen-only supervised finetuning against direct preference optimization for a number of open LLM models in the 4B-8B range. Through a curated selected of finetuning dataset with specific ratios of stage 1 SFT prediction alignment against a preference dataset, the effect on corrected, preserved and corrupted samples are observed. The results indicate that DPO shows more robust behavior in terms of less corruption of stage 1 correct results when the stage 2 dataset contains a higher ratio of samples that the initial SFT checkpoint gets wrong.
Critical strengths
The question of when and why CSFT can match DPO is relevant. The evaluation protocol of clearly separating along the three axes of corrected, preserved and corrupted while varying the stage 2 dataset mass is informative and gives insight into the mechanisms of DPO and CSFT.
The evaluation is robust and includes a number of different frontier architectures that shows that the conclusions are model agnostic.
Critical weaknesses
If i understand correctly, all stage 2 training is done using LoRA. This opens the question of whether these results has any bearing on full model fine tuning. This should be clearly stated in the claims in the manuscript that currently only has a mention in passing in section 3.1.
The manuscript, although easy to follow, fails to coherently tie together the main points through abstract, contributions, results and conclusions. It is hard to verify the exact statements of e.g. the contributions against the bulk material.
To the best of my reading of the supplied code, the CSFT stage 2 training is using the full context as training target, whereas DPO only trains on the answer part. I couldn’t find a clear mention of this in the manuscript. This seems to make it difficult to separate the effect of CSFT vs DPO from the change in training protocol.
Questions for authors
Can the authors clarify if CSFT is using unmasked training on the context, and if so, why the results should be comparable to DPO?
“Evidence that DPO’s additional correction in natural ranking settings is often partly offset by preservation loss, which aggregate accuracy conceals.” (“43185_When_Does_DPO_Add_Value_”, p. 2) I find it hard to trace this contribution statement to the results. It would be helpful for the reader if the manuscript clearly connects the results, discussion and conclusion to the chosen formulations in the contribution points.
In particular "... often partly offset by preservation loss" is hard to connect to any specific quantitative result. In the bulk it is often referred to that the CSFT shows preservation loss under an increased mass. Is this what the statement refers to?
“Unless varied explicitly, all stages use LoRA” (“43185_When_Does_DPO_Add_Value_”, p. 3) Do you expect any transfer of these results to full model SFT? And how do you view the relevance of these results as a general CSFT vs DPO comparison?
“mi(θ)” (“43185_When_Does_DPO_Add_Value_”, p. 4) Why has notation switched between subscript and argument compared to section 3.1?
“An exploratory location study found the clearest late frontier separation in the intermediate-margin stratum, which was then fixed as the focal population; the exploratory comparison is not pooled with the intervention.” (“43185_When_Does_DPO_Add_Value_”, p. 4) This is opaque.What is "exploratory location study", "late frontier separation", "intermediate-margin stratum"?
“Quantitative claims draw on the experimental results” (“43185_When_Does_DPO_Add_Value_”, p. 10) "draw on"? Is there any reason to believe that the quantitative claims do not strictly match the experimental results?
“The 2, 5, 10, and 20% focal sets are nested prefixes of this master after stratification by margin and length tertiles;” (“43185_When_Does_DPO_Add_Value_”, p. 17) What margin and length tertiles?
“covariate-only pooled-centroid rule was declared and then applied” (“43185_When_Does_DPO_Add_Value_”, p. 17) what is the rule?