- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper learns the use of model-generated "rejected answers" to construct DPO data for knowledge updating. The authors observe that many rejected answers are factually correct but differ in expression from the chosen gold answers. They propose style-debiased DPO (SD-DPO), which scores whether the rejected response of each pair is factually correct, inverts the preference of such pairs, and weights them so that the learning signal due to differences in style cancels out as a whole.
Critical strengths
The motivation behind the issue is clear and universal.
The method attempts to separate facts from style in a structured manner.
The baseline and mechanism ablation studies are extensive.
Critical weaknesses
-
The theory covers only idealized initial gradients and does not fully explain the actual SD-DPO loss. The main motivation is about decompose the margin into and . But the actual loss have confidence weighting, group rescaling, knowledge-preservation hinge... Those add-on looks like engineering parts and not discussed in theory part. The performance cannot be explained by the motivation and theory.
-
The baselines are not convincing. In Appendix, the length of Llama output increases from 6 to 272 after using SD-DPO. If accuracy is calculated based on whether the ground truth answer appears in the response, then it is very unfair to the baseline. Also, all baselines were tested using the same learning rate, and no hyperparameter experiments regarding the learning rate were conducted. Authors can not demonstrate that this learning rate was not the result of cherry-picking. Although the authors consider this issue a topic for future work, a fair comparison against baselines is a fundamental requirement.
-
Although the authors promised to release the code, they did not provide an anonymous link; however, there are currently many common websites and tools available for sharing code. Furthermore, while this paper provides highly detailed parameters and prompts, it fails to present any training loss or gradient descent curves, making it challenging to substantiate the theory of "style-gradient cancellation."
Questions for authors
Please see Weaknesses above. I have no more questions. Thank you.
Dear reviewer, as the deadline is approaching, please check the rebuttal and consider participating in the discussion.