- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents a method to evaluate whether a method used to select the training set used for DPO is effective beyond mitigating DPO's length bias. Several
Critical strengths
- Understanding the value of a data selection method after controlling for known confounds is an important problem to tackle.
- There is a human evaluation and the section describing the human evaluation protocol is easy to follow.
Critical weaknesses
- The writing of the paper leaves me largely unable to understand what has been done or why it has been done. Several aspects of the writing contribute to my challenges in following what the authors are saying. This is not due to unfamiliarity with DPO nor data selection techniques having written papers on both. (1) The authors appear to pull precise and technical language from the literature, but do not provide definitions or explanations for what the words mean. They are then used to modify each other or in a list (e.g. "length-controlled, shared, multi-metrics audit", "the paired bootstrap resamples aligned policy contrasts", "UltraFeedback TruthfulQA prompt slates"). Individually the words make sense, but put together I cannot always understand what exactly they are saying about the work done in the paper. (2) The paper does not dedicate space to making sure the reader is on the correct page about how data selection is done, what standard practices are, and what domain-specific words mean. (3) Several metrics and pieces of information are listed, but the information they provide about the proposed method is not made clear. For example, a git commit ID is provided, but no repo is linked and it is not clear why that git commit detail is needed.
- The abstract ends with this text, "The contribution is not a universal selector; it is a falsification standard for deciding when DPO data-selection gains survive a shared, confound-aware audit." This makes it sound like the proposed method should apply to arbitrary confounds, but only length appears to be explored.
Questions for authors
I generally struggled to understand what the paper is doing, so am not able to articulate specific questions to ask. To improve overall clarity, it would be helpful to provide extra context. For example, start with an overview of standard data selection approach in Section 3 to help the reader more quickly pick up what the maybe technical terms mean.