- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Pairwise preferences discard important information such as the aspects that separate the response, the size of the gap and limitations of the rejected response. They re-audit Skywork-80K, an existing preference dataset. Specifically, it uses a per-aspect ordinal distribution, as well as prompt-gated verifiers. Using these methods, it shows improvements on Llama and Qwen based models on RewardBench 2.
Critical strengths
- The improvements over Bradley-Terry baseline models are consistent and good.
Critical weaknesses
- The approach of the method is very similar to ArmoRM “Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts” (EMNLP, 2024), which the authors cite but do not clearly explain the difference. Specifically, ArmoRM trains a prompt-gated head to determine how various HelpSteer attributes with 5 levels of distribution (NAACL 2024) are considered, which is exactly what’s considered in Figure 3. Even the concept of preference strengths (1/2/3) in Table 1 is directly taken from HelpSteer2-Preference “HelpSteer2-Preference: Complementing Ratings with Preferences” (ICLR, 2025) without acknowledgement or differentiation. Similarly, the concept behind “why rejected loses” in Figure 3 was also done in HelpSteer2-Preference paper. The poor positioning of this work among prior works makes it challenging to understand the genuine contributions of the paper.
- The paper claims to use 5 other benchmarks alongside RewardBench 2 (see line 148) but only primarily reports results on RewardBench 2 (there was a small mention at line 375 but it does not show how the scores compare to baseline models in a table that is typically expected) It also claims to “summarize(s) the technical differences among these paradigms (of evaluation benchmarks)” in Appendix A but did not do that.
Questions for authors
- (Not a question but some general advice for authors) Overall, the paper reads to me like it was either primarily generated by an LLM (way too many em-dashs and weird use of italics) and/or it was written by authors who have not had much experience writing ML papers. Both are fine (e.g. LLM use is disclosed and all authors had to start from somewhere) but I think the current paper is not written well enough to be accepted (see weaknesses, especially weakness 1).