- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for comparing preference optimisation methods at four levels of equivalence. It derives theoretical conditions for these equivalences and conducts experiments to examine the relationships between different objectives.
Critical strengths
The authors attend to explore a fundamental question in RLHF.
Critical weaknesses
The paper reads as though the authors are constantly trying to convince the reader of a complete research narrative: “We pose a question, we solve it, we provide a certificate, we validate it, and we draw practical implications”. So auto-research, so sad.
I find the main takeaway from Table 5 very puzzling. The authors report that: 22 losses fall into 20 equivalence classes, with only two pairs sharing the same loss function, despite being trained on different data. However, why exact loss equivalence should be expected among methods that were specifically proposed with different objectives? This is confuse. And, identifying that two methods use the same loss function but different training data seems to offer limited insight into whether switching between them would meaningfully change the optimization process or the resulting policy.
The paper does not appear to present (in my opinion, the problem is the way to present, I do not say there have no new findings) any new, valuable findings from the perspective of scientific research.
Questions for authors
Many of the Table 5 reported equivalence classes follow immediately from the definitions of the underlying scores and losses, while most loss classes contain only a single method. What nontrivial insights this classification provides beyond a systematic restatement of existing objective formulations?