- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper tackles three fundamental problem in Reinforcement Learning from Human Feedback. Namely,
- Auditing existing, published preference datasets for intransitive properties finding that most don't provide sufficient information to answer the question whether preferences are cyclic or not
- Recovering missing preference structures from the partial available information in these datasets.
- Measuring model performance on intransitive datasets violating most algorithms' fundamental assumptions. Downstream evaluations show that they offer minor post-training benefits but more significant for inference-time scaling.
Critical strengths
- The paper addresses fundamental problems and questions in RLHF which underpin many results but not investigated sufficiently.
Critical weaknesses
- Missing background section makes the paper hard to follow for readers not familiar with the domain. Many crucial definitions are missing such as ECE or regret.
- Too much information is deferred to the Appendix is Section 3 making the main content of the paper less self-contained and readers have to go back-and-forth to read the section.
- I could not find measurements on how well InTransit reproduce human judgement in Section 3.1. As I understand, the authors use PRISM annotator information to estimate preference signals of the annotators using an LLM. However, I could not find any evidence that it actually recover well the annotators' observed signals.
- The paper investigates interesting problems but results are trivial. Showing that the Bradley-Terry model does not work well on intransitive datasets is clear as it violates the preference model's main assumption. This means that most of the paper is just underlying issues that are common knowledge without a clear resolution.
Questions for authors
- What was the selection criteria for the datasets evaluated in Section 2? How comprehensive is it?
- Can you provide more information about PersonalLLM? There is no citation or reference for it in the paper.
- In Appendix A.1.4, how was gemini-3.1-flash-lite chosen? Why this model in particular and have the authors carried out any ablation on this?
- In Section 4.1, the authors claim "we see that training on completed datasets improves performance when compared to the same model trained on the original dataset". Could this be only a result of the findings in Section 3.3, i.e., that the completed preferences are biased towards transitive preferences?
- In Section 4.2, were each model tuned for the right hyperparameters? What was the grid and how do results differ with the hyperparameter selection?
- In Section 4.2 and 4.3, how do the better performances transfer to benchmark results?