- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how preference aggregation and observation dependence affect measured preference cyclicity. It provides theoretical analyses showing that aggregation can introduce nonadditive log-odds and that observation dependence can change the variance of estimated curl. Experiments on human preference data further demonstrate sensitivity to worker assignment, regularization, and smoothing.
Critical strengths
- The paper carefully distinguishes population-level nonadditivity from finite-sample measurements.
- The analysis of observation dependence and the matched-worker reconstruction provide concrete and interesting statistical insights.
- The authors acknowledge important limitations and investigate alternative explanations involving calibration and regularization.
Critical weaknesses
- Insufficient connection to downstream RLHF performance.
This is my main concern. The paper is motivated by whether observed preference cyclicity justifies moving beyond scalar reward models. However, its experiments focus primarily on measured curl and preference prediction loss, without establishing how these quantities relate to downstream policy performance.
Even when population preferences cannot be represented exactly by a BT model, it remains unclear under what conditions this limitation actually harms RLHF training. The paper does not provide theoretical or empirical evidence connecting its statistical findings to this question.
I do not think large-scale LLM training is necessarily required, but some evidence connecting these findings to meaningful learning outcomes would substantially strengthen the paper's significance.
- Limited novelty and empirical implications.
Several underlying ideas, including preference heterogeneity, aggregation effects, and the distinction between transitivity and additive representations, have been studied previously. The paper offers useful quantitative refinements, but the theoretical contribution appears relatively incremental.
The human-data experiments demonstrate that measured curl depends on worker assignment, but do not establish population-level cyclicity or its consequences for downstream learning. The prediction experiments are also limited in scope and sensitive to regularization.
Questions for authors
Under what conditions does population-level preference nonadditivity actually lead to worse downstream policy performance when using BT-based reward models? Can the authors provide theoretical analysis or empirical evidence connecting the identified statistical effects to RLHF outcomes, rather than only preference prediction loss?