acceptodds
Under review as a conference paper at ICLR 2027

Manufactured Curl in Preference Learning

Abstract

Scalar reward models underpin reinforcement learning from human feedback (RLHF) for large language models (LLMs). But do cycles in human preferences imply that scalar rewards are inadequate? We examine how preference aggregation, prediction targets, and judgment dependence shape the answer. Averaging choice probabilities can create nonadditive log-odds even when every annotator follows scalar utilities. Concentrating disagreement on one response can make this distortion substantially larger. Recovering mean utilities and predicting choices can favor different estimators; calibration and smoothing affect their comparison. Observation design matters independently: shared rankings and independent judgments can have identical Bradley–Terry probabilities but threefold different asymptotic variances in estimated curl. Reassembling human summarization judgments across 363 contexts into disjoint rather than shared worker groups increases mean absolute curl by 24.4% at matched counts. Simulations at the observed comparison counts quantify detection limits, while held-out prediction tests sensitivity to regularization. These results show why measured preference cycles require an audit of aggregation and observation design before they can justify replacing scalar reward models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
preference learning, reinforcement learning from human feedback, reward modeling, large language models, preference aggregation, annotator heterogeneity, cyclic preferences

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KF7DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how preference aggregation and observation dependence affect measured preference cyclicity. It provides theoretical analyses showing that aggregation can introduce nonadditive log-odds and that observation dependence can change the variance of estimated curl. Experiments on human preference data further demonstrate sensitivity to worker assignment, regularization, and smoothing.

Critical strengths
  1. The paper carefully distinguishes population-level nonadditivity from finite-sample measurements.
  2. The analysis of observation dependence and the matched-worker reconstruction provide concrete and interesting statistical insights.
  3. The authors acknowledge important limitations and investigate alternative explanations involving calibration and regularization.
Critical weaknesses
  1. Insufficient connection to downstream RLHF performance.

This is my main concern. The paper is motivated by whether observed preference cyclicity justifies moving beyond scalar reward models. However, its experiments focus primarily on measured curl and preference prediction loss, without establishing how these quantities relate to downstream policy performance.

Even when population preferences cannot be represented exactly by a BT model, it remains unclear under what conditions this limitation actually harms RLHF training. The paper does not provide theoretical or empirical evidence connecting its statistical findings to this question.

I do not think large-scale LLM training is necessarily required, but some evidence connecting these findings to meaningful learning outcomes would substantially strengthen the paper's significance.

  1. Limited novelty and empirical implications.

Several underlying ideas, including preference heterogeneity, aggregation effects, and the distinction between transitivity and additive representations, have been studied previously. The paper offers useful quantitative refinements, but the theoretical contribution appears relatively incremental.

The human-data experiments demonstrate that measured curl depends on worker assignment, but do not establish population-level cyclicity or its consequences for downstream learning. The prediction experiments are also limited in scope and sensitive to regularization.

Questions for authors

Under what conditions does population-level preference nonadditivity actually lead to worse downstream policy performance when using BT-based reward models? Can the authors provide theoretical analysis or empirical evidence connecting the identified statistical effects to RLHF outcomes, rather than only preference prediction loss?

Reviewer_hsvMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether observed preference cycles necessarily indicate the inadequacy of scalar reward models in RLHF. Theoretically, it shows that aggregating individual preferences can introduce nonzero log-odds curl, while observation dependence can substantially affect its estimation. Empirically, the authors analyze human preference data and find that using disjoint rather than shared workers increases mean absolute curl by 24.4% across 363 contexts. Additional simulations and prediction experiments examine detection limits and the effects of regularization. The findings highlight the importance of accounting for preference aggregation and observation design before concluding that more expressive reward models are necessary.

Critical strengths
  1. The paper provides a systematic analysis of distinct sources of preference cyclicity.
  2. The paper connects theoretical observations to real human preference data through empirical analysis.
Critical weaknesses

1.The prediction experiments are limited to small response sets, and their results are sensitive to regularization and smoothing. 2. Limited practical significance for reward model selection. Although the paper shows that observation dependence can change measured curl by 24.4%, it remains unclear whether this difference materially affects the choice between scalar and non-scalar reward models. The experiments do not demonstrate whether accounting for observation dependence would materially change the conclusions about the adequacy of scalar reward models, leaving its practical implications for RLHF unclear.

Questions for authors

Could the authors further clarify how the paper's findings on preference aggregation and observation dependence differ from prior work on marginal versus conditional logistic models [1, 2] and dependence in paired comparisons [3]? Such clarification would help better understand the paper's novel contributions. [1] Zeger, S. L., Liang, K. Y., & Albert, P. S. (1988). Models for Longitudinal Data: A Generalized Estimating Equation Approach. Biometrics, 44(4), 1049–1060. [2] Neuhaus, J. M., Kalbfleisch, J. D., & Hauck, W. W. (1991). A Comparison of Cluster-Specific and Population-Averaged Approaches for Analyzing Correlated Binary Data. International Statistical Review, 59(1), 25–35. [3] Dittrich, R., Francis, B., Hatzinger, R., & Katzenbeisser, W. (2006). Modelling Dependency in Multivariate Paired Comparisons: A Log-Linear Approach. Mathematical Social Sciences, 52(2), 197–209.

Reviewer_3f2EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether cycles measured in human preference data justify replacing a scalar Bradley–Terry reward model. It makes three arguments. First, averaging choice probabilities over annotators who all follow scalar utilities gives population log-odds that are not additive. With Gaussian heterogeneity the curl of a triangle is, to leading order in small utility gaps, cubic in the gaps, and with unequal edge variances it has a term linear in the gaps, for example when the disagreement is concentrated on one response. Second, two observation protocols with identical BT edge probabilities can give different noise in the estimated curl: its variance is about three times larger for independent judgments than for judgments taken from one ranking. Third, on the Stiennon et al. summarization comparisons, a conditional test against independent BT finds less curl than the null predicts at modest counts, and reassembling contexts into disjoint rather than shared worker groups at matched vote counts raises the mean absolute curl by 24.4%. Simulations give detection limits at the observed counts, and prediction comparisons are used to separate representation from calibration and regularization. The authors conclude that aggregation and observation design should be audited before a measured cycle is used to justify a non-scalar preference model.

Critical strengths
  • The paper separates a majority cycle, a log-odds cycle, and an estimated cycle, and the data analysis keeps the three apart. Majority cycles appear in 1.07% of triangles, and strength consistency is treated as a separate question.
  • I recomputed the closed-form quantities and they agree with the paper, and I found no error in the derivations I checked. The paper does not report its ten-item utility vector, but with a vector drawn from N(0, 0.8²) I also reproduced the calibration quantities and the cyclic-fraction curve.
  • The matched-count reconstruction fixes the number of votes per edge and changes whether the same or different workers supply them. The authors also state what the analyses cannot show: the independent-BT null is questioned by their own worker audit, the thresholds were set after looking at the data, and the test has low power for small curl.
Critical weaknesses
  1. The human-data results do not test the aggregation mechanism behind the title, and they show no clear excess curl over the independent-BT conditional null. Sec. 2 argues that averaging over scalar-utility annotators creates population log-odds curl, but the analyses of human judgments address a different quantity. Apart from one construction in Sec. 6.2, the observed mean absolute curl is below the mean of the conditional null: 0.968 against 1.097 at minimum count 2 (Table 1), and similarly in the all-triangle census (App. A.9) and at every threshold in Fig. 5b. The 24.4% increase is also not a difference in population curl. Workers are randomly permuted before assignment, so under both constructions each edge receives a random set of workers and only the dependence across edges differs. This is the noise effect of Sec. 4 reproduced on real votes, which is useful, but it does not show that aggregation or observation design creates a curl that could be mistaken for a population cycle, and no analysis of an actual case is given. The observed-minus-null value in Sec. 6.2 ("changes from −0.1503 to +0.0235"; I read these as shared and disjoint workers, which the paper does not label) has no uncertainty attached, and the interval [0.1230, 0.3582] describes allocation variability between the constructions, as the authors state. I would like to see (a) a bootstrap over contexts for the observed-minus-null value in both constructions, and (b) the audit applied to at least one analysis in which a measured cycle was used to argue for a non-scalar model.

  2. The practical relevance of the warning is not shown. With equal edge variances, the curl for a triangle with gaps (2g, −g, −g) has a maximum over V in [0.25, 8] of 0.004 at g = 0.5, 0.03 at g = 1 and 0.18 at g = 2, by my numerical integration. Unequal edge variances give larger values (0.172 at g = 0.5 and 0.538 and 0.782 at g = 1 in Table 4), but there one response has a utility standard deviation of 1 to 2 across annotators while the other two do not vary at all, and Sec. 2 states that these are specified populations, not estimates of human utility variance. By comparison, the standard deviation of an estimated curl from independent judgments is √(12/n) when every edge has win probability 1/2, which is 0.35 at n = 100, and the paper's own power simulations show low power for small curl (8.0% rejection at curl 0.172 and 29.5% at 0.538, against 89.5% at 0.782 when every context carries it). What is missing is evidence that such sizes arise in preference data or matter for training; no neural reward model is trained and no policy is optimized. The audit in Sec. 8 is a checklist. Its only diagnostic with a null distribution, the independent-BT conditional test, has no power at one judgment per edge and rests on an independence assumption that the paper itself questions on its main corpus, and Sec. 5.1 argues that an acyclic encoding such as best-of-four feedback cannot validate transitivity but names no corpus. I therefore cannot tell whether the audit changes a conclusion in practice.

  3. The contrast between mean-utility recovery and prediction in Fig. 2a,b appears to be largely a scale effect. The recovery side is the RMSE between the centered estimate and the true mean utilities, which depends on the scale of the estimate, and the paper shows in one population example that a single scale removes 99.66–99.998% of the excess log loss (App. A.4) but does not show how much of Fig. 2a,b is a scale effect. I re-implemented the Sec. 3 simulation (the paper does not give the ridge form for Sec. 3, so I used λ/2·‖θ‖² on the summed log-likelihood, following Eq. 15; details may differ from the authors' code) and reproduced the pattern: pooling has lower expected log loss in all 36 cells but a larger RMSE, with a pooled-to-individual ratio at V = 4, A = 160 of 1.41–1.66 at ridge 0.001 and 2.34–2.66 at ridge 0.1 over three runs, against 1.36 and 2.44 in the paper. After rescaling each estimate by its least-squares scale against the true mean utilities, the RMSE ratio lies between 0.88 and 1.09 in all 36 cells at both ridge values, and the ratio of Pearson correlations (pooled over individual) between 0.96 and 1.04. If this holds for the authors' code, the crossover reflects scale, not what the two estimators recover about the mean utilities. The comparison that bears on the paper's question is BT against the pairwise predictor with matched pseudo-counts (−0.0060 in favor of BT), which uses three-item menus, so the prediction experiments add little to the central argument.

Questions for authors
  1. In Sec. 6.2, can the observed-minus-null value of +0.0235 (disjoint workers) and −0.1503 (shared workers) be distinguished from 0 with a bootstrap over contexts?
  2. Does the region in Fig. 2a,b where individual averaging recovers the mean utilities better remain if the estimates are compared after rescaling, or with a rank correlation?
  3. Can you point to a published analysis in which a measured cycle was used to argue for a non-scalar preference model, and say whether the audit would change its conclusion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.