acceptodds
Under review as a conference paper at ICLR 2027

The Comparison Graph Determines What RLHF Can Learn: Spectral Identifiability, Error Bounds, and a Pre-Training Diagnostic for Preference Data

Abstract

Reward models for **RLHF** are trained on pairwise human comparisons, and the field treats that data as a quantity to be increased. We show the binding constraint is its structure, measurable before training. A preference dataset induces a *comparison graph* on prompt–response pairs, so reward learning under Bradley–Terry is estimation on that graph; lifting it through the reward model's feature map yields one matrix, computable in a single pass, whose spectrum settles which reward directions the data identifies, how fast error falls, and which responses can be scored reliably. We characterise the identifiable directions, bound finite-sample error pointwise by an effective resistance, and give a minimax lower bound with the same dependence on conditioning, so the limits bind every estimator. Two consequences are properties of the graph alone and hold at every scale and under every feature map. First, none of six widely used public datasets contains a single comparison between responses to *different* prompts, and each fragments into to components, so every reward component varying across prompts is unidentifiable at any dataset size, model size or embedding, including inside two trained reward models' own representations. This is a gauge that within-prompt **RLHF** may ignore and that pooled best-of-, reward-thresholded filtering, a shared value baseline and any reported mean reward may not: on the audited corpus, two rewards the data cannot tell apart, with identical log-likelihood in double precision, agree on 1.000 of within-prompt decisions while a pooled top- selection overlaps by only 0.385 and prompt ranking by mean reward moves at Spearman +0.728. Second, where the feature dimension no longer binds the rank the unlearnable subspace closes, but per-response conditioning spreads by to : the constraint moves rather than lifts. Linear, margin, **MLP** and transformer heads return bit-identical estimates along a direction the data cannot identify, and varying only topology at fixed budget changes error by , a gap that does not close over a range of budgets. We also report where our own selection objective fails, and argue the diagnostic rather than any selection rule is the contribution.

Primary area:
learning theory
Keywords:
RLHF, Reward Modeling, Preference Learning, Comparison Graphs, Spectral Identifiability, Bradley–Terry Models, Effective Resistance, LLM Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Fr51OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the statistical structure of RLHF preference data through its comparison graph. For a fixed feature map ϕ\phi, the authors define the feature-weighted comparison Laplacian Lϕ=Φ⊤B⊤WBΦL_\phi=\Phi^\top B^\top W B\Phi and argue that three spectral quantities provide a pre-training diagnostic: null⁡(Lϕ)\operatorname{null}(L_\phi) characterizes unidentifiable reward directions, λmin⁡+(Lϕ)\lambda_{\min}^+(L_\phi) controls the difficulty of estimating the weakest identifiable direction, and feature effective resistance measures response-specific scoreability. The theoretical development includes an identifiability characterization, a prompt-level gauge result for datasets containing only within-prompt comparisons, pointwise and global error bounds, and several minimax lower-bound constructions. The paper further discusses which RLHF computations are invariant to prompt-dependent offsets and which operations, such as pooled filtering or reporting mean reward, may depend on them. The empirical study includes controlled Bradley–Terry experiments, comparisons across graph topologies at fixed label budgets, tests with several reward-head classes, audits of five public preference datasets and one synthetic control, prompt-gauge perturbation experiments, and a spectral comparison-selection procedure called SPEC. The experiments generally support the structural identifiability claims and show that topology can substantially affect recovery. The paper also reports useful negative results: the theoretically certified log-determinant objective is not the strongest empirical selector, uncertainty sampling can underperform random selection, and maximizing the weakest eigenvalue can be inappropriate for dataset repair.

Critical strengths

The mapping from the null space, the smallest positive eigenvalue, and effective resistance to identifiability, global conditioning, and response-level reliability is conceptually clean. The prompt-level gauge analysis also identifies a consequential distinction between within-prompt RLHF decisions and computations that place different prompts on a common reward scale.

Critical weaknesses

The central statistical guarantees are not currently stated on a well-defined finite-sample parameter and estimator class. Theorem 3 defines θ^\widehat\theta as a maximizer of the unregularized Bradley–Terry likelihood, but finite preference samples can be separable, in which case a finite MLE need not exist. Bounded true margins alone do not rule out sample separation. Moreover, the proof invokes a self-concordant linearization with an unspecified lower-order remainder, without stating the sample-size, leverage, or local-curvature conditions needed to control it. The displayed theorem nevertheless gives an unconditional finite-sample bound without this remainder. Theorem 4 has a related issue: if the supremum is over all of Rd\mathbb R^d, logistic saturation makes the global minimax Euclidean risk infinite even for full-rank designs. Consequently, the finite lower bounds do not by themselves justify the claimed minimax floor unless an explicit bounded parameter class is introduced and the dependence on that bound is analyzed. These issues concern two of the paper’s main theoretical contributions.

The evidence for practical downstream harm remains indirect. The large nullities in Table 2 are mostly forced by the chosen rank-binding operating point d>∣V∣−cd>|V|-c, as the appendix acknowledges; at the production-scale operating point, the matrices are full rank and the argument becomes one about weak rather than unidentifiable directions. The real-corpus resistance experiment uses planted Bradley–Terry rewards, while the gauge experiment deliberately perturbs a fitted reward inside an exact or weak eigenspace. These experiments establish possibility, but they do not show that a normally trained end-to-end reward model has a substantial head projection or uncertainty in those directions. Likewise, the RewardBench selection experiment uses frozen representations and operates far below trained-model accuracy. Thus, the diagnostic may be useful, but the prevalence and magnitude of the claimed deployment consequences are not yet established.

Questions for authors

Could the authors restate Theorem 3 for a precisely defined estimator and give the conditions under which that estimator exists? In particular, how is sample separation excluded for the unregularized MLE, and what sample-size, leverage, or curvature condition controls the “lower-order” remainder used in the proof? If the experiments use a constrained or ridge-regularized estimator, does the stated resistance bound hold for that estimator?

What is the parameter class in Theorem 4? If the supremum is over all θ⋆∈Rd\theta^\star\in\mathbb R^d, the minimax Euclidean risk of a finite logistic experiment appears to be infinite even when LϕL_\phi is full rank, because arbitrarily large parameters along a fixed sign cone generate nearly indistinguishable saturated observations. Can the authors provide a bounded-margin or bounded-diameter minimax statement and show how the lower bound depends jointly on this radius and the spectrum of LϕL_\phi?

For the two trained reward-model representations, could the authors report the norm of the learned scalar head projected onto the prompt-constant null space or the weakest eigenspace, together with the resulting change in actual cross-prompt decisions? This would distinguish the existence of an ambiguous direction from whether a normally trained reward model materially uses that direction.

Could the authors more sharply distinguish the new theoretical contribution from existing topology-aware Bradley–Terry bounds, covariate-assisted ranking theory, and recent work on what pairwise preference learning recovers? In particular, comparison with Shah et al. (2015), Fan et al. (2024), Saha and Rajkumar (2024), and Pukdee et al. (2026) would materially affect my assessment of novelty.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Fr51 (1/2)

Thank you for checking the theory so closely. Both soundness points were correct. The submitted Theorem 3 assumed an MLE that need not exist and dropped a remainder without conditions, and Theorem 4 took suprema over Rd\mathbb R^d, where the risk is infinite. Both theorems are now stated over a bounded-margin class (lines 260 to 319, proofs in Appendix A).

Let Ln=∑enexexe⊤L_n=\sum_e n_e x_e x_e^\top have rank rr and smallest nonzero eigenvalue λmin⁡\lambda_{\min}, let κB=σ′(B)\kappa_B=\sigma'(B) and ΘB={θ∈row(Ln):∣⟨θ,xe⟩∣≤B\Theta_B=\lbrace\theta\in\mathrm{row}(L_n):|\langle\theta,x_e\rangle|\le B for all e}e\rbrace. The likelihood maximiser θ^B\hat\theta_B over this compact polytope exists and is unique, as the likelihood is strictly concave on row(Ln)\mathrm{row}(L_n). We bound margins rather than the norm because on ΘB\Theta_B the Hessian is at least κBLn\kappa_B L_n. Here θ⋆\theta^\star is the true parameter projected onto row(Ln)\mathrm{row}(L_n), RvR_v the resistance of response vv, and cvc_v the mean of r^−r⋆\hat r-r^\star over its component.

Theorem 3(i) holds at every sample size. Let εδ=(r+2log⁡(1/δ))/(2κB)\varepsilon_\delta=(\sqrt{r}+\sqrt{2\log(1/\delta)})/(2\kappa_B). On one event of probability at least 1−δ1-\delta, which does not depend on BB, for every BB with θ⋆∈ΘB\theta^\star\in\Theta_B and every vv,

∣r^(v)−r⋆(v)−cv∣≤Rvεδ,∥θ^B−θ⋆∥Ln≤εδ.|\hat r(v)-r^\star(v)-c_v|\le\sqrt{R_v}\varepsilon_\delta,\qquad \lVert\hat\theta_B-\theta^\star\rVert_{L_n}\le\varepsilon_\delta .

Dividing by λmin⁡\sqrt{\lambda_{\min}} gives the Euclidean bound, and for fixed BB both hold in expectation with r/(2κB)\sqrt{r}/(2\kappa_B). The proof of (i) uses no Taylor expansion, so there is no remainder. Optimality and curvature give κB∥Δ∥Ln≤∥g∥Ln+\kappa_B\lVert\Delta\rVert_{L_n}\le\lVert g\rVert_{L_n^+} for the error Δ\Delta and the score gg, and Hsu, Kakade and Zhang (2012) bound ∥g∥Ln+\lVert g\rVert_{L_n^+}.

Theorem 3(ii) is about the MLE. Let ρn=max⁡exe⊤Ln+xe\rho_n=\max_e x_e^\top L_n^+x_e, let θ⋆∈ΘB′\theta^\star\in\Theta_{B'} with B′<BB'<B, and let η=ρnεδ/2<min⁡{1,B−B′}\eta=\sqrt{\rho_n}\varepsilon_{\delta/2}<\min\lbrace1,B-B'\rbrace. With probability at least 1−δ1-\delta no constraint is active, so the MLE exists and equals θ^B\hat\theta_B, and

∣r^(v)−r⋆(v)−cv∣≤RvκB′[2κB′log⁡2∣V∣δ+23ρnlog⁡4∣V∣δ+12(eη−1)(r+2log⁡2δ)].|\hat r(v)-r^\star(v)-c_v|\le\frac{\sqrt{R_v}}{\kappa_{B'}}\Big[2\sqrt{\kappa_{B'}\log\tfrac{2|V|}{\delta}}+\tfrac23\sqrt{\rho_n}\log\tfrac{4|V|}{\delta}+\tfrac12(e^{\eta}-1)\big(\sqrt{r}+\sqrt{2\log\tfrac2\delta}\big)\Big].

The first term is the submitted bound. The other two are the remainder it left implicit.

Q1. For the MLE, separation is excluded only on the event of (ii), and the condition on η\eta is the leverage and curvature condition you asked for. Since ρn≥r/n\rho_n\ge r/n, it needs n>r(r+2log⁡(2/δ))2/(4κB2)n>r(\sqrt{r}+\sqrt{2\log(2/\delta)})^2/(4\kappa_B^2). An LP check (Albert and Anderson, 1984) finds no quasi-separation in any of F3's 36 one-comparison-per-edge fits (line 289). Those fits use a ridge τ=10−8\tau=10^{-8}, and yes, (i) covers it: with BB the largest fitted or planted margin, the bound holds with εδ+τ∥θ⋆∥Ln+/κB\varepsilon_\delta+\tau\lVert\theta^\star\rVert_{L_n^+}/\kappa_B (line 1026). Table 2 needs no fit, since its operating point has no finite MLE. No experiment depends on either part or tests their constants: at our sizes (i) is loose to vacuous and the condition of (ii) is never met. F3 checks only whether errors are ordered by resistance, as the form of the bound suggests.

Q2. The class is ΘB\Theta_B, built from the realised design. Lower bounds on inf⁡θ^sup⁡ΘBE∥θ^−θ⋆∥2\inf_{\hat\theta}\sup_{\Theta_B}\mathbb E\lVert\hat\theta-\theta^\star\rVert_2:

  • (a) Le Cam: 12(1−18)min⁡(2,B)/λmin⁡\tfrac12(1-\tfrac1{\sqrt8})\min(\sqrt2,B)/\sqrt{\lambda_{\min}}, which is 0.457/λmin⁡0.457/\sqrt{\lambda_{\min}} for B≥2B\ge\sqrt2.
  • (b) Fano, 2≤k≤r2\le k\le r, correcting a step of the submitted proof: 12(34−log⁡2log⁡(k+1))min⁡(2log⁡(k+1),B)/λk\tfrac12(\tfrac34-\tfrac{\log2}{\log(k+1)})\min(\sqrt{2\log(k+1)},B)/\sqrt{\lambda_k}.
  • (c) Assouad, squared loss: s24∑i≤kλi−1\tfrac{s^2}{4}\sum_{i\le k}\lambda_i^{-1} with s=min⁡(1,B/kρn)s=\min(1,B/\sqrt{k\rho_n}).
  • (d) Over ΘB+null(Ln)\Theta_B+\mathrm{null}(L_n) the risk is infinite exactly when null(Ln)≠{0}\mathrm{null}(L_n)\ne\lbrace0\rbrace.
  • (e) For B≥1B\ge1 (Eq. 11, line 1107): 14min⁡{e(c1(B−1)−1)/2/λmin⁡,1/m1}\tfrac14\min\lbrace e^{(c_1(B-1)-1)/2}/\sqrt{\lambda_{\min}},1/m_1\rbrace, with m1=max⁡e∣⟨v1,xe⟩∣m_1=\max_e|\langle v_1,x_e\rangle| for a bottom eigenvector v1v_1 and a design constant c1∈(0,1]c_1\in(0,1], which is 1 on a bipartite item-level graph.

The radius enters (a) to (c) through factors that stop growing once BB passes a constant. Part (e) grows as eB/2e^{B/2} for moderate BB when c1=1c_1=1, but it beats (a) only if B>3.2B>3.2 and λmin⁡>3.34m12\lambda_{\min}>3.34m_1^2, and for large BB the risk is linear in BB. With (i), for B≥2B\ge\sqrt2 the minimax risk lies between 0.457/λmin⁡0.457/\sqrt{\lambda_{\min}} and r/(2κBλmin⁡)\sqrt{r}/(2\kappa_B\sqrt{\lambda_{\min}}). Both scale as λmin⁡−1/2\lambda_{\min}^{-1/2}, but their ratio is only bounded by 4.4reB4.4\sqrt{r}e^B, so we claim no matching rate.

AuthorsOpenReview ↗

Response to Reviewer Fr51 (2/2)

The revision adds F5 (lines 451 to 467, Appendices R and U), with the head projection you asked for in Q3.

Q3. A reward model with a scalar head is linear in the head's input hh, so with ϕ:=h\phi:=h we built LϕL_\phi from 1,987 of F3's 2,000 within-prompt UltraFeedback comparisons for deberta and two Skywork-Reward-V2 decoders, at 0.6B and 1.7B. internlm2 (submitted Section 5.1) is now in Appendix U, post hoc, checked against its published RewardBench accuracies instead of its model card. Table 11 (line 2171) and Appendix U give the head's share ∥Pkw∥/∥w∥\lVert P_kw\rVert/\lVert w\rVert in the k=16,64,256k=16,64,256 weakest eigendirections:

  • deberta (d=1024d=1024, full rank): 0.11, 0.21, 0.37, against 0.125, 0.25, 0.50 by chance;
  • Skywork 0.6B (d=1024d=1024): 0.12, 0.26, 0.47, same chance values;
  • Skywork 1.7B (d=2048d=2048, more than the comparisons): 0.168 of the head in the forced 61-dimensional prompt-constant null space, against 0.173 by chance.
  • internlm2 (d=2048d=2048; Appendix U): 0.03, 0.05, 0.10 against 0.09, 0.18, 0.35, and 0.051 in its 62-dimensional null space against 0.174.

Removing these components moves the reward by at most 1.6% of its standard deviation and leaves every pooled top-10% selection at Jaccard 0.975 or more. In all four models the head's weight on the directions this dataset leaves weak is near or below chance, so the answer to Q3 is negative.

On the offsets the graph does predict something. No comparison crosses prompts, so each model sets them by extrapolation, and two models can differ there without disagreeing on any within-prompt decision. A full-rank spectrum cannot show this; F5 measures it in function space. On all four rated completions of 1,835 prompts, the prompt accounts for 0.73, 0.40 and 0.34 of the models' score variance, against 0.29 for GPT-4's ratings (paired differences +0.05 to +0.44, all above zero). Removing the offsets lowers the mean GPT-4 rating of a pooled top-10% selection for every model (deberta: 7.85 to 6.97), so they carry signal. Yet for both Skywork models they lower within-prompt pick quality (Appendix U; deberta is mixed). A model given another family's offsets, rescaled, keeps every within-prompt decision, but its pooled selection overlaps the original only at Jaccard 0.32 to 0.37. With their own offsets, the selections cover 334 to 487 prompts against 573 under GPT-4's ratings, and the median pick runs 1,900 to 2,552 characters against 1,179 (Table 3). All of this is post hoc and on one corpus (three models here, a fourth from a third family in Appendix U); no policy was trained.

Second weakness. F4's perturbation shows only possibility, as you note. Section 5.3 now calls cross-component differences unidentifiable only along directions constant within each component, and the submitted "8 to 35× spread in the error floor" is a 2.9× to 5.9× spread in a per-response upper bound. F3 uses planted rewards, as the corpus has no true reward, so the paper calls its result an empirical regularity (rank correlation +0.17 to +0.36). The selection results are claims about conditioning on frozen representations. The synthetic control in your summary was our error: the sixth dataset is the public argilla/distilabel-intel-orca-dpo-pairs, labelled by a language model (Appendix I).

Q4. Section 2 (lines 108 to 142) covers all four. Shah et al. (2015; JMLR 2016) prove item-level minimax bounds over a bounded class and compare topologies; Theorems 3(i) and 4 carry their argument to feature designs. Fan et al. (2024, Lemma 15) analyse LϕL_\phi for ϕ(i)=(ei,zi)\phi(i)=(e_i,z_i); our linear reward is their model without per-item scores. Saha and Rajkumar (2024) bound error through the smallest nonzero eigenvalue of a feature-weighted Laplacian and show that features can make a disconnected graph recoverable. Pukdee et al. (2026) define a connectivity degree that controls within-prompt margin error and compute a diagnostic on real data, three of our corpora included. We no longer present LϕL_\phi or the bounding argument as new. We add a per-direction reading of which reward differences a within-prompt design leaves undetermined under a given feature map, per-response resistance in feature space, and the RLHF accounting and measurements of Sections 4.2 and 5.4. A single eigenvalue cannot name the missing direction: in Section 5.5's planted instance, Corollary 2 names the missing axis, eight random cross-prompt pairs lift cross-prompt accuracy from 0.633 to 0.974, and eight chosen to maximise λmin⁡\lambda_{\min} give 0.935 (lines 1790 to 1797). On the audited data the graph adds to rank, λmin⁡\lambda_{\min} and leverage only on SHP (line 414).

Do the restated theorems and F5 answer both of your concerns?

Thank you for the thorough response and additional theorem statements, which largely address my concerns. I am therefore raising my score to 3 and encourage the authors to incorporate these additions into the revisions.

Thank you for reading our response and for updating your review. The restated Theorems 3 and 4 are already in the revised PDF (lines 260 to 276 and 302 to 319, with proofs at lines 864 to 1228), and they will stay in the final version, along with the other additions described in our response. If any concern is still open, we are glad to answer it before the discussion closes.

Reviewer_UjB2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reframes reward-model learning under Bradley–Terry as estimation on a comparison graph whose vertices are prompt–response pairs.

Critical strengths

Lifting the graph through the reward model’s feature map yields the “feature-weighted comparison Laplacian”, which is computable without training. The prompt-level gauge framework can separate RLHF practices that are safe from the ones that are exposed.

Critical weaknesses

The novelty of this paper is questionable. Is the result a measurement or a restatement of the annotation design?

Questions for authors

On the six audited datasets, what portion of edges fall outside the regime?

1 comment
AuthorsOpenReview ↗

Response to Reviewer UjB2

Thank you for the review, and for singling out the feature-weighted Laplacian and the split between safe and exposed RLHF steps. In short: no edge falls outside the regime at d=16d=16, 1% to 21% do at d=512d=512 on five datasets, and all of SHP's do; and the audit is partly a restatement of the design and partly measurement, as below.

On the six audited datasets, what portion of edges fall outside the regime?

We took "the regime" to be the well-conditioned logistic regime to which the submission's Limitations restricted Theorem 3, that is, fits that do not saturate. The new Appendix S (line 2182, Table 12) refits each dataset on its own labels, on the graphs of Appendix N (up to 4,000 responses and 2,000 comparisons each):

  • At $d=16$, where LϕL_\phi has full rank, no dataset is separable and no edge is saturated (fitted win probability above 0.99). These fits are weak, though: 26% to 46% of edges are fitted with the wrong sign. They were added for your question; Section 5.3's full-rank audit reads resistance off LϕL_\phi and needs no fit (lines 407 to 410).
  • At d=512d=512, 1.0% to 20.8% of edges are saturated on five datasets. SHP becomes separable, and all of its edges saturate.
  • At Table 2's operating point (d≥∣V∣d\ge|V|), no dataset has a finite MLE, which is why Table 2 reports only quantities that need no fit.

One caveat: Table 12 uses the offline word-hashing map, not the hosted Titan embeddings of Table 2 and Appendix N. Appendix J traces the failure of a bootstrap error proxy under Titan features on UltraFeedback to saturating fits, so with those features the saturated share is likely larger; Appendix S says so, and Section 5.3 gives the saturation shares (lines 415 to 416). Appendix A extends Theorem 3(i) to these 10−810^{-8} ridge fits (line 288), and its bounded-margin estimator exists even under separation (line 193); with no true parameter known, Table 12 checks none of its conditions.

If you meant the within-prompt setting of Corollary 2, no edge falls outside it: every comparison in all six datasets is between two responses to the same prompt.

Is the result a measurement or a restatement of the annotation design?

Partly a restatement, and you are right about that part. The zero cross-prompt fraction follows from how the data were collected, the free per-prompt offset it implies is Lemma 1 of Rafailov et al. (2023), and Table 2's nullity is forced by counting at the operating point we chose. The Discussion now says this (lines 488 to 493). Three findings are measurements:

  • F5 (Section 5.4, post hoc). On all four rated completions of 1,835 UltraFeedback prompts, the prompt accounts for 0.34 to 0.73 of the score variance of three public reward models, against 0.29 for GPT-4's ratings. This puts numbers on the split you highlighted: giving a model another family's offsets leaves every within-prompt decision unchanged but changes about half of a pooled top-10% selection (Jaccard 0.32 to 0.37; 0.60 to 0.63 within the Skywork pair).
  • Topology at a fixed budget (Section 5.2, semi-synthetic). On UltraFeedback text with a rubric reward, changing only which pairs are compared changes recovery error by up to 22.4×.
  • Resistance at full rank (Section 5.3). Its 90th-to-5th percentile ratio is 8.4× to 35.4× within a corpus, a 2.9× to 5.9× spread in the per-response upper bound. Where comparisons share responses, as in SHP or a designed collection, it adds to rank, λmin⁡\lambda_{\min} and leverage; on the five datasets made almost entirely of isolated pairs, a response's resistance is a quarter of its comparison's leverage (line 414).

On novelty: the tools are prior work, as the paper says (line 84). With count weights the matrix is the data covariance of Zhu et al. (2023) times the number of comparisons, related feature-weighted Laplacians appear in Fan et al. (2024) and Saha and Rajkumar (2024), and Theorems 3 and 4 follow Shah et al. (2016) and Zhu et al. (2023). What is new is what they show about RLHF data before training. For a given feature map, Corollary 2 names the reward directions a within-prompt design leaves undetermined, which no single eigenvalue can: in Section 5.5's planted instance it names the missing axis, eight cross-prompt pairs then lift cross-prompt accuracy from 0.633 to 0.974, and pairs chosen to maximise λmin⁡\lambda_{\min} do worse (lines 478 to 484 and 1790). Section 4.2 sorts RLHF steps into those the offset cannot change and those it can, the split you singled out. At full rank the spectrum cannot show the offset, so F5 measures it, and the exposed side turns out to be real in trained reward models.

Does Appendix S answer the regime question as you meant it, or did you mean another regime?

Reviewer_JmZaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper treats preference datasets as a comparison graph, wherein the prompt-response pair is a node and comparison between these instances correspond to edges. The paper uses the feature weighted comparison laplacian of this graph, to characterize what reward information the data can or cannot determine, irrespective of the training or model size. The paper provides theoretical analysis of the comparison graph and provides identifiability conditions and error bounds. Empirically, they observe that existing preference datasets do not have cross-prompt comparisons, so prompt level differences in reward cannot be learned.

Critical strengths
  • The framing about what a preference dataset can determine about the reward, irrespective of training and model scale, is appealing.
Critical weaknesses
  • The writing is dense and hard to follow. Many claims are hedged and the paper looks heavily AI-edited. The presentation alone would require substantial revision to meet the standard for ICLR.
  • The theoretical contribution is itself limited. The authors themselves concede that the design matrix and rank deficiency are known ("We claim neither, and Theorem 1 is that design’s rank condition" in Sec 2). Furthermore, Corollary 2 is the already know DPO's reward equivalence class result. The paper applies this graph-based formulation to preference datasets in RLHF, which is a much narrower contribution.
  • Adding to the earlier point, the paper's central RLHF finding that prompt level offsets are unidentifiable from within prompt comparisions is already well known. This is the DPO reward equivalence class result repackaged differently.
  • The paper offers no new practical takeaways. Its main recommendation on not comparing rewards across prompts is already the common practice. The problem that matters in practice, as to whether reward models carry harmful prompt level offsets is not addressed and is left open.
  • The paper recommends collecting cross-prompt comparisons. But does not talk about how feasible it is to compare responses to different prompts and if that is even practical. Eg: How do you compare an elegant poem vs a solution to math problem.
Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer JmZa

Thank you for the review. Your fourth point raised the question that matters in practice, whether reward models carry harmful prompt-level offsets, so we measured the offsets in three public reward models (F5; a fourth in Appendix U). The prompt carries 0.34 to 0.73 of their score variance, against 0.29 for GPT-4's ratings of the same completions. A model given another family's offsets, rescaled to its spread, keeps every within-prompt decision, but about half of its pooled top-10% selection changes (Jaccard 0.32 to 0.37). DPO's equivalence class says the offset is free, not how large trained models make it or how much of a pooled selection it moves. We also agree with two of your points outright: the submitted text was hard to read, and Corollary 2 is DPO's reward equivalence class, which the submission should have said. The revision fixes both.

Writing. We rewrote the abstract and Sections 1 to 6 in shorter sentences and replaced the term "gauge" with "prompt-level offset". Our AI use statement (line 537) lists how we used AI tools, including for drafting and editing; we checked all of their output and take responsibility for the text.

Novelty (your second and third points). We agree. Theorem 1 is the standard rank condition, and Corollary 2 is Lemma 1 of Rafailov et al. (2023) for a linear reward on a finite dataset. The revision says so directly after Corollary 2 (line 227) and in Sections 1 and 2. What the paper adds is a tool and a measurement. For a given feature map, one matrix shows before training which reward directions a dataset cannot determine and which responses it determines poorly. For RLHF, we list the pipeline steps that depend on the offset and measure its size in real reward models.

Practical value. F5 scores all four rated completions of 1,835 UltraFeedback prompts (Section 5.4, Table 3; post hoc). The offsets carry signal: removing them lowers the mean GPT-4 rating of a pooled top-10% selection, from 7.85 to 6.97 for deberta. But no comparison in the data constrains them. With their own offsets, the models' pooled selections cover 334 to 487 prompts (573 for GPT-4's ratings), and their median pick is 1.6 to 2.2 times as long as GPT-4's. We also checked, post hoc, whether the offsets make the picks worse within their own prompt (Appendix U). For both Skywork models they do: 59% and 63% of pooled top-10% picks are their prompt's best by GPT-4's rating, against 71% and 74% without the offsets; for deberta, 46% against 49% is within noise, so the result is mixed. We have not shown that a policy trained on such a selection is worse.

On common practice, we agree that within-prompt use is the norm, and RAFT's authors switched to within-prompt ranking for this reason. Common pipelines still have steps that put prompts on one scale, though. Appendix M (lines 1820 to 1841) lists cases we checked in the cited versions: OpenRLHF clips rewards to (-10, 10) by default, and TRL's PPO trainer can clip every score at one value; Llama 2 switches to its safety reward model for safety-related prompts and whenever the safety score is below 0.15, one threshold for all prompts, although per-response safety labels give that score some anchor. The paper adds how large the effect is and a cheap check for it. Section 6 recommends:

  1. If you filter or rank across prompts with a Bradley-Terry reward model, check how many prompts the selection covers; if it must cover the prompt set, select within prompts or use a model trained on absolute ratings.
  2. To put prompts on one scale, keep absolute ratings or add cross-prompt comparisons.
  3. Before buying labels, compute λmin⁡+\lambda^+_{\min} and each response's resistance, which tracked recovery error in our tests; we have not shown that choosing comparisons this way improves a trained reward model.

Poem versus proof. Such judgements are harder, and we have no human data on them. The comparison we mean asks which of two responses better does what its own prompt asked: in your example, whether the poem is a better poem than the proof is a proof. Absolute rating scales already ask this of every response (InstructGPT, HelpSteer2), so they supply such links when raters use the scale consistently. With a free offset per prompt, linking $P$ prompts takes at least P−1P-1 cross-prompt comparisons. Cross-prompt comparisons have helped when reward models were the annotators (Shen et al., 2025; Sun et al., 2025), and in a planted instance, eight cross-prompt pairs raised cross-prompt accuracy from 0.633 to 0.974 (Section 5.5). Whether people make them reliably is not established.

Does F5 answer your question about offsets in real reward models? If any point, the writing included, still falls short, tell us which and we will answer it here.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision

Thank you for the reviews. The revision keeps the submission's six sections in the same order. The prose of the abstract and Sections 1 to 6 was rewritten for readability, and the claims are the submission's except for the changes listed below, so this comment gives each change with its line numbers rather than marking it in colour. Each reviewer's questions are answered in their own thread: Theorems 3 and 4, F5 and related work for Fr51; Appendix S and the Discussion for UjB2; the writing, novelty, F5 and cross-prompt comparisons for JmZa.

Theorems 3 and 4 (Fr51). The submitted Theorem 3 assumed that the unregularised MLE exists and left a linearisation remainder unstated, and Theorem 4 took a supremum over all of Rd\mathbb R^d, where the minimax risk is infinite. Both are now stated over parameters with bounded edge margins: Theorem 3 at lines 260 to 276, Theorem 4 at lines 302 to 319, proofs at lines 864 to 1228. Part (i) of Theorem 3 holds at every sample size, and part (ii) states when the MLE exists, with the remainder written out. In Theorem 4, part (b) corrects a step of the submitted Fano argument and part (e) shows how the floor grows with the margin bound. An LP check (line 289) finds that the MLE exists in all 36 one-comparison-per-edge F3 fits.

F5, a new measurement (lines 451 to 467, Table 3, Appendix R from line 1983). We scored all four rated completions of 1,835 UltraFeedback prompts with three public reward models. The prompt accounts for 0.73, 0.40 and 0.34 of their score variance, and for 0.29 of GPT-4's ratings. Giving a model another family's offsets leaves its within-prompt decisions unchanged and changes about half of a pooled top-10% selection (Jaccard 0.32 to 0.37 across families, 0.60 to 0.63 between the two Skywork models). The analysis is post hoc. It also contains the head projection Fr51 asked for in Q3, which came out negative.

Credit. Corollary 2 is Lemma 1 of Rafailov et al. (2023), and the paper now says so right after it (line 227). Theorems 3 and 4 follow Shah et al. (2016) and Zhu et al. (2023), and Section 2 (lines 108 to 142) adds Fan et al. (2024), Saha and Rajkumar (2024) and Pukdee et al. (2026).

Also new. Appendix S (line 2182) reports separation and saturation on all six audited datasets, for UjB2's question. The Discussion ends with three recommendations for practice (lines 497 to 509). To stay within ten pages, the F4 figure moved to Appendix O. Appendix U (post hoc, added in the author response) adds internlm2 as a fourth F5 model, checked against its published RewardBench accuracies, with its head projection near or below chance like the other three, and reports whether the offsets lower the within-prompt quality of pooled top-10% picks: they do for the two Skywork models, not detectably for deberta.

Corrections to the submission.

  • Section 5.3: the "8 to 35× spread in the error floor" is a 2.9× to 5.9× spread in an upper bound (footnote, line 431). Figure 1 now also says "error bound" where it said "error".
  • Abstract and Section 5.3: the submission said that every reward component varying across prompts is unidentifiable at any dataset size, model size or embedding. That holds only along feature directions that are constant within each prompt. At full rank a model sets each prompt's level by extrapolation.
  • Section 5.1: the prompt-constant null directions we reported inside two trained reward models are forced by counting at that 400-node operating point, so the revision states this for any representation. The statement never used the models' scores. One of the two, internlm2, is not among the main F5 models: no setting we tried reproduces the two example scores on its model card, and the same two scores appear on the 7B and 20B models' cards (line 2004); Appendix U now adds it.
  • Appendix I: the sixth dataset is the public argilla/distilabel-intel-orca-dpo-pairs, whose preferences a language model produced. We did not generate it. No number changes.
  • Section 4.2: a mean-reward difference between two policies on the same prompts, and a critic that sees the prompt, are no longer listed as offset-dependent.
  • Section 5.5: "a few anchors suffice" held only for one planted direction. With a free offset per prompt, P prompts need at least P − 1 cross-prompt comparisons (line 485).
  • Section 5.3: F3 is described as an empirical regularity on planted rewards, consistent with the form of Theorem 3 but not a test of it.
  • Section 1 (line 37): the opening now says what the cited work shows, that reward models are over-optimised even with more comparisons, rather than that more comparisons stop helping.
  • The selection results are stated as claims about conditioning on frozen representations, not end-to-end accuracy.
AuthorsOpenReview ↗

Reply to JmZa

Thank you for reading our response and for replying. We agree with you on the theory: the unidentifiability is DPO's reward equivalence result, and the revision credits it to Rafailov et al. (2023) right after Corollary 2 (line 227) and does not claim it as new. What we add here is the measurement you mention: how much the offsets of trained reward models change what gets picked across prompts. On whether those picks are worse, Appendix U is the closest evidence we have. For both Skywork models, pooled top-10% picks are less often their prompt's best response by GPT-4's rating (59% and 63%, against 71% and 74% without the offsets); for deberta the difference is within noise. We have not trained a policy, and we will say so in the Limitations paragraph. Your other two questions are open, and the paper says so (lines 505 and 509). We will also keep working on the writing. We would be grateful if you would take these points into account in your final recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.