- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the statistical structure of RLHF preference data through its comparison graph. For a fixed feature map , the authors define the feature-weighted comparison Laplacian and argue that three spectral quantities provide a pre-training diagnostic: characterizes unidentifiable reward directions, controls the difficulty of estimating the weakest identifiable direction, and feature effective resistance measures response-specific scoreability. The theoretical development includes an identifiability characterization, a prompt-level gauge result for datasets containing only within-prompt comparisons, pointwise and global error bounds, and several minimax lower-bound constructions. The paper further discusses which RLHF computations are invariant to prompt-dependent offsets and which operations, such as pooled filtering or reporting mean reward, may depend on them. The empirical study includes controlled Bradley–Terry experiments, comparisons across graph topologies at fixed label budgets, tests with several reward-head classes, audits of five public preference datasets and one synthetic control, prompt-gauge perturbation experiments, and a spectral comparison-selection procedure called SPEC. The experiments generally support the structural identifiability claims and show that topology can substantially affect recovery. The paper also reports useful negative results: the theoretically certified log-determinant objective is not the strongest empirical selector, uncertainty sampling can underperform random selection, and maximizing the weakest eigenvalue can be inappropriate for dataset repair.
Critical strengths
The mapping from the null space, the smallest positive eigenvalue, and effective resistance to identifiability, global conditioning, and response-level reliability is conceptually clean. The prompt-level gauge analysis also identifies a consequential distinction between within-prompt RLHF decisions and computations that place different prompts on a common reward scale.
Critical weaknesses
The central statistical guarantees are not currently stated on a well-defined finite-sample parameter and estimator class. Theorem 3 defines as a maximizer of the unregularized Bradley–Terry likelihood, but finite preference samples can be separable, in which case a finite MLE need not exist. Bounded true margins alone do not rule out sample separation. Moreover, the proof invokes a self-concordant linearization with an unspecified lower-order remainder, without stating the sample-size, leverage, or local-curvature conditions needed to control it. The displayed theorem nevertheless gives an unconditional finite-sample bound without this remainder. Theorem 4 has a related issue: if the supremum is over all of , logistic saturation makes the global minimax Euclidean risk infinite even for full-rank designs. Consequently, the finite lower bounds do not by themselves justify the claimed minimax floor unless an explicit bounded parameter class is introduced and the dependence on that bound is analyzed. These issues concern two of the paper’s main theoretical contributions.
The evidence for practical downstream harm remains indirect. The large nullities in Table 2 are mostly forced by the chosen rank-binding operating point , as the appendix acknowledges; at the production-scale operating point, the matrices are full rank and the argument becomes one about weak rather than unidentifiable directions. The real-corpus resistance experiment uses planted Bradley–Terry rewards, while the gauge experiment deliberately perturbs a fitted reward inside an exact or weak eigenspace. These experiments establish possibility, but they do not show that a normally trained end-to-end reward model has a substantial head projection or uncertainty in those directions. Likewise, the RewardBench selection experiment uses frozen representations and operates far below trained-model accuracy. Thus, the diagnostic may be useful, but the prevalence and magnitude of the claimed deployment consequences are not yet established.
Questions for authors
Could the authors restate Theorem 3 for a precisely defined estimator and give the conditions under which that estimator exists? In particular, how is sample separation excluded for the unregularized MLE, and what sample-size, leverage, or curvature condition controls the “lower-order” remainder used in the proof? If the experiments use a constrained or ridge-regularized estimator, does the stated resistance bound hold for that estimator?
What is the parameter class in Theorem 4? If the supremum is over all , the minimax Euclidean risk of a finite logistic experiment appears to be infinite even when is full rank, because arbitrarily large parameters along a fixed sign cone generate nearly indistinguishable saturated observations. Can the authors provide a bounded-margin or bounded-diameter minimax statement and show how the lower bound depends jointly on this radius and the spectrum of ?
For the two trained reward-model representations, could the authors report the norm of the learned scalar head projected onto the prompt-constant null space or the weakest eigenspace, together with the resulting change in actual cross-prompt decisions? This would distinguish the existence of an ambiguous direction from whether a normally trained reward model materially uses that direction.
Could the authors more sharply distinguish the new theoretical contribution from existing topology-aware Bradley–Terry bounds, covariate-assisted ranking theory, and recent work on what pairwise preference learning recovers? In particular, comparison with Shah et al. (2015), Fan et al. (2024), Saha and Rajkumar (2024), and Pukdee et al. (2026) would materially affect my assessment of novelty.
4 comments
Response to Reviewer Fr51 (2/2)
The revision adds F5 (lines 451 to 467, Appendices R and U), with the head projection you asked for in Q3.
Q3. A reward model with a scalar head is linear in the head's input , so with we built from 1,987 of F3's 2,000 within-prompt UltraFeedback comparisons for deberta and two Skywork-Reward-V2 decoders, at 0.6B and 1.7B. internlm2 (submitted Section 5.1) is now in Appendix U, post hoc, checked against its published RewardBench accuracies instead of its model card. Table 11 (line 2171) and Appendix U give the head's share in the weakest eigendirections:
- deberta (, full rank): 0.11, 0.21, 0.37, against 0.125, 0.25, 0.50 by chance;
- Skywork 0.6B (): 0.12, 0.26, 0.47, same chance values;
- Skywork 1.7B (, more than the comparisons): 0.168 of the head in the forced 61-dimensional prompt-constant null space, against 0.173 by chance.
- internlm2 (; Appendix U): 0.03, 0.05, 0.10 against 0.09, 0.18, 0.35, and 0.051 in its 62-dimensional null space against 0.174.
Removing these components moves the reward by at most 1.6% of its standard deviation and leaves every pooled top-10% selection at Jaccard 0.975 or more. In all four models the head's weight on the directions this dataset leaves weak is near or below chance, so the answer to Q3 is negative.
On the offsets the graph does predict something. No comparison crosses prompts, so each model sets them by extrapolation, and two models can differ there without disagreeing on any within-prompt decision. A full-rank spectrum cannot show this; F5 measures it in function space. On all four rated completions of 1,835 prompts, the prompt accounts for 0.73, 0.40 and 0.34 of the models' score variance, against 0.29 for GPT-4's ratings (paired differences +0.05 to +0.44, all above zero). Removing the offsets lowers the mean GPT-4 rating of a pooled top-10% selection for every model (deberta: 7.85 to 6.97), so they carry signal. Yet for both Skywork models they lower within-prompt pick quality (Appendix U; deberta is mixed). A model given another family's offsets, rescaled, keeps every within-prompt decision, but its pooled selection overlaps the original only at Jaccard 0.32 to 0.37. With their own offsets, the selections cover 334 to 487 prompts against 573 under GPT-4's ratings, and the median pick runs 1,900 to 2,552 characters against 1,179 (Table 3). All of this is post hoc and on one corpus (three models here, a fourth from a third family in Appendix U); no policy was trained.
Second weakness. F4's perturbation shows only possibility, as you note. Section 5.3 now calls cross-component differences unidentifiable only along directions constant within each component, and the submitted "8 to 35× spread in the error floor" is a 2.9× to 5.9× spread in a per-response upper bound. F3 uses planted rewards, as the corpus has no true reward, so the paper calls its result an empirical regularity (rank correlation +0.17 to +0.36). The selection results are claims about conditioning on frozen representations. The synthetic control in your summary was our error: the sixth dataset is the public argilla/distilabel-intel-orca-dpo-pairs, labelled by a language model (Appendix I).
Q4. Section 2 (lines 108 to 142) covers all four. Shah et al. (2015; JMLR 2016) prove item-level minimax bounds over a bounded class and compare topologies; Theorems 3(i) and 4 carry their argument to feature designs. Fan et al. (2024, Lemma 15) analyse for ; our linear reward is their model without per-item scores. Saha and Rajkumar (2024) bound error through the smallest nonzero eigenvalue of a feature-weighted Laplacian and show that features can make a disconnected graph recoverable. Pukdee et al. (2026) define a connectivity degree that controls within-prompt margin error and compute a diagnostic on real data, three of our corpora included. We no longer present or the bounding argument as new. We add a per-direction reading of which reward differences a within-prompt design leaves undetermined under a given feature map, per-response resistance in feature space, and the RLHF accounting and measurements of Sections 4.2 and 5.4. A single eigenvalue cannot name the missing direction: in Section 5.5's planted instance, Corollary 2 names the missing axis, eight random cross-prompt pairs lift cross-prompt accuracy from 0.633 to 0.974, and eight chosen to maximise give 0.935 (lines 1790 to 1797). On the audited data the graph adds to rank, and leverage only on SHP (line 414).
Do the restated theorems and F5 answer both of your concerns?
Thank you for the thorough response and additional theorem statements, which largely address my concerns. I am therefore raising my score to 3 and encourage the authors to incorporate these additions into the revisions.
Thank you
Thank you for reading our response and for updating your review. The restated Theorems 3 and 4 are already in the revised PDF (lines 260 to 276 and 302 to 319, with proofs at lines 864 to 1228), and they will stay in the final version, along with the other additions described in our response. If any concern is still open, we are glad to answer it before the discussion closes.
Response to Reviewer Fr51 (1/2)
Thank you for checking the theory so closely. Both soundness points were correct. The submitted Theorem 3 assumed an MLE that need not exist and dropped a remainder without conditions, and Theorem 4 took suprema over Rd, where the risk is infinite. Both theorems are now stated over a bounded-margin class (lines 260 to 319, proofs in Appendix A).
Let Ln=∑enexexe⊤ have rank r and smallest nonzero eigenvalue λmin, let κB=σ′(B) and ΘB={θ∈row(Ln):∣⟨θ,xe⟩∣≤B for all e}. The likelihood maximiser θ^B over this compact polytope exists and is unique, as the likelihood is strictly concave on row(Ln). We bound margins rather than the norm because on ΘB the Hessian is at least κBLn. Here θ⋆ is the true parameter projected onto row(Ln), Rv the resistance of response v, and cv the mean of r^−r⋆ over its component.
Theorem 3(i) holds at every sample size. Let εδ=(r+2log(1/δ))/(2κB). On one event of probability at least 1−δ, which does not depend on B, for every B with θ⋆∈ΘB and every v,
∣r^(v)−r⋆(v)−cv∣≤Rvεδ,∥θ^B−θ⋆∥Ln≤εδ.
Dividing by λmin gives the Euclidean bound, and for fixed B both hold in expectation with r/(2κB). The proof of (i) uses no Taylor expansion, so there is no remainder. Optimality and curvature give κB∥Δ∥Ln≤∥g∥Ln+ for the error Δ and the score g, and Hsu, Kakade and Zhang (2012) bound ∥g∥Ln+.
Theorem 3(ii) is about the MLE. Let ρn=maxexe⊤Ln+xe, let θ⋆∈ΘB′ with B′<B, and let η=ρnεδ/2<min{1,B−B′}. With probability at least 1−δ no constraint is active, so the MLE exists and equals θ^B, and
∣r^(v)−r⋆(v)−cv∣≤κB′Rv[2κB′logδ2∣V∣+32ρnlogδ4∣V∣+21(eη−1)(r+2logδ2)].
The first term is the submitted bound. The other two are the remainder it left implicit.
Q1. For the MLE, separation is excluded only on the event of (ii), and the condition on η is the leverage and curvature condition you asked for. Since ρn≥r/n, it needs n>r(r+2log(2/δ))2/(4κB2). An LP check (Albert and Anderson, 1984) finds no quasi-separation in any of F3's 36 one-comparison-per-edge fits (line 289). Those fits use a ridge τ=10−8, and yes, (i) covers it: with B the largest fitted or planted margin, the bound holds with εδ+τ∥θ⋆∥Ln+/κB (line 1026). Table 2 needs no fit, since its operating point has no finite MLE. No experiment depends on either part or tests their constants: at our sizes (i) is loose to vacuous and the condition of (ii) is never met. F3 checks only whether errors are ordered by resistance, as the form of the bound suggests.
Q2. The class is ΘB, built from the realised design. Lower bounds on infθ^supΘBE∥θ^−θ⋆∥2:
The radius enters (a) to (c) through factors that stop growing once B passes a constant. Part (e) grows as eB/2 for moderate B when c1=1, but it beats (a) only if B>3.2 and λmin>3.34m12, and for large B the risk is linear in B. With (i), for B≥2 the minimax risk lies between 0.457/λmin and r/(2κBλmin). Both scale as λmin−1/2, but their ratio is only bounded by 4.4reB, so we claim no matching rate.