acceptodds
Under review as a conference paper at ICLR 2027

Pointwise or Pairwise: When Do Pairwise Losses Help Reward Learning, Provably?

Abstract

Pairwise losses are increasingly used for reward learning even when pointwise rewards are observed, with mixed empirical results. *When and why do pairwise losses outperform pointwise losses?* We study this question in a grouped offline contextual-bandit setting allowing multiple actions per context, capturing many reward learning scenarios. We compare Value Regression (VR), which regresses observed rewards pointwise, with Value Difference Regression (VDR), which regresses reward differences between a pair of actions sampled under the same context. We consider a semiparametric model where the mean reward is the sum of a learnable action-dependent component and an arbitrary context-dependent yet *action-independent* nuisance, capturing context-specific disturbances. Using a unified localized analysis, we prove finite-sample regression guarantees for finite and linear function classes and translate them into offline-regret bounds. For finite classes, VDR eliminates the misspecification term in the VR bound and improves a reward-scale-dependent error term by averaging over actions within each context, a benefit absent from the corresponding VR term. For linear classes, neither method uniformly dominates: within-context differencing removes nuisance-induced bias but may increase estimation variance relative to using absolute rewards when the misspecification is sufficiently low. This yields a feature geometry-dependent *bias–variance tradeoff*, which we corroborate with numerical experiments.

Primary area:
learning theory
Keywords:
Reward modeling, pairwise regression, offline contextual bandits, semiparametric models, function approximation, bias–variance tradeoffs, grouped data

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jWajOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when learning from numerical reward differences is preferable to regressing absolute rewards. It considers grouped offline contextual-bandit data consisting of NxN_x independent context draws and NaN_a action–reward observations per context. Under the weak realizability assumption r⋆(x,a)=f⋆(x,a)+b⋆(x)r^\star(x,a)=f^\star(x,a)+b^\star(x), where f⋆f^\star belongs to the hypothesis class and b⋆b^\star is an arbitrary action-independent nuisance, the authors compare pointwise Value Regression (VR) with squared-loss Value Difference Regression (VDR).

The paper derives finite-sample prediction guarantees for finite and linear function classes, distinguishing fluctuations governed by NxN_x from those reduced by all N=NxNaN=N_xN_a observations. It translates centered prediction error into coverage-dependent offline regret and gives a finite-class instance in which VR incurs persistent regret while VDR succeeds. The linear analysis incorporates raw and centered feature geometry. Controlled synthetic experiments illustrate a nuisance–variance tradeoff, and additional experiments compare regularized reward models for LLM response selection and preference models for mathematical-reasoning search.

Critical strengths

1. The grouped finite-sample analysis provides a useful distinction between two sources of statistical uncertainty.

The paper explicitly accounts for dependence among observations sharing a context and among pairs formed from the same observations. Its analysis separates fluctuations governed by the number of contexts from those reduced by repeated action observations. The linear results additionally account for the identifiable centered-feature subspace and explain how repeated actions affect empirical-covariance concentration.

These distinctions are consequential because grouped data collection is common in the motivating reward-learning settings. The results clarify why collecting more observations for existing contexts need not have the same statistical effect as observing additional contexts. The treatment of both finite and linear classes, together with the refined centered VR bound, makes this a coherent theoretical contribution.

2. The finite-class separation establishes an actual downstream difference between the estimators.

Theorem 3.2 constructs an instance satisfying weak realizability on which empirical VR followed by greedy selection incurs constant regret with high probability, whereas VDR succeeds on the same instance. This result is stronger than a comparison between upper bounds: it demonstrates that fitting an action-independent nuisance through a restricted model class can distort action rankings.

The controlled synthetic experiments provide complementary evidence. They vary nuisance magnitude and the informativeness of within-context comparisons separately, exhibit both VR-favored and VDR-favored regimes, and report uncertainty across independently generated problem instances. The positive-coverage results support the proposed mechanism without relying solely on the degenerate zero-coverage setting.

Critical weaknesses

1. The central comparative interpretation needs a sharper distinction between intrinsic estimator differences and differences in analytical tightness.

For linear classes, Section 4.3 primarily establishes that separately derived upper bounds are not uniformly ordered. The text acknowledges this with the qualification “upper bound-wise,” but the examples making αC/α0\alpha_C/\alpha_0, or its inverse, large do not by themselves establish corresponding advantages in actual prediction error or regret.

Appendix F.1.3 makes this distinction particularly important. The sharper VR bound has a leading noise term involving deff≤dCd_{\mathrm{eff}}\le d_C, and its decision-relevant population bias is εVR,C\varepsilon_{\mathrm{VR},C}, which can vanish even when the absolute approximation error εaprx\varepsilon_{\mathrm{aprx}} is positive. These quantities should be central to the comparison. In the correctly specified, unregularized, conditionally homoscedastic case, when the relevant centered functionals are estimable, Gauss–Markov gives OLS no larger conditional expected centered squared error than VDR. This special case should be distinguished from the broader heteroscedastic setting allowed by the paper.

A related issue arises in the finite-class reward-scale comparison. Under ordinary realizability, r⋆∈Fr^\star\in\mathcal F, separately concentrating the empirical quadratic term over context blocks and the reward-noise multiplier conditional on the full design yields a VR bound of order

\frac{F^2}{N_x} + \frac{\sigma^2+FR_{\max}}{N} \right] \log\frac{2|\mathcal F|}{\delta}.$$ Thus, the $FR_{\max}/N$ dependence is also available to VR in this special case using the decomposition already employed for VDR. The published VR bound remains valid, and the nuisance-induced separation remains meaningful, but the displayed reward-scale improvement is not established as an independent advantage of pairing. The authors should clarify which differences remain intrinsic under weak realizability and which reflect the sharpness of the analyses. These issues limit the strength of the paper’s answer to its central comparative question, although they do not invalidate the grouped guarantees or Theorem 3.2. **2. The application experiment changes the estimator and does not isolate the mechanism emphasized by the theory.** The response-selection experiment in Appendix H.2 uses ridge-regularized estimators in a 3,072-dimensional feature space, whereas the linear theorems analyze unregularized estimators under lower-isometry conditions. Moreover, VDR uses all pairs only for $N_a\le32$ and switches to $16N_a$ sampled pairs for larger budgets. The consistent advantage across all three reward models is then highlighted from $N_a=64$ onward. An exact centered-ridge implementation would provide a more direct comparison. The paper already gives the identity allowing the all-pairs squared objective to be evaluated through within-context centering, so this check does not require enumerating all pairs. Regularization and effective rank also matter at the smaller budgets. The centered feature matrix has rank at most $N_x(N_a-1)$, which equals 800 and 2,400 for $N_a=2$ and $N_a=4$, respectively. Unless the relevant population rank is correspondingly smaller, these regimes cannot satisfy the unregularized analysis’s invertibility conditions. Reporting effective-rank diagnostics, the regularization search grid, selected penalties, and sensitivity to exact versus sampled pairs would help explain the crossover. The matched samples, prompt-level holdout, separate regularization selection, and paired confidence intervals are positive features of the experiment. However, it does not establish weak realizability or show that nuisance cancellation explains the observed gains. The supplementary MCTS experiment provides limited additional evidence: it uses a binary logistic preference objective and reports gains of one and three solved problems on 100 tests with one search seed. These results should remain explicitly exploratory.
Questions for authors

1. Clarification of the linear comparative claim.

Beyond the finite-class separation in Theorem 3.2, which result establishes an ordering or separation of actual linear-estimator risks, rather than non-ordering of upper bounds? Please explain how Section 4.3 should be interpreted when εVR,C\varepsilon_{\mathrm{VR},C} and deffd_{\mathrm{eff}} from Appendix F.1.3 are used throughout. A direct comparison in an appropriate special case would help distinguish the well-specified homoscedastic setting from the more general noise assumptions.

2. Clarification of the finite-class reward-scale dependence.

In the realizable case, let hf=f−r⋆h_f=f-r^\star and q^f=N−1∑i,jhf(xi,aij)2\widehat q_f=N^{-1}\sum_{i,j}h_f(x_i,a_{ij})^2. Block concentration controls the population quadratic error through q^f\widehat q_f and an F2/NxF^2/N_x term, while conditional concentration controls (2/N)∑i,jhf(xi,aij)ηij(2/N)\sum_{i,j}h_f(x_i,a_{ij})\eta_{ij} through q^f\widehat q_f and (σ2+FRmax⁡)/N(\sigma^2+FR_{\max})/N terms. Empirical optimality then gives the refined VR rate stated above.

How does this refinement change the interpretation of the second finite-class advantage? Under weak realizability, can the remaining context-scale range contribution be expressed through the approximation-residual envelope, thereby separating misspecification effects from observation-noise averaging?

3. Validation of the reported response-selection comparison.

Do the conclusions persist when VDR uses the exact all-pairs centered objective at every budget, with consistent loss normalization and separately selected regularization strengths? Please clarify the regularization grids, the candidate and label budget used for validation regret, and any available effective-rank or regularization-sensitivity diagnostics.

These requests concern the interpretation and validation of the existing results rather than extensions to new applications.

3 comments

We sincerely thank the reviewer for their exceptionally careful review, for recognizing the value of our grouped finite-sample analysis, and for highlighting the regret separation between VR and VDR. We address their points below.

W1 and Q1. Where the linear bounds lose information

Thank you for emphasizing this distinction. We agree that the original geometric examples compare upper bounds and do not establish an ordering of actual risks. The revised §3.3 explicitly separates these comparisons, and Appendix F.3 provides direct risk comparisons for the unregularized linear estimators.

We have also incorporated sharper quantities into the main VR theorem and its interpretation. Bounding centered error through uncentered error loses information about both the population approximation error and the feature geometry. The revised proof works directly in the centered prediction norm, yielding the centered population error εVR,C\varepsilon_{\rm VR,C} and leading noise dependence σ2deff/N\sigma^2d_{\rm eff}/N, with deff≤dCd_{\rm eff}\leq d_{\rm C}. In particular, εVR,C\varepsilon_{\rm VR,C} can vanish even when εaprx>0\varepsilon_{\rm aprx}>0. Appendix F.1.2 gives the corresponding proof. The examples involving α0\alpha_0 and αC\alpha_{\rm C} now explicitly compare the N−2N^{-2} correction terms.

For actual risks, the revised discussion distinguishes three cases:

  • Ordinary realizability and homoskedastic. Conditional on the logged design, suppose the reward noises are independent with common variance and the centered test features are estimable from the within-context differences. Both centered prediction rules are linear in the observed rewards and unbiased. Gauss–Markov therefore gives VR no larger variance, and hence no larger centered risk, than VDR. VR retains information in the block means that VDR discards.

  • Weak realizability. Under the same design and noise conditions, VDR remains unbiased for centered predictions, while VR can incur bias from fitting the baselines. VDR has lower conditional risk exactly when VR’s squared conditional bias exceeds its variance savings. We illustrate this with XX uniform on {−1,1}\{-1,1\}, A=XA=X with probability 3/43/4, ϕ(X,A)=A\phi(X,A)=A, and r⋆(X,A)=θ⋆A−BXr^\star(X,A)=\theta^\star A-BX, where B≥0B\geq0. Consider two context blocks with two observations each: one contains both actions, and the other contains A=XA=X twice. The exact conditional centered risks are

    \qquad \frac{3}{8}\sigma^2\quad\text{for VDR}.$$ Thus, VR has lower risk for $B<\sigma$, and VDR for $B>\sigma$. Appendix F.3.2 also establishes a crossover after averaging over the logged design. For the same population, as $N_x$ grows with $N_a=2$, the limiting greedy regrets are $2\theta^\star$ and zero, respectively, whenever $B/2>\theta^\star>0$.
  • Ordinary realizability and heteroskedastic. The variance ordering can reverse even under ordinary realizability. The example in §3.3 and Appendix F.3.3 gives exact conditional centered risks 9σ2/259\sigma^2/25 for VR and σ2/4\sigma^2/4 for VDR, so VDR has strictly lower risk.

W1 and Q2. The finite-class proof and reward-scale term

Thank you very much for this observation. Through a tighter proof, we have indeed separated misspecification effects from observation-noise averaging and corrected the interpretation of the finite-class comparison.

Let f∘f^\circ be a population VR minimizer, F=sup⁡f∈F∥f∥∞F=\sup_{f\in\mathcal F}\|f\|_\infty, and

=\left(\frac{F^2}{N_x}+\frac{\sigma^2+FR_{\max}}{N}\right) \log\frac{|\mathcal F|}{\delta}.$$ Under weak realizability and $N_a\ge2$, the revised Theorem 2.1 gives, with probability at least $1-\delta$, for any $\eta>0$, $$\mathbb E[(\mathcal C\hat f_N^{\rm VR}-\mathcal C r^\star)^2] \le (1+\eta)\varepsilon_{\rm aprx} +\mathcal{O}\left( (1+\eta^{-1})\mathcal E_\delta +\frac{F\|f^\circ-r^\star\|_\infty}{N_x} \log\frac{|\mathcal F|}{\delta} \right), \quad \mathbb E[(\mathcal C\hat f_N^{\rm VDR}-\mathcal C r^\star)^2] \lesssim \mathcal E_\delta.$$ Thus, **we no longer claim an independent advantage in the observation-noise dependence on $R_{\max}$**: both bounds contain $FR_{\max}/N$. Under ordinary realizability, $\varepsilon_{\rm aprx}$ and $\|f^\circ-r^\star\|_\infty$ vanish, and the bounds have the same order. Under weak realizability, these additional misspecification terms can remain for VR, whereas VDR avoids them by canceling the shared baseline.

The proof improvement is to separate the noiseless excess loss from the reward-noise multiplier before applying concentration. We concentrate the former over the NxN_x context blocks and the latter over the NN observations conditional on the sampled contexts and actions. Empirical optimality and localization then yield the bounds above; revised Appendix D.1 gives the full proof.

We have updated the finite-class statement and discussion in §2 accordingly. These are comparisons of upper bounds; the actual greedy-regret separation is established separately in Theorem 2.2.

W2 and Q3. Exact pairs, ridge, and interpretation

Exact all-pairs objective. We agree that exact within-context centering provides a cleaner comparison. We additionally evaluated exact VDR at every response budget by centering features and rewards within each prompt, avoiding pair subsampling without explicitly enumerating all pairs. The larger-budget conclusion persists: exact VDR achieves lower mean held-out regret than VR for all three reward models at Na=64,128,256,512N_a=64,128,256,512, consistent with our original findings. Figure 1 now shows the exact fit alongside the original VDR results.

To clarify normalization, Appendix H.2 uses summed squared losses, with ridge penalties defined on that scale. Writing uiju_{ij} for a prediction residual and uˉi=Na−1∑juij\bar u_i=N_a^{-1}\sum_j u_{ij}, the exact all-pairs ridge objective satisfies ∑i∑j<k(uij−uik)2+λ∣θ∣22=Na∑i,j(uij−uˉi)2+λ∣θ∣22.\sum_i\sum_{j<k}(u_{ij}-u_{ik})^2+\lambda\\|\theta\\|_2^2=N_a\sum_{i,j}(u_{ij}-\bar u_i)^2+\lambda\\|\theta\\|_2^2. This gives the identical regularized objective without enumerating pairs. Equivalently, using the centered sum without the factor NaN_a requires the penalty λ/Na\lambda/N_a. The grid reported below refers to the summed-loss convention; relative to an averaged loss, the corresponding penalties are λVR/N\lambda_{\rm VR}/N for VR and λVDR/Q\lambda_{\rm VDR}/Q for VDR, where QQ is the total number of pairs used.

Regularization and rank. We agree that the smallest response budgets can prevent direct application of our unregularized analysis. The centered covariance estimated from the full response pool has numerical rank 3,072, whereas the empirical VDR Gram matrices on 800 training prompts have rank at most 800 and 2,400 for Na=2N_a=2 and 44, respectively. This matters since Theorem 3.2 requires lower isometry on the identifiable population subspace.

To examine this limitation, we computed (i) eigenvalue decay, (ii) cumulative spectral energy, and (iii) ridge effective dimension. Despite the full numerical rank, 1,468 and 1,974 directions explain 90% and 95% of the spectral energy, respectively. Both are below 2,400, so at Na=4N_a=4, the rank restriction alone does not rule out capturing most of the feature variation in the final refit. However, these diagnostics do not establish lower isometry or show that the sampled designs actually capture those directions. The rank restriction remains more severe at Na=2N_a=2.

On the same splits and samples, the zero-penalty controls reproduce the tuned original fits’ test regrets and confidence intervals at all four budgets from 64 to 512. At Na=2,4N_a=2,4, however, tuning improves VR substantially and can change the ordering. Thus, regularization affects the comparison at the smallest budgets, while the advantage of the original VDR fits at larger budgets persists without a positive ridge penalty. As noted in Appendix H.2, some smaller-budget runs select the largest tested penalty, so sensitivity to a wider grid remains unresolved.

We have updated Appendix H.2 to report the regularization search grid, the validation-based selection procedure, and the main patterns in the selected penalties. Penalties are selected independently for each reward model, response budget, method—including exact VDR—and random seed, from 0,10−6,10−5,10−4,10−3,10−2,10−1,1\\{0,10^{-6},10^{-5},10^{-4},10^{-3},10^{-2},10^{-1},1\\}. We use 640 prompts for fitting in the inner split, 160 for validation regret, 800 for the refit, and 200 for held-out testing. Validation regret uses all 1,000 labeled candidate responses per validation prompt at every budget. Thus, NaN_a specifies the responses used per prompt for fitting, while validation uses the fixed full candidate pool.

We also believe that our theoretical framework can be naturally extended to ridge regularization by accounting for regularization bias and the corresponding spectral quantities, along the lines of Hsu et al. (2014). Ridge regularization addresses empirical singularity, while the effective dimension curve shows how it reduces the contribution of directions with smaller eigenvalues. Hence, although our current theorems do not directly cover these regularized experiments, we believe that they remain closely connected to the setting we analyze.

Scope of the empirical evidence. The response-selection study compares regularized linear fits on frozen features, while the synthetic study directly examines the role of the additive baseline and comparison geometry. We do not claim that the response-selection results alone establish weak realizability or identify baseline cancellation as the source of the gains. In §3.4, we now describe the MCTS study in Appendix H.3 as exploratory. Specifically, it uses a binary logistic objective and one search seed, with one and three additional solved problems out of 100 for the linear and MLP heads, respectively.

Reviewer_vRgPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on theoretically understanding the performance of value regression (VR) and value-difference regression (VDR) for reward learning. It shows that VDR removes shared context effects but can lose useful information. And they illustrate the tradeoff by synthetic and LLM experiments.

Critical strengths

1)The theoretical analysis is sound, with some new technical contributions.

2)The mechanism of how the expressiveness of the model class influences which approach can train the reward model better is novel to me.

Critical weaknesses

1)Assumption 1 already ensures that reward differences are realizable in the model class, while absolute rewards may be misspecified. Therefore, my concern is that the absence of the model expressiveness term(or misspecification error) in the VDR bound could directly come from this assumption about the model class. In other words, I am worried that it's something that you can directly expect to happen.

2)Though I understand the complexity of the theory, I hope the authors can state their theory even more clearly(or add some intuition for their proofs if possible)

Questions for authors

1)Most importantly, please respond my first concern in the weaknesses.

2)Can you explain why previous works do not solve the case where N_a>=2? For example, do they face some technical difficulty, or is the motivation different?

3)Can you tell me the single most important takeaway(from your perspective) about how your theoretical work can guide the practice of LLM reward model training?

3 comments

We sincerely thank the reviewer for their careful review, for recognizing the technical contributions of our analysis, and for finding our characterization of the role of model expressiveness novel. We address their points below.

W1 and Q1. What follows from Assumption 1

We agree that the absence of an approximation-error term in the VDR bound is expected under Assumption 1, which ensures exact realizability of within-context reward gaps.

Our main question is when pairwise regression can provably outperform pointwise regression, even when pointwise rewards are available. Instead of throwing in speculations about how it may happen, we have identified a set of conditions (including our model assumption) under which pairwise can outperform pointwise. This itself is a meaningful theoretical contribution: setting aside whether our condition is true or not, we have verified that our assumption implies that pairwise loss can provably beat pointwise in a mathematically rigorous manner. This is a valuable learning theoretic advance.

Now, is weak realizability too strong? You might be worried that when weak realizability is false, then there is no difference between VDR and VR due to the misspecification. This is not true in general. Even if the weak realizability is false, the degree of misspecification (i.e., non-realizability) can be significantly different, and our theory extends naturally -- please see Remark 1 in our updated pdf.

So, regardless of weak realizability being true or not, what matters for VDR vs VR is often the strength of the context-dependent term. Figure 1 shows the strength of VDR and we believe this reflect that there exists a nontrivial context-dependent term in our real-world experiment. To see this, note that one can show that solving VDR is equivalent to solving arg⁡min⁡θmin⁡{αx}∑x∑i=1Nr(rxi−θ⊤ϕ(x,i)−αx)2+λ∥θ∥22\arg \min_{\theta} \min_{\{\alpha_x\}} \sum_{x} \sum_{i=1}^{N_r} (r_{xi} - \theta^\top \phi(x,i) - \alpha_x)^2 + \lambda \|\theta\|^2_2 Looking at the objective function, if the context-dependent term is zero, it is highly unlikely that this method will be better than VR. We will add details and the discussion in the final version.

W2. Intuition for the analysis

Thank you for this suggestion. We have reorganized the main text to emphasize the interpretation of the results and added a dedicated Proof Sketch (§4), moving the formal framework to Appendix C. We also expanded the explanations and examples accompanying the bounds in §§2–3, and significantly expanded on the proof intuition in the newly written §4, which we describe here.

The central intuition is to keep the two sampling scales separate. Variation between context means averages over NxN_x independent contexts, whereas reward noise, conditional on the sampled contexts and actions, averages over all N=NxNaN=N_xN_a observations. Although VDR uses dependent pairs, within-context centering rewrites its noise term using the original NN reward noises, preserving this averaging benefit. VDR also cancels the shared baseline, while fitting that baseline can introduce approximation error and additional context-level fluctuations for VR.

Our localized analysis controls these fluctuations near the population fit, exploiting the quadratic structure of squared loss to relate their magnitude to the prediction error. Solving the resulting fixed-point inequality yields fast 1/N1/N-type estimation terms, alongside the separate 1/Nx1/N_x contributions from context sampling. For finite classes, we use scalar concentration and a union bound. For linear classes, the normal equations and feature covariances further allow us to track estimation error specifically in the directions relevant to reward gaps.

Q2. Why the grouped case differs from prior work

Thank you for asking us to clarify. Both the learning problem and the analysis differ, although our approach builds on tools from these works.

Appendix A discusses classical least squares and semiparametric methods. Hsu et al. (2014) analyze least squares with independent covariate–response observations. Semiparametric contextual bandits (Krishnamurthy et al. (2018)) and treatment-effect oracles (Carranza et al. (2023)) study contextual bandits with single-outcome feedback per context occurrence and remove action-independent baseline effects using orthogonalization or outcome residualization. Zhu et al. (2023) study reward estimation and policy learning from pairwise or KK-wise preference feedback under Bradley–Terry and Plackett–Luce models. Ji et al. (2023) provide a theoretical analysis of numerical rating-based and preference-based pessimistic offline policy learning.

These works consider different sampling schemes or feedback modalities, from numerical rewards to preference labels. Our question is instead how to learn from the same grouped numerical reward observations: given NxN_x contexts and Na≥2N_a\ge 2 action–reward observations per context, how do VR and VDR compare? Thus, the distinction is not simply whether Na≥2N_a\ge 2, since prior preference-learning work already considers multiple responses per context, but how the choice of regression objective affects learning from the same numerical data.

The grouped setting also introduces a technical issue: observations sharing a context are generally dependent, and constructed pairs reuse rewards. Our analysis separates context-level fluctuations, which average over NxN_x contexts, from conditional reward noise, which averages over all N=NxNaN=N_xN_a observations. We provide policy-regret guarantees that distinguish the roles of NxN_x and NaN_a and characterize how model misspecification affects this comparison. The purple edits in §§1.1 and 2 bring this distinction into the main text, while the new §4 explains the proof intuition and Appendix A provides the detailed comparison with prior work.

Q3. Implication for practical LLM reward-model training

The single most important takeaway is to match the training objective to both prompt-specific nuisance effects and the information retained by within-prompt comparisons. VDR can be preferable when rewards contain substantial prompt-specific, response-independent effects that are difficult to model, provided that response differences remain informative along the directions needed to rank responses. VR can be preferable when such effects are small or when differencing leaves those directions poorly observed. Thus, high overall within-prompt variability alone is insufficient: the variability must occur along directions that affect response selection.

The controlled experiment in Appendix H.1 illustrates this tradeoff. In the crossover experiment, reward-relevant comparisons require actions from opposite faces of the cube. The parameter δ\delta controls how often the less frequent face is sampled: small δ\delta makes informative cross-face comparisons rare, disadvantaging VDR. The parameter β\beta controls the magnitude of an action-independent context effect, which can bias VR but cancels from every VDR target. Figure 2 shows the resulting crossover: VR is favored when informative comparisons are scarce and the nuisance is small, while increasing the nuisance can make VDR preferable. The accompanying alignment experiment further shows that the usefulness of within-prompt variation depends on its alignment with the reward-relevant directions, beyond its total amount.

When this structure is a-priori unknown, a practical approach is to train a family of mixed objectives,

+\alpha\widehat L_{\mathrm{VDR}},$$ with the losses on a common scale, and select $\alpha$ using prompts excluded from training. When numerical rewards are available for the validation candidates, one can use response-selection regret within that candidate pool. Alternatively, when numerical reward differences are available, a within-prompt squared difference loss provides a nuisance-invariant surrogate, although it cannot assess reward directions absent from the observed comparisons.

Thanks for your efforts and clarification. I have updated my score to reflect my positive attitude towards your paper and explanation.

Reviewer_DhyhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper compares pointwise (VR) and pairwise (VDR) reward regression in offline contextual bandits. Assuming realizable action gaps (and an arbitrary context baseline), VDR converges to the optimal policy while VR suffers the constant misspecification regret. For linear classes, VDR removes baseline bias but can cost information when the logged actions have little within-context variation. VDR overtakes VR as baseline bias grows in the synthetic bandit and achieves lower selection regret on all three LLM reward models at 64 or more training responses per prompt.

Critical strengths

The paper gives a detailed finite-sample comparison of pointwise and pairwise reward regression for finite and linear classes. The upper bounds show which terms improve with more actions per context. Theorem 3.2 gives a clean example where VR incurs constant regret and VDR recovers the optimal policy. For finite classes with a fixed number of logged actions per context, the agnostic minimax lower bound matches the leading approximation and statistical terms in the regret upper bound.

Critical weaknesses

While it is a weaker assumption, it’s unclear why realizability of reward differences is much more realistic in practice than realizability of rewards. Assumption 1 allows a prompt-specific offset, but one scorer still has to recover ALL within-prompt gaps. This is the main weakness in my opinion, because it is unclear how practically useful this comparison between VR and VDR is; this is why I give a 2 and not a 3 in informativeness.

In terms of presentation I also found the paper to be very algebra heavy and hard to parse. Moreover, the experiment section, especially the LLM section, is not very self-contained and requires many passes with the appendix to make sense of key details. This is why I give a 2 and not a 3 for presentation.

Questions for authors
  1. In Theorem 3.1, suppose the best scorer makes a small but nonzero error on within-context reward gaps. How would VDR handle that misspecification error? In this regime, how would VDR compare to VR?
  2. In Fig. 1’s LLM experiment, how well do the fitted models predict reward gaps at 512 training responses per prompt? On the held-out prompts, what fraction of VR’s squared error comes from prompt offsets, and how much within-prompt reward variance do VR and VDR each explain?
2 comments

We sincerely thank the reviewer for their careful review, for recognizing the technical contributions of our analysis. We address their points below.

W1. Motivation and scope of weak realizability

Requiring scorer to recover all within-prompt reward gaps is not a weakness or being restrictive because the gaps are not independent! They are all coupled through the linear model, so the difficulty of learning is related to the size of the hypothesis class (dd-dimensional vector for the linear case) rather than the number of response pairs. As our theoretical results reveals, VDR does not lead to a worse bound in general (though it depends on the problem at hand for the linear class). Note that VDR does not incur larger computational complexity either in learning due to the trick we mentioned at the end of Section 1.1.

Moreover, even if we remove the realizability of within-context reward gap, we can still show that misspecification under reward gap is less or equal than misspecification under reward level. We added some details in Remark 1, §2.

Weak realizability helps characterize when fitting action-independent reward components with a restricted model class can lead to suboptimal decision-making. Our motivation is consistent with existing work on treatment-effect learning, which separates the estimation of treatment effects from outcome components (Nie and Wager (2021)), including in contextual bandits (Carranza et al. (2023)), which focuses on reward differences relevant to action selection. A related motivation in LLM reward modeling is that a prompt-only shift preserves both response rankings and Bradley–Terry preference probabilities (Rafailov et al. (2023), §5.1). More details about related work is included in Appendix A.

Our separation, however, concerns the two squared-loss ERMs trained on the same grouped numerical-reward data. We use VR and VDR as theoretical testbeds for two ways of using the data. The purple edits in §1.1 connect Assumption 1 to the prior work and state the same-data comparison explicitly.

W2. Presentation and experimental setup

Thank you for this feedback. We agree that the original presentation was too algebra-heavy and that the LLM experimental discussion required too much consultation of the appendix. Given the paper’s theoretical focus and the main-text space limit, we have prioritized clearer explanations of the results and proof ideas in this revision:

  • Separating intuition from technical details. The formal localized-analysis framework is presented in Appendix C, with the detailed proofs of the finite-class error bounds in Appendix D. Section 4 provides a proof roadmap explaining why context-level fluctuations average over NxN_x independent blocks, while reward-noise terms benefit from all N=NxNaN=N_xN_a observations, and how the finite- and linear-class arguments differ.
  • Explaining the results through concrete examples. In §2, we walk through a two-context example showing how a restricted VR model can absorb context-dependent baselines into action predictions and reverse the correct ranking, while VDR cancels those baselines through within-context differences. Section 3 explains the error terms and feature geometry. Section 3.3 explicitly distinguishes comparisons of upper bounds from comparisons of actual risks and gives a scalar example in which increasing the baseline magnitude reverses the ordering of the estimators’ conditional prediction risks.

For the LLM experiment, §3.4 describes the response-selection task, frozen Llama-3.2-3B-Instruct representations and regularized linear scorers, matched training data for both methods, the split into 800 training and 200 held-out prompts, and the three reward models evaluated across five seeds. It also specifies the range of labeled responses per training prompt. These setup details were already present in the submission.

To make the main text more self-contained, we have now moved key evaluation and validation details from Appendix H.2 into §3.4. These clarify that each scorer selects from all 1,000 cached responses per prompt, define regret relative to the best reward-model score in that pool, and describe that the ridge penalty is selected per setting. The full fitting and evaluation protocol remains in Appendix H.2. We would be happy to incorporate further suggestions for making the main-text experimental discussion clearer and more self-contained.

Q1. Bounds for both methods under gap misspecification

Yes. Our analysis extends to nonzero gap misspecification. Let C\mathcal C denote within-context centering and define

:=\min_{f\in\mathcal F}\mathbb E[(\mathcal C(f-r^\star))^2] \leq \varepsilon_{\rm aprx}.$$ The finite-class bounds have the same structure: for any $\eta>0$, the approximation contribution is $(1+\eta)\varepsilon_{\rm aprx}$ for VR and $(1+\eta)\varepsilon_{\rm gap}$ for VDR, with estimation terms depending on their respective residuals. These bounds appear in Theorem 2.1 and the new Remark 1, with the gap-misspecification proof in Appendix D.3. VDR still cancels the action-independent nuisance exactly, but misspecification in within-context reward gaps remains, as captured by $\varepsilon_{\rm gap}$. At the population level, VDR minimizes the centered prediction error and achieves $\varepsilon_{\rm gap}$, which is no larger than the centered error of the population VR minimizer. Thus, VDR can retain an approximation advantage when reward gaps are easier to approximate than absolute rewards. However, this does not guarantee smaller finite-sample error, since the estimation terms for VR and VDR also depend on $\\| f^\circ_{\rm VR}-r^\star \\|_\infty$ and $\\| \mathcal C(f^\circ_{\rm VDR}-r^\star) \\|_\infty$, respectively. We provide a more detailed comparison in Appendix D.3. ## Q2. Held-out gap prediction at $N_a=512$ To answer the reviewer's question, we evaluated reward gap prediction, the contribution of prompt offsets to VR's squared error, and the reward variance explained within each prompt at $N_a=512$. Specifically, we used $P=200$ held-out prompts with $K=1000$ candidate responses per prompt. For each held-out prompt $i$ and candidate response $j$, let $r_{ij}$ denote the reward model score and $\hat{r}_{ij}$ the model prediction. Define the prediction error and its mean within each prompt: $$e_{ij}=\hat{r}_{ij} - r_{ij}, \qquad \bar{e}_i = \frac{1}{K}\sum_{j=1}^{K}e_{ij}.$$ The total prediction error then decomposes exactly as: $$\underbrace{\frac{1}{PK}\sum_{i,j}e_{ij}^{2}}_{\text{total MSE}} = \underbrace{\frac{1}{P}\sum_i\bar{e}_i^{2}}_{\text{prompt-offset MSE}} + \underbrace{\frac{1}{PK}\sum_{i,j}(e_{ij} - \bar{e}_i)^{2}}_{\text{within-prompt MSE}}.$$ Since $\bar{e}_i = \overline{\hat{r}}_i-\bar r_i$, the prompt-offset component measures errors in predicted prompt means. We report its fraction of VR's total squared error: $$F_{\mathrm{offset}} := \frac{\frac{1}{P}\sum_i\bar e_i^2}{\frac{1}{PK}\sum_{i,j}e_{ij}^2}.$$ To measure within-prompt variance explained, we first center predictions and reward model scores separately within each prompt: $$\tilde r_{ij}=r_{ij}-\bar r_i, \qquad \widetilde{\hat r}_{ij}=\hat r_{ij}-\overline{\hat r}_i.$$ This removes differences in prompt means while preserving all reward gaps between responses. We then compute the pooled within-prompt variance explained: $$R_{\mathrm{within}}^2 := 1 - \frac{\sum_{i,j}(\widetilde{\hat r}_{ij}-\tilde r_{ij})^2}{\sum_{i,j}\tilde r_{ij}^{\,2}}.$$ Here, $R_{\mathrm{within}}^2=0$ means the same squared error as predicting no within-prompt variation; a negative value indicates worse performance than this baseline. We also report reward gap RMSE: $$\mathrm{RMSE}_{\mathrm{gap}} := \sqrt{\frac{1}{P\binom{K}{2}}\sum_{i=1}^{P}\sum_{j<k}\left[(\hat r_{ij}-\hat r_{ik})-(r_{ij}-r_{ik})\right]^2}.$$ Here reward gap RMSE directly evaluates predictions of differences between responses: | Reward model | $\mathrm{RMSE}_{\mathrm{gap}}$ of VR | $\mathrm{RMSE}_{\mathrm{gap}}$ of VDR | VR error due to prompt-mean errors, $F_{\mathrm{offset}}$ | Within-prompt variance explained $R_{\mathrm{within}}^2$ of VR | Within-prompt variance explained $R_{\mathrm{within}}^2$ of VDR | |---|---:|---:|---:|---:|---:| | Llama | 6.0632 | 5.6751 | 67.1% | 11.5% | 22.5% | | Qwen3 | 3.5243 | 3.2621 | 69.9% | 17.3% | 29.1% | | ArmoRM | 0.0237 | 0.0188 | 80.4% | −13.5% | 28.8% | Metrics are computed separately for each split seed and averaged over five seeds. Each split contains 200 held-out prompts with 1,000 candidate responses per prompt. Gap RMSE evaluates all distinct response pairs within each prompt, and its units differ across reward models. According to the attached table, errors in prompt means account for 67.1-80.4% of VR's total squared prediction error. Even after removing prompt means, VDR achieves lower reward gap RMSE and explains more reward variance within each prompt across all three reward models. Therefore, VDR's advantage seems also evident in predicting reward differences, beyond differences in predicted prompt means.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.