acceptodds
Under review as a conference paper at ICLR 2027

Learning Reward Distributions from Implicit Feedback with Sparse Expert Supervision

Abstract

Reward modeling is a key component of reinforcement learning from human feedback. Existing methods mainly rely on explicitly collected expert preferences, which provide high-quality supervision but are costly to obtain. In contrast, implicit feedback is abundant in real-world systems, but it is noisy and thus cannot be directly treated as expert supervision. Moreover, conventional reward models typically represent human preferences with a single expected reward, limiting their ability to capture complex variation in expert assessments. To address these challenges, we propose Reward Conditional Mean Embedding (RCME), a kernel-based framework that learns conditional expert-reward distributions from sparse expert preferences and abundant implicit feedback. Equipped with a characteristic kernel, RCME uniquely characterizes the conditional reward distribution and can be reused for different downstream reward utilities without retraining. We establish an identification result for RCME and develop a doubly robust estimator, which further show that the learning loss is Neyman-orthogonal to nuisance-estimation errors. For downstream policy optimization, we propose Group Kernel Relative Policy Optimization (GKRPO), which uses a kernel alignment score relative to a maximum-reward anchor and recovers mean-reward GRPO as a special case. We demonstrate the utility of our framework on synthetic datasets, PKU-Safety, and a real-world e-commerce application.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reward Modeling, Conditional mean embedding, RLHF, Implicit feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t82FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The authors study reward modeling when expert labels are sparse and selectively observed, while noisy implicit feedback is available for every sample. They propose the Reward Conditional Mean Embedding (RCME), which represents the conditional distribution of expert rewards in an RKHS. Assuming that expert review is missing at random given the observed features and the implicit feedback, they identify the RCME by regressing a nuisance embedding on the prompt and response, and estimate it with a cross-fitted doubly robust pseudo-outcome that is shown to be Neyman-orthogonal. For policy optimization, they propose GKRPO which scores each response by the kernel similarity between its learned reward embedding and the feature of the maximum reward, and uses this score as the reward in group relative policy optimization (GRPO). Experiments cover a synthetic heteroskedastic design, a semi-synthetic PKU-SafeRLHF benchmark with downstream safety alignment, and an e-commerce inquiry generation task.

Critical strengths

The problem setting is practically relevant, and the doubly robust correction is consistently supported. Combining sparse expert supervision with abundant but biased implicit feedback is common in deployed systems, and the authors formalize it cleanly under a missing at random assumption. Across various reward noise distributions and the label rate sweep in Figure 1, the orthogonal estimator improves over training on expert labels alone (PE) and over the naive proxy baselines set up by the authors. The same gap between PE and the authors' method carries over to PKU-SafeRLHF in Tables 1 and 3.

Critical weaknesses
  1. Once the GKRPO score is written out, the reproducing property gives q(x)=E[u(R)∣X=x]q(x) = \mathbb{E}[u(R) \mid X = x] with u(r)=kR(r,Rmax⁡)u(r) = k_R(r, R_{\max}). In other words, the learned distribution is collapsed into a single expected utility before the policy update, and what follows is a standard GRPO step on a transformed reward. I could not find what the distributional view contributes at the policy optimization stage beyond the choice of uu.

  2. With the ordered Li–Racine kernel, λ=0.5\lambda = 0.5, and anchor r=3r = 3 used in the PKU experiments, the utility is u(r)=2r−3u(r) = 2^{r-3}, so the comparison between the authors' CME and Mean variants is really a comparison between E[2R∣X]\mathbb{E}[2^{R} \mid X] and E[R∣X]\mathbb{E}[R \mid X]. The first puts most of its weight on P(R=3∣X)P(R = 3 \mid X), and since the safety benchmarks are scored by a binary judge, I would expect it to do better even without any distributional modeling. Given this, my main concern is whether the learned embedding can deliver results for a different kernel or anchor without retraining, or at least at a small computational cost, since this is where it would have a real advantage over a scalar reward model. I would be willing to raise my score depending on the authors' explanation of this point.

  3. Remark 4.6 acknowledges that this scalar could be estimated directly by applying the same doubly robust construction to kR(R,Rmax⁡)k_R(R, R_{\max}). The remaining argument for RCME is that one embedding can be reused across many utilities without retraining, but every experiment uses a single kernel and a single anchor. So the main advantage the paper claims for keeping the full distribution is never actually tested, and I do not think RQ4 is supported beyond what a utility transform would give.

  4. When both nuisance models are misspecified in Table 2, the bias of the CME estimator is much larger than that of the mean estimator (for example, 6.84 versus 0.99 under Gaussian noise). The text states that the bias stays close to zero under the other specifications but does not discuss this row.

Minor Issues

  • Corollary 4.5 uses μ(Z)\mu(Z) without definition in the main text.
  • The Li–Racine kernel should be defined and cited.
  • The italic labels next to (RQ1) to (RQ4), such as "Reward recovery" and "Robustness", do not reflect the research questions well and could be removed.
  • Line 040: "whose annotations is costly" →\rightarrow "whose annotation is costly"
  • Line 120: "the distribution of R" →\rightarrow "the distribution of V"
  • Line 323: "the bias of the our method" →\rightarrow "the bias of our method"
Questions for authors
  1. Could the authors report a baseline that applies the scalar doubly robust estimator of Corollary 4.5 to kR(R,3)=0.53−Rk_R(R, 3) = 0.5^{3-R} and trains GRPO on it?

  2. Can one learned RCME be evaluated under several kernels or anchors (for example, risk averse and risk seeking utilities) without retraining the reward model? If so, how do the resulting policies and the total computational cost compare against retraining a scalar reward model for each utility?

  3. In Table 2, are the biases of the mean and CME estimators measured on a comparable scale? If so, what explains the much larger degradation of the CME estimator when both nuisances are misspecified, and how should practitioners interpret it?

Rebuttal update

After carefully reading the author's response and the revised manuscript, my initial concerns have been mostly addressed. Therefore, I have raised my Informativeness score from 1 to 2 and my overall rating from 2 to 3.

5 comments
AuthorsOpenReview ↗

Response to Reviewer t82F (part 1)

Thank you for the positive assessment and valuable suggestions about our paper. We address your questions below. We will correct typos and make the number of lemmas consistent in the updated version.

W1. The RCME score reduces to an expected utility followed by standard GRPO. What does the distributional view contribute beyond the choice of utility?

Response to Comment. The contribution of the distributional perspective lies in learning a reusable conditional reward representation rather than modifying the GRPO update itself. A single frozen RCME can support different downstream utilities without retraining, allowing different aspects of the conditional reward distribution to influence policy optimization. For the ordered Li–Racine kernel used in PKU, the score gives the largest weight to P(R=3∣X=x)P(R=3\mid X=x), emphasizing the safest reward grade. Thus, responses with identical conditional means but different reward distributions can receive different scores. As detailed in our response to Comment 2, we further provide a matched Scalar-DR baseline and demonstrate that the same frozen RCME supports policy training with both an alternative anchor-based readout and a Gaussian-kernel utility, validating its flexibility across different reward preferences.

W2, Q1, Q2. With u(r)=2r−3u(r)=2^{r-3}, the gains may reflect alignment with binary safety evaluation. Can the same embedding support different kernels or anchors without retraining, and at what computational cost?

Response to Comment. We conducted three additional experiments on PKU to address the reviewer's concern.

(a) Direct scalar regression. We fit a Scalar-DR reward model directly to the original utility u+(R)=2R−3u_+(R)=2^{R-3}.

(b) Frozen RCME with a lower-tail readout. We used q−(x)=⟨m^k(x),−φk(0)⟩Hkq_-(x)=\langle \hat m_k(x),-\varphi_k(0)\rangle_{\mathcal H_k}, which penalizes predicted similarity to the least safe grade.

(c) Frozen RCME with a Gaussian readout. We used qG(x)=⟨m^k(x),gG⟩Hkq_G(x)=\langle \hat m_k(x),g_G\rangle_{\mathcal H_k}, where gGg_G represents the Gaussian utility on the four PKU reward grades. This readout reweights weights for different ratings.

As shown in Table, Scalar-DR achieves performance comparable to the original RCME across all three benchmarks. In contrast, both alternative underperform Mean-DR on all three benchmarks for Qwen2.5-7B. One possible explanation is that these readouts place greater emphasis on improvements among lower safety grades, which may not translate into changes in binary safety outcomes. Additionally, we have revised the experimental section to include the results for Qwen3-4B in the main text.

MethodStrongRejectHarmBenchWildGuardMixTraining Time
Base (no alignment)40.7961.2071.850 s
Ours (Mean)81.4772.6683.38~30 min
Direct regression u+u_+87.0477.0886.52~30 min
Frozen RCME ⋅,u−\cdot\\,u_-75.0871.8082.36~1 s
Frozen RCME ⋅,uG\cdot\\,u_G78.1872.4681.50~1 s
RCME ⋅,u+\cdot\\,u_+87.6176.6386.34~30 min
AuthorsOpenReview ↗

Response to Reviewer t82F (part 2)

W3. The same utility can be estimated directly using Scalar-DR. Experiments with a single kernel and anchor do not establish the claimed reuse advantage or sufficiently support RQ4.

Response to Comment. The new experiments reported in Comment 2 directly test the reuse claim. We froze the original RCME checkpoint and changed only its readout weights to train separate GRPO policies with u−=−kR(⋅,0)u_-=-k_R(\cdot,0) and uG=kG(⋅,3)u_G=k_G(\cdot,3), requiring no additional reward-model fits. The matched Scalar-DR baseline also closely matches RCME under the original utility u+u_+, showing that direct scalar estimation recovers most of the gain over Mean-DR. We will revise RQ4 to distinguish these findings. By measuring the runtime, we find that once RCME is frozen, new utility functions can be applied without retraining the reward model, substantially reducing computational time.

W4, Q3. Under joint nuisance misspecification, Table 2 reports much larger CME bias than mean-estimator bias. Are these measures comparable, and what explains this result?

Response to Comment. The two bias measures in Table 2 are not directly comparable in magnitude, as they belong to different spaces. Specifically, within each simulation run, we first average the difference between the estimator and its oracle target over the sampled inputs, and then compute the magnitude of the average:

Mean: 1n∑i=1n∣r^(xi)−r0(xi)∣,CME: 1n∑i=1n∥∣m^(xi)−m0(xi)∥∣H.\text{Mean: } \frac{1}{n}\sum_{i=1}^n \left|\widehat r(x_i)-r_0(x_i)\right|,\qquad \text{CME: } \frac{1}{n}\sum_{i=1}^n \left\||\widehat m(x_i)-m_0(x_i)\right\||_{\mathcal H}.

By Eq. (11), the population bias vanishes when either nuisance model is correctly specified. When both are misspecified, however, the bias depends on the interaction of their errors and need not vanish. Moreover, the RKHS norm may capture errors across multiple feature directions, whereas scalar errors may partially cancel.

AuthorsOpenReview ↗

Follow-up on Review for Submission

Dear Reviewer t82F,

We hope this message finds you well. Thank you again for your valuable time and thoughtful comments on our work.

We have posted our detailed response addressing your concerns. Specifically, we have provided further analyses and clarifications on the benefits of distribution-aware reward modeling for downstream policy optimization, the flexibility of RCME across different utility functions and kernel choices, and the comparison with scalar doubly robust alternatives. We have also further discussed the robustness of our method under nuisance model misspecification.

We wanted to gently follow up to see whether our clarifications have adequately addressed your concerns, particularly regarding the practical advantages of learning the full reward distribution, or if there are any remaining questions we could further clarify.

We remain actively engaged in the Author-Reviewer discussion period and would be happy to provide any additional explanations or evidence that may be helpful.

Best regards,
The Authors of Submission #24552

Reviewer_kVP2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Reward Conditional Mean Embedding (RCME) for learning the conditional distribution of expert rewards from sparse, selectively observed expert feedback together with abundant but biased implicit feedback, and then uses this learned distribution for downstream policy optimization.

The method is evaluated on:

  1. Synthetic data with Gaussian, Laplace, and Student-t rewards;

  2. A semi-synthetic benchmark built from PKU-Safe RLHF;

  3. A real-world e-commerce Inquiry Card application.

The proposed RCME improves conditional mean reward estimation and conditional distribution recovery over non-oracle baselines. The paper also shows that directly using proxy/implicit rewards introduces substantial bias. The gains are largest when expert labels are scarce, supporting the benefit of orthogonal estimation with abundant auxiliary feedback. Some of the empirical results support the double-robustness property.

Critical strengths
  1. An RCME-based framework for learning conditional expert-reward distributions from sparse expert supervision and abundant implicit feedback.

  2. An identification result and a doubly robust, Neyman-orthogonal learning objective.

  3. The integration into GKRPO, which uses kernel utilities from the learned embedding for policy optimization and recovers GRPO as a special case.

  4. Empirical validation on synthetic, semi-synthetic, safety-alignment, and real-world e-commerce settings.

Critical weaknesses

The paper’s core idea is sound, but its contributions are relatively incremental. Mean embedding and doubly robust approaches are relatively standard. Although their integration in the context of reward modeling seemss to be new. I also list the following perceived weaknesses for the authors to address in their rebuttal

  1. The method relies on a missing-at-random expert-review mechanism. These are plausible only if expert review is driven entirely by observed prompt–response features and implicit feedback. In practice, auditors may select examples based on unobserved content quality, safety concerns, or true reward, violating MAR. Overlap can also fail in rare safety-critical strata, and propensity truncation introduces bias. The paper suggests random auditing to ensure overlap, but that changes the data-collection process rather than solving the observational problem.

  2. The identification result assumes a characteristic kernel and the full RKHS embedding. In implementation, the method uses a finite set of M=50M=50 reward anchors and a neural-kernel parameterization. But fixed anchors may poorly represent multimodal or heavy-tailed reward distributions, and results likely depend on kernel choice, bandwidth, and anchor placement

  3. GKRPO is a reasonable extension of GRPO, but the distribution-aware score is reduced to a scalar before policy optimization. Also gains over mean-reward variants are sometimes small, especially on PKU-SafeRLHF reward metrics

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Response to Reviewer kVP2 (part 1)

We appreciate your efforts and suggestions in reviewing our paper. We address your concerns below.

W1. The method relies on a missing-at-random expert-review mechanism. These are plausible only if expert review is driven entirely by observed prompt–response features and implicit feedback. In practice, auditors may select examples based on unobserved content quality, safety concerns, or true reward, violating MAR. Overlap can also fail in rare safety-critical strata, and propensity truncation introduces bias. The paper suggests random auditing to ensure overlap, but that changes the data-collection process rather than solving the observational problem.

Response to Comment. MAR is a standard identification assumption in the missing-data literature. To our knowledge, our work is among the first to introduce MAR-based identification into LLM reward modeling, relaxing the MCAR assumption adopted by prior work [1]. In particular, MAR allows the review probability to depend on observed covariates and surrogate feedback, making it substantially less restrictive than MCAR. Regarding overlap, the coarse-grained nature of surrogate feedback helps maintain sufficient review coverage across different feedback categories. While propensity truncation may introduce some bias, it can substantially reduce variance, reflecting a standard bias–variance trade-off. Importantly, mild truncation affects only observations with extreme propensity scores, and the resulting bias can be limited when such observations are rare [2]. Extending our framework to the more challenging MNAR setting, where review selection may depend on unobserved true rewards, is an important direction for future work.

W2. The identification result assumes a characteristic kernel and the full RKHS embedding. In implementation, the method uses a finite set of M=50M=50 reward anchors and a neural-kernel parameterization. But fixed anchors may poorly represent multimodal or heavy-tailed reward distributions, and results likely depend on kernel choice, bandwidth, and anchor placement.

Response to Comment. We evaluated anchor numbers $M\in (25,50,100) $, uniform versus quantile-based anchor placement, bandwidths h0/2h_0/2, h0h_0, and 2h02h_0, and Gaussian versus Laplace kernels on heavy-tailed t5t_5 and bimodal rewards with 20 repetitions per setting. To disentangle estimation error from representation error, we considered the decomposition

∥∣m^(x)−m0(x)∥∣=∥∣m^(x)−ΠAm0(x)∥∣+∥∣ΠAm0(x)−m0(x)∥∣,\||\hat m(x)-m_0(x)\||=\||\hat m(x)-\Pi_A m_0(x)\|| + \||\Pi_A m_0(x)-m_0(x)\||,

where the first term is estimation error and the second term is projection error. We recorded the projection error and the MMD under a fixed reference kernel. For fixed MM, both Gaussian and Laplace kernels yielded near-zero projection errors. Moreover, for a fixed kernel, varying anchor number, anchor placement or bandwidth resulted in only minor changes in the fitted discrepancy. These findings suggest that the selected anchors adequately represent both reward distributions and that the results are robust to reasonable choices of anchor configuration. This observation is consistent with [3], who demonstrated the effectiveness of fixed-anchor modeling for non-Gaussian distributions.

Configurationt5t_5: Proj. errort5t_5: MMDG2\mathrm{MMD}_G^2Bimodal: Proj. errorBimodal: MMDG2\mathrm{MMD}_G^2
Base (M=50M=50)(1.3±4.0)×10−9(1.3\pm4.0)\times10^{-9}0.1887±0.00220.1887\pm0.0022(0.3±4.3)×10−9(0.3\pm4.3)\times10^{-9}0.2203±0.00230.2203\pm0.0023
M=25M=25(9.0±9.7)×10−9(9.0\pm9.7)\times10^{-9}0.1775±0.00200.1775\pm0.0020(2.5±6.1)×10−9(2.5\pm6.1)\times10^{-9}0.2047±0.00220.2047\pm0.0022
M=100M=100(3.2±2.1)×10−9(3.2\pm2.1)\times10^{-9}0.2045±0.00240.2045\pm0.0024(0.04±1.2)×10−9(0.04\pm1.2)\times10^{-9}0.2376±0.00260.2376\pm0.0026
Quantile anchors(6.8±1.4)×10−6(6.8\pm1.4)\times10^{-6}0.1846±0.00210.1846\pm0.0021(1.1±0.2)×10−6(1.1\pm0.2)\times10^{-6}0.2237±0.00230.2237\pm0.0023
Bandwidth h0/2h_0/2(4.5±1.4)×10−8(4.5\pm1.4)\times10^{-8}0.1926±0.00220.1926\pm0.0022(5.0±3.6)×10−9(5.0\pm3.6)\times10^{-9}0.2263±0.00270.2263\pm0.0027
Bandwidth 2h02h_0(1.2±3.4)×10−9(1.2\pm3.4)\times10^{-9}0.1982±0.00160.1982\pm0.0016(2.2±3.0)×10−9(2.2\pm3.0)\times10^{-9}0.2202±0.00200.2202\pm0.0020
Laplace kernel(5.38±0.26)×10−6(5.38\pm0.26)\times10^{-6}0.1850±0.00210.1850\pm0.0021(4.83±0.11)×10−6(4.83\pm0.11)\times10^{-6}0.2168±0.00220.2168\pm0.0022

[1] Xie, Tong, et al. "A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design." arXiv preprint arXiv:2606.11189 (2026).

[2] Lee, Brian K., Justin Lessler, and Elizabeth A. Stuart. "Weight trimming and propensity score weighting." PloS one 6.3 (2011): e18174.

[3] Shimizu, Eiki, Kenji Fukumizu, and Dino Sejdinovic. "Neural-kernel conditional mean embeddings." arXiv preprint arXiv:2403.10859 (2024).

AuthorsOpenReview ↗

Response to Reviewer kVP2 (part 2)

W3. GKRPO is a reasonable extension of GRPO, but the distribution-aware score is reduced to a scalar before policy optimization. Also gains over mean-reward variants are sometimes small, especially on PKU-SafeRLHF reward metrics.

Response to Comment. Mean-reward recovery and distribution-reward recovery target different aspects of reward modeling and are evaluated using different metrics, so their improvements should be interpreted separately. For mean-reward recovery, evaluated using MSE and MAE, our method achieves the best performance by effectively leveraging observed feedback. For distribution-reward recovery, evaluated using MMD and embedding risk (ER), our method captures information beyond the conditional mean. On PKU, RCME substantially reduces ER from 0.923 (AO) and 0.588 (EO) to 0.147. Although GKRPO uses a scalar score for policy optimization, this score is derived from the learned conditional reward distribution and can therefore capture information beyond the mean. On PKU, the ordered Li–Racine gives the largest weight to P(R=3∣X=x)P(R=3\mid X=x), emphasizing the safest grade. Moreover, RCME also supports changing this preference after reward-model training. On the PKU benchmark, we compare two readout strategies for Frozen RCME: (a). We use $ q_-(x)=\langle \hat m_k(x),\varphi_k(0)\rangle_{\mathcal H_k} $, which penalizes predicted similarity to the least safe reward grade. (b). We use $ q_G(x)=\langle \hat m_k(x),g_G\rangle_{\mathcal H_k} $, where $ g_G $ encodes a Gaussian-shaped utility function over the four PKU reward grades, allowing different grades to contribute with different weights.

MethodStrongRejectHarmBenchWildGuardMixTraining Time
Base (no alignment)40.7961.2071.850 s
Ours (Mean)81.4772.6683.38~30 min
Frozen RCME ⋅,u−\cdot\\,u_-75.0871.8082.36~1 s
Frozen RCME ⋅,uG\cdot\\,u_G78.1872.4681.50~1 s
Ours (CME)87.6176.6386.34~30 min
AuthorsOpenReview ↗

Follow-up on Review for Submission

Dear Reviewer kVP2,

We hope this message finds you well. Thank you again for your valuable time and thoughtful comments on our work.

We have posted our detailed response addressing your concerns. Specifically, we have provided further clarifications on the assumptions underlying our framework, the practical considerations of finite reward anchors and kernel choices, and the benefits of distribution-aware reward modeling for downstream policy optimization.

We wanted to gently follow up to see whether our clarifications have adequately addressed your concerns, or if there are any remaining questions or points that would benefit from further discussion.

We remain actively engaged in the Author-Reviewer discussion period and would be happy to provide any additional explanations or evidence that may be helpful.

Best regards,
The Authors of Submission #24552

Reviewer_X8ABOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackles the problem of learning to align AI systems with human preferences via small amounts of expert preference data. The authors present a method based on learning a reward conditional mean embedding which uniquely represents the distribution over the reward function, and is learned via an objective that uses the implicit feedback to learn the reward model along with an error term to the expert data to ensure that its robust to the noise in implicit feedback. The authors further present GRKPO, a policy optimization method to use this learned kernel distribution to train the model. Overall, while the idea is potentially interesting, but the presentation is very weak. The paper does not motivate or explain the problem setup and terminology clearly. The introduced method is not explained intuitively and the paper requires a substantial improvement in writing to clearly understand the contributions.

Critical strengths
  • The paper presents an interesting problem of learning from implicit feedback with sparse supervision.
  • The introduced method is a framework that learns a reward conditional mean embedding (RCME) to predict the distribution of rewards under uncertainty.
  • The paper shows its robust under the variance and uncertainty introduced by sparse expert supervision and noisy implicit feedback.
  • The introduced method shows improvement in reward modelling for didactic experiments as well scaled to LLMs on a safety dataset.
Critical weaknesses
  • The introduction is very weak. Its discusses conditional reward learning -- however doesnt not discuss what the setting is.
  • It does not discuss the motivation or intuition behind using RKHS for reward modeling and the downstream benefits.
  • Overall, the paper is very hard to follow and understand the claims made, so I request the authors to please provide a. clear motivation for the setup, b. an explanation of their problem setting, c. clearly consolidate all the assumptions (expert review sampling independence, anchor scores etc..) and d. finally, present a pseudocode or something to explain the workings of the method.
Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer X8AB

The introduction is very weak. Its discusses conditional reward learning -- however doesnt not discuss what the setting is. It does not discuss the motivation or intuition behind using RKHS for reward modeling and the downstream benefits. Overall, the paper is very hard to follow and understand the claims made, so I request the authors to please provide a. clear motivation for the setup, b. an explanation of their problem setting, c. clearly consolidate all the assumptions (expert review sampling independence, anchor scores etc..) and d. finally, present a pseudocode or something to explain the workings of the method.

Response to Comment. We appreciate your efforts and suggestions in reviewing our paper. We have revised the Introduction to clarify the motivation and problem setting, and to highlight the key challenges and contributions of our method. We provide further details below.

(a) Motivation. Traditional reward modeling relies on costly expert annotations, limiting the scale of training data. In contrast, observational feedback (e.g., clicks, upvotes, or automated safety signals) is abundant and inexpensive but may be noisy and biased. This raises the first challenge: how to combine abundant proxy feedback with limited expert rewards for reliable reward learning. Moreover, expert preferences may exhibit uncertainty and heterogeneity that scalar reward models cannot fully capture. This raises the second challenge: how to capture the conditional distribution information of expert rewards rather than only their mean. To address both challenges, we propose learning conditional expert reward distributions by integrating abundant proxy feedback with limited expert annotations, enabling more informative and flexible reward signals for downstream RL.

(b) Problem setting. We consider a setting where abundant proxy feedback is available for all prompt–response pairs, while costly expert rewards are observed only for a subset. Specifically, let X=(P,Y)X=(P,Y) denote a prompt–response pair, R~\widetilde R the proxy feedback, and let SS be an indicator of whether the expert reward is observed. We observe (X,R~,S,SR)(X,\widetilde R,S,SR) and and aim to estimate the conditional reward distribution. To this end, we introduce the conditional mean embedding m0(x)=E[kR(R,⋅)∣X=x]m_0(x)=\mathbb E[k_R(R,\cdot)\mid X=x], whose advantages are discussed in detail in Item c.

(c) RKHS intuition. The key intuition is to characterize a reward distribution through its expectations, rather than explicitly estimating its density. CME offers three key advantages:

  1. It preserves conditional distributional information beyond the mean under a characteristic kernel;
  2. it turns distribution learning into a regression problem without explicit density estimation;
  3. it enables flexible downstream utility through inner products, without numerical integration or retraining the reward model.

(d) Assumptions. We require the following assumptions.

  1. The samples are i.i.d.;
  2. R⊥S∣(X,R~)R\perp S\mid (X,\widetilde R);
  3. ϵ≤P(S=1∣X,R~)≤1−ϵ\epsilon\le P(S=1\mid X,\widetilde R)\le1-\epsilon;
  4. The feature map is measurable and E[kR(R,R)]<∞\mathbb E[k_R(R,R)]<\infty;
  5. For the maximum-reward setting, RR is bounded.

For further explanations of these assumptions, please refer to Lines 164–173.

(e) End-to-end algorithm. The complete pseudocode is provided in Algorithm 1 of Appendix A. Here, we briefly outline the key steps of the proposed procedure.

  1. Fit two nuisance models.

    Estimate π(X,R~)=P(S=1∣X,R~)\pi(X,\widetilde R) = P(S=1\mid X,\widetilde R) and ν(X,R~)=E[φ(R)∣X,R~,S=1]\nu(X,\widetilde R) = \mathbb E[\varphi(R)\mid X,\widetilde R, S = 1].

  2. Construct pseudo-outcomes.

    Set

    Ψ^i=ν^(Xi,R~i)+Siφ(Ri)−ν^(Xi,R~i)π^(Xi,R~i).\hat\Psi_i=\hat\nu(X_i,\widetilde R_i)+ S_i \frac{\varphi(R_i)-\hat\nu(X_i,\widetilde R_i)}{\hat\pi(X_i,\widetilde R_i)}.

  3. Fit RCME by regression.

    Learn the reward embedding using

    \in \arg\min_\theta \frac1n\sum_{i=1}^n \||m_\theta(X_i)-\hat\Psi_i\||_{\mathcal H}^{2}.$$
  4. Optimize the policy with GRPO.

    Freeze m^\hat m, sample a group of responses for each prompt, and score each prompt–response pair xx by

    q(x)=⟨m^(x),kR(Rmax⁡,⋅)⟩.q(x)=\langle\hat m(x),k_R(R_{\max},\cdot)\rangle.

    Compute group-relative advantages from these scores and update the policy using GRPO.

AuthorsOpenReview ↗

Follow-up on Review for Submission

Dear Reviewer X8AB,

We hope this message finds you well. Thank you again for your valuable time and constructive feedback on our work.

We have posted our detailed response addressing your concerns. Specifically, we have clarified the motivation and problem setting, elaborated on the intuition behind using RKHS for reward modeling, and provided further explanations of the key assumptions and methodology to improve the clarity and accessibility of our work.

We wanted to gently follow up to see whether our clarifications have adequately addressed your concerns, or if there are any remaining questions or points that would benefit from further explanation.

We remain actively engaged in the Author-Reviewer discussion period and would be happy to provide any additional clarifications.

Best regards, The Authors of Submission #24552

Dear Reviewer X8AB,

We hope you are doing well. As fewer than 24 hours remain before the Author-Reviewer discussion period concludes, we would like to kindly follow up on our previous response.

We sincerely hope that our clarifications and revisions, particularly regarding the motivation, problem setting, and methodological details, have adequately addressed your concerns and improved the overall clarity of our work.

If you have had a chance to review our response, we would greatly appreciate any further thoughts or feedback you may have. We would also be more than happy to provide any additional clarification if needed.

Once again, we sincerely thank you for your thoughtful feedback and constructive suggestions. Your comments have been very helpful in improving the clarity of our presentation and strengthening our work.

Best regards, The Authors of Submission #2552

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We thank the reviewers for their careful feedback. Their comments have helped us improve the clarity, theoretical justification, and empirical validation of our work. In response, we have revised the manuscript and conducted additional analyses and experiments. We organize our revisions into four main modules, as summarized below.

  1. Clarification of the scientific motivation, problem setting, and methodological contributions: We substantially revised the Introduction to clarify the motivation, problem setting, and contributions of our work. We also consolidated the key assumptions, explained the intuition behind RKHS-based Conditional Mean Embedding (CME), and added a more explicit description of the proposed algorithm (Reviewer X8AB).

  2. Theoretical clarification and justification: We discussed in detail the MAR and overlap assumptions, the role of proxy feedback, and the doubly robust identification property. We also clarified the limitations under joint nuisance-model misspecification and the potential challenges in more general MNAR settings (Reviewers kVP2 and t82F).

  3. Additional experiments and robustness analysis: We conducted additional experiments on reward anchor configurations, kernel choices, and bandwidths, including heavy-tailed and multimodal reward distributions. We further evaluated the robustness of RCME under different settings and provided additional analyses of estimation and projection errors (Reviewers kVP2 and t82F).

  4. Further validation of distributional modeling and policy optimization: We introduced a direct Scalar-DR baseline and additional utility readout experiments using frozen RCME representations. These results further clarify the distinction between scalar reward estimation and reusable distributional reward representations, as well as their respective benefits for downstream policy optimization (Reviewer t82F).

Thank you again to the Area Chair for your time and thoughtful consideration. We warmly welcome further questions and discussion, and would be happy to clarify any aspect of the theory, experiments, or revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.