- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors study reward modeling when expert labels are sparse and selectively observed, while noisy implicit feedback is available for every sample. They propose the Reward Conditional Mean Embedding (RCME), which represents the conditional distribution of expert rewards in an RKHS. Assuming that expert review is missing at random given the observed features and the implicit feedback, they identify the RCME by regressing a nuisance embedding on the prompt and response, and estimate it with a cross-fitted doubly robust pseudo-outcome that is shown to be Neyman-orthogonal. For policy optimization, they propose GKRPO which scores each response by the kernel similarity between its learned reward embedding and the feature of the maximum reward, and uses this score as the reward in group relative policy optimization (GRPO). Experiments cover a synthetic heteroskedastic design, a semi-synthetic PKU-SafeRLHF benchmark with downstream safety alignment, and an e-commerce inquiry generation task.
Critical strengths
The problem setting is practically relevant, and the doubly robust correction is consistently supported. Combining sparse expert supervision with abundant but biased implicit feedback is common in deployed systems, and the authors formalize it cleanly under a missing at random assumption. Across various reward noise distributions and the label rate sweep in Figure 1, the orthogonal estimator improves over training on expert labels alone (PE) and over the naive proxy baselines set up by the authors. The same gap between PE and the authors' method carries over to PKU-SafeRLHF in Tables 1 and 3.
Critical weaknesses
-
Once the GKRPO score is written out, the reproducing property gives with . In other words, the learned distribution is collapsed into a single expected utility before the policy update, and what follows is a standard GRPO step on a transformed reward. I could not find what the distributional view contributes at the policy optimization stage beyond the choice of .
-
With the ordered Li–Racine kernel, , and anchor used in the PKU experiments, the utility is , so the comparison between the authors' CME and Mean variants is really a comparison between and . The first puts most of its weight on , and since the safety benchmarks are scored by a binary judge, I would expect it to do better even without any distributional modeling. Given this, my main concern is whether the learned embedding can deliver results for a different kernel or anchor without retraining, or at least at a small computational cost, since this is where it would have a real advantage over a scalar reward model. I would be willing to raise my score depending on the authors' explanation of this point.
-
Remark 4.6 acknowledges that this scalar could be estimated directly by applying the same doubly robust construction to . The remaining argument for RCME is that one embedding can be reused across many utilities without retraining, but every experiment uses a single kernel and a single anchor. So the main advantage the paper claims for keeping the full distribution is never actually tested, and I do not think RQ4 is supported beyond what a utility transform would give.
-
When both nuisance models are misspecified in Table 2, the bias of the CME estimator is much larger than that of the mean estimator (for example, 6.84 versus 0.99 under Gaussian noise). The text states that the bias stays close to zero under the other specifications but does not discuss this row.
Minor Issues
- Corollary 4.5 uses without definition in the main text.
- The Li–Racine kernel should be defined and cited.
- The italic labels next to (RQ1) to (RQ4), such as "Reward recovery" and "Robustness", do not reflect the research questions well and could be removed.
- Line 040: "whose annotations is costly" "whose annotation is costly"
- Line 120: "the distribution of R" "the distribution of V"
- Line 323: "the bias of the our method" "the bias of our method"
Questions for authors
-
Could the authors report a baseline that applies the scalar doubly robust estimator of Corollary 4.5 to and trains GRPO on it?
-
Can one learned RCME be evaluated under several kernels or anchors (for example, risk averse and risk seeking utilities) without retraining the reward model? If so, how do the resulting policies and the total computational cost compare against retraining a scalar reward model for each utility?
-
In Table 2, are the biases of the mean and CME estimators measured on a comparable scale? If so, what explains the much larger degradation of the CME estimator when both nuisances are misspecified, and how should practitioners interpret it?
Rebuttal update
After carefully reading the author's response and the revised manuscript, my initial concerns have been mostly addressed. Therefore, I have raised my Informativeness score from 1 to 2 and my overall rating from 2 to 3.
5 comments
Response to Reviewer t82F (part 2)
W3. The same utility can be estimated directly using Scalar-DR. Experiments with a single kernel and anchor do not establish the claimed reuse advantage or sufficiently support RQ4.
Response to Comment. The new experiments reported in Comment 2 directly test the reuse claim. We froze the original RCME checkpoint and changed only its readout weights to train separate GRPO policies with and , requiring no additional reward-model fits. The matched Scalar-DR baseline also closely matches RCME under the original utility , showing that direct scalar estimation recovers most of the gain over Mean-DR. We will revise RQ4 to distinguish these findings. By measuring the runtime, we find that once RCME is frozen, new utility functions can be applied without retraining the reward model, substantially reducing computational time.
W4, Q3. Under joint nuisance misspecification, Table 2 reports much larger CME bias than mean-estimator bias. Are these measures comparable, and what explains this result?
Response to Comment. The two bias measures in Table 2 are not directly comparable in magnitude, as they belong to different spaces. Specifically, within each simulation run, we first average the difference between the estimator and its oracle target over the sampled inputs, and then compute the magnitude of the average:
By Eq. (11), the population bias vanishes when either nuisance model is correctly specified. When both are misspecified, however, the bias depends on the interaction of their errors and need not vanish. Moreover, the RKHS norm may capture errors across multiple feature directions, whereas scalar errors may partially cancel.
Follow-up on Review for Submission
Dear Reviewer t82F,
We hope this message finds you well. Thank you again for your valuable time and thoughtful comments on our work.
We have posted our detailed response addressing your concerns. Specifically, we have provided further analyses and clarifications on the benefits of distribution-aware reward modeling for downstream policy optimization, the flexibility of RCME across different utility functions and kernel choices, and the comparison with scalar doubly robust alternatives. We have also further discussed the robustness of our method under nuisance model misspecification.
We wanted to gently follow up to see whether our clarifications have adequately addressed your concerns, particularly regarding the practical advantages of learning the full reward distribution, or if there are any remaining questions we could further clarify.
We remain actively engaged in the Author-Reviewer discussion period and would be happy to provide any additional explanations or evidence that may be helpful.
Best regards,
The Authors of Submission #24552
Response to Reviewer t82F (part 1)
Thank you for the positive assessment and valuable suggestions about our paper. We address your questions below. We will correct typos and make the number of lemmas consistent in the updated version.
W1. The RCME score reduces to an expected utility followed by standard GRPO. What does the distributional view contribute beyond the choice of utility?
Response to Comment. The contribution of the distributional perspective lies in learning a reusable conditional reward representation rather than modifying the GRPO update itself. A single frozen RCME can support different downstream utilities without retraining, allowing different aspects of the conditional reward distribution to influence policy optimization. For the ordered Li–Racine kernel used in PKU, the score gives the largest weight to P(R=3∣X=x), emphasizing the safest reward grade. Thus, responses with identical conditional means but different reward distributions can receive different scores. As detailed in our response to Comment 2, we further provide a matched Scalar-DR baseline and demonstrate that the same frozen RCME supports policy training with both an alternative anchor-based readout and a Gaussian-kernel utility, validating its flexibility across different reward preferences.
W2, Q1, Q2. With u(r)=2r−3, the gains may reflect alignment with binary safety evaluation. Can the same embedding support different kernels or anchors without retraining, and at what computational cost?
Response to Comment. We conducted three additional experiments on PKU to address the reviewer's concern.
(a) Direct scalar regression. We fit a Scalar-DR reward model directly to the original utility u+(R)=2R−3.
(b) Frozen RCME with a lower-tail readout. We used q−(x)=⟨m^k(x),−φk(0)⟩Hk, which penalizes predicted similarity to the least safe grade.
(c) Frozen RCME with a Gaussian readout. We used qG(x)=⟨m^k(x),gG⟩Hk, where gG represents the Gaussian utility on the four PKU reward grades. This readout reweights weights for different ratings.
As shown in Table, Scalar-DR achieves performance comparable to the original RCME across all three benchmarks. In contrast, both alternative underperform Mean-DR on all three benchmarks for Qwen2.5-7B. One possible explanation is that these readouts place greater emphasis on improvements among lower safety grades, which may not translate into changes in binary safety outcomes. Additionally, we have revised the experimental section to include the results for Qwen3-4B in the main text.