- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SURE, which adapts a reward model to a shifted target distribution using a small set of comparisons labeled by a stronger LLM. Comparisons are selected by combining low reward margins with representation distance from in-distribution anchors. The teacher's labels are used to retrain the RM, and its rationales are summarized into general and domain-specific rubrics. The authors then freeze the RM and refine these rubrics through candidate search guided by group-relative feedback. Experiments on RM-Bench and RewardBench evaluate accuracy at different annotation budgets, and an MT-Bench experiment examines downstream policy optimization.
Critical strengths
- The paper addresses an important practical problem: improving an imperfect RM under distribution shift with a limited external-annotation budget. Reusing each queried judgment for parameter adaptation and its rationale for explicit evaluation criteria is a coherent design.
- The experiments show that SURE can improve preference accuracy with a limited annotation budget, particularly on difficult comparisons. The ablation studies suggest that both uncertainty-guided selection and rubric conditioning contribute to these gains.
Critical weaknesses
- Both scores reuse RM training data for percentile calibration, while OOD calibration also reuses covariance-fitting samples. This may inflate uncertainty on unseen in-distribution data. Separate sets for RM training, distance fitting, and percentile calibration would provide a cleaner evaluation.
- The evaluation is appropriate for the stated target-pool adaptation setting, but the extent of transfer remains unclear. Reported accuracy includes queried examples, and RM-Bench contains multiple response-style variants per prompt. Separating gains on queried pairs from those on unqueried, prompt-disjoint examples would clarify how much of the improvement reflects reusable preference knowledge.
- SURE is evaluated with only one RM backbone trained on one source dataset, leaving its generality across model families and source distributions unclear. Moreover, using different training and evaluation datasets does not by itself establish meaningful distribution shift.
Questions for authors
- Sections 3.1 and D.2 describe scoring each prompt–response pair separately. How is the comparison-level representation (h_\theta(x,y_a,y_b)) in Section 4.1.2 constructed?
- How much of the reported improvement remains on unqueried examples whose prompts do not appear in the queried set?
- Could the authors evaluate SURE with an additional RM backbone and an alternative source training dataset, varying each separately under matched adaptation budgets? Please also provide evidence of source–target distribution differences, such as domain, task, and response-style composition, rather than relying solely on dataset identity.
Response (1/3)
Thank you very much for your review comments! Your comments are extremely valuable and important, and they are also easy to address. We have added the following experiments and hope to convince you to raise your rating.
W1. & Q1. About Evaluation
This is a very good question. In fact, the experiments in this part follow the original work of SNGP [1]. In real-world scenarios, these annotated samples can also be used for continued training. We believe that for this purpose, the original paper included annotated samples for testing. We understand that you are very concerned about the additional generalization brought by annotated samples. To examine whether SURE's improvements extend beyond the annotated comparisons, we additionally evaluate on the unselected portion of RM-Bench.
[1] Ask a Strong LLM Judge when Your Reward Model is Uncertain, NeurIPS'25
For each annotation budget (2%, 5%, 10%, 20%, and 30%), we exclude every selected prompt together. At each budget, we evaluate the original reward model and the corresponding SURE method on exactly the same held-out set. Preference judgments and rationales used for adaptation are restricted to the selected samples.
As can be seen, compared with the original reward model evaluated on the same held-out samples, our method achieves better results, highlighting that our method provides improved generalization through training/evolving on the selected samples.
Broken down by difficulty, the results are as follows. Our method shows more significant improvements on Normal and Hard.
Q2. When Facing Incorrect Annotations
This is a very good question. This issue is actually beyond our main focus and is a common problem in reward models. We understand this as similar to noisy data in interactive systems. Since the labels evaluated by the stronger large model used in our tests have high consistency and ground-truth agreement, manually corrupting labels and rationales would not faithfully simulate real-world scenarios and would not allow us to draw conclusions about rubrics. Since our work and the baseline [2] you provided also conduct related research under the premise that data annotations are reliable, we can only offer some provisional views:
That is, do not rely entirely on the same Strong LLM; instead, introduce evaluators from different model families for evaluation. Samples with strong consistency are recommended for adoption, while samples with large disagreement should be sent to human experts for evaluation. In this way, variance can be reduced and data usability improved. We also look forward to future work selecting appropriate datasets to validate our work and further improve the method's robustness to biases.
[2]Active Preference Learning for Large Language Models, ICML'24.