acceptodds
Under review as a conference paper at ICLR 2027

SURE: Evolving Pairwise Reward Models with Rubrics from Uncertain Comparisons

Abstract

Pairwise reward models (RMs) are widely used as scalable proxies for human preferences in language model alignment, representing the relative quality of two responses through their reward difference. However, their judgments can become unreliable under distribution shift, particularly for comparisons near the model's decision boundary or requiring criteria and domain knowledge insufficiently covered during training. We propose \SURE, a sample-efficient framework for evolving pairwise RMs with rubrics learned from uncertain comparisons. SURE identifies informative comparisons from an unlabeled target set using two complementary signals calibrated against in-distribution anchors: reward-margin uncertainty and representation-space deviation. It queries a stronger evaluator only for the selected pairs and uses the resulting judgments to re-train the RM, while deriving hierarchical rubrics containing general principles and domain-specific criteria from the accompanying rationales. The rubrics are further refined through Group Relative Policy Optimization (GRPO)-inspired textual optimization, which converts group-relative agreement with the stronger evaluator into natural-language feedback rather than parameter gradients. At inference time, the evolved rubrics provide explicit criteria for preference judgment, complementing the adapted RM. Experiments show that, under matched selection ratios or external-query budgets, SURE identifies more informative uncertain cases, improves pairwise preference accuracy under distribution shift, and delivers reasonable gains on downstream tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reward model, Rubrics Evolving

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cEknOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SURE, which adapts a reward model to a shifted target distribution using a small set of comparisons labeled by a stronger LLM. Comparisons are selected by combining low reward margins with representation distance from in-distribution anchors. The teacher's labels are used to retrain the RM, and its rationales are summarized into general and domain-specific rubrics. The authors then freeze the RM and refine these rubrics through candidate search guided by group-relative feedback. Experiments on RM-Bench and RewardBench evaluate accuracy at different annotation budgets, and an MT-Bench experiment examines downstream policy optimization.

Critical strengths
  1. The paper addresses an important practical problem: improving an imperfect RM under distribution shift with a limited external-annotation budget. Reusing each queried judgment for parameter adaptation and its rationale for explicit evaluation criteria is a coherent design.
  2. The experiments show that SURE can improve preference accuracy with a limited annotation budget, particularly on difficult comparisons. The ablation studies suggest that both uncertainty-guided selection and rubric conditioning contribute to these gains.
Critical weaknesses
  1. Both scores reuse RM training data for percentile calibration, while OOD calibration also reuses covariance-fitting samples. This may inflate uncertainty on unseen in-distribution data. Separate sets for RM training, distance fitting, and percentile calibration would provide a cleaner evaluation.
  2. The evaluation is appropriate for the stated target-pool adaptation setting, but the extent of transfer remains unclear. Reported accuracy includes queried examples, and RM-Bench contains multiple response-style variants per prompt. Separating gains on queried pairs from those on unqueried, prompt-disjoint examples would clarify how much of the improvement reflects reusable preference knowledge.
  3. SURE is evaluated with only one RM backbone trained on one source dataset, leaving its generality across model families and source distributions unclear. Moreover, using different training and evaluation datasets does not by itself establish meaningful distribution shift.
Questions for authors
  1. Sections 3.1 and D.2 describe scoring each prompt–response pair separately. How is the comparison-level representation (h_\theta(x,y_a,y_b)) in Section 4.1.2 constructed?
  2. How much of the reported improvement remains on unqueried examples whose prompts do not appear in the queried set?
  3. Could the authors evaluate SURE with an additional RM backbone and an alternative source training dataset, varying each separately under matched adaptation budgets? Please also provide evidence of source–target distribution differences, such as domain, task, and response-style composition, rather than relying solely on dataset identity.
Reviewer_7gyAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SURE, which improves pairwise reward models under distribution shift using a limited budget of queries to a stronger LLM. It selects uncertain comparisons, retrins the reward model on the stronger LLM's judgments and turn LLM's rationales into evolving rubrics. Experiments on different benchmarks show gains over random and SNGP-based selection.

Critical strengths
  • The problem is practical and interesting. Spending a limited annotation budget on uncertain comparison is efficient.
  • The two uncertainty signals are complementary and well-motivated, making the reward model's judgements more interpretable.
Critical weaknesses
  • The annotated comparisons are selected from the test set itself, and the model is then evaluated on full test set again, which is quite strange for evaluation step. How are the results when performing on unselected samples ?
  • The technical novelty is quite limited. The method combines existing techniques such as OOD detection, LLM distillation and there is no actual policy optimization technique.
  • The paper compares SURE only against random selection and SNGP. Please include stronger and more relevant baselines, especially other active preference learning methods for LLMs such as [1].
  • As I understand, the method assumes domain labels are available. In a realistic unlabeled target distribution, this information may be unavailable. Please clarify how domains are obtained and how the method peforms when domain labels are missing.

[1] Active Preference Learning for Large Language Models, ICML 2024.

Questions for authors
  • How does the proposed method perform under unselected test samples ?
  • How robust is SURE when the stronger LLM gives systematically incorrect judgments in a particular domain? Could the rubrics amplify its biases ?
4 comments
AuthorsOpenReview ↗

Response (1/3)

Thank you very much for your review comments! Your comments are extremely valuable and important, and they are also easy to address. We have added the following experiments and hope to convince you to raise your rating.

W1. & Q1. About Evaluation

This is a very good question. In fact, the experiments in this part follow the original work of SNGP [1]. In real-world scenarios, these annotated samples can also be used for continued training. We believe that for this purpose, the original paper included annotated samples for testing. We understand that you are very concerned about the additional generalization brought by annotated samples. To examine whether SURE's improvements extend beyond the annotated comparisons, we additionally evaluate on the unselected portion of RM-Bench.

[1] Ask a Strong LLM Judge when Your Reward Model is Uncertain, NeurIPS'25

For each annotation budget (2%, 5%, 10%, 20%, and 30%), we exclude every selected prompt together. At each budget, we evaluate the original reward model and the corresponding SURE method on exactly the same held-out set. Preference judgments and rationales used for adaptation are restricted to the selected samples.

As can be seen, compared with the original reward model evaluated on the same held-out samples, our method achieves better results, highlighting that our method provides improved generalization through training/evolving on the selected samples.

Selection RatioMethodChatMathCodeSafetyAvg
2%Original75.6170.2860.0887.0473.25
2%SURE82.5569.0261.0191.1375.93
5%Original75.9670.9759.2687.1073.32
5%SURE79.4270.6660.9891.1575.55
10%Original76.3871.8359.3287.5173.76
10%SURE80.2172.9463.6592.8977.42
20%Original77.7873.3359.5088.7274.83
20%SURE78.0974.0466.0994.4278.16
30%Original79.6577.0159.7689.6476.52
30%SURE78.4877.3466.7596.3879.74

Broken down by difficulty, the results are as follows. Our method shows more significant improvements on Normal and Hard.

Selection RatioMethodEasyNormalHard
2%Original88.4076.2855.07
2%SURE83.9677.5466.29
5%Original88.4476.4855.05
5%SURE80.9676.8668.84
10%Original88.5676.9255.80
10%SURE83.9479.1969.13
20%Original88.5677.8658.07
20%SURE85.5380.4968.44
30%Original88.5179.0162.03
30%SURE87.3482.1469.74

Q2. When Facing Incorrect Annotations

This is a very good question. This issue is actually beyond our main focus and is a common problem in reward models. We understand this as similar to noisy data in interactive systems. Since the labels evaluated by the stronger large model used in our tests have high consistency and ground-truth agreement, manually corrupting labels and rationales would not faithfully simulate real-world scenarios and would not allow us to draw conclusions about rubrics. Since our work and the baseline [2] you provided also conduct related research under the premise that data annotations are reliable, we can only offer some provisional views:

That is, do not rely entirely on the same Strong LLM; instead, introduce evaluators from different model families for evaluation. Samples with strong consistency are recommended for adoption, while samples with large disagreement should be sent to human experts for evaluation. In this way, variance can be reduced and data usability improved. We also look forward to future work selecting appropriate datasets to validate our work and further improve the method's robustness to biases.

[2] Active Preference Learning for Large Language Models, ICML'24.

AuthorsOpenReview ↗

Response (2/3)

W2. Novelty

Regarding novelty: first, our paper is not about policy optimization per se, but about how to use costly annotation budgets on samples that serve as golden data and how to maximize their utility. The framework revolves around two closely related questions: which comparison samples are worth annotating, and how to maximize the reusable knowledge that each annotation can provide.

First, regarding comparison selection, uncertainty estimation and OOD detection capture different sources of annotation value. Reward-margin uncertainty identifies ambiguous comparisons near the RM's decision boundary, whereas representation-based OOD detection identifies comparisons that are unfamiliar to the current RM, even when its predictions are confident. Their combination therefore enables SURE to identify comparisons that are informative from both decision-uncertainty and distribution-shift perspectives, rather than relying on either signal alone.

Second, regarding annotation utilization, we go beyond conventional preference label distillation. Each selected comparison provides not only a pairwise preference judgment for RM fine-tuning, but also a rationale that can be transformed into reusable evaluation knowledge. Specifically, we organize the extracted knowledge into hierarchical general and domain-specific rubrics, which are further refined through feedback-driven, GRPO-inspired textual optimization. Consequently, a single annotation contributes to both local preference supervision and reusable evaluation criteria, allowing the acquired knowledge to benefit subsequent, previously unannotated comparisons.

In addition, we provide a bonus: we study the evolving loop (in Appendix C.4). We further design a continuously evolving feedback loop (Evolving Loop), which is an important aspect that distinguishes SURE from one-off data selection and knowledge distillation pipelines. In each iteration, the current reward model and its evaluation criteria are used to identify new high-value comparison samples; the preference judgments and rationales provided by the strong evaluator are used to update the reward model and further enrich and optimize the hierarchical evaluation criteria. The updated reward model and evaluation criteria in turn guide the selection of comparison samples and knowledge extraction in the next round. This forms a continuous feedback process from comparison selection and knowledge acquisition to criteria optimization and model updating. This mechanism enables the preference knowledge accumulated from each round of annotation not only to serve the current model but also to flow back and influence subsequent data selection and model improvement, thereby transforming one-off annotation gains into a sustainably accumulating knowledge asset. We believe that this continuous data and knowledge feedback mechanism is particularly important for coping with evolving preference distributions, and it also provides a foundation for the continuous evolution of reward models toward recursive self-improvement (RSI) in the future.

We hope our response addresses your concerns.

AuthorsOpenReview ↗

Response (3/3)

W3. Baseline

In fact, the recent work SNGP [1] that we follow already compares with the random scenario. Following your suggestion, we have added APL [2] as a baseline method and compare it with the main table. As can be seen, our method still achieves significant improvements over other baselines and the original model in most cases.

RatioMethodChatMathCodeSafetyEasyNormalHardAverage
0Original75.570.260.386.288.276.254.773.1
2%Random75.670.259.586.188.276.054.472.9
SNGP73.070.260.089.088.876.653.873.0
APL82.367.861.292.785.077.365.676.0
SURE82.569.161.390.984.077.666.376.0
5%Random76.070.860.487.188.576.655.773.6
SNGP73.272.260.189.188.877.155.173.6
APL79.168.661.891.485.077.363.275.2
SURE79.371.162.591.181.277.269.676.0
10%Random75.371.462.488.488.777.756.774.4
SNGP73.075.460.189.288.877.956.774.4
APL79.269.163.098.288.279.664.377.4
SURE79.974.267.792.784.680.470.978.6
20%Random75.472.163.292.088.578.859.875.7
SNGP73.778.662.989.789.179.859.976.2
APL80.270.764.397.588.180.266.278.2
SURE75.776.771.393.786.081.570.679.4
30%Random71.672.265.091.888.377.759.375.1
SNGP74.281.563.790.789.080.762.977.5
APL80.070.869.797.482.281.075.279.5
SURE76.580.273.295.388.383.771.981.3

[1] Ask a Strong LLM Judge when Your Reward Model is Uncertain, NeurIPS'25
[2] Active Preference Learning for Large Language Models, ICML 2024.


W4. About Domain Label

We get to know that many companies divide domains before data collection. This also reflects a task-oriented and weakness-oriented mindset. For example, Google [3] disclosed that their pipeline is:

  • Created a methodology for collecting and validating data about salient domains (e.g., health, education, finance) with relevant experts (i.e., people with domain-specific professional or academic expertise, such as health workers and teachers).
  • Identified rewards for data creation (e.g., compensation, certificates).
  • Established an ecosystem using an app to collect data.

In particular, they mentioned:

Before beginning data collection, the team ... identify which specific domains are most important to the region.

Therefore, this information is often available in practical scenarios. To further address your concern, we conducted a similar experiment. For labels, we used the original Qwen2.5-7B-Instruct to read the prompt of each selected sample and summarize its category with one English word. The prompt does not provide predefined categories; the model first reuses existing categories and then merges similar categories to reduce over-fine-grained divisions. At this stage, it does not read responses, preference judgments, or original domain labels; the remaining steps remain unchanged.

As can be seen, even without label prior information, we can still outperform models trained with the original model and the Random method.

RatioMethodChatMathCodeSafetyEasyNormalHardAverage
0Original75.570.260.386.288.276.254.773.1
5%SURE w/o original domain labels78.969.964.890.683.177.867.276.0
10%SURE w/o original domain labels74.973.965.992.284.479.066.876.7

[3]: Amplify Initiative: Localized Data for Globalized AI (2025)

Reviewer_7gyAOpenReview ↗

Response to Authors' Rebuttal

I would like to thank the authors for their detailed response and additional experiments, which have addressed several of my initial concerns. However, the technical novelty remains somewhat limited, as the method mainly combines existing uncertainty-based selection, LLM supervision, and rubric refinement techniques. Moreover, the reliability of the evolving rubrics under potentially biased LLM judgments remains unclear. A clearer discussion of the main contribution and the limitations of rubric evolution would help strengthen the paper. Therefore, I will maintain my original score.

Reviewer_t7QoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for improving pairwise reward models under distribution shift. SURE combines reward margin uncertainty and representation-based OOD detection to select informative comparisons for annotation by a stronger LLM. The judgments are then used to fine-tune the reward model, while the rationales are used to build general and domain-specific rubrics, which are further refined through GRPO-inspired textual optimization. Experiments on RM-Bench and RewardBench show improvements in preference accuracy.

Critical strengths

No substantive strengths to support the acceptance of this paper.

Critical weaknesses
  • Main: The method mainly combines several separate stages of existing techniques into a relatively complicated pipeline. It is difficult to identify a clear method novelty.

  • The experiment baseline comparisons and model backbones are not broad enough. It makes it difficult to assess how well the method generalizes in different settings.

Questions for authors
  • The main experiments only compare with Random and SNGP. Could the authors include other recent methods to better demonstrate the effectiveness of SURE?

  • The experiments only use Qwen2.5-7B as the reward model backbone. Could the authors provide results on other model families or larger models to demonstrate generalizability?

1 comment

Thank you for your review comments. We would like to clarify the positioning of our paper and provide additional supporting experiments in order to persuade you to raise your rating~

Contribution

Thank you for your thoughtful comments. We would like to take this opportunity to clarify the underlying design philosophy of SURE, particularly why we formulate it as a unified framework rather than a combination of techniques.

The practical problem that motivates this work is that obtaining high-quality preference annotations for reward modeling is expensive, especially under distribution shifts. Thus, our goal is to make every annotation count. Our philosophy is similar to "making two dishes from one chicken": rather than using a comparison only to obtain a binary preference label, we aim to fully exploit each comparison as a rich source of supervision, extracting multiple complementary and reusable forms of preference knowledge. This leads to two fundamental questions that jointly define our research:

(1) Which comparisons should be annotated?

(2) How can we maximize the utility of each annotation?

These two questions are sequentially dependent, but more importantly, they are intrinsically connected in our framework.

For the first question, we combine reward-margin uncertainty and representation-based OOD detection to capture two complementary sources of uncertainty: ambiguous preference boundaries and distributional deviations. This allows us to identify comparisons that are not only difficult for the current reward model but also potentially underrepresented in its training distribution.

For the second question, we argue that a valuable annotation should not be reduced to a binary preference label. A stronger teacher can provide both a preference judgment and its underlying rationale. SURE explicitly exploits these two complementary forms of supervision: preference judgments are used to improve the parametric reward model through pairwise fine-tuning, while rationales are used to construct hierarchical general and domain-specific rubrics. Furthermore, rather than treating rubric generation as a one-time extraction process, we introduce group-relative textual optimization to iteratively refine the evaluation criteria.

Therefore, the key idea is not to stack several existing techniques together, but to establish a budget-aware framework that connects informative comparison acquisition with the dual utilization of instance-level preference supervision and criterion-level evaluation knowledge. The selection stage determines which supervision signals can be acquired, while the utilization stage extracts complementary knowledge from annotations. Our contribution lies in the selection and utilization of comparison samples; at a high level, the core idea centers on the samples as the main thread.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Thanks for your patience!

Dear Reviewers,

Thank you for your patience. Due to personal matters and GPU scheduling issues, we have only just updated the PDF (with revisions highlighted in blue) and responded to your questions. We look forward to further discussion and to any improvement in the rating.

Best,

The Authors of SURE

Thanks for the additional results. The prompt-disjoint evaluation helps address my concern about generalization. I still have two questions:

  1. My calibration concern is about reusing the same samples for fitting and calibration. Would the results change with separate calibration data?

  2. Since the RM scores each response separately, how is the pair-level representation used for OOD detection constructed?

I find this reply too cursory and difficult to follow. Please take the time to organize your response clearly and address the questions directly. If AI tools are used to draft the response, please check the wording carefully before posting.

The additional experiment results address some of my evaluation concerns. My main concern about methodological novelty remains, as the method largely combines existing techniques. I maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.