acceptodds
Under review as a conference paper at ICLR 2027

Transfer One to All: Anchor Query-Specific Rubric Set Learning with Coarse Preferences

Abstract

Recent work generates query-specific rubric sets to provide fine-grained criteria for evaluating responses in LLM post-training. When scores assigned under rubric sets are used as rewards, the policy learns to favor higher-scoring responses. Therefore, the contribution of rubric sets to policy improvement should be an explicit criterion for rewarding rubric-set generation. Existing methods learn rubric sets based on metrics such as diversity, non-redundancy, and response discrimination, but these measures do not directly assess whether the induced policy updates encourage the desired overall behavior. In this paper, we are the first to study how a coarse preference can serve as an anchor for generating fine-grained, query-specific rubric sets. Although coarse preferences and generated rubric sets evaluate responses using different criteria, they both provide rewards for training the same policy. Therefore, we derive a reference policy-update direction from the coarse preference and augment the rubric generator's existing training reward with an alignment term measuring how closely each rubric set's induced update matches this reference. For each query with a positively aligned candidate, the best-aligned rubric set supplies response scores for updating the policy. Extensive experiments on multiple benchmarks demonstrate improvements in both response discrimination using rubric sets and downstream policy performance. Our code is available on anonymous GitHub.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, reward modeling, rubrics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CEkSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes using policy-gradient alignment to guide query-specific rubric-set learning with coarse preferences. The authors train an evaluator on existing coarse-grained scores to construct a reference policy gradient, then compare it with the gradients induced by candidate rubric sets. The alignment signal both augments the rubric generator’s training reward and selects the rubric set used to train the response policy. Experiments on two existing frameworks report improvements in rubric-set discrimination and downstream policy performance.

Critical strengths

The method offers some novelty by connecting rubric-set evaluation to its effect on policy updates, and it can be incorporated into existing rubric-learning frameworks. The experiments cover two training frameworks, multiple benchmarks, and three judge models, with additional results across datasets and model backbones. The reported improvements are fairly consistent, providing some support for the method’s effectiveness.

Critical weaknesses
  1. The effectiveness of the alignment signal and the reference gradient still lacks direct validation. Although the cross-dataset experiments report positive results, the evidence connecting local gradient alignment to more effective rubric learning remains limited. The signal is based on gradients computed from a small number of responses generated by the current policy. The paper does not directly examine whether candidate rubric sets with higher alignment lead to greater actual policy improvement, nor does it systematically assess rubric validity or the preservation of query-specific requirements. Moreover, the entire method depends on the quality of the reference gradient, while RQ4 mainly shows plausible trends in predicted scores and intervals as model size increases. It does not report the evaluator’s pairwise ranking accuracy, empirical interval coverage, or the stability of the reference gradient across batches and random seeds.
  2. The source of the performance gains remains unclear, with key controls and component ablations missing. The method jointly introduces additional coarse-preference supervision, an alignment reward for the generator, selection of the best-aligned rubric set, and filtering that retains only queries with at least one positively aligned candidate. These factors are not sufficiently isolated in the experiments. Although the authors match the policy-update budget, comparisons using the same coarse-preference evaluator are still missing, including directly training the policy on its scores and simply combining coarse-preference rewards with rubric-based rewards. It is therefore difficult to determine whether the gains primarily come from the proposed rubric-learning mechanism, additional supervision, or training-query selection. Ablating the individual components and reporting the proportion and types of filtered queries would help establish the necessity of the method and clarify its potential task-selection bias.
Questions for authors

Please see the Weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to W1

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

We use Qwen3-8B for the additional experiments. RQA and UF denote ResearchQA and UltraFeedback, respectively. GPT and DeepSeek denote GPT-5.6-Luna and DeepSeek V4-Flash, respectively.


[W1.1] The paper lacks direct evidence linking the degree of gradient alignment to improvements in final policy performance.

Response: Thank you for requesting direct evidence of gradient alignment. Gradient alignment encourages policy updates that support both human preferences and query-specific rubric criteria.

  • Human preferences: The coarse reference guides the policy toward preferences learned from human annotations.
  • Query-specific details: Rubrics provide fine-grained criteria tailored to each question, capturing distinctions that coarse feedback may overlook.

We add a new experiment to score a large sample pool during alignment, then construct equal-sized training batches using only samples within each specified cosine interval. All settings start from the same checkpoint.

Alignment intervalRQA · GPT ↑UF · GPT ↑
Cosine in (0.0, 0.1)50.1072.36
Cosine in [0.1, 0.2)51.0472.61
Cosine in [0.2, 0.3)54.2173.45
Cosine in [0.3, 0.4)56.2875.69

The results show that the highest-alignment group performs best on both benchmarks.

[W1.2] Moreover, the entire method depends on the quality of the reference gradient, while RQ4 mainly shows plausible trends in predicted scores and intervals as model size increases.

Response: Thank you for highlighting rubric quality. We directly evaluate the quality of the learned rubrics. On 20 queries, GPT rates rubrics on a 1–5 scale for validity, requirement preservation, and judgeability.

MethodValidity ↑Requirement preservation ↑Judgeability ↑
EvoLM3.754.003.25
EvoLM + Coarse (Ours)4.004.504.00

The results show that our method receives higher ratings on all three dimensions.

[W1.3] It does not report the evaluator’s (1) pairwise ranking accuracy, (2) empirical interval coverage, or (3) the stability of the reference gradient across batches and random seeds.

Response: Thank you for requesting these diagnostics. We report each measurement below.

  1. Pairwise ranking accuracy

    On HelpSteer 2, we compare evaluator rankings with human scores averaged over helpfulness, correctness, and coherence, excluding tied pairs.

    Non-tied pairspairs agrees with human rankingRanking accuracy (%)
    41338593.22

    The results show that the evaluator agrees with human rankings on 93.22% of non-tied pairs, providing an informative preference signal.

  2. Empirical interval coverage

    We measure whether human mean scores fall within the predicted intervals for 1,038 HelpSteer 2 answers.

    Covered answersTotal answersCoverage (%)
    934103889.98

    The results show that empirical coverage is 89.98%, indicating overcoverage in this evaluation.

  3. Stability across batches and random seeds

    We evaluate the reference signal’s sensitivity across batches and random seeds.

    • Across batches: We sample different reference-query batches at M=16,32,64,128M=16,32,64,128 and measure gradient cosine and rubric-selection agreement.
    • Across seeds: We repeat each batch-size setting with five random seeds and report the mean ± standard deviation across seeds.
    Reference queriesGradient cosineSelection agreement (%)
    M = 160.270 ± 0.02479.40 ± 2.15
    M = 320.270 ± 0.02179.85 ± 1.86
    M = 640.275 ± 0.01980.00 ± 1.73
    M = 1280.280 ± 0.02080.35 ± 1.91

    The results show that rubric-selection agreement is high, with small standard deviations across the random seeds.

AuthorsOpenReview ↗

Response to W2 & W3

[W2.1] The source of the performance gains remains unclear, with key controls and component ablations missing. The method jointly introduces (a) additional coarse-preference supervision, (b) an alignment reward for the generator, (c) selection of the best-aligned rubric set, and (d) filtering that retains only queries with at least one positively aligned candidate. These factors are not sufficiently isolated in the experiments.

Response: Thank you for identifying these missing controls. For (a), we include direct reward combination using the same coarse evaluator. For (b)–(d), we remove one component at a time while retaining the others.

MethodRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
EvoLM-only53.8672.9248.7668.80
Coarse-only47.9867.4843.2765.99
EvoLM + Coarse (a) direct combination53.9773.2048.4767.98
EvoLM + Ours w/o (b) alignment reward54.1273.8749.4268.87
EvoLM + Ours w/o (c) rubric selection53.3572.7849.0167.45
EvoLM + Ours w/o (d) update filtering50.7172.5550.0764.33
EvoLM + Ours (Full)55.2075.6650.7170.12
  • (a) Direct reward combination: Normalize and combine rubric and coarse rewards for policy training, controlling for access to additional coarse supervision.
  • (b) Without alignment reward: Retain the generator’s base reward but remove its alignment reward, testing alignment-guided rubric learning.
  • (c) Without rubric selection: Select a random candidate while retaining filtering, testing candidate selection.
  • (d) Without update filtering: Keep all queries while retaining alignment learning and best-aligned selection.

The results show that our method outperforms direct reward combination, while removing any component in (b)–(d) lowers all four scores.

[W2.2] Although the authors match the policy-update budget, comparisons using the same coarse-preference evaluator are still missing, including directly training the policy on its scores and simply combining coarse-preference rewards with rubric-based rewards.

Response: Thank you for suggesting these comparisons. We compare coarse-only training, reward combination (Combine), and alternating updates (Joint) using the same coarse evaluator and matched policy-update budgets.

MethodRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
EvoLM-only53.8672.9248.7668.80
Coarse-only47.9867.4843.2765.99
EvoLM + Coarse (Combine)53.9773.2048.4767.98
EvoLM + Coarse (Joint)54.0973.4847.3968.80
EvoLM + Coarse (Ours)55.2075.6650.7170.12

The results show that our method outperforms all compared baselines across all four evaluations.

[W2.3] ...reporting the proportion and types of filtered queries would help establish the necessity of the method and clarify its potential task-selection bias.

Response: Thank you for requesting this breakdown. We report filtering statistics from 384 query occurrences, including repeated queries, in the available training-prefix logs.

  • Filtering rule: Skip an update when none of the available candidates is positively aligned with the coarse reference.
  • Counting: Report policy and generator filtering separately, with each task’s occurrences as the denominator.
Task groupQuery occurrencesPolicy filtered, n (%)Generator filtered, n (%)
All tasks384172 (44.79%)181 (47.14%)
Math / reasoning9644 (45.83%)49 (51.04%)
Code6427 (42.19%)29 (45.31%)
Knowledge / general QA16070 (43.75%)74 (46.25%)
Other6431 (48.44%)29 (45.31%)

The results show that filtering rates are broadly similar across the reported task groups.


Please let us know if we have properly addressed your questions and we are more than happy to discuss more!

Dear Reviewer CEkS,

As the discussion deadline approaches, we are wondering whether our responses have properly addressed your concerns? Your feedback would be extremely helpful to us. If you have further comments or questions, we hope for the opportunity to respond to them.

Many thanks,

The authors of #30356

Reviewer_dapMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper works on rubric-based reward modeling for LLM alignment. The authors argued that existing rubric generators' training objectives do not directly consider their effects on policy updates, and proposed to incorporate this information. To this end, they first trained a scalar RM to provide coarse preference scores. These scores were used to construct a reference policy gradient as a proxy for the desired policy update direction. The authors then used the cosine similarity between this reference gradient and the policy gradients induced by candidate rubric sets to guide rubric generator training and rubric selection. The proposed method was applied to two representative rubric-based RM frameworks and evaluated on several benchmarks for preference discrimination, downstream policy performance, and cross-domain transfer.

Critical strengths
  • The problem studied in this paper is important and well-motivated.
  • The paper is generally well presented and easy to follow.
  • The proposed method is conceptually interesting, with some promising empirical results.
Critical weaknesses

My concerns are mostly about the conceptual and empirical validity of the proposed method. Specifically, after reading through the paper, I feel that the high-level rationale is to enhance rubric RM training using another scalar (coarse) RM. In this sense, it resembles a form of RM distillation: by aligning the policy gradients induced by the two RMs, the rubric RM may implicitly learn from the coarse RM as an additional source of supervision. This leads to the following questions:

  • The idea of using an existing RM to guide rubric learning has been explored before. For example, ARGO [1] uses black-box RM predictions to train a rubric generator, aiming to provide interpretability for the black-box model. The key difference here is that the authors estimate and align policy gradients induced by the two RMs. However, it is not clear to me why gradient alignment should work better. The authors use an averaged gradient from different queries to guide rubric learning for each individual query. I am not fully convinced why these gradients should align, especially since different queries may have different requirements.

  • Another simpler way to combine the two RMs for policy training is to directly combine their reward predictions. I am also not sure whether the proposed method is conceptually better than this simple ensemble baseline.

  • I feel that my concerns cannot be fully addressed without additional experiments to distinguish whether the improvements come from gradient alignment or simply the additional supervision from the scalar RM. Some simpler baselines should be compared, such as combining the two RM rewards for policy optimization. Also, the effects of rubric learning, rubric selection, and update filtering are not separately studied. This leaves the sources of the reported performance gains unclear.

[1] ARGO. Interpreting Black Box Reward Models.

Questions for authors

Please see my comments above.

4 comments
AuthorsOpenReview ↗

Response to W1

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

We use Qwen3-8B for the additional experiments. RQA and UF denote ResearchQA and UltraFeedback, respectively. GPT and DeepSeek denote GPT-5.6-Luna and DeepSeek V4-Flash, respectively.


[W1] (1) How does the method differ from prior RM-guided rubric learning, such as ARGO? (2) Why should gradient alignment work better? (3) Why align gradients across different queries?

Response: Thank you for these suggestions. We provide point-by-point explanations and experiments addressing differ from prior work, gradient alignment, and the cross-query alignment.

  1. What's the different from prior work?

    • On method:
      • ARGO [2] optimizes rubrics so a fixed judge’s predictions match the RM.
      • We learn and select rubrics by aligning their induced policy gradients with a coarse-reward reference.
    • On motivation:
      • ARGO [2] explains the RM’s criteria and biases.
      • We learn query-specific rubrics to improve downstream policy training.
  2. Why using gradients:

    • Advantages, gradients are two distinct levels: (1) advantages measure their relative quality within a sampled group -> (2) gradients determine how the current policy is updated.
    • Similar advantages can induce different gradients. For Example: Two signals may agree closely on most responses but assign +0.1 and −0.1 to one response. In a one-parameter illustration, if that response’s log-probability gradient is 10, its update contributions are +1 and −1. If the other contributions largely cancel, the two updates point in opposite directions.
  3. Why align gradients cross query?

    • Using same samples for both signals can make alignment specific to the sampled subgroup.
    • Although we use both batches come from the same distribution, their empirical composition varies with sampling.
    • This variation during training improve robustness to distribution shifts.
  4. New experiments:

    • (1) For what's the different from prior work:

    We compare our method with LSC [1] and ARGO [2] using the same coarse evaluator.

    MethodRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
    EvoLM53.8672.9248.7668.80
    EvoLM + LSC [1]54.4872.6150.0870.04
    EvoLM + ARGO [2]54.7174.9747.9068.80
    EvoLM + Ours55.2075.6650.7170.12

    The results show that our method outperforms both baselines across all four evaluations.

    • (2) For why using gradients: Following the reviewer’s suggestion, we compare gradient alignment with advantage cosine and Spearman alignment, using the same query and sampled responses for all three methods.
    Alignment criterion (Same Query)RQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
    Advantage cosine55.2475.6950.8570.20
    Advantage Spearman55.3375.8550.7470.32
    Gradient alignment55.6776.3151.0770.76

    The results show that gradient alignment outperforms both advantage-based alignment methods across all four evaluations.

    • (3) For why align gradients cross query:

    We compare same-query and cross-query gradient alignment under distribution shifts measured by MMD2MMD^2.

    Distribution shift (MMD2MMD^2)Query settingRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
    0.0–0.3Same Query55.6176.2451.0270.69
    Cross Query55.4876.1550.9770.73
    0.3–0.6Same Query54.3274.7149.6569.12
    Cross Query54.9375.3650.2869.87
    0.6–1.0Same Query52.8472.9548.0767.52
    Cross Query54.1774.5849.6169.18

    The results show that cross-query alignment performs better under the tested distribution shifts.


References

[1] Zihao Wang et al.Transforming and Combining Rewards for Aligning Large Language Models. ICML 2024.

[2] Paloma Sodhi et al. Interpreting Black Box Reward Models. OpenAI Alignment Research Blog, 2026.

AuthorsOpenReview ↗

Response to W2 & W3

[W2, W3.1] Another simpler way to combine the two RMs for policy training is to directly combine their reward predictions. I am also not sure whether the proposed method is conceptually better than this simple ensemble baseline. ... Some simpler baselines should be compared, such as combining the two RM rewards for policy optimization.

Response: Thank you for suggesting this comparison. We consider two straightforward ways to use both RMs for policy training: direct reward combination and alternating updates.

  • Compared with direct reward combination: We use coarse feedback to improve rubric learning, rather than only mixing rewards for policy training, helping the generator produce rubrics whose induced updates support coarse preferences.
  • Compared with alternating updates: Alternating rubric-driven and coarse-driven updates can cause one update to counteract the other. We select positively aligned rubrics to reduce these conflicts and improve policy optimization.

We compare coarse-only training, reward combination (Combine), and alternating updates (Joint) using the same coarse evaluator.

MethodRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
EvoLM-only53.8672.9248.7668.80
Coarse-only47.9867.4843.2765.99
EvoLM + Coarse (Combine)53.9773.2048.4767.98
EvoLM + Coarse (Joint)54.0973.4847.3968.80
EvoLM + Coarse (Ours)55.2075.6650.7170.12

The results show that our method achieves the highest scores in all baselines.

[W3.2] Also, the effects of (a) rubric learning, (b) rubric selection, and (c) update filtering are not separately studied. This leaves the sources of the reported performance gains unclear.

Response: Thank you for identifying these missing controls. We remove one component at a time while retaining the others.

MethodRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
EvoLM53.8672.9248.7668.80
EvoLM + Ours w/o (a) rubric learning54.1273.8749.4268.87
EvoLM + Ours w/o (b) rubric selection53.3572.7849.0167.45
EvoLM + Ours w/o (c) update filtering50.7172.5550.0764.33
EvoLM + Ours (Full)55.2075.6650.7170.12
  • (a) Without rubric learning: Retain the generator’s base reward but remove its alignment reward, testing alignment-guided rubric learning.
  • (b) Without rubric selection: Select a random candidate while retaining filtering, testing candidate selection.
  • (c) Without update filtering: Keep all queries while retaining alignment learning and best-aligned selection.

The results show that removing any component lowers all four scores relative to the full method.


Please let us know if we have properly addressed your questions and we are more than happy to discuss more!

Dear Reviewer dapM,

As the discussion deadline approaches, we are wondering whether our responses have properly addressed your concerns? Your feedback would be extremely helpful to us. If you have further comments or questions, we hope for the opportunity to respond to them.

Many thanks,

The authors of #30356

I thank the authors for their efforts in providing the additional results. I confirm that I have read through the rebuttal and other reviews. However, I still feel that the current results do not fully justify the empirical benefit of the proposed alignment-guided rubric learning. To me, the filtering mechanism seems to play a major role instead. Therefore, I feel that further exploration of this dynamic would benefit the paper. For now, I will maintain my current score.

Reviewer_fakbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The setting here is RL where we are jointly training the policy model and a rubric defining model. The idea is that in addition to the usual standard used to select good rubrics—e.g., how well do they discriminate between responses from the policy model—we may also want rubrics that satisfy less legible criterion (e.g., responses that are helpful or simple should be preferred). The basic methodological idea of the paper is to compute the policy gradient induced by a candidate rubric, the policy gradient induced by a reward model capturing the additional desiderata, and then use the cosine similarity between these as a heuristic for selecting good rubric sets.

Critical strengths

The problem is reasonably well motivied and the exposition is clear (at least, relative to other papers in this area I read during my review batch). The experiments seem ably done and show a consistent gain.

Critical weaknesses

The below are copy-and-paste summaries of weaknesses from Gemini, but all of them seem like pretty critical things to address to me!

• The central methodological question goes unanswered: Why align gradients in parameter space across disjoint queries instead of aligning rewards/advantages on the same query?

At training time, they have a fast, frozen coarse evaluator m(q, a) (FsfairX-LLaMA3-RM-v0.1 trained on HelpSteer) that can score any (q, a) pair. On query q, the policy already samples G = 4 responses a₁, …, a_G. Why not just evaluate m(q, aₙ) on those same 4 responses? That immediately gives two dead-simple, zero-overhead baselines using the exact same supervision: 1. Reward-space alignment on query q: Measure the cosine similarity (or rank correlation) between candidate rubric rₖ's advantage vector A_{k,:}(q) ∈ ℝᴳ and the coarse RM's advantage vector A_{coarse}(q) ∈ ℝᴳ on the same responses. 2. Direct reward combination (cf. Wang et al., 2024): Simply combine the rubric reward and the coarse RM reward in the policy's GRPO advantage! Instead, they compute K backward passes per query in LoRA parameter space θ and take the inner product with a gradient g_{ref} averaged over 64 unrelated queries. Because gₖ(q) and g_{ref} are evaluated on different prompts and responses, their inner product in parameter space is a very noisy proxy—and neither of the two simple baselines above is ever tested.

• Missing ablations:

  • There is no ablation table in the paper. We never learn whether the gains come from: (a) the quantile-regression interval weighting (Eq. 6–9), (b) training the rubric generator with Rₖ^{align},
  (c) selecting k^*(q) at policy update time, or (d) dropping queries with maxₖ Rₖ^{align}(q) ≤ 0.

• Uncontrolled supervision and compute advantage: • Only the +Ours methods get access to the 37,120 human preference labels in HelpSteer and the pretrained FsfairX-LLaMA3-RM-v0.1 encoder, plus 4 × more judge evaluations per query (K × G = 16 vs. 4) and 2 × more policy rollouts per step.

• Vacuous theory (Theorem 4.1, p. 6):

  • Assumes by fiat that the true task gradient is g_⋆ = d + z with z ∼ 𝒩(0, τ²I) and that g_{ref} = z + ξ is an unbiased Gaussian measurement of z. Under that assumption, it is a textbook Gaussian conditioning calculation showing that using g_{ref} helps when its SNR is large enough (A_q(M) > Δ_q). It offers no insight into why single-query LLM gradients and batch-averaged reference gradients would satisfy this model.
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to W1

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

We use Qwen3-8B for the additional experiments. RQA and UF denote ResearchQA and UltraFeedback, respectively. GPT and DeepSeek denote GPT-5.6-Luna and DeepSeek V4-Flash, respectively.


[W1] 1. Why use cross-query rather than same-query alignment? 2. Why align gradients rather than rewards or advantages? 3. Is the inner product between gradients from different queries a noisy proxy?

Response: Thank you for these suggestions. We provide point-by-point explanations and experiments addressing cross-query alignment, gradient alignment, and the noisy-proxy concern.

  1. Why align gradients cross query:

    • Using same samples for both signals can make alignment specific to the sampled subgroup.
    • Although we use both batches come from the same distribution, their empirical composition varies with sampling.
    • This variation during training improve robustness to distribution shifts.
  2. Why using gradients:

    • Advantages, gradients are two distinct levels: (1) advantages measure their relative quality within a sampled group -> (2) gradients determine how the current policy is updated.
    • Similar advantages can induce different gradients. For Example: Two signals may agree closely on most responses but assign +0.1 and −0.1 to one response. In a one-parameter illustration, if that response’s log-probability gradient is 10, its update contributions are +1 and −1. If the other contributions largely cancel, the two updates point in opposite directions.
  3. Little noise:

    • We average coarse gradients over multiple samples. With sufficiently many samples, individual sampling differences have less influence on the reference direction, reducing noise in the alignment signal.
  4. New experiments:

    • (1) For why align gradients cross query: Following the reviewer’s suggestion, we compare same-query and cross-query alignment using the same gradient criterion under distribution shifts.
    Distribution shift (MMD2MMD^2)Query settingRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
    0.0–0.3Same Query55.6176.2451.0270.69
    Cross Query55.4876.1550.9770.73
    0.3–0.6Same Query54.3274.7149.6569.12
    Cross Query54.9375.3650.2869.87
    0.6–1.0Same Query52.8472.9548.0767.52
    Cross Query54.1774.5849.6169.18

    The results show that cross-query gradient alignment performs better than same-query gradient alignment under the tested distribution shifts.

    • (2) For why using gradients: Following the reviewer’s suggestion, we compare gradient alignment with advantage cosine and Spearman alignment, using the same query and sampled responses for all three methods.
    Alignment criterion (Same Query)RQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
    Advantage cosine55.2475.6950.8570.20
    Advantage Spearman55.3375.8550.7470.32
    Gradient alignment55.6776.3151.0770.76

    The results show that gradient alignment outperforms both advantage-based alignment methods across all four evaluations.

    • (3) For noisy-proxy concern: We score a large sample pool during alignment, then construct equal-sized training batches using only samples within each specified cosine interval. All settings start from the same checkpoint.
    Alignment intervalRQA · GPT ↑UF · GPT ↑
    Cosine in (0.0, 0.1)50.1072.36
    Cosine in [0.1, 0.2)51.0472.61
    Cosine in [0.2, 0.3)54.2173.45
    Cosine in [0.3, 0.4)56.2875.69

    The results show that policy scores increase consistently across higher-cosine groups, supporting cosine alignment as an informative signal for guiding policy updates.

AuthorsOpenReview ↗

Response to W2 & W3

[W2] There is no ablation table in the paper. We never learn whether the gains come from: (a) the quantile-regression interval weighting (Eq. 6–9), (b) training the rubric generator with RkalignR_k^{align}, (c) selecting k∗(q)k^*(q) at policy update time, or (d) dropping queries with max⁡kRkalign(q)≤0\max_k R_k^{align}(q)\leq 0.

Response: Thank you for identifying these components. Following reviewer’s suggestion, We add ablations corresponding to (a)–(d), removing one component at a time while retaining the others.

MethodRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
EvoLM53.8672.9248.7668.80
EvoLM + Ours w/o (a): interval weighting54.2874.2950.1469.37
EvoLM + Ours w/o (b): alignment reward54.1273.8749.4268.87
EvoLM + Ours w/o (c): best-aligned selection53.3572.7849.0167.45
EvoLM + Ours w/o (d): query filtering50.7172.5550.0764.33
EvoLM + Ours (Full)55.2075.6650.7170.12
  • (a) Without interval weighting: Use uniform pairwise weights to test the value of accounting for uncertain comparisons.
  • (b) Without alignment reward: Retain the generator's base reward but remove its alignment reward, testing alignment-guided rubric learning.
  • (c) Without best-aligned selection: Select a random candidate while retaining filtering, testing candidate selection.
  • (d) Without query filtering: Keep all queries while retaining alignment learning and best-aligned selection.

The results show that removing any component lowers all four scores relative to the full method.

[W3] Uncontrolled supervision and compute advantage: (1) more data and a pretrained model; (2) more rollouts and judge evaluations; (3) more optimization steps.

Response: Thank you for raising this concern. We provide point-by-point responses and experiments addressing the concerns about additional data, rollouts, and optimization steps.

  1. Additional data and a pretrained model

    We do not directly train the policy or rubric generator on HelpSteer annotations. Instead, we use these annotations to train a coarse evaluator built on a pretrained model. To examine whether this additional supervision alone explains the gains, we compare coarse-only training, reward combination (Combine), and alternating updates (Joint) using the same coarse evaluator. Following reviewer suggestion, we also include LSC [1] as a reward-combination baseline and additionally compare with ARGO [2]. All methods using coarse supervision share the same evaluator.

    MethodRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
    EvoLM-only53.8672.9248.7668.80
    Coarse-only47.9867.4843.2765.99
    EvoLM + Coarse (Combine)53.9773.2048.4767.98
    EvoLM + Coarse (Joint)54.0973.4847.3968.80
    EvoLM + LSC [1]54.4872.6150.0870.04
    EvoLM + ARGO [2]54.7174.9747.9068.80
    EvoLM + Coarse (Ours)55.2075.6650.7170.12

    The results show that our method achieves the highest scores in all baselines. This supports the importance of how coarse supervision is used, beyond access to additional data and a pretrained model alone.

  2. Additional rollouts and judge evaluations

    For each query, our method generates four candidate rubrics and four policy responses, totaling eight rubric/answer rollouts, and performs 4×4=164\times4=16 judge evaluations. We therefore add an EvoLM control that increases the number of sampled responses from 4 to 16, with 16 judge evaluations per query.

    MethodRQA · GPT ↑UF · GPT ↑RQA · DeepSeek ↑UF · DeepSeek ↑
    EvoLM, rollouts=4, evaluations=453.8672.9248.7668.80
    EvoLM, rollouts=16, evaluations=1654.1372.6748.6869.60
    EvoLM + Ours, rollouts=4+4, evaluations=4×455.2075.6650.7170.12

    The results show that providing more rollouts and judge evaluations does not lead to consistent performance improvements. Our method still achieves higher scores than baseline.

  3. Additional optimization steps

    Our experiments use exactly the same number of optimization steps as Baselines. EvoLM also updates the rubric generator and the policy model once per iteration.


References

[1] Zihao Wang et al.Transforming and Combining Rewards for Aligning Large Language Models. ICML 2024.

[2] Paloma Sodhi et al. Interpreting Black Box Reward Models. OpenAI Alignment Research Blog, 2026.

AuthorsOpenReview ↗

Response to W4

[W4] Vacuous theory (Theorem 4.1, p. 6): … It offers no insight into why single-query LLM gradients and batch-averaged reference gradients would satisfy this model.

Response: We thank the reviewer for highlighting this gap. The original theorem is a conditional analysis under an idealized Gaussian model, not an established distributional description of actual LLM gradients. We will clarify its scope and add a complementary analysis allowing task-reference mismatch.

  • The reference connects rubric selection to a cross-query coarse-preference objective. At the current policy, candidate rubrics reweight the same sampled response-gradient features: gk=ΨqAk+cqg_k=\Psi_q A_k+c_q, where AkA_k contains rubric-specific advantages and cqc_q is the common KL-gradient term. The reference estimates the population coarse-surrogate gradient μ\mu, so ⟨μ,gk/∥gk∥⟩\langle\mu,g_k/\|g_k\|\rangle measures a candidate direction's first-order surrogate improvement per unit step. In Figure 2, the coarse-preference gradient has positive average cosine similarity to the physician-annotated rubric gradient on HealthBench. This supports directional association with an expert proxy, without establishing unbiasedness or preservation of every candidate-pair ordering.

  • A non-Gaussian bound separates reference mismatch from sampling uncertainty. Fix the policy, evaluator, and candidates, and consider an independent reference batch gref=M−1∑jhjg_{\mathrm{ref}}=M^{-1}\sum_j h_j, with i.i.d. gradients of mean μ\mu and finite covariance. Let uk=gk/∥gk∥u_k=g_k/\|g_k\| for nonzero candidates, with u0=0u_0=0 for skipping. For a fixed pair, define

    \gamma=\langle g_\star,v\rangle,\qquad b=\langle g_\star-\mu,v\rangle,\qquad \nu^2=\operatorname{Var}(\langle h_j,v\rangle).$$ Here, $\gamma$ is the true task-direction margin and $b$ its signed distortion under the surrogate. If $\gamma>0$ and $b<\gamma$, the population reference preserves this ordering because $\gamma-b=\langle\mu,v\rangle>0$. The one-sided variance inequality gives $$\Pr\left(\langle g_{\mathrm{ref}},v\rangle\le 0\right) \le \frac{\nu^2}{\nu^2+M(\gamma-b)^2}.$$
  • Larger batches reduce sampling uncertainty, not systematic mismatch. Increasing MM tightens this bound but leaves bb unchanged. If b>γb>\gamma, larger batches instead converge to the incorrect ordering. Thus, the result quantifies ranking reliability when the population reference preserves the task ordering; it does not establish that this condition always holds. Its guarantee concerns first-order task improvement per unit step, not finite-step improvement for the unnormalized policy update.

We will incorporate this result and its proof alongside the clarified scope of Theorem 4.1.


Please let us know if we have properly addressed your questions and we are more than happy to discuss more!

Dear Reviewer fakb,

As the discussion deadline approaches, we are wondering whether our responses have properly addressed your concerns? Your feedback would be extremely helpful to us. If you have further comments or questions, we hope for the opportunity to respond to them.

Many thanks,

The authors of #30356

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank the reviewers for their valuable comments. Additions and substantive updates are highlighted in red in the revised PDF file.

  1. Alignment design and policy improvement.
    • Gradient versus advantage alignment (§5.4, Table 5). We compare gradient alignment with advantage cosine and Spearman using identical queries and responses. Gradient alignment achieves higher scores in all four evaluations.
    • Cross-query robustness (Appendix A.11, Table 9). We compare same-query and cross-query alignment across three MMD2MMD^2 ranges. Scores are similar at small shifts; cross-query alignment has a growing advantage at larger shifts.
    • Alignment strength (Appendix A.11, Table 10). We train on equal-sized batches from different cosine intervals, starting from the same checkpoint. Higher-alignment groups achieve higher policy scores on both benchmarks.
  2. Strengthened theoretical analysis (§4.5, Proposition 4.2; Appendix B.1). We complement Theorem 4.1 with a non-Gaussian bound on candidate-pair ranking errors. Under the stated ordering condition, larger reference batches reduce sampling uncertainty but not systematic mismatch. The result concerns first-order improvement per unit step.
  3. Same-supervision controls and additional baselines (§5.3, Table 4; Appendix A.12, Table 12). We compare coarse-only training, reward combination, alternating updates, LSC, and ARGO using the same evaluator. The first three controls also match policy-update budgets. Our method outperforms all five alternatives across four evaluations.
  4. Component contributions and filtering behavior.
    • Component ablations (§5.3, Table 3). We separately remove interval weighting, generator alignment reward, best-aligned selection, and positive-alignment filtering. Each removal lowers all four policy scores.
    • Filtering distribution (Appendix A.11, Table 11). We analyze 384 query occurrences, including repeats, in the available training-prefix logs. Filtering rates are similar across the reported task groups.
  5. Evaluator quality and reference stability.
    • Human agreement and interval coverage (Appendix A.13, Table 13). Ranking accuracy is 93.22% on 413 non-tied HelpSteer 2 pairs. Coverage is 89.98% over 1,038 answers. The evaluator shows high ranking agreement and interval overcoverage.
    • Reference stability (Appendix A.13, Table 14). We vary M=16,32,64,128M=16,32,64,128 and repeat each setting with five seeds. Selection agreement remains around 80%, with small standard deviations across seeds.
  6. Direct rubric-quality evaluation (Appendix A.14, Table 15). GPT rates validity, requirement preservation, and judgeability on 20 queries. Our method receives higher ratings on all three dimensions.
  7. Sampling-budget control (Appendix A.15, Table 17). We increase EvoLM to 16 responses, matching judge-evaluation counts and optimization steps. More sampling gives no consistent improvement; our method retains higher scores in all four evaluations.
  8. Evaluation clarifications (§§5.1–5.2). We clarify benchmark coverage, score aggregation, and binary win rates without ties.

We hope these revisions can fully address reviewers' concerns, and we are very eager to hear the feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.