- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes using policy-gradient alignment to guide query-specific rubric-set learning with coarse preferences. The authors train an evaluator on existing coarse-grained scores to construct a reference policy gradient, then compare it with the gradients induced by candidate rubric sets. The alignment signal both augments the rubric generator’s training reward and selects the rubric set used to train the response policy. Experiments on two existing frameworks report improvements in rubric-set discrimination and downstream policy performance.
Critical strengths
The method offers some novelty by connecting rubric-set evaluation to its effect on policy updates, and it can be incorporated into existing rubric-learning frameworks. The experiments cover two training frameworks, multiple benchmarks, and three judge models, with additional results across datasets and model backbones. The reported improvements are fairly consistent, providing some support for the method’s effectiveness.
Critical weaknesses
- The effectiveness of the alignment signal and the reference gradient still lacks direct validation. Although the cross-dataset experiments report positive results, the evidence connecting local gradient alignment to more effective rubric learning remains limited. The signal is based on gradients computed from a small number of responses generated by the current policy. The paper does not directly examine whether candidate rubric sets with higher alignment lead to greater actual policy improvement, nor does it systematically assess rubric validity or the preservation of query-specific requirements. Moreover, the entire method depends on the quality of the reference gradient, while RQ4 mainly shows plausible trends in predicted scores and intervals as model size increases. It does not report the evaluator’s pairwise ranking accuracy, empirical interval coverage, or the stability of the reference gradient across batches and random seeds.
- The source of the performance gains remains unclear, with key controls and component ablations missing. The method jointly introduces additional coarse-preference supervision, an alignment reward for the generator, selection of the best-aligned rubric set, and filtering that retains only queries with at least one positively aligned candidate. These factors are not sufficiently isolated in the experiments. Although the authors match the policy-update budget, comparisons using the same coarse-preference evaluator are still missing, including directly training the policy on its scores and simply combining coarse-preference rewards with rubric-based rewards. It is therefore difficult to determine whether the gains primarily come from the proposed rubric-learning mechanism, additional supervision, or training-query selection. Ablating the individual components and reporting the proportion and types of filtered queries would help establish the necessity of the method and clarify its potential task-selection bias.
Questions for authors
Please see the Weaknesses above.
3 comments
Response to W2 & W3
[W2.1] The source of the performance gains remains unclear, with key controls and component ablations missing. The method jointly introduces (a) additional coarse-preference supervision, (b) an alignment reward for the generator, (c) selection of the best-aligned rubric set, and (d) filtering that retains only queries with at least one positively aligned candidate. These factors are not sufficiently isolated in the experiments.
Response: Thank you for identifying these missing controls. For (a), we include direct reward combination using the same coarse evaluator. For (b)–(d), we remove one component at a time while retaining the others.
| Method | RQA · GPT ↑ | UF · GPT ↑ | RQA · DeepSeek ↑ | UF · DeepSeek ↑ |
|---|---|---|---|---|
| EvoLM-only | 53.86 | 72.92 | 48.76 | 68.80 |
| Coarse-only | 47.98 | 67.48 | 43.27 | 65.99 |
| EvoLM + Coarse (a) direct combination | 53.97 | 73.20 | 48.47 | 67.98 |
| EvoLM + Ours w/o (b) alignment reward | 54.12 | 73.87 | 49.42 | 68.87 |
| EvoLM + Ours w/o (c) rubric selection | 53.35 | 72.78 | 49.01 | 67.45 |
| EvoLM + Ours w/o (d) update filtering | 50.71 | 72.55 | 50.07 | 64.33 |
| EvoLM + Ours (Full) | 55.20 | 75.66 | 50.71 | 70.12 |
- (a) Direct reward combination: Normalize and combine rubric and coarse rewards for policy training, controlling for access to additional coarse supervision.
- (b) Without alignment reward: Retain the generator’s base reward but remove its alignment reward, testing alignment-guided rubric learning.
- (c) Without rubric selection: Select a random candidate while retaining filtering, testing candidate selection.
- (d) Without update filtering: Keep all queries while retaining alignment learning and best-aligned selection.
The results show that our method outperforms direct reward combination, while removing any component in (b)–(d) lowers all four scores.
[W2.2] Although the authors match the policy-update budget, comparisons using the same coarse-preference evaluator are still missing, including directly training the policy on its scores and simply combining coarse-preference rewards with rubric-based rewards.
Response: Thank you for suggesting these comparisons. We compare coarse-only training, reward combination (Combine), and alternating updates (Joint) using the same coarse evaluator and matched policy-update budgets.
| Method | RQA · GPT ↑ | UF · GPT ↑ | RQA · DeepSeek ↑ | UF · DeepSeek ↑ |
|---|---|---|---|---|
| EvoLM-only | 53.86 | 72.92 | 48.76 | 68.80 |
| Coarse-only | 47.98 | 67.48 | 43.27 | 65.99 |
| EvoLM + Coarse (Combine) | 53.97 | 73.20 | 48.47 | 67.98 |
| EvoLM + Coarse (Joint) | 54.09 | 73.48 | 47.39 | 68.80 |
| EvoLM + Coarse (Ours) | 55.20 | 75.66 | 50.71 | 70.12 |
The results show that our method outperforms all compared baselines across all four evaluations.
[W2.3] ...reporting the proportion and types of filtered queries would help establish the necessity of the method and clarify its potential task-selection bias.
Response: Thank you for requesting this breakdown. We report filtering statistics from 384 query occurrences, including repeated queries, in the available training-prefix logs.
- Filtering rule: Skip an update when none of the available candidates is positively aligned with the coarse reference.
- Counting: Report policy and generator filtering separately, with each task’s occurrences as the denominator.
| Task group | Query occurrences | Policy filtered, n (%) | Generator filtered, n (%) |
|---|---|---|---|
| All tasks | 384 | 172 (44.79%) | 181 (47.14%) |
| Math / reasoning | 96 | 44 (45.83%) | 49 (51.04%) |
| Code | 64 | 27 (42.19%) | 29 (45.31%) |
| Knowledge / general QA | 160 | 70 (43.75%) | 74 (46.25%) |
| Other | 64 | 31 (48.44%) | 29 (45.31%) |
The results show that filtering rates are broadly similar across the reported task groups.
Please let us know if we have properly addressed your questions and we are more than happy to discuss more!
Dear Reviewer CEkS,
As the discussion deadline approaches, we are wondering whether our responses have properly addressed your concerns? Your feedback would be extremely helpful to us. If you have further comments or questions, we hope for the opportunity to respond to them.
Many thanks,
The authors of #30356
Response to W1
We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.
We use Qwen3-8B for the additional experiments. RQA and UF denote ResearchQA and UltraFeedback, respectively. GPT and DeepSeek denote GPT-5.6-Luna and DeepSeek V4-Flash, respectively.
Response: Thank you for requesting direct evidence of gradient alignment. Gradient alignment encourages policy updates that support both human preferences and query-specific rubric criteria.
We add a new experiment to score a large sample pool during alignment, then construct equal-sized training batches using only samples within each specified cosine interval. All settings start from the same checkpoint.
The results show that the highest-alignment group performs best on both benchmarks.
Response: Thank you for highlighting rubric quality. We directly evaluate the quality of the learned rubrics. On 20 queries, GPT rates rubrics on a 1–5 scale for validity, requirement preservation, and judgeability.
The results show that our method receives higher ratings on all three dimensions.
Response: Thank you for requesting these diagnostics. We report each measurement below.
Pairwise ranking accuracy
On HelpSteer 2, we compare evaluator rankings with human scores averaged over helpfulness, correctness, and coherence, excluding tied pairs.
The results show that the evaluator agrees with human rankings on 93.22% of non-tied pairs, providing an informative preference signal.
Empirical interval coverage
We measure whether human mean scores fall within the predicted intervals for 1,038 HelpSteer 2 answers.
The results show that empirical coverage is 89.98%, indicating overcoverage in this evaluation.
Stability across batches and random seeds
We evaluate the reference signal’s sensitivity across batches and random seeds.
The results show that rubric-selection agreement is high, with small standard deviations across the random seeds.