- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ARES, an automated pipeline that converts raw documents into reinforcement learning data by jointly generating self-contained questions, reference answers, and question-specific weighted rubrics. The pipeline incorporates persona-based question generation and quality filtering, producing approximately 100K training instances across ten domains. During training, an LLM judge evaluates responses against individual rubric criteria, and the weighted scores serve as rewards for standard GRPO. Experiments with Qwen3-4B-Base across seven benchmarks show higher average performance than continued pretraining, SFT on the same generated data, and binary-reward RL.
However, two concerns remain. First, although rubric generation is grounded in source documents and accompanied by jointly generated reference answers, it lacks explicit calibration against independently validated examples of good and bad responses. Consequently, the generated answers and rubrics may share factual errors or unsupported assumptions, and their mutual consistency does not establish correctness. Second, the ablation results provide limited evidence for the incremental value of question-specific rubrics: ARES improves the average score over generic rubrics by only 0.90 points, with gains concentrated in instruction following, while performing worse on HealthBench and WritingBench. These results leave the broader benefits of the proposed rubric-generation design insufficiently established.
Critical strengths
-
Practical and well-motivated framework. ARES provides a practical approach to scaling rubric-based RL data. Generating questions from source documents uses external information to construct training tasks, offering a promising way to expand training coverage and develop model capabilities.
-
Human validation of data quality and judge reliability. The appendix reports a human audit of sampled questions, reference answers, and rubrics, together with agreement analyses between human annotators and the LLM judge. These assessments provide evidence supporting both the quality of the generated data and the reliability of the reward signal.
-
Extensive comparative evaluation. The paper compares its approach against continued pretraining, SFT, binary-reward RL, and alternative reward designs. The proposed method achieves the highest average score across the seven evaluated benchmarks, supporting the overall effectiveness of the framework.
Critical weaknesses
Limited calibration of generated rubrics against response quality. Although rubrics are grounded in source documents and jointly generated with reference answers, they are not explicitly calibrated against independently validated good and bad responses. The generated answers and criteria may therefore share unsupported assumptions or factual errors. While the human audit provides useful aggregate evidence, the paper lacks side-by-side examples showing how individual criteria distinguish strong from weak responses, or where rubric-based judgments fail.
Inconsistent gains across benchmarks in the reward ablation. Question-specific rubrics improve the average score over generic rubrics by only 0.90 points, with gains concentrated in IFEval, while performing worse on HealthBench and WritingBench. Thus, although ARES achieves the best overall average, the ablation provides limited evidence that its question-specific rubric design offers consistent benefits across task types.
Questions for authors
-
Would calibrating generated rubrics against independently validated good and bad responses improve their quality and downstream RL performance?
-
Could you provide side-by-side response examples with criterion-level scores, including failure cases where rubric-based judgments disagree with human assessments?
-
Why do question-specific rubrics improve IFEval substantially but underperform generic rubrics on HealthBench and WritingBench?
-
Is the 0.90-point average improvement over generic rubrics consistent across training seeds and statistically significant?
2 comments
Thank you
Thank you for the clarification. The 0.90-point average gain over the General Rubric baseline is driven by IFEval, and this comparison is reported for a single training run. Given the limited evidence that question-specific rubrics provide consistent gains across tasks and training seeds, I will keep my original score.
Dear Reviewer,
Thank you for recognizing the practical framework and comparative evaluation. Our human audit, cross-judge experiments, and reward ablations provide complementary evidence for reward quality and training effectiveness.
W1 / Q1: Independent response-quality assessment
Appendix J audits 200 instances, stratified across ten domains, with two independent annotators and third-person adjudication. Eight candidate responses per question yield 1,600 responses for comparing rubric judgments with human assessments.
These results support the correspondence between rubric rewards and independent judgments. Rescoring the same outputs with Qwen and Llama gives κ = 0.69 and reward correlation ρ = 0.79 (Appendix K). Replacing the online judge with Llama yields HealthBench 40.80, close to Qwen's 41.45, supporting training with either tested judge.
Independent response contrasts offer a natural refinement mechanism: labels can identify criteria that discriminate strong from weak responses. ARES's explicit criterion representation supports this extension, complementing approaches such as OpenRubrics.
Q2: Criterion-level interpretation and examples
Our audit evaluates eight responses per question at both criterion and total-score levels. Criterion agreement κ = 0.72 measures alignment on individual requirements; total-score correlation ρ = 0.81 measures alignment after aggregation. Together, they assess both local scoring decisions and overall response quality.
Figure 5 illustrates the decomposition: identifying HSV-1 receives +10, the specified environmental exposures +8, and the stated transmission criterion +6. Responses satisfying different subsets receive different rewards, making partial success inspectable. The dietary-omission penalty also highlights a criterion requiring independent clinical justification. The audit separately evaluates criterion validity and scoring agreement, enabling targeted diagnosis of both criterion selection and judge application. This transparency is a practical advantage of explicit rubric rewards.
The weighted representation also connects interpretability to training utility. Under the same data, policy, judge, and GRPO settings, Table 18 improves the average from 51.88 with uniform criteria to 52.69 with weighted criteria, and IFEval from 52.40 to 54.88. Each response receives a scalar reward aggregated from criterion-level judgments, allowing partial satisfaction to contribute to the GRPO update. This supplies a practical mechanism for turning explicit assessment requirements into a useful optimization signal.
W2 / Q3: Benefits across task types
Table 5 shows ARES's strongest instance-specific advantage on instruction following:
IFEval uses strict prompt-level accuracy, requiring all constraints to be satisfied. The +9.22 gain is consistent with tailored criteria supplying useful supervision for varying format, keyword, language, and structural requirements. General Rubric's medical and writing results suggest that shared quality dimensions transfer well in those settings.
ARES achieves the highest average in this ablation and ties or exceeds General Rubric on five benchmarks. Both rubric approaches also outperform Blind Judge (49.53 average), supporting structured reward decomposition across the evaluation suite.
Q4: Statistical evidence and training stability
Table 10 reports ARES-RL at 52.69 [51.32, 54.04] and General Rubric at 51.79 [50.39, 53.21]. The +0.90 point estimate and +9.22 IFEval gain characterize the instance-specific advantage in this run, with the intervals quantifying evaluation uncertainty.
The main training comparisons additionally show consistent ordering across three seeds (Table 22):
Table 21 reports paired-bootstrap p < 0.001 for ARES-RL versus Webscale-style RL on the seven-task mean, HealthBench, and IFEval. These results support stable overall training gains, complementing the task-level reward analysis in Table 5.