acceptodds
Under review as a conference paper at ICLR 2027

ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning

Abstract

Rubric-based rewards offer a promising way to extend reinforcement learning (RL) for large language models beyond tasks with automatically verifiable answers. However, scaling rubric-based RL remains challenging: existing approaches often rely on expert-written rubrics and manually constructed question sets, while fixed task-level rubrics may fail to capture the evaluation requirements of individual questions. We propose ARES (Automated Rubric synthEsis for Scalable RL), a framework for automatically constructing rubric-based RL data at scale. Starting from raw pretraining documents, ARES converts source knowledge into self-contained question-answer pairs and co-generates question-specific weighted rubrics, enabling instance-level reward supervision for open-ended responses. To improve diversity and quality, ARES conditions generation on domain labels and persona information, and applies validation filters for question self-containment, answer faithfulness, and rubric validity. Using ARES, we construct 100K rubric-annotated instances across ten domains. Experiments on seven benchmarks show that rubric-based RL trained with ARES, outperforms continual pretraining, supervised fine-tuning, and binary-reward RL, with the largest gains on multi-dimensional open-ended tasks such as healthcare and instruction following.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llms, scaling, data-centric ai

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QAb2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ARES, an automated pipeline that converts raw documents into reinforcement learning data by jointly generating self-contained questions, reference answers, and question-specific weighted rubrics. The pipeline incorporates persona-based question generation and quality filtering, producing approximately 100K training instances across ten domains. During training, an LLM judge evaluates responses against individual rubric criteria, and the weighted scores serve as rewards for standard GRPO. Experiments with Qwen3-4B-Base across seven benchmarks show higher average performance than continued pretraining, SFT on the same generated data, and binary-reward RL.

However, two concerns remain. First, although rubric generation is grounded in source documents and accompanied by jointly generated reference answers, it lacks explicit calibration against independently validated examples of good and bad responses. Consequently, the generated answers and rubrics may share factual errors or unsupported assumptions, and their mutual consistency does not establish correctness. Second, the ablation results provide limited evidence for the incremental value of question-specific rubrics: ARES improves the average score over generic rubrics by only 0.90 points, with gains concentrated in instruction following, while performing worse on HealthBench and WritingBench. These results leave the broader benefits of the proposed rubric-generation design insufficiently established.

Critical strengths
  • Practical and well-motivated framework. ARES provides a practical approach to scaling rubric-based RL data. Generating questions from source documents uses external information to construct training tasks, offering a promising way to expand training coverage and develop model capabilities.

  • Human validation of data quality and judge reliability. The appendix reports a human audit of sampled questions, reference answers, and rubrics, together with agreement analyses between human annotators and the LLM judge. These assessments provide evidence supporting both the quality of the generated data and the reliability of the reward signal.

  • Extensive comparative evaluation. The paper compares its approach against continued pretraining, SFT, binary-reward RL, and alternative reward designs. The proposed method achieves the highest average score across the seven evaluated benchmarks, supporting the overall effectiveness of the framework.

Critical weaknesses

Limited calibration of generated rubrics against response quality. Although rubrics are grounded in source documents and jointly generated with reference answers, they are not explicitly calibrated against independently validated good and bad responses. The generated answers and criteria may therefore share unsupported assumptions or factual errors. While the human audit provides useful aggregate evidence, the paper lacks side-by-side examples showing how individual criteria distinguish strong from weak responses, or where rubric-based judgments fail.

Inconsistent gains across benchmarks in the reward ablation. Question-specific rubrics improve the average score over generic rubrics by only 0.90 points, with gains concentrated in IFEval, while performing worse on HealthBench and WritingBench. Thus, although ARES achieves the best overall average, the ablation provides limited evidence that its question-specific rubric design offers consistent benefits across task types.

Questions for authors
  1. Would calibrating generated rubrics against independently validated good and bad responses improve their quality and downstream RL performance?

  2. Could you provide side-by-side response examples with criterion-level scores, including failure cases where rubric-based judgments disagree with human assessments?

  3. Why do question-specific rubrics improve IFEval substantially but underperform generic rubrics on HealthBench and WritingBench?

  4. Is the 0.90-point average improvement over generic rubrics consistent across training seeds and statistically significant?

2 comments

Dear Reviewer,

Thank you for recognizing the practical framework and comparative evaluation. Our human audit, cross-judge experiments, and reward ablations provide complementary evidence for reward quality and training effectiveness.

W1 / Q1: Independent response-quality assessment

Appendix J audits 200 instances, stratified across ten domains, with two independent annotators and third-person adjudication. Eight candidate responses per question yield 1,600 responses for comparing rubric judgments with human assessments.

AssessmentResult
Criterion factual validity93.2%
Key-aspect coverage89.5%
Human–human criterion agreementκ = 0.78
Qwen–human criterion agreementκ = 0.72
Reward–human total score correlationρ = 0.81

These results support the correspondence between rubric rewards and independent judgments. Rescoring the same outputs with Qwen and Llama gives κ = 0.69 and reward correlation ρ = 0.79 (Appendix K). Replacing the online judge with Llama yields HealthBench 40.80, close to Qwen's 41.45, supporting training with either tested judge.

Independent response contrasts offer a natural refinement mechanism: labels can identify criteria that discriminate strong from weak responses. ARES's explicit criterion representation supports this extension, complementing approaches such as OpenRubrics.

Q2: Criterion-level interpretation and examples

Our audit evaluates eight responses per question at both criterion and total-score levels. Criterion agreement κ = 0.72 measures alignment on individual requirements; total-score correlation ρ = 0.81 measures alignment after aggregation. Together, they assess both local scoring decisions and overall response quality.

Figure 5 illustrates the decomposition: identifying HSV-1 receives +10, the specified environmental exposures +8, and the stated transmission criterion +6. Responses satisfying different subsets receive different rewards, making partial success inspectable. The dietary-omission penalty also highlights a criterion requiring independent clinical justification. The audit separately evaluates criterion validity and scoring agreement, enabling targeted diagnosis of both criterion selection and judge application. This transparency is a practical advantage of explicit rubric rewards.

The weighted representation also connects interpretability to training utility. Under the same data, policy, judge, and GRPO settings, Table 18 improves the average from 51.88 with uniform criteria to 52.69 with weighted criteria, and IFEval from 52.40 to 54.88. Each response receives a scalar reward aggregated from criterion-level judgments, allowing partial satisfaction to contribute to the GRPO update. This supplies a practical mechanism for turning explicit assessment requirements into a useful optimization signal.

W2 / Q3: Benefits across task types

Table 5 shows ARES's strongest instance-specific advantage on instruction following:

BenchmarkGeneral RubricARES-RLDifference
MMLU-Pro49.0049.36+0.36
GSM8K86.0586.96+0.91
HumanEval+34.7634.760.00
MBPP+61.9063.16+1.26
HealthBench45.9741.45−4.52
WritingBench39.1938.24−0.95
IFEval45.6654.88+9.22
Average51.7952.69+0.90

IFEval uses strict prompt-level accuracy, requiring all constraints to be satisfied. The +9.22 gain is consistent with tailored criteria supplying useful supervision for varying format, keyword, language, and structural requirements. General Rubric's medical and writing results suggest that shared quality dimensions transfer well in those settings.

ARES achieves the highest average in this ablation and ties or exceeds General Rubric on five benchmarks. Both rubric approaches also outperform Blind Judge (49.53 average), supporting structured reward decomposition across the evaluation suite.

Q4: Statistical evidence and training stability

Table 10 reports ARES-RL at 52.69 [51.32, 54.04] and General Rubric at 51.79 [50.39, 53.21]. The +0.90 point estimate and +9.22 IFEval gain characterize the instance-specific advantage in this run, with the intervals quantifying evaluation uncertainty.

The main training comparisons additionally show consistent ordering across three seeds (Table 22):

MethodAverage, mean ± standard deviation
Webscale-style RL48.34 ± 0.25
ARES-SFT49.73 ± 0.21
ARES-RL52.64 ± 0.27

Table 21 reports paired-bootstrap p < 0.001 for ARES-RL versus Webscale-style RL on the seven-task mean, HealthBench, and IFEval. These results support stable overall training gains, complementing the task-level reward analysis in Table 5.

Reviewer_QAb2OpenReview ↗

Thank you

Thank you for the clarification. The 0.90-point average gain over the General Rubric baseline is driven by IFEval, and this comparison is reported for a single training run. Given the limited evidence that question-specific rubrics provide consistent gains across tasks and training seeds, I will keep my original score.

Reviewer_7Af3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ARES, a framework for automatically constructing rubric-based RL data from raw pretraining documents. It jointly generates self-contained questions, reference answers, and question-specific weighted rubrics, using domain and persona conditioning and quality filtering to construct training instances. The authors generate 101,847 examples across ten domains and use them for GRPO training. Across seven benchmarks, ARES achieves a higher average score than continual pretraining, SFT on the same question–answer data, and Webscale-style RL. The appendices provide additional experiments involving multiple training seeds, human auditing, reward-judge replacement, and an additional policy model.

Critical strengths
  1. The automated pipeline has practical value. ARES converts raw documents into complete supervision suitable for RL, reducing dependence on existing question–answer datasets and manually constructed rubrics. Its construction of approximately 100K instances across ten domains demonstrates the practical scalability of the pipeline.

  2. The overall approach produces actual training gains. Across seven benchmarks, ARES improves the average score by 2.98 points over SFT on the same question–answer data and by 4.39 points over Webscale-style RL. Results across three training seeds preserve the ranking of the main methods, supporting the stability of these gains.

  3. The paper includes useful checks of data quality and judge consistency. Independent annotation of 200 instances by two annotators and an RL experiment replacing the online reward judge help assess the usability of the synthetic data and reward signals. Reporting synthesis costs also helps readers assess the practical requirements of adopting the pipeline.

Critical weaknesses
  1. The methodological novelty is limited. Automatically generating question-specific weighted rubrics and using them for RL already have precedents. ARES primarily integrates these ideas with document-driven question–answer synthesis into a unified pipeline. The paper insufficiently discusses closely related work such as OpenRubrics and RubricHub, leaving its substantive advance over existing automated synthesis methods unclear.

  2. The experiments do not adequately isolate the value of the central new design. There is no matched comparison against generating question–answer pairs from the same documents and then generating rubrics separately. Consequently, it remains unclear whether joint generation independently improves quality, cost, or downstream performance. The appendix comparison with RaR changes both dataset size and domain coverage, so it does not resolve this attribution issue.

  3. A matched comparison with existing automated rubric-generation methods is missing. Table 5 primarily compares question-specific rubrics against general rubrics and other scoring strategies; it does not establish that ARES produces better rubrics than existing methods for generating question-specific criteria. The appendix comparison with RaR also changes dataset size and domain coverage. Consequently, the current comparisons cannot adequately distinguish improvements in rubric construction from differences in training data.

  4. The substantive validity of some generated criteria remains questionable. The audit in Appendix J reports a 6.8% incorrect-criterion rate. Figure 5 also assigns a −10 penalty for failing to mention avoiding arginine and/or consuming lysine, without adequately establishing the medical basis for making this advice mandatory. Source support and structural checks are insufficient to guarantee appropriate scoring objectives, and these criteria directly determine training rewards.

Questions for authors
  1. What is ARES's central incremental contribution relative to RaR, OpenRubrics, and RubricHub? Beyond the data source and pipeline integration, which design choices yield demonstrable advantages?

  2. With the documents, generation model, and approximate budget controlled, does jointly generating question–answer pairs and rubrics improve data quality, reduce cost, or improve downstream training compared with generating them sequentially?

  3. Could the authors construct rubrics using ARES and an existing method such as RaR for the same questions and reference answers, then compare them under the same policy model, reward judge, and training budget? Comparing criterion correctness, coverage of key requirements, and downstream training performance would help establish whether ARES's construction method provides additional value.

  4. What evidence supports the arginine/lysine penalty in Figure 5? How does the validation pipeline prevent insufficiently supported recommendations in source documents from becoming mandatory scoring requirements?

2 comments

Dear Reviewer,

Thank you for recognizing the pipeline's practical value and training gains. ARES's central advantage is turning source documents into complete task and reward supervision at scale, supported by controlled training comparisons and independent quality assessments.

W1 / Q1: Contribution and related work

ARES jointly constructs self-contained questions, reference answers, and weighted criteria from documents, producing 101,847 instances across ten domains for both SFT and RL.

MethodContribution
RaRRubric rewards on curated science and medicine questions
OpenRubricsContrastive rubric generation and preference-consistency filtering
RubricHubCoarse-to-fine rubric synthesis with multi-model aggregation and difficulty evolution
ARESJoint document-conditioned task, answer, and weighted-rubric construction, followed by validation and training

This construction expands the supply of training tasks and associated reward criteria from domain corpora. Its supervision produces measurable gains: ARES-RL improves over ARES-SFT by 2.98 points on the same QA pairs. Criterion weighting adds 0.81 points under fixed data, policy, judge, and training settings (Table 18). These controls demonstrate the value of rubric-based optimization and weighting within ARES.

W2 / Q2: Co-generation quality and cost

Co-generation specifies the task, intended answer, and evaluation criteria together in one source-conditioned call. Domain/persona conditioning extends this process across topics and audiences. We measure its practical effectiveness through output quality and accepted-instance cost:

MeasureResult
Accepted instances / retention101,847 / 71.6%
Question self-containment97.0%
Reference-answer support94.5%
Criterion factual validity93.2%
Cost per 1,000 accepted instancesApproximately US$42 standard / US$21 batch

The human-audit results come from Appendix J. Historical cost estimates in Appendix N include generation, validation, retries, and rejected candidates. These measurements demonstrate a practical route to large-scale, quality-assessed supervision at a quantified cost. Downstream training establishes that the resulting rewards translate into model improvements.

ARES starts from DCLM, FineWeb-Edu, and FinePDFs, turning approximately 0.1B source tokens into the 101,847-instance collection. Each document can be paired with up to three personas, allowing task construction to reflect different audiences and information needs. The final dataset contains 1,108,163 criteria, including 817,047 positive and 291,116 negative criteria (Table 3). This quantifies the breadth and granularity of the supervision produced by the pipeline.

W3 / Q3: Controlled reward evidence and RaR

Table 5 holds the ARES questions, policy, judge, and GRPO setup fixed:

RewardSeven-task average
Blind Judge49.53
General Rubric51.79
Reference Answer46.25
ARES-RL52.69

ARES achieves the highest average, supporting explicit, question-specific reward supervision on these tasks. Table 18 further isolates weighting (+0.81 average). Independent auditing assesses criterion correctness and key-aspect coverage at 93.2% and 89.5%.

Table 19 adds a system-level comparison using the same 4B policy, training framework, and evaluation suite: ARES scores 52.69 versus RaR's 50.28. The respective datasets contain 101,847 examples across ten domains and 36.2K science/medicine examples. The fixed-question ablations establish the value of the reward design; this comparison demonstrates the competitiveness of the complete pipeline.

W4 / Q4: Substantive validity and Figure 5

ARES combines source-faithfulness checks, rubric validation, and independent human auditing. Appendix J assesses 200 instances with two annotators each and third-person adjudication, including relevant domain expertise. It reports 93.2% criterion validity, 89.5% key-aspect coverage, 97.1% weight-sign correctness, and human–human agreement κ = 0.78. These assessments substantiate quality beyond structural checks and identify criteria for targeted refinement.

Figure 5 illustrates the value of inspecting individual reward requirements: the answer and rubric share the arginine/lysine recommendation, while assigning a mandatory −10 omission penalty requires independent clinical justification. We will use this case to clarify source faithfulness versus clinical validity. The audit explicitly quantifies this distinction, reporting incorrect-criterion rates of 6.8% overall and 8.1% in the medical subset. ARES's explicit criteria and weights make these issues traceable and amenable to expert correction.

Thank you for the response. The clarifications on data quality, cost, and reward ablations address some of my concerns. However, the experiments still do not sufficiently isolate the benefit of document-conditioned joint construction over existing or sequential construction methods. My main concerns about novelty and its empirical support therefore remain, and I maintain my rating of 2 .

Reviewer_xZs7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ARES, a framework for automatically constructing rubric-based RL data at scale from raw pretraining documents. The framework consists of three phases: source preparation, rubric-augmented question–answer co-generation, and quality control. Using this pipeline, the authors construct approximately 100K QA instances with question-specific weighted rubrics across ten domains. The authors train Qwen3-4B-Base using GRPO with rubric-based rewards provided by a Qwen3-32B judge. ARES-RL outperforms other baselines on average across diverse domains, including baselines using different datasets and those using different training algorithms on the same training data. The paper also includes an ablation study in which the LLM judge receives either general rubrics instead of question-specific rubrics or reference answers. ARES rubric outperforms these alternatives on average as well.

Critical strengths
  1. The paper tackles the important problem of generating high-quality, prompt-specific rubrics at scale for RL, especially in open-ended domains.

  2. The paper is well presented and easy to follow.

Critical weaknesses
  1. Limited model families and scales. The main experiments use Qwen3-4B-Base, with an additional experiment using Qwen2.5-7B-Instruct. Evaluation across model families and scales remains limited.

  2. Limited baseline comparisons. It would be helpful to compare different datasets, such as NaturalReasoning and Webscale, using the same rubric-based GRPO setup. Currently, NaturalReasoning uses SFT, while Webscale uses binary-reward GRPO without subsequently generated rubrics. Comparisons of different rubric-generation methods on the same ARES QA pairs would also be valuable. A basic and particularly important baseline would be to directly ask the LLM judge to generate its own question-specific rubrics and provide both criterion-level rewards and an overall reward. Beyond this basic baseline, existing rubric-generation methods [1, 2] would also be relevant comparisons.

  3. Limited analysis of co-generation. The benefit of jointly generating QA pairs and rubrics over generating rubrics afterward or using other rubric-generation methods remains unclear without a direct comparison.

References

[1] Shen et al. Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks.

[2] Liu et al. OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment.

Questions for authors
  1. Could the authors clarify the choice of Qwen2.5-7B-Instruct rather than Qwen3-8B-Base for the additional experiment?

  2. Would it be possible to generate rubrics for the Webscale QA pairs and apply the same rubric-based GRPO setup? Similarly, could the authors clarify the choice to evaluate NaturalReasoning with SFT rather than the same rubric-based RL setup?

  3. Could the authors elaborate on the benefit of co-generating QA pairs and rubrics compared with generating rubrics afterward? How does this approach compare with other rubric-generation methods [1, 2], including methods that use multiple candidate responses to construct rubrics?

  4. Since the approach generates rubrics during the initial construction of QA pairs, could it also be applied to an existing prompt–response dataset?

  5. Beyond the confidence intervals in Tables 9–10, would it be possible to provide paired-bootstrap comparisons or standard deviations across training seeds for the Table 5 ablations?

  6. Could the authors offer further insight into the lower performance of ARES-RL relative to General Rubric on HealthBench and WritingBench? Additional evidence explaining the variation in gains across benchmarks would be helpful.

Minor clarifications and presentation suggestions:

  1. It would be helpful to explicitly define \delta in Equation (8).

  2. Adding labels to the clusters in Figures 2–3 would make them easier to interpret. Could the authors clarify what types of questions the central cluster represents and what the three apparent RaR clusters in Figure 2 correspond to?

1 comment

Dear Reviewer,

Thank you for recognizing the importance of scalable rubric construction. ARES combines document-driven task synthesis, weighted reward supervision, and validated training gains across seven benchmarks.

W1 / Q1: Policy checkpoints and judges

The Qwen2.5-7B-Instruct experiment tests an instruction-tuned initialization alongside Qwen3-4B-Base. With the same ARES data, reward pipeline, and evaluation suite, it achieves a reported average of 62.99 (Table 19), demonstrating applicability to both initialization regimes. This complements the scale-focused question posed by a Qwen3-8B-Base comparison. Replacing the Qwen reward judge with Llama-3.1-70B-Instruct yields HealthBench 40.80 versus 41.45 with Qwen (Appendix K), supporting robustness to the tested judge replacement.

W2 / Q2: Baseline design

Our comparisons provide complementary controls:

ComparisonControlled factorsSeven-task average
NaturalReasoning-SFT → ARES-SFTPolicy, SFT recipe, subset size/token count45.91 → 49.71
Webscale-style RL → ARES-RLPolicy, source pool, example count, GRPO framework48.30 → 52.69
ARES-SFT → ARES-RLPolicy and ARES QA pairs49.71 → 52.69
General Rubric → ARES-RLQuestions, policy, judge, GRPO setup51.79 → 52.69

NaturalReasoning supplies an external SFT data baseline; Webscale supplies a source-matched binary-reward pipeline. The same-QA comparisons directly demonstrate the value of ARES reward supervision. Applying rubric-based GRPO to existing Webscale or NaturalReasoning pairs is compatible with our modular training interface.

W2–W3 / Q3: Co-generation and rubric methods

ARES constructs the question, answer, and weighted criteria in one source-conditioned generation pass. Its practical advantage is producing complete RL supervision directly from documents: 101,847 instances across ten domains, with 94.5% answer support and 93.2% criterion validity in the human audit.

Table 5 evaluates reward strategies on identical questions. Table 18 further isolates weighting: the average increases from 51.88 to 52.69 and IFEval from 52.40 to 54.88. Table 19 compares the complete systems on their respective datasets, with ARES scoring 52.69 versus RaR's 50.28. These results support the utility of structured, weighted rewards and the document-driven pipeline.

Direct judge-generated rubrics fit the fixed-QA interface. RRD and OpenRubrics contribute refinement and response-contrast mechanisms; ARES supplies complete document-grounded tasks and criteria that such methods can further refine.

The construction also makes scaling concrete: Appendix N reports 71.6% retention and approximately US$42 per 1,000 accepted instances at historical standard API prices. The dataset contains 1,108,163 criteria, averaging 10.88 per question (Table 3). Positive criteria reward desired content, while negative criteria encode specific failure modes. This provides reusable, inspectable reward supervision across domains, with each question carrying its own evaluation requirements.

Q4: Existing QA datasets

Each ARES instance contains a prompt, reference answer, rubric, and metadata (Appendix C). Existing QA pairs can retain their questions and answers, acquire rubrics using available source context, and enter the same validation and GRPO interfaces. The evaluated document-driven setting generates all these components together.

Q5: Statistical evidence

Tables 9–10 quantify uncertainty for the reward ablation: ARES-RL averages 52.69 [51.32, 54.04], versus General Rubric's 51.79 [50.39, 53.21]. The reported advantage is +0.90, with +9.22 on IFEval.

For the main training comparison, ARES-RL exceeds Webscale-style RL by +4.39 with paired p < 0.001. Three-seed averages are 52.64 ± 0.27 for ARES-RL, 49.73 ± 0.21 for ARES-SFT, and 48.34 ± 0.25 for Webscale-style RL (Appendix M), supporting reproducible training gains.

Q6: Task-level differences

ARES's strongest advantage over General Rubric is IFEval (+9.22), consistent with explicit criteria helping satisfy varying instruction constraints. General Rubric scores higher on HealthBench (45.97 vs. 41.45) and WritingBench (39.19 vs. 38.24), suggesting broad quality dimensions transfer well there. ARES achieves the highest seven-task average in Table 5; both rubric approaches exceed Blind Judge (49.53), supporting structured evaluation across the suite.

Q7–Q8: Notation and figures

In Eq. (8), δ\delta denotes the assigned domain and pp the selected persona. Table 2 quantifies ARES's ten-domain coverage; the RaR data span science and medicine. We will clarify the notation and annotate Figures 2–3 using domain metadata and representative examples to explain the central region and apparent groups.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.