acceptodds
Under review as a conference paper at ICLR 2027

Rubrics Beyond Evaluation:Exploiting Rubric Structure in Reinforcement Learning

Abstract

Rubrics are widely used to evaluate open-ended question answering by breaking overall quality into explicit criteria (Kim et al., 2024; Arora et al., 2025). These explicit criteria make rubrics a rich source of learning guidance, yet their potential for guiding the training process has been largely overlooked. Existing methods use rubrics mainly as direct feedback signals, without explicitly modeling the capabilities behind their criteria (Gunjal et al., 2025; Li et al., 2026). This leaves their capability structure largely unused in guiding the training process. We introduce Rubric-Structured Reinforcement Learning (RSRL), a RL framework that unlocks the full potential of rubrics by exploiting their structure throughout the training process, rather than treating them simply as direct reward signals. At its core, RSRL builds on three key designes: (1) Rubric-to-Capability Structuring first recovers the capability structure hidden in rubric criteria, (2) SelfSufficient Capability Learning then uses this structure to bootstrap underdeveloped capabilities into effective learning, (3) Capability-Residual Credit Routing finally preserves capability-specific gains during optimization. Experiments show that RSRL consistently improves both overall performance and underdeveloped capabilities, boosting downstream scientific question-answering performance by up to 7.43 over the initial policy and 7.60 over non-rubric structured RL baselines, showing that reinforcement learning can substantially benefit from the capability structure encoded in rubrics

Primary area:
reinforcement learning
Keywords:
Rubric-Based Reinforcement Learning,Capability Learning,Targeted Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HETnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Rubric-Structured Reinforcement Learning (RSRL), which uses rubric-derived capability structure to coordinate weakness diagnosis, guided exploration, and credit assignment. It groups question-specific criteria into shared capability categories and combines local scores with historical performance to identify weaknesses. Its RL method steers trajectory generation by contrasting capability-guided and control-prompt token distributions, then learns from guided and unguided trajectories under the original question context using importance weighting. A residual credit-routing mechanism further adjusts response-level advantages to emphasize improvements in the targeted capability. Experiments report gains on scientific question-answering benchmarks.

However, the framework introduces numerous interacting hyperparameters, including diagnostic thresholds, history-pooling coefficients, guidance strengths, clipping bounds, and residual projection and scaling parameters. These choices jointly affect the sampled trajectories, importance weights, and advantages, thereby influencing the magnitude and direction of policy-gradient updates. Although the reported experiments support the method’s effectiveness in the evaluated setting, the limited sensitivity analysis and evaluation primarily on a single model size and task domain leave its training stability and generalizability insufficiently established. Broader robustness studies would help determine whether the gains persist across models and domains without substantial retuning.

Critical strengths

An interesting capability-oriented perspective. Identifying and addressing specific capability deficits is a compelling direction for rubric-based RL. Using a shared capability diagnosis to coordinate both exploration and policy updates offers a potentially novel contribution beyond aggregate reward optimization.

A conceptually motivated optimization mechanism. Capability-guided sampling seeks to produce useful behaviors that ordinary rollouts may rarely discover, while residual credit routing aims to prevent improvements in weak capabilities from being diluted by aggregate advantages. Importance weighting further accounts for the change in the sampling distribution. These components provide a coherent rationale for the proposed approach.

Supporting ablation evidence. The paper separately removes guided exploration and residual credit routing, with both ablations reducing performance on the capability development probe. These results support the contribution of both components to the gains observed in the evaluated setting.

Critical weaknesses

Sensitivity to interacting hyperparameters and limited evidence of robustness. RSRL combines historical capability estimates, diagnostic thresholds, guided sampling, importance weighting, and residual advantage routing. The associated hyperparameters jointly affect which trajectories are sampled and how they contribute to policy-gradient updates. Because capability estimates and the policy evolve together during online training, the interactions among these mechanisms may introduce considerable tuning complexity. This concern is particularly relevant when each question’s local diagnosis relies on only four sampled responses. The reported component ablations support the usefulness of guided exploration and residual routing, but do not establish robustness to the choices governing these mechanisms. More systematic sensitivity analysis and evaluation across models and rubric domains would help substantiate the framework’s broader applicability.

Unclear incremental benefit over directly fine-tuning on guided trajectories. A more direct baseline would apply SFT to high-quality capability-guided trajectories under the original question context. Comparing RSRL against this targeted SFT baseline would clarify whether its credit-assignment and RL machinery provides sufficient additional gains to justify the added complexity and computational cost. The existing generic SFT baseline does not answer this question, as it does not use the same targeted guided trajectories. Furthermore, the experiments focus on Qwen3-4B and do not report variability across independent training seeds. The four GPQA evaluation repetitions measure inference variability rather than training stability. Reporting training-seed variability and uncertainty in benchmark differences would strengthen confidence in the improvements, especially where margins over Rubric-RL are small.

Questions for authors
  1. How does RSRL compare with directly applying SFT to high-quality capability-guided trajectories under the original question context?

  2. How sensitive are the gains to guidance strength, gating thresholds, and residual-routing hyperparameters?

  3. Does cross-question capability history improve performance over using only question-local diagnostics?

  4. What is the additional computational cost relative to Rubric-RL, including diagnosis, guided sampling, and importance-weight computation?

  5. Are the improvements consistent across independent training seeds, particularly on benchmarks with small gains?

  6. Can the same hyperparameter settings transfer to larger models or other rubric-based tasks without substantial retuning?

3 comments

We thank the reviewer for the thoughtful review and for recognizing the capability-oriented perspective and the connection between guided exploration and credit assignment. We also appreciate the reviewer's detailed questions, which have been very helpful in improving the rigor and completeness of our paper. We address Q1-Q2 here and Q3-Q6 in Comment 2.

[W2 & Q1] How does RSRL compare with directly applying SFT to high-quality capability-guided trajectories under the original question context?

Re: We thank the reviewer for this valuable suggestion and agree that Guided SFT is an important comparison. We have conducted an additional SFT experiment using high-quality training rollouts. We first explain how our existing SFT baseline provides evidence for the effectiveness of credit assignment and RL machinery from two aspects: (1) training data quality and (2) capability-based credit assignment. We then present (3) the evaluation results of SFT on high-quality capability-guided trajectories under the original question context.

(1) Our existing SFT baseline already uses high-quality responses selected from multiple candidates. Similarly, capability-guided rollouts are generated to satisfy the question-specific rubrics. Since both types of responses are evaluated using the same rubric criteria, we expect their overall quality to be comparable. Thus, our existing SFT baseline provides a useful comparison for assessing the benefits of high-quality training data.

(2) Both standard SFT and SFT on high-quality capability-guided responses learn by imitating selected responses. They cannot identify weak capabilities and adjust their training focus based on new feedback during training. In contrast, RSRL identifies weak capabilities during training, guides exploration toward them, and uses capability-residual credit routing to help the model improve these capabilities. Our existing SFT baseline also provides evidence for the benefits of this learning process.

(3) We further conducted an SFT experiment using 1,000 high-quality rollouts selected from our training data, each with a rubric score of at least 0.75. These responses cover more than 600 questions. We trained an SFT model on these responses and evaluated it on four benchmarks.

Table R1. SFT results using high-quality training rollouts.

MethodResearchQASQAv2GPQA-DiamondRubricHub
RSRL71.9285.8447.6055.01
Guided SFT69.8284.3843.6050.50

These results further support the effectiveness of our credit assignment and reinforcement learning mechanisms.

[W1 & Q2] How sensitive are the gains to guidance strength, gating thresholds, and residual-routing hyperparameters?

Re: Our parameters were selected based on theoretical analysis of our method, rather than task-specific tuning. We further present (1) comparisons of three guidance strengths (1×, 3×, and 4×) in Appendix E and (2) downstream ablations of adaptive gating and residual credit routing to examine the effects of these design choices.

(1) Appendix E reports 1×, 3×, and 4× guidance on the same 99 valid triggered questions.

Table R2. Guidance-strength analysis on 99 triggered questions (Appendix E). Target gain is on the 0–1 scale.

Measure1×3×4×
Mean target gain0.00175−0.004920.01282
Median ESS/G0.9910.9200.866

In the update comparison, the capability-macro scores are 50.29 (1×) and 50.82 (4×), differing by only 0.53 points. The final weakest-capability scores are 42.27 and 40.97, respectively. These results suggest that the overall performance remains relatively stable across the tested settings, while guidance strength affects the balance between targeted exploration and sampling overlap.

(2) We also compared full RSRL with w/o Adaptive Gating and w/o Residual Routing on the four benchmarks.

Table R3. Downstream scores with different gating and routing rules.

MethodResearchQASQAv2GPQA-DiamondRubricHub
RSRL71.9285.8447.6055.01
w/o Adaptive Gating69.1682.5844.6152.20
w/o Residual Routing66.4483.8645.6251.76

Both alternative settings achieve lower scores than full RSRL across all four benchmarks. These results support our choices of adaptive guidance and residual credit routing.

[Q3] Does cross-question capability history improve performance over using only question-local diagnostics?

Re: We added w/o History which uses only question-local scores to select weak capabilities.

Table R4. Effect of cross-question history on the four benchmarks

MethodResearchQASQAv2GPQA-DiamondRubricHub
RSRL71.9285.8447.6055.01
w/o History67.0584.6046.4651.81

RSRL scores higher on all four benchmarks when cross-question capability history is used together with question-local diagnosis.

[Q4] What is the additional computational cost relative to Rubric-RL, including diagnosis, guided sampling, and importance-weight computation?

Re: We report the GPU-hours for both methods under the same optimization rollout budget.

Table R5. Approximate total training cost.

MethodGPU-hours
Rubric-RL69
RSRL103

RSRL uses about 34 additional GPU-hours. The added cost comes mainly from diagnostic rollouts, guided sampling and additional distribution evaluations, and importance-weight calculations.

[W2 & Q5] Are the improvements consistent across independent training seeds, particularly on benchmarks with small gains?

Re: We thank the reviewer for this valuable suggestion and agree that independent training seeds are important for evaluating the stability of our method. Since our training and evaluation require considerable time and computational resources, we are currently conducting additional experiments with multiple training seeds.

[W1 & Q6] Can the same hyperparameter settings transfer to larger models or other rubric-based tasks without substantial retuning?

Re: We thank the reviewer for this valuable suggestion and agree that extending RSRL to larger models is important. Our parameter choices are based on theoretical analysis of the method rather than tuning for a specific model size. We are also conducting additional experiments on larger models and other rubric-based tasks to further evaluate the generalizability of RSRL.

Reviewer_HETnOpenReview ↗

Thank you for your comments

Thank you for the additional experiments and for reporting the training costs. I share other reviewers’ concerns about how well RSRL generalizes to larger models and other rubric-based tasks. Given the limited evidence on that question and the roughly 50% increase in GPU-hours relative to Rubric-RL under the same optimization rollout budget, I will keep my original score.

Reviewer_eCYiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors extend rubric-based RL with three components: Rubric-to-Capability, Self-Sufficient Capability Learning, and retention of capability-specific gains. The resulting method improves performance over the original approach.

Critical strengths

None.

Critical weaknesses
  • The ablation study should be extended to cover the experimental settings in Table 1. The current results do not establish how much each component contributes to the overall improvement.
  • The method’s effectiveness should be demonstrated on at least two models, preferably from different model families.
  • The three stages introduce several hyperparameters, including κ\kappa, τ\tau, βD\beta_D, and ξ\xi. How were these hyperparameters selected?
Questions for authors
  • What does “learning structure” mean? Could the authors provide a concrete example?
  • In line 78, where do the figures of 11.52% and 11.81% come from? Why is the improvement over the RL baseline greater than the improvement over the initial policy?
8 comments

We sincerely thank the reviewer for the detailed review and insightful questions which have greatly helped us improve the completeness of our method and experimental analysis. We have added downstream ablations and clarified the parameter choices and reported gains. We address W1-W3 here and Q1-Q2 in Comment2.

[W1] The ablation study should be extended to cover the experimental settings in Table 1. The current results do not establish how much each component contributes to the overall improvement.

Re: We added three ablations evaluated on ResearchQA, SQAv2, GPQA-Diamond, and RubricHub, using the Table 1 evaluation settings.

  • w/o History selects weak capabilities using question-local scores.
  • w/o Adaptive Gating applies guided sampling more broadly, while keeping the deficit-based strength rule.
  • w/o Residual Routing uses simple capability-weighted advantages in place of residual credit routing.

Table R1. Additional component ablations on the four benchmarks.

MethodResearchQASQAv2GPQA-DiamondRubricHub
RSRL71.9285.8447.6055.01
w/o History67.0584.6046.4651.81
w/o Adaptive Gating69.1682.5844.6152.20
w/o Residual Routing66.4483.8645.6251.76

RSRL scores higher than each variant across the four benchmarks. The results support using cross-question history, adaptive gating, and residual credit routing together. The Simple weighted comparison also shows the gain from our routing rule over direct capability weighting.

[W2] The method’s effectiveness should be demonstrated on at least two models, preferably from different model families.

Re: We thank the reviewer for this valuable suggestion and acknowledge this limitation. Training and evaluating additional models require considerable time and computational resources. We are currently conducting experiments on other model families.

[W3] The three stages introduce several hyperparameters. How were these hyperparameters selected?

Re: We selected these parameters based on theoretical analysis of their roles in our method. We explain the reasons for each parameter choice below.

  • For τ=0.75\tau=0.75, rubric scores range from 0 to 1 with 0.5 as the middle point. We consider scores above 0.75 to indicate strong capabilities. Thus, we use 0.75 to identify weak capabilities that need improvement.

  • For κ=8\kappa=8, each question has only four diagnostic responses so the scores may not be stable. We set 8, twice the number of responses. This gives one-third of the weight to current scores and two-thirds to historical scores to make the diagnosis more stable.

  • For βD=0.9\beta_D=0.9, our training has 100 steps. We do not want the historical capability scores to change too quickly because of a few batches or to remain almost unchanged throughout training. if we choose 0.9 , about 35% of the initial historical weight remains after 10 updates. This provides a balance between stability and adaptation.

  • For ξ=0.5\xi=0.5, we first calculate the difference between the target weak capability advantage and the overall advantage. Since the overall advantage may already contain useful learning signals for weak capabilities, we want to keep part of the shared information. If we choose 0, no overlap is removed from this difference while if we choose 1, all the overlap is removed, which may lead to the loss of useful information. Therefore, we choose 0.5 to remove half of the overlap while keeping the other half.

We use these settings throughout the reported training experiments.

[Q1] What does “learning structure” mean? Could the authors provide a concrete example?

Re: We group rubric criteria from different questions that require the same capability into a shared channel. The channel scores are then used for diagnosis, guided exploration, and learning credit.

For example, one question may ask the model to discuss the limits of an experiment, while another asks it to explain the assumptions behind a conclusion. Both involve assessing evidence and assumptions, so their related rubric criteria are grouped into one capability channel. When this capability is weak, RSRL uses it to guide sampling and give more learning credit to responses that improve it.

[Q2] In line 78, where do the figures of 11.52% and 11.81% come from? Why is the improvement over the RL baseline greater than the improvement over the initial policy?

Re: We explain (1) the source of the reported numbers and (2) why the improvement over the RL baseline is larger than that over the initial policy.

(1) The numbers refer to relative improvements on ResearchQA. Table 1 reports 71.92 for RSRL, 64.49 for Initial, and 64.32 for the RL baseline. The absolute improvements are 7.43 and 7.60 score points, corresponding to relative improvements of (71.92 − 64.49) / 64.49 × 100% = 11.52% and (71.92 − 64.32) / 64.32 × 100% = 11.82%.

(2) The RL baseline scores 64.32, slightly below the initial score of 64.49 on ResearchQA. Our standard RL baseline uses an LLM judge to give one overall score from the question and response without question-specific rubric criteria. This gives less detailed feedback about individual capabilities.

  1. The ablation studies for one of the paper’s central innovations should appear in the main text, along with the experimental details, including how the hyperparameters were selected.

  2. I did not find the authors’ example of a “learning structure” convincing. A concrete example drawn from the current evaluation set would help clarify what this structure adds. In the spirit of Occam’s razor, I remain skeptical that the rubric needs this additional layer of structure. My impression is that these design choices are primarily tuned to the benchmarks used for the leaderboard.

Reviewer_17dpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Rubric-Structured Reinforcement Learning (RSRL), which uses rubric criteria to guide capability development rather than treating them only as reward signals. RSRL consists of three components: capability structuring, adaptive guided exploration, and capability-specific credit routing. Experiments on scientific question-answering benchmarks show improvements over standard RL and rubric-based RL baselines.

Critical strengths

1.The paper explores how rubric structure can guide capability-level learning beyond conventional reward aggregation. 2.RSRL improves performance across several scientific QA benchmarks. The component ablations also show the potential benefits of guided exploration and capability-specific credit assignment.

Critical weaknesses

1.RSRL trains on guided responses under the original question context, which may cause semantic inconsistencies when responses depend on the additional guidance. Importance weighting does not directly address this issue. Moreover, the ablations do not isolate capability mapping, adaptive gating, or individual components of residual credit routing, making their necessity unclear. 2.RSRL requires additional diagnostic rollouts, multiple conditional distributions, and importance-weight calculations. However, the experiments only match optimization rollout budgets, not total computational cost. Without GPU-hour or wall-clock comparisons, it is difficult to assess whether the gains justify the overhead.

Questions for authors

1.How do the authors ensure that guided responses remain coherent under the original question context? 2.Could the authors report total training costs and provide compute-matched comparisons with simpler rubric-based RL methods? 3.Could the authors provide separate ablations for capability mapping, adaptive gating, and the components of residual credit routing? In particular, how does the proposed routing compare with a simple capability-weighted advantage baseline? 4.The experiments focus on scientific question-answering tasks, and it remains unclear whether RSRL generalizes to other domains.

1 comment

Thank you for your careful review and for recognizing the capability-level learning perspective of RSRL, the benefits of guided exploration and credit routing. Your comments have helped us make the method and the new results clearer. We respond to each question below.

[W1 & Q1] How do the authors ensure that guided responses remain coherent under the original question context?

Re: We explain our approach from (1) the guided sampling strategy and (2) the source of guidance.

(1) We do not directly add capability guidance to the original question prompt. Instead, we combine the original policy distribution with capability-guided information to construct a new sampling strategy. We then use both guided and unguided sampling to generate training trajectories.

(2) The guidance comes from the rubric criteria of the original question rather than additional information. These criteria describe what a good response should satisfy. Thus, guided and unguided responses follow the same task requirements and are evaluated using the same rubrics.

[W2 & Q2] Could the authors report total training costs and provide compute-matched comparisons with simpler rubric-based RL methods?

Re: We have added the training costs of RSRL and Rubric-RL.

Table R1. Training costs and results.

MethodGPU-hourResearchQASQAv2GPQA-DiamondRubricHub
Rubric-RL6969.3784.0245.2054.93
RSRL10371.9285.8447.6055.01

RSRL uses about 34 more GPU-hours, mainly for diagnostic rollouts, guided sampling, evaluating additional token distributions, and importance-weight calculations.

[W1 & Q3] Could the authors provide separate ablations for capability mapping, adaptive gating, and the components of residual credit routing? In particular, how does the proposed routing compare with a simple capability-weighted advantage baseline?

Re: We added three ablations on the four benchmarks in Table 1. We describe (1) the ablation settings and (2) the experimental results.

(1) We set three ablation settings:

  • w/o History uses only question-local scores to select weak capabilities.
  • w/o Adaptive Gating applies guidance more broadly while keeping the deficit-based guidance strength.
  • w/o Residual Routing replaces our routing rule with simple capability-weighted advantages.

(2) We evaluate these three ablations on the four downstream benchmarks.

Table R2. Additional downstream ablation results.

MethodResearchQASQAv2GPQA-DiamondRubricHub
RSRL71.9285.8447.6055.01
w/o History67.0584.6046.4651.81
w/o Adaptive Gating69.1682.5844.6152.20
w/o Residual Routing66.4483.8645.6251.76

RSRL achieves the highest scores on all four benchmarks. Removing cross-question history or adaptive gating reduces performance. Replacing residual routing with simple capability weighting also leads to lower scores across all four benchmarks.

[Q4] The experiments focus on scientific question-answering tasks, and it remains unclear whether RSRL generalizes to other domains.

Re: We thank the reviewer for this valuable suggestion and acknowledge the current limitation. We are conducting additional experiments on other domains. We believe RSRL can be extended to other domains for two reasons: (1) shared capabilities across questions within a domain and (2) the applicability of our method across different domains.

(1) Different questions within the same domain often require shared capabilities. We groups these capabilities into channels and uses them to guide learning so RSRL is not limited to scientific question answering.

(2) Our guidance is based on the rubrics of each question rather than rules designed specifically for scientific tasks. Questions in other domains can also have rubrics that describe the capabilities needed for a good response. Therefore, the same method can be used to identify weak capabilities, guide exploration, and assign learning credit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.