- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Rubric-Structured Reinforcement Learning (RSRL), which uses rubric-derived capability structure to coordinate weakness diagnosis, guided exploration, and credit assignment. It groups question-specific criteria into shared capability categories and combines local scores with historical performance to identify weaknesses. Its RL method steers trajectory generation by contrasting capability-guided and control-prompt token distributions, then learns from guided and unguided trajectories under the original question context using importance weighting. A residual credit-routing mechanism further adjusts response-level advantages to emphasize improvements in the targeted capability. Experiments report gains on scientific question-answering benchmarks.
However, the framework introduces numerous interacting hyperparameters, including diagnostic thresholds, history-pooling coefficients, guidance strengths, clipping bounds, and residual projection and scaling parameters. These choices jointly affect the sampled trajectories, importance weights, and advantages, thereby influencing the magnitude and direction of policy-gradient updates. Although the reported experiments support the method’s effectiveness in the evaluated setting, the limited sensitivity analysis and evaluation primarily on a single model size and task domain leave its training stability and generalizability insufficiently established. Broader robustness studies would help determine whether the gains persist across models and domains without substantial retuning.
Critical strengths
An interesting capability-oriented perspective. Identifying and addressing specific capability deficits is a compelling direction for rubric-based RL. Using a shared capability diagnosis to coordinate both exploration and policy updates offers a potentially novel contribution beyond aggregate reward optimization.
A conceptually motivated optimization mechanism. Capability-guided sampling seeks to produce useful behaviors that ordinary rollouts may rarely discover, while residual credit routing aims to prevent improvements in weak capabilities from being diluted by aggregate advantages. Importance weighting further accounts for the change in the sampling distribution. These components provide a coherent rationale for the proposed approach.
Supporting ablation evidence. The paper separately removes guided exploration and residual credit routing, with both ablations reducing performance on the capability development probe. These results support the contribution of both components to the gains observed in the evaluated setting.
Critical weaknesses
Sensitivity to interacting hyperparameters and limited evidence of robustness. RSRL combines historical capability estimates, diagnostic thresholds, guided sampling, importance weighting, and residual advantage routing. The associated hyperparameters jointly affect which trajectories are sampled and how they contribute to policy-gradient updates. Because capability estimates and the policy evolve together during online training, the interactions among these mechanisms may introduce considerable tuning complexity. This concern is particularly relevant when each question’s local diagnosis relies on only four sampled responses. The reported component ablations support the usefulness of guided exploration and residual routing, but do not establish robustness to the choices governing these mechanisms. More systematic sensitivity analysis and evaluation across models and rubric domains would help substantiate the framework’s broader applicability.
Unclear incremental benefit over directly fine-tuning on guided trajectories. A more direct baseline would apply SFT to high-quality capability-guided trajectories under the original question context. Comparing RSRL against this targeted SFT baseline would clarify whether its credit-assignment and RL machinery provides sufficient additional gains to justify the added complexity and computational cost. The existing generic SFT baseline does not answer this question, as it does not use the same targeted guided trajectories. Furthermore, the experiments focus on Qwen3-4B and do not report variability across independent training seeds. The four GPQA evaluation repetitions measure inference variability rather than training stability. Reporting training-seed variability and uncertainty in benchmark differences would strengthen confidence in the improvements, especially where margins over Rubric-RL are small.
Questions for authors
-
How does RSRL compare with directly applying SFT to high-quality capability-guided trajectories under the original question context?
-
How sensitive are the gains to guidance strength, gating thresholds, and residual-routing hyperparameters?
-
Does cross-question capability history improve performance over using only question-local diagnostics?
-
What is the additional computational cost relative to Rubric-RL, including diagnosis, guided sampling, and importance-weight computation?
-
Are the improvements consistent across independent training seeds, particularly on benchmarks with small gains?
-
Can the same hyperparameter settings transfer to larger models or other rubric-based tasks without substantial retuning?
3 comments
[Q3] Does cross-question capability history improve performance over using only question-local diagnostics?
Re: We added w/o History which uses only question-local scores to select weak capabilities.
Table R4. Effect of cross-question history on the four benchmarks
| Method | ResearchQA | SQAv2 | GPQA-Diamond | RubricHub |
|---|---|---|---|---|
| RSRL | 71.92 | 85.84 | 47.60 | 55.01 |
| w/o History | 67.05 | 84.60 | 46.46 | 51.81 |
RSRL scores higher on all four benchmarks when cross-question capability history is used together with question-local diagnosis.
[Q4] What is the additional computational cost relative to Rubric-RL, including diagnosis, guided sampling, and importance-weight computation?
Re: We report the GPU-hours for both methods under the same optimization rollout budget.
Table R5. Approximate total training cost.
| Method | GPU-hours |
|---|---|
| Rubric-RL | 69 |
| RSRL | 103 |
RSRL uses about 34 additional GPU-hours. The added cost comes mainly from diagnostic rollouts, guided sampling and additional distribution evaluations, and importance-weight calculations.
[W2 & Q5] Are the improvements consistent across independent training seeds, particularly on benchmarks with small gains?
Re: We thank the reviewer for this valuable suggestion and agree that independent training seeds are important for evaluating the stability of our method. Since our training and evaluation require considerable time and computational resources, we are currently conducting additional experiments with multiple training seeds.
[W1 & Q6] Can the same hyperparameter settings transfer to larger models or other rubric-based tasks without substantial retuning?
Re: We thank the reviewer for this valuable suggestion and agree that extending RSRL to larger models is important. Our parameter choices are based on theoretical analysis of the method rather than tuning for a specific model size. We are also conducting additional experiments on larger models and other rubric-based tasks to further evaluate the generalizability of RSRL.
Thank you for your comments
Thank you for the additional experiments and for reporting the training costs. I share other reviewers’ concerns about how well RSRL generalizes to larger models and other rubric-based tasks. Given the limited evidence on that question and the roughly 50% increase in GPU-hours relative to Rubric-RL under the same optimization rollout budget, I will keep my original score.
We thank the reviewer for the thoughtful review and for recognizing the capability-oriented perspective and the connection between guided exploration and credit assignment. We also appreciate the reviewer's detailed questions, which have been very helpful in improving the rigor and completeness of our paper. We address Q1-Q2 here and Q3-Q6 in Comment 2.
Re: We thank the reviewer for this valuable suggestion and agree that Guided SFT is an important comparison. We have conducted an additional SFT experiment using high-quality training rollouts. We first explain how our existing SFT baseline provides evidence for the effectiveness of credit assignment and RL machinery from two aspects: (1) training data quality and (2) capability-based credit assignment. We then present (3) the evaluation results of SFT on high-quality capability-guided trajectories under the original question context.
(1) Our existing SFT baseline already uses high-quality responses selected from multiple candidates. Similarly, capability-guided rollouts are generated to satisfy the question-specific rubrics. Since both types of responses are evaluated using the same rubric criteria, we expect their overall quality to be comparable. Thus, our existing SFT baseline provides a useful comparison for assessing the benefits of high-quality training data.
(2) Both standard SFT and SFT on high-quality capability-guided responses learn by imitating selected responses. They cannot identify weak capabilities and adjust their training focus based on new feedback during training. In contrast, RSRL identifies weak capabilities during training, guides exploration toward them, and uses capability-residual credit routing to help the model improve these capabilities. Our existing SFT baseline also provides evidence for the benefits of this learning process.
(3) We further conducted an SFT experiment using 1,000 high-quality rollouts selected from our training data, each with a rubric score of at least 0.75. These responses cover more than 600 questions. We trained an SFT model on these responses and evaluated it on four benchmarks.
Table R1. SFT results using high-quality training rollouts.
These results further support the effectiveness of our credit assignment and reinforcement learning mechanisms.
Re: Our parameters were selected based on theoretical analysis of our method, rather than task-specific tuning. We further present (1) comparisons of three guidance strengths (1×, 3×, and 4×) in Appendix E and (2) downstream ablations of adaptive gating and residual credit routing to examine the effects of these design choices.
(1) Appendix E reports 1×, 3×, and 4× guidance on the same 99 valid triggered questions.
Table R2. Guidance-strength analysis on 99 triggered questions (Appendix E). Target gain is on the 0–1 scale.
In the update comparison, the capability-macro scores are 50.29 (1×) and 50.82 (4×), differing by only 0.53 points. The final weakest-capability scores are 42.27 and 40.97, respectively. These results suggest that the overall performance remains relatively stable across the tested settings, while guidance strength affects the balance between targeted exploration and sampling overlap.
(2) We also compared full RSRL with w/o Adaptive Gating and w/o Residual Routing on the four benchmarks.
Table R3. Downstream scores with different gating and routing rules.
Both alternative settings achieve lower scores than full RSRL across all four benchmarks. These results support our choices of adaptive guidance and residual credit routing.