- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how rubric feedback can guide reinforcement learning. RITR uses the judge's confidence and differences between answers to select and weight criteria. It can rewrite one answer when all trusted answers fail a criterion. The judge must mark the rewrite as passing its targets and preserving earlier passes on other criteria. The extra guidance is later removed. Separate policies are trained for medicine, science, and finance. The authors evaluate seven benchmarks and study triage, rewriting, and the guidance removal schedule.
Critical strengths
-
The method addresses two useful problems. It selects rubric comparisons for learning. It also seeks a better example when trusted answers all fail a criterion. The rewrite is checked on criteria that its parent already passes. This helps detect fixes that break other requirements.
-
The reported gains are promising. RITR improves on six of seven benchmarks over Base Rubric RL. It shows gains of 4.66 points on HealthBench-500 and 4.95 on LLMEval-Med. The comparison shares the initial policy, data, reward judge, and eight-response budget. Adding repair improves all four medical scores over triage alone.
Critical weaknesses
-
Table 3 changes several things together. It is hard to tell how much each part helps.
-
RITR uses the judge's probability scores, while the baseline only uses pass/fail labels. This difference could explain some of the gains.
-
The baseline is also not tested with the same rewriting and preservation checks. A fairer comparison would help show what the new reward design adds.
-
Each experiment has only one training run. It is hard to know if the smaller gains are consistent.
-
The paper does not clearly state which graders are used for the open-ended evaluations.
Questions for authors
- How does Base Rubric RL perform with continuous judge scores instead of pass/fail labels? Please keep the rubric importance weights unchanged. Within RITR, how much changes when only the reliability multipliers are set to 1?
- How does Base Rubric RL perform with the same rewrite targets, parent choice, preservation check, and removal schedule? Please keep the eight-response budget and explain how accepted and rejected rewrites are scored.
- Which graders and decoding settings produced the open-ended evaluation scores? Were the graders also used for training rewards? Could the central medical comparison be repeated across training seeds to show whether the gains are stable?
4 comments
Response to Reviewer YUUX
B. Same-protocol rewrite control and scoring rules (W3, Q2)
Scoring in RITR works as follows. A rewrite that passes the gate is scored by Eq. (13), that is, the parent's main-channel score plus a net-improvement term that is clipped, multiplied by the absolute value of the parent score, and scaled by the schedule. A rewrite that fails the gate receives a main-channel score of 0. Because centered scores of the ordinary responses sum to zero within the group, a rejected rewrite has zero reward advantage under Eq. (14), while still entering group normalization (Appendix B.5).
Both branches occur in training. In stage one, for queries with a DEGRADED target, the trigger rate is 100%; in stage two it follows the schedule. Of the rewrites actually generated, the gate accepts about 40%, and the other six in ten are set to zero by the rule above.
Applying the same trigger, target selection, parent choice, validation gate, two-stage schedule, and eight-response budget to Base Rubric RL gives the fair control you ask for.
C. Statistical stability, evaluation grader, and overhead (W4, W5, Q3)
Evaluation uses Qwen3-235B-Instruct as grader, and training uses Qwen3-14B as reward judge. The two differ, so the open-ended comparison is not obtained on the training judge. Grader decoding and scoring settings will be listed per benchmark in the final version.
On grader preference: all three RL methods use the same actor initialization (Qwen3-8B), the same reward judge (Qwen3-14B), and the same evaluation grader (Qwen3-235B-Instruct). They also share the training data and the budget of eight responses per prompt. Because actor, reward model, and grader are held fixed, the differences between methods come from the training method itself. The relative gains in the table therefore reflect, to a reasonable degree, improvements from the method. A single grader may still prefer the outputs of one method over another, for example by favoring longer or differently styled answers. The current results therefore cannot fully rule out method-specific evaluation bias. If sufficient resources become available, we will add a multi-grader consistency study to test whether these relative gains depend on a single evaluator.
On overhead: training steps in the rewrite-active phase take about 10% longer on average than standard eight-response rollouts. Rewriting is active only in the first 200 steps, so the overhead is confined to that phase. A detailed cost discussion will appear in the final version.
On additional experiments: Due to time and resource constraints, we cannot complete this experiment during the rebuttal period. If the paper is accepted, we will add it in the final version. This round we completed the closed-ended medical evaluation of continuous probabilities plus rewriting (table above). Still missing are the two controls of pure continuous scores without rewriting and r_k = 1, the same-protocol rewrite control, and multi-seed retraining. HealthBench-500 and LLMEval-Med require a deployed open-ended grader, and our current resources are severely limited. The current results support the overall performance of the full method and the gain from triage. The independent contribution of each sub-component and its stability remain unverified.
Thank you for clarifying the grader setup and adding the new baseline. The additional results provide useful support for the full method on the closed-ended tasks. However, I still cannot tell how much reliability weighting itself contributes, or how much of the open-ended gain remains against a baseline with the same rewriting setup. With one training run per configuration, the stability of the gains also remains unclear. I appreciate the authors’ efforts, but these concerns remain important to my assessment, so I will maintain my current rating.
Response to the follow-up comment
We thank the reviewer for the follow-up. For one of the remaining attribution questions, we found a controlled comparison in our existing training logs.
The comparison changes a single factor. Both runs use Qwen3-8B as reward model and Qwen3-235B-A22B-Instruct-2507 as evaluation grader, with the same training data and the same budget of eight responses per prompt. One run uses the triage reward without boundary normalization. The other adds boundary normalization. Everything else is identical.
| Setting | HealthBench-500 |
|---|---|
| baseline | 57.05 |
| reliability triage, without boundary normalization | 58.56 |
| reliability triage, with boundary normalization | 59.88 |
Both variants are above the baseline, by 1.51 and 2.83 points. In this controlled comparison, boundary normalization alone accounts for 1.32 points. The two runs share four-state routing, confidence margins, and reliability weighting. The 1.51-point gain from the baseline to the non-normalized variant is therefore the joint effect of these shared steps. Boundary normalization adds 1.32 points on top.
We hope this comparison helps in judging the contribution of the individual steps inside triage. It further decomposes the triage components and shows that boundary normalization alone accounts for 1.32 points. If these results are helpful to your assessment, we would be grateful if you would take them into account.
Response to Reviewer YUUX
We thank the reviewer for the comments and constructive suggestions. We group our reply by topic.
A. Component attribution: Table 3, continuous scores, and r_k = 1 (W1, W2, Q1)
Table 3 shows that, without rewriting, triage-only gives higher point estimates on the two open-ended medical benchmarks (HealthBench-500 +3.20, LLMEval-Med +0.60), while MedQA is -1.41 and MedMCQA is +0.17. The independent contributions of continuous probabilities replacing 0/1 labels and of the reliability multiplier are not separated.
The steps inside triage are complementary by design. Four-state routing selects criteria with trusted cross-boundary verdicts. Boundary normalization builds the contrast statistic of each criterion. The reliability multiplier adjusts the relative contribution of multiple ACTIVE criteria. This is why we first evaluate triage as a whole, namely the triage-only row of Table 3. Each step can still be tested with explicit controls. A continuous-probability aggregation baseline with the original rubric weights tests continuous scores. Setting r_k = 1 in full RITR tests the increment of the reliability multiplier. We do not have either result this round, so the claim of Table 3 is limited to the empirical gain of triage as a whole (see our reply to WvNc W1 for the reasoning).
One analytic property: group standardization removes common scale factors. With a single ACTIVE criterion and no rewriting, r_k is a common positive factor on the whole score vector. Ignoring the stability term ε, this factor cancels exactly; with ε kept, it cancels approximately when the score standard deviation is well above ε (§5.6, Appendix B.6). The reliability multiplier therefore mainly sets the relative contribution across multiple ACTIVE criteria, rather than an absolute attenuation.
The two controls you suggest, namely continuous-score aggregation with the original rubric weights and r_k = 1 in full RITR, are exactly the identifying experiments.
For the first one, we completed a simplified version this round. It aggregates judge probabilities directly with the original rubric weights and reuses the same targeted rewriting, without the reliability multiplier, boundary normalization, or four-state routing. It is not the pure continuous-score version without rewriting. Closed-ended medical results follow.
This combination is above Base Rubric RL (MedQA +0.57, MedMCQA +0.93) and below full RITR (-2.03 / -1.39). Two points follow. First, continuous probabilities plus rewriting alone already give a small positive transfer on closed-ended medical tasks. Second, the full method adds further gain on top of it. The two gains therefore cannot be explained by switching to continuous probabilities or by adding targeted rewriting alone. Note that this combination contains both, so it cannot separate their contributions. HealthBench-500 and LLMEval-Med require a deployed open-ended grader. Our current resources are severely limited, so we cannot run these two evaluations this round.