acceptodds
Under review as a conference paper at ICLR 2027

RITR: Reliability-Informed Triage and Constrained Rewriting for Rubric-Based Reinforcement Learning

Abstract

Rubric-based reinforcement learning (RL) extends training to open-ended tasks, but standard importance-weighted aggregation ignores the reliability of grading evidence. Group normalization can turn judge fluctuations into unit-scale advantages when responses lack genuine quality differences, while uniformly failed criteria offer no successful examples. We propose RITR, a diagnose-before-repair framework that decomposes judge scores into decision direction and confidence. A criterion-level reliability proxy and four-state triage admit only sufficiently confident cross-boundary criteria to the main reward and control their relative weights. For criteria failed by all trusted responses, RITR allocates one sample from a fixed response budget to a targeted rewrite of the best-scoring response. The rewrite is gated on target satisfaction and preservation of previously satisfied criteria, and its scaffold is removed in two stages. Starting from Qwen3-8B, we train separate policies on medical, science, and finance rubric data. RITR improves over uniform-aggregation rubric RL on six of seven benchmarks, by up to 4.95 points on open-ended medical evaluation and 2.60 points on medical multiple-choice tasks. Medical ablations show open-ended gains from triage and further gains from constrained repair.

Primary area:
reinforcement learning
Keywords:
rubric-based reinforcement learning, large language models, open-ended generation, judge reliability, targeted rewriting.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YUUXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how rubric feedback can guide reinforcement learning. RITR uses the judge's confidence and differences between answers to select and weight criteria. It can rewrite one answer when all trusted answers fail a criterion. The judge must mark the rewrite as passing its targets and preserving earlier passes on other criteria. The extra guidance is later removed. Separate policies are trained for medicine, science, and finance. The authors evaluate seven benchmarks and study triage, rewriting, and the guidance removal schedule.

Critical strengths
  • The method addresses two useful problems. It selects rubric comparisons for learning. It also seeks a better example when trusted answers all fail a criterion. The rewrite is checked on criteria that its parent already passes. This helps detect fixes that break other requirements.

  • The reported gains are promising. RITR improves on six of seven benchmarks over Base Rubric RL. It shows gains of 4.66 points on HealthBench-500 and 4.95 on LLMEval-Med. The comparison shares the initial policy, data, reward judge, and eight-response budget. Adding repair improves all four medical scores over triage alone.

Critical weaknesses
  • Table 3 changes several things together. It is hard to tell how much each part helps.

  • RITR uses the judge's probability scores, while the baseline only uses pass/fail labels. This difference could explain some of the gains.

  • The baseline is also not tested with the same rewriting and preservation checks. A fairer comparison would help show what the new reward design adds.

  • Each experiment has only one training run. It is hard to know if the smaller gains are consistent.

  • The paper does not clearly state which graders are used for the open-ended evaluations.

Questions for authors
  • How does Base Rubric RL perform with continuous judge scores instead of pass/fail labels? Please keep the rubric importance weights unchanged. Within RITR, how much changes when only the reliability multipliers are set to 1?
  • How does Base Rubric RL perform with the same rewrite targets, parent choice, preservation check, and removal schedule? Please keep the eight-response budget and explain how accepted and rejected rewrites are scored.
  • Which graders and decoding settings produced the open-ended evaluation scores? Were the graders also used for training rewards? Could the central medical comparison be repeated across training seeds to show whether the gains are stable?
4 comments
AuthorsOpenReview ↗

Response to Reviewer YUUX

We thank the reviewer for the comments and constructive suggestions. We group our reply by topic.

A. Component attribution: Table 3, continuous scores, and r_k = 1 (W1, W2, Q1)

Table 3 shows that, without rewriting, triage-only gives higher point estimates on the two open-ended medical benchmarks (HealthBench-500 +3.20, LLMEval-Med +0.60), while MedQA is -1.41 and MedMCQA is +0.17. The independent contributions of continuous probabilities replacing 0/1 labels and of the reliability multiplier are not separated.

The steps inside triage are complementary by design. Four-state routing selects criteria with trusted cross-boundary verdicts. Boundary normalization builds the contrast statistic of each criterion. The reliability multiplier adjusts the relative contribution of multiple ACTIVE criteria. This is why we first evaluate triage as a whole, namely the triage-only row of Table 3. Each step can still be tested with explicit controls. A continuous-probability aggregation baseline with the original rubric weights tests continuous scores. Setting r_k = 1 in full RITR tests the increment of the reliability multiplier. We do not have either result this round, so the claim of Table 3 is limited to the empirical gain of triage as a whole (see our reply to WvNc W1 for the reasoning).

One analytic property: group standardization removes common scale factors. With a single ACTIVE criterion and no rewriting, r_k is a common positive factor on the whole score vector. Ignoring the stability term ε, this factor cancels exactly; with ε kept, it cancels approximately when the score standard deviation is well above ε (§5.6, Appendix B.6). The reliability multiplier therefore mainly sets the relative contribution across multiple ACTIVE criteria, rather than an absolute attenuation.

The two controls you suggest, namely continuous-score aggregation with the original rubric weights and r_k = 1 in full RITR, are exactly the identifying experiments.

For the first one, we completed a simplified version this round. It aggregates judge probabilities directly with the original rubric weights and reuses the same targeted rewriting, without the reliability multiplier, boundary normalization, or four-state routing. It is not the pure continuous-score version without rewriting. Closed-ended medical results follow.

ModelMedQAMedMCQA
Initial (Qwen3-8B)68.1159.28
Base Rubric RL67.7158.69
Continuous prob. + rewriting (new)68.2859.62
RITR (full method)70.3161.01

This combination is above Base Rubric RL (MedQA +0.57, MedMCQA +0.93) and below full RITR (-2.03 / -1.39). Two points follow. First, continuous probabilities plus rewriting alone already give a small positive transfer on closed-ended medical tasks. Second, the full method adds further gain on top of it. The two gains therefore cannot be explained by switching to continuous probabilities or by adding targeted rewriting alone. Note that this combination contains both, so it cannot separate their contributions. HealthBench-500 and LLMEval-Med require a deployed open-ended grader. Our current resources are severely limited, so we cannot run these two evaluations this round.

AuthorsOpenReview ↗

Response to Reviewer YUUX

B. Same-protocol rewrite control and scoring rules (W3, Q2)

Scoring in RITR works as follows. A rewrite that passes the gate is scored by Eq. (13), that is, the parent's main-channel score plus a net-improvement term that is clipped, multiplied by the absolute value of the parent score, and scaled by the schedule. A rewrite that fails the gate receives a main-channel score of 0. Because centered scores of the ordinary responses sum to zero within the group, a rejected rewrite has zero reward advantage under Eq. (14), while still entering group normalization (Appendix B.5).

Both branches occur in training. In stage one, for queries with a DEGRADED target, the trigger rate is 100%; in stage two it follows the schedule. Of the rewrites actually generated, the gate accepts about 40%, and the other six in ten are set to zero by the rule above.

Applying the same trigger, target selection, parent choice, validation gate, two-stage schedule, and eight-response budget to Base Rubric RL gives the fair control you ask for.

C. Statistical stability, evaluation grader, and overhead (W4, W5, Q3)

Evaluation uses Qwen3-235B-Instruct as grader, and training uses Qwen3-14B as reward judge. The two differ, so the open-ended comparison is not obtained on the training judge. Grader decoding and scoring settings will be listed per benchmark in the final version.

On grader preference: all three RL methods use the same actor initialization (Qwen3-8B), the same reward judge (Qwen3-14B), and the same evaluation grader (Qwen3-235B-Instruct). They also share the training data and the budget of eight responses per prompt. Because actor, reward model, and grader are held fixed, the differences between methods come from the training method itself. The relative gains in the table therefore reflect, to a reasonable degree, improvements from the method. A single grader may still prefer the outputs of one method over another, for example by favoring longer or differently styled answers. The current results therefore cannot fully rule out method-specific evaluation bias. If sufficient resources become available, we will add a multi-grader consistency study to test whether these relative gains depend on a single evaluator.

On overhead: training steps in the rewrite-active phase take about 10% longer on average than standard eight-response rollouts. Rewriting is active only in the first 200 steps, so the overhead is confined to that phase. A detailed cost discussion will appear in the final version.

On additional experiments: Due to time and resource constraints, we cannot complete this experiment during the rebuttal period. If the paper is accepted, we will add it in the final version. This round we completed the closed-ended medical evaluation of continuous probabilities plus rewriting (table above). Still missing are the two controls of pure continuous scores without rewriting and r_k = 1, the same-protocol rewrite control, and multi-seed retraining. HealthBench-500 and LLMEval-Med require a deployed open-ended grader, and our current resources are severely limited. The current results support the overall performance of the full method and the gain from triage. The independent contribution of each sub-component and its stability remain unverified.

Thank you for clarifying the grader setup and adding the new baseline. The additional results provide useful support for the full method on the closed-ended tasks. However, I still cannot tell how much reliability weighting itself contributes, or how much of the open-ended gain remains against a baseline with the same rewriting setup. With one training run per configuration, the stability of the gains also remains unclear. I appreciate the authors’ efforts, but these concerns remain important to my assessment, so I will maintain my current rating.

AuthorsOpenReview ↗

Response to the follow-up comment

We thank the reviewer for the follow-up. For one of the remaining attribution questions, we found a controlled comparison in our existing training logs.

The comparison changes a single factor. Both runs use Qwen3-8B as reward model and Qwen3-235B-A22B-Instruct-2507 as evaluation grader, with the same training data and the same budget of eight responses per prompt. One run uses the triage reward without boundary normalization. The other adds boundary normalization. Everything else is identical.

SettingHealthBench-500
baseline57.05
reliability triage, without boundary normalization58.56
reliability triage, with boundary normalization59.88

Both variants are above the baseline, by 1.51 and 2.83 points. In this controlled comparison, boundary normalization alone accounts for 1.32 points. The two runs share four-state routing, confidence margins, and reliability weighting. The 1.51-point gain from the baseline to the non-normalized variant is therefore the joint effect of these shared steps. Boundary normalization adds 1.32 points on top.

We hope this comparison helps in judging the contribution of the individual steps inside triage. It further decomposes the triage components and shows that boundary normalization alone accounts for 1.32 points. If these results are helpful to your assessment, we would be grateful if you would take them into account.

Reviewer_oS58OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on challenges in rubric-based reinforcement learning training and proposes RITR, a method that aims to improve how LLMs learn from rubric-based rewards by filtering out unhelpful grading signals and focusing on cases where all trusted model responses fail a particular criterion. Specifically, the method introduces four routing states: NOISE, ACTIVE, MASTERED, and DEGRADED. These states determine which criteria should contribute to the reward and which should receive additional attention during RL training. The main contribution of this paper is the idea that not every graded rubric criterion should be used as a training reward. Additionally, when the model fails certain criteria, the method provides guidance to generate an improved response and uses that response during RL training. The authors train Qwen3-8B with RITR on medical, science, and finance rubric datasets and report better performance than their standard rubric-RL baseline on six out of seven benchmark tasks.

Critical strengths

The paper points out an important problem in current rubric-based RL training. Specifically, some rubric criteria are not informative when all sampled responses fail those criteria, as they provide no successful examples for group-relative learning. RITR proposes a framework that addresses this issue through reward selection and targeted rewriting. I like that the authors provide a structured approach to deciding how different rubric criteria should be handled during training. Instead of treating all criteria equally, they introduce four routing states to distinguish between criteria that provide useful training signals and those that may require additional exploration. This makes the overall approach more interpretable. The paper shows promising empirical results across different domains. It also shows improvements on medical multiple-choice benchmarks, even though the medical RL training data does not contain multiple-choice supervision. This suggests that the method may help with generalization beyond the original training task format.

Critical weaknesses
  • The results are mainly compared against Base Rubric RL, RuscaRL, and reference models. However, there appear to be other algorithms more closely related to RITR that the authors did not include in their experimental comparisons. Therefore, it is unclear whether RITR actually achieves better performance than these closely related approaches. Also, although the paper shows improvements across several benchmarks, it does not demonstrate whether these improvements are consistent across different random seeds or provide confidence intervals (CIs). Some of the reported gains are also relatively small.

  • Another concern I have is that the authors introduce many different components. While this is interesting, the ablation studies do not sufficiently isolate these components to help us understand which ones are actually responsible for the improvements. It's possible that some of these components are not necessary. It's not clear to me how much the confidence scores and reliability-based components contribute to the overall performance improvements. To me, it seems possible that rewriting alone could account for much of the performance gain.

Questions for authors
  1. I wonder how important the confidence-weighted scores are in this algorithm. If we remove confidence-based weighting and simply use standard rubric-based GRPO with targeted rewriting on failed criteria, how much of RITR's improvement remains? For example, providing feedback about what the model missed and allowing it to revise its previous response can often lead to better answers, as explored in existing work.

  2. In RITR, the authors use yes/no token probabilities as a measure of confidence. However, LLM judges can also be confidently incorrect. How do we know whether this confidence correlates with actual rubric-grading accuracy?

  3. What does the distribution of each routing state (ACTIVE, MASTERED, NOISE, DEGRADED) look like at the beginning of training, at intermediate training steps, and at the end of training?

  4. Could excluding MASTERED criteria from the main reward lead to forgetting or regression on previously satisfied criteria during training?

  5. The method uses fixed thresholds of (m = 0.5) and (\theta_c = 0.6). Did the authors perform a hyperparameter search to select these values, or what is the rationale behind these choices?

  6. Is the comparison fair in terms of hyperparameter tuning?

2 comments
AuthorsOpenReview ↗

Response to Reviewer oS58

We thank the reviewer for the comments and sharp insight. We group our reply by topic.

A. Strength of evidence and attribution of rewriting (W1, W2, Q1)

Table 3 already provides the control without rewriting. triage-only improves HealthBench-500 by 3.20 and LLMEval-Med by 0.60, so the open-ended gains are not produced by rewriting alone. Adding rewriting to triage raises the four medical scores by a further 1.46 / 4.35 / 4.01 / 2.15. The triage row tests the reward design as a whole. Its internal steps are complementary by design and can be tested separately with explicit controls (see our reply to WvNc W1).

Rewriting is used frequently in training. In stage one, for queries with a DEGRADED target, the trigger rate is 100%; in stage two it follows the cosine schedule. Of the rewrites actually generated, the gate accepts about 40%; the other six in ten fail the joint target-satisfaction and preservation checks.

To separate rewriting from the reward design, we ran a simplified combination this round. It aggregates judge probabilities directly with the original rubric weights and reuses the same targeted rewriting, without the reliability multiplier, boundary normalization, or four-state routing. We evaluated it on the closed-ended medical tasks.

ModelMedQAMedMCQA
Initial (Qwen3-8B)68.1159.28
Base Rubric RL67.7158.69
Continuous prob. + rewriting (new)68.2859.62
RITR (full method)70.3161.01

This combination is above Base Rubric RL (MedQA +0.57, MedMCQA +0.93) and below full RITR (-2.03 / -1.39). Two points follow. First, continuous probabilities plus rewriting alone already give a small positive transfer on closed-ended medical tasks. Second, the full method adds further gain on top of it. The two gains therefore cannot be explained by switching to continuous probabilities or by adding targeted rewriting alone. Note that this combination contains both, so it cannot separate their contributions. HealthBench-500 and LLMEval-Med require a deployed open-ended grader. Our current resources are severely limited, so we cannot run these two evaluations this round.

Two controls remain unfinished: standard binary Rubric GRPO with the same rewriting protocol, and same-protocol comparisons with the other close baselines. The existing comparison of Base Rubric RL, RuscaRL, and RITR shares training data, initialization, reward judge, and response budget. Each configuration is a single training run, so the table reports point estimates.

B. Dynamics and forgetting of the four states (Q3, Q4)

The four states are recomputed for every rollout group and are not persistent labels. Their distribution therefore reflects dynamic routing during training. As training proceeds, MASTERED rises from the low sixties to about three quarters, ACTIVE and DEGRADED decline overall, and NOISE stays low. This is consistent with more criteria being confidently and consistently passed, and with fewer criteria needing within-group separation or targeted repair. Curves will be added to the appendix before the rebuttal deadline.

Because states are recomputed per group, MASTERED is not a permanent mask. If trusted verdicts on both sides reappear in a later group, the criterion returns to ACTIVE and again provides within-group signal. If all trusted verdicts fail and the quality gate passes, it becomes DEGRADED and a candidate for rewriting. The gate also requires that non-target criteria the parent already satisfied remain satisfied. Long-term retention of a single criterion cannot be measured from aggregate scores.

AuthorsOpenReview ↗

Response to Reviewer oS58

C. Threshold choice, tuning fairness, and reliability of the grading signal (Q2, Q5, Q6)

We set m = 0.5 and θ_c = 0.6 exploratorily from the four-state distribution on offline grading data, not from benchmark scores. With m = 0.5, a single judgment enters the trusted set only if g_ik < 0.25 or g_ik > 0.75. With θ_c = 0.6, the mean squared margin must exceed 0.6, which is equivalent to a mean g(1 - g) below 0.10. The two thresholds constrain the routing direction of a single judgment and the evidence quality of a whole group.

All methods run with their original settings. Training configuration is in Table 4, and data and sampling settings are in §6.1. All methods share data, initialization, reward judge, and the budget of eight responses per prompt.

On the grading signal behind these statistics: h_ik = |2g_ik - 1| is the confidence the judge expresses, not its accuracy. The expected-agreement reading of Lemma 1 needs a calibration assumption, which we do not test here. Table R1 (in our reply to WvNc) shows 95.60% at the two ends and only 0.79% in [0.4, 0.6). This indicates that the judge is decisive on this data, but it is not evidence that the verdicts are correct. On grader preference: all three RL methods use the same actor initialization (Qwen3-8B), the same reward judge (Qwen3-14B), and the same evaluation grader (Qwen3-235B-Instruct). They also share the training data and the budget of eight responses per prompt. Because actor, reward model, and grader are held fixed, the differences between methods come from the training method itself. The relative gains in the table therefore reflect, to a reasonable degree, improvements from the method. A single grader may still prefer the outputs of one method over another, for example by favoring longer or differently styled answers. The current results therefore cannot fully rule out method-specific evaluation bias. If sufficient resources become available, we will add a multi-grader consistency study to test whether these relative gains depend on a single evaluator.

On additional experiments: Due to time and resource constraints, we cannot complete this experiment during the rebuttal period. If the paper is accepted, we will add it in the final version. This round we completed the closed-ended medical evaluation of continuous probabilities plus rewriting (table above). Still missing are the control of standard rubric GRPO with the same rewriting, the same-protocol comparison with close baselines, and multi-seed and threshold-sensitivity experiments. HealthBench-500 and LLMEval-Med require a deployed open-ended grader, and our current resources are severely limited. The current results show that the rewriting branch triggers frequently in training and that the gate rejects a substantial share of candidates. The standalone contribution of rewriting and stability across seeds remain unverified.

Reviewer_WvNcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework, RITR (Reliability-Informed Triage and Constrained Rewriting), for optimizing Rubric-Based Reinforcement Learning (Rubric-RL) under GRPO. The authors outline the deficiencies of standard importance-weighted reward aggregation: noise-dominated group normalization (where standardizing rollouts amplifies within-class judge fluctuations) and zero-contrast exploration bottlenecks (where all trusted responses fail a criterion). To mitigate these problems, they propose decomposing aligned token probabilities into a decision direction and confidence margin, triaging criteria into four states (ACTIVE, MASTERED, DEGRADED, NOISE), and applying a reliability proxy. For DEGRADED criteria, the framework uses a single targeted rewrite of the best diagnostic response, guarded by a cross-criterion validation gate. They perform an empirical study across medical, science, and finance domains to conclude that RITR outperforms uniform-aggregation baselines on open-ended tasks and exhibits strong positive transfer to closed-ended multiple-choice benchmarks.

Overall, this paper could be a significant algorithmic contribution to Reinforcement Learning from Human Feedback, offering a highly principled formulation and impressive cross-format generalization, with the caveat for some clarifications on the theoretical edge cases, ablation design, and evaluation setup.

Critical strengths

1- The mathematical exposition is rigorous and exceptionally transparent. It clearly distinguishes idealized calibration assumptions from observable heuristics and thoroughly documents edge cases, such as sign inversions under centering and single-criterion scale cancellation.

2- The boundary normalization elegantly ensures same-side probability fluctuations automatically yield zero main-channel signal.

3- Anchoring the single rewrite on the best diagnostic response, requesting minimal edits, and enforcing a preservation gate on non-target criteria effectively prevents multidimensional reward hacking and capability forgetting.

4- The method consistently improves open-ended rubric adherence across multiple domains (healthcare, science, finance) and impressively reverses the performance degradation seen in Base Rubric RL on closed-ended medical exams (MedQA, MedMCQA) despite using zero multiple-choice supervision.

Critical weaknesses

1- The ablation study bundles continuous confidence margins, boundary normalization, four-state routing, and reliability weighting into a single configuration, compared against a binary-thresholded baseline. It is difficult to isolate the effect of the reliability proxy itself versus the benefit of simply providing continuous verifier probabilities. It would have been better to also show the performance of a continuous-score uniform-aggregation baseline.

2- The paper cites recent methods for dynamic criterion reweighting and unmet-criterion-guided exploration, but does not compare against them empirically. Direct comparisons to these adaptive baselines would strengthen the argument for RITR's specific triage and repair mechanics.

3- The paper lacks logged internal training telemetry (e.g., the proportion of criteria routed to each of the four states, validation gate acceptance rates, and frequency of degenerate groups), which would help explain edge-case behaviors, such as the observed regression on dense PRBench rubrics.

Questions for authors

1- The paper does not specify the LLM judge model or decoding hyperparameters used at test time to grade the open-ended benchmarks. If Qwen3-14B was used for both training and evaluation, the results might be confounded by same-judge overfitting. Do the gains hold when evaluated by an independent judge (e.g., GPT-4o)?

2- Can you provide more context on the diagnostic telemetry? Specifically, reporting the empirical percentages of criteria routed to the four states, the rewrite trigger rate, and the validation gate pass rate would help explain edge-case behaviors, such as the observed regression on PRBench.

3- What is the computational cost of the sequential rewrite dependency? I would like to see some discussion on the actual wall-clock training time per step and total token overhead of RITR relative to Base Rubric RL, especially during the steps where rewrites are active.

4- There appears to be a mismatch in the reliability proxy between equations 7 and 8. The between-class variance is confidence-weighted, whereas the within-class uncertainty uses a uniform measure. Could you discuss how this proxy behaves under extreme polarization or single-confident-response degeneracy?

5- How frequently do passing verdicts actually receive negative centered contributions in practice? Did you consider using a sign-preserving centering scheme instead?

4 comments
AuthorsOpenReview ↗

Response to Reviewer WvNc

We thank the reviewer for the comments and constructive suggestions. We group our reply by topic.

A. Contribution attribution and internal training diagnostics (W1, W3, Q2)

The triage-only row of Table 3 (HealthBench-500 +3.20, LLMEval-Med +0.60, MedQA -1.41, MedMCQA +0.17) shows that the open-ended gains do not come from rewriting.

You identify the attribution limit of our ablation. The triage-only row of Table 3 jointly tests continuous confidence margins, boundary normalization, four-state routing, and reliability weighting. Without rewriting, it improves HealthBench-500 by 3.20 and LLMEval-Med by 0.60 points. This shows that the complete triage reward design already helps on the two open-ended medical benchmarks. It does not show how much of the gain comes from continuous scores and how much from the reliability proxy.

These steps are complementary by design. Routing selects criteria with trusted cross-boundary verdicts. Boundary normalization builds the contrast statistic of each criterion. The reliability multiplier then adjusts the relative contribution of multiple ACTIVE criteria. This relation explains why we evaluate the design as a whole first. It does not prevent testing each step with explicit controls. A continuous-probability aggregation baseline that keeps the original rubric weights isolates the effect of continuous scores. Setting r_k = 1 in full RITR isolates the increment of the reliability multiplier.

We do not have either result this round. We therefore limit the claim of Table 3 to the empirical gain of triage as a whole, and we withhold judgment on the independent contribution of the reliability proxy. §6.3 and the Limitations state this scope.

For the continuous-score part, we ran a simplified combination this round. It aggregates judge probabilities directly with the original rubric weights and reuses the same targeted rewriting, without the reliability multiplier, boundary normalization, or four-state routing. We evaluated it on the closed-ended medical tasks.

ModelMedQAMedMCQA
Initial (Qwen3-8B)68.1159.28
Base Rubric RL67.7158.69
Continuous prob. + rewriting (new)68.2859.62
RITR (full method)70.3161.01

This combination is above Base Rubric RL (MedQA +0.57, MedMCQA +0.93) and below full RITR (-2.03 / -1.39). Two points follow. First, continuous probabilities plus rewriting alone already give a small positive transfer on closed-ended medical tasks. Second, the full method adds further gain on top of it. The two gains therefore cannot be explained by switching to continuous probabilities or by adding targeted rewriting alone. Note that this combination contains both, so it cannot separate their contributions. HealthBench-500 and LLMEval-Med require a deployed open-ended grader. Our current resources are severely limited, so we cannot run these two evaluations this round.

Internal training statistics (medical, 500 steps) show the overall trend of the four states. MASTERED rises from the low sixties to about three quarters. ACTIVE and DEGRADED decline overall. NOISE stays low (about 2%). The step-by-step curves will appear in the updated paper we submit after the rebuttal. State shares are recomputed for every rollout group.

On the frequency of degenerate groups: we counted, per rollout group, whether the group contains any ACTIVE criterion. The share of groups with no ACTIVE criterion is 0. The degenerate case discussed in §5.5, where an empty ACTIVE set makes every main-channel score zero and leaves no reward-driven gradient, therefore did not occur in the groups covered by this statistic. Every group contains at least one criterion with a trusted cross-boundary verdict. Note that ACTIVE describes a trusted cross-boundary verdict from the judge, not a verified difference in true quality.

On the rewriting statistics: in stage one, for queries with a DEGRADED target, the rewrite trigger rate is 100% (ρ_0 = 1.0 in this stage). In stage two, the rewrite probability follows the cosine decay described in the paper (about 0.5 at the midpoint of stage two), and rewriting stops afterwards. For rewrites actually generated, the validation gate acceptance rate is about 40%. That is, about four in ten rewrites pass both the target-satisfaction check and the preservation check. A rewrite that passes the gate is scored by Eq. (13). Its net-improvement term and final advantage still depend on the parent anchor and on score changes of the other criteria. About six in ten rewrites fail this joint gate, so the gate does filter a substantial share of candidate rewrites. The aggregate acceptance rate does not separate target failures from preservation failures.

The PRBench gap (33.57 vs 34.79) still requires a per-criterion analysis; due to time constraints, this will be conducted in future work.

AuthorsOpenReview ↗

Response to Reviewer WvNc

B. Comparison scope, evaluation judge, and compute overhead (W2, Q1, Q3)

Direct comparisons are Base Rubric RL and RuscaRL, with shared data, initialization, reward judge, and eight responses per prompt. The two closest lines of work are saturation-driven dynamic reweighting and guidance for unmet criteria. RITR decides from within-group confidence margins and from whether a trusted cross-boundary contrast exists. Rewriting is triggered by DEGRADED criteria and checked by the preservation gate.

Evaluation uses Qwen3-235B-Instruct as grader, and training uses Qwen3-14B as reward judge. The two differ in scale and checkpoint, so the reported gains are not obtained on the training judge. Decoding settings will be listed per benchmark in the final version.

On grader preference: all three RL methods use the same actor initialization (Qwen3-8B), the same reward judge (Qwen3-14B), and the same evaluation grader (Qwen3-235B-Instruct). They also share the training data and the budget of eight responses per prompt. Because actor, reward model, and grader are held fixed, the differences between methods come from the training method itself. The relative gains in the table therefore reflect, to a reasonable degree, improvements from the method. A single grader may still prefer the outputs of one method over another, for example by favoring longer or differently styled answers. The current results therefore cannot fully rule out method-specific evaluation bias. If sufficient resources become available, we will add a multi-grader consistency study to test whether these relative gains depend on a single evaluator.

On overhead: on the same hardware and batch size, training steps in the rewrite-active phase take about 10% longer on average than standard eight-response rollouts. This is consistent with the sequential generation dependency and the longer rewrite input. Rewriting is active only in the first 200 steps (ρ(t) > 0), so the overhead is confined to that phase. We do not have per-module timing. Details will be reported in the final version after the measurement is re-run.

C. Reliability proxy and edge cases of centering (Q4, Q5)

The asymmetry in Eq. (7)–(8) is a design choice. The dispersion term is weighted by c_ik = h_ik², so high-margin judgments dominate the question of whether a group has real separability. The uncertainty term g_ik(1 - g_ik) uses a uniform average, so every valid judgment contributes to the noise estimate. λ_k is therefore an observable proxy. The exact interpretation in Proposition 2 covers only the calibrated case with uniform weights on both terms.

Extreme cases: if valid scores sit at both 0 and 1, dispersion is positive and the within-group uncertainty u_k = 0, so λ_k = 1. This reflects confident disagreement, and the verdicts may still be wrong. If only one response has a nonzero margin and all others sit exactly at 0.5, the weighted dispersion is 0. Even with two valid judgments, the mean squared margin is at most 0.5, below θ_c = 0.6, so the quality gate routes the criterion to NOISE.

A negative contribution for a passing verdict follows from the explicit centering in Eq. (11). A passing response receives a negative contribution when its pre-centering boundary statistic is below the criterion's group mean. Example C in Appendix B.4 shows this algebraic property before routing (the statistics change from [0.896, 0.090, -0.201] to [0.634, -0.172, -0.462]). Table R1 shows that judgments concentrate at the two ends: 95.60% fall in [0, 0.1) and [0.9, 1], and only 0.79% fall in [0.4, 0.6). This marginal distribution keeps no rollout grouping and is not restricted to ACTIVE criteria, so it cannot determine the actual sign-flip rate within the ACTIVE subset. We do not yet have that rate, nor a comparison with a sign-preserving variant.

Table R1: distribution of criterion-level p(yes) over 1,970,344 judgments, bin width 0.1.

IntervalCountShare (%)
[0.0, 0.1)770,00839.08
[0.1, 0.2)15,7030.80
[0.2, 0.3)10,3850.53
[0.3, 0.4)10,1150.51
[0.4, 0.5)5,1810.26
[0.5, 0.6)10,3300.52
[0.6, 0.7)9,9590.51
[0.7, 0.8)10,1420.51
[0.8, 0.9)14,9250.76
[0.9, 1.0]1,113,59656.52
Total1,970,344100.00

On additional experiments: Due to time and resource constraints, we cannot complete this experiment during the rebuttal period. If the paper is accepted, we will add it in the final version. This round we completed the closed-ended medical evaluation of continuous probabilities plus rewriting (table above). HealthBench-500 and LLMEval-Med require a deployed open-ended grader, and our current resources are severely limited. The current results support the comparison of the full method against the listed baselines, and the incremental gain of triage and of adding rewriting.

I appreciate the authors' engagement and for providing more experiments to address my concerns. However, considering the overall quality and contribution of the paper, I'll maintain my original score.

AuthorsOpenReview ↗

Response to the follow-up comment

Thank you for the careful reading and for giving our response fair consideration. We understand your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.