acceptodds
Under review as a conference paper at ICLR 2027

CriPO: Enhancing Rubric-based RL via Self-Distillation

Abstract

Rubric-based Reinforcement Learning (RL) has recently shown promise in improving Large Language Models (LLMs) on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (*Unexplored Criteria*) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout generation, yet they introduce a *train-inference mismatch*: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term *Suppressed Criteria*—criteria that are satisfied by some rollouts yet whose learning signals might be lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that suppressed criteria constitute a persistent and non-negligible failure mode—over 25% of samples exhibit this issue throughout training. To simultaneously address both unexplored and suppressed criteria without introducing training-inference mismatch, we propose **Criterion-Distilled Policy Optimization (CriPO)**, which enhances rubric-based RL via on-policy self-distillation. For unexplored criteria, CriPO constructs a behavior-injection teacher and computes a filtered forward-KL loss to inject missing behaviors into the policy. For suppressed criteria, CriPO uses a counterfactual teacher to locate criterion-relevant tokens in negative-advantage rollouts, and corrects their advantages in GRPO to preserve useful patterns. Experiments on medicine and science benchmarks demonstrate that CriPO outperforms existing rubric-based RL methods, e.g., achieving an average gain of **3.3** points over GRPO on Qwen3-4B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Rubric-based Reinforcement Learning, Large Language Models, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3TFCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CriPO, which combines GRPO with on-policy self-distillation for rubric-based training. It distinguishes criteria that no sampled response satisfies from criteria whose satisfying responses receive non-positive aggregate advantages. A rubric-conditioned self-teacher provides auxiliary supervision for the former, while a counterfactual teacher identifies tokens for advantage correction in the latter. Experiments evaluate the method on medicine and science QA tasks.

Critical strengths
  1. Useful problem formulation: Distinguishing unexplored and suppressed criteria makes the limitations of aggregate rubric rewards more concrete. The two targeted modules provide a clear connection between the diagnosis and the method.
  2. Substantial empirical support: The experiments include in-domain and cross-domain tasks, multiple model sizes, component ablations, different judges, seed variation, and a random-token control. Together, these provide useful evidence for the overall improvement.
Critical weaknesses
  1. Token localization needs more direct validation: Low probability under the counterfactual teacher may reflect rewriting or formatting changes rather than criterion-specific content. The random-token control is helpful, but a control matched by token type and position would better establish the specificity of the selected tokens (Section 4.3; Appendix D.2).
  2. Criterion trade-offs need clearer analysis: A criterion-satisfying response can have a negative advantage because it performs poorly on more important criteria. It is unclear when local advantage correction preserves useful behavior without weakening these intended trade-offs. Per-criterion gains and regressions would help clarify this.
Questions for authors
  1. Does editing the selected tokens affect the intended criterion more than editing control tokens matched by type and position?
  2. When rubric criteria conflict, how does advantage correction affect the higher-weighted criteria? Please report both criterion-level improvements and regressions.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 3TFC (1/2)

Thank you for acknowledging our work and providing the constructive comments! We hope the following responses can address your remaining concerns.

Q1: Effect of alternative SC token localization strategy (controlled type and position)

To address your concern, we compare CriPO's SC token selection strategy with a type-and-position-matched control, and evaluate them along three dimensions: (i) whether deleting the selected tokens changes the rubric judge's decision, (ii) how much the advantage correction (AC) module mitigates the suppressed criteria after one update training step, and (iii) end-to-end performance after full training. These experiments show that CriPO's localization method is more closely associated with SC-related behaviors and enables more effective mitigation of their suppression when used with AC.

The compared variant: Type-and-position-matched localization. For each token selected by CriPO, we choose the nearest eligible token in the same response that is not selected by CriPO, has not already been selected yet, and belongs to the same one of the five token types used in the paper (in Table 3): formatting/structural boundaries, knowledge/semantic content, logical/discourse connectors, grammatical function words, and numbers/option labels. Categories are assigned by deterministic lexical rules over decoded token pieces.

All experiments use Qwen3-4B as the policy model on RaR-Medicine and Qwen3-32B as the LLM judge. Experiments 1 and 2 use the first five training batches (64 queries per batch, 320 queries in total). In addition to type-and-position-matched localization, we report random token selection with the same token budget as CriPO. We refer to these two variants as CriPO-Type/Position and CriPO-Random, respectively.

Experiment 1: Direct deletion. Do the selected tokens affect the judge's decision?

We edit the original negative-advantage responses at positions selected by CriPO, CriPO-Type/Position, or CriPO-Random and ask the rubric judge to re-evaluate the target SC. We use two editing modes. Token only deletes only the selected token pieces. Through delimiter deletes each selected token and the subsequent local segment up to punctuation, a newline, or a Markdown boundary. We include the latter because deleting a single token may not fully capture its influence on the subsequent content of the response. Loss rate is the fraction of originally satisfied target criteria that become unmet after editing. The pooled evaluation contains 273 responses and 194 target criteria judged satisfied before editing. As shown in the table below, deleting tokens selected by CriPO causes the greatest loss of SC satisfaction under both editing modes, suggesting that CriPO's localized tokens are more closely associated with the target SCs.

Selection StrategyEditing ModeLost / originally satisfiedLoss rate
CriPOToken only19/1949.79%
CriPO-Type/PositionToken only11/1945.67%
CriPO-RandomToken only14/1947.22%
CriPOThrough delimiter31/19415.98%
CriPO-Type/PositionThrough delimiter29/19414.95%
CriPO-RandomThrough delimiter22/19411.34%

However, the methods compared in Experiment 1 directly delete tokens from the original rollouts, which may leave incomplete sentences and affect the LLM judge's accuracy. To evaluate CriPO's token localization more reliably, we conduct the following two additional experiments.

Experiment 2: One-update mitigation of suppressed criteria

We measure how the different AC variants affect the satisfaction of the same 117 diagnosed SCs after one update. For each training batch, we (i) sample 32 responses per query before the update to estimate each criterion's pre-update satisfaction rate, (ii) use the original eight training rollouts per query to diagnose SCs and perform one update for each variant, and (iii) independently sample 32 responses per query from each updated variant to estimate post-update satisfaction rates. We use a relatively large one-step learning rate of 10−510^{-5} to make the effect of the update measurable. As shown in the table below, the average satisfaction rate of the suppressed criteria decreases by 6.28 percentage points when training with CriPO w/o AC. All variants alleviate this suppression to some extent after adding the advantage correction module, with CriPO achieving the most pronounced mitigation.

MethodSC satisfaction rate changeMitigation gain
CriPO w/o AC-6.28--
CriPO-4.73+1.55
CriPO-Type/Position-6.06+0.21
CriPO-Random-5.61+0.67
AuthorsOpenReview ↗

Response to Reviewer 3TFC (2/2)

Experiment 3. End-to-end training

The completed end-to-end comparison shows that CriPO achieves the highest score among these variants on both benchmarks, outperforming CriPO-Type/Position by 1.1 percentage points on RaR-Medicine and 0.3 points on HealthBench.

MethodRaR-MedicineHealthBench
CriPO w/o AC61.464.2
CriPO63.165.6
CriPO-Type/Position62.065.3
CriPO-Random61.665.1

We have added this analysis in our revised paper in Appendix C.13 (where we also compares a teacher-student probability-difference selection strategy).

Q2: Trade-offs with conflict high-weight criteria

Thank you for raising this insightful question. Our additional experiments reveal a trade-off between optimizing suppressed criteria and improving conflict high-weight criteria. Even so, the reduction in the latter's satisfaction-rate gain is small compared with the improvement on suppressed criteria.

Specifically, we define conflict high-weight criteria as criteria with weight 9 or 10 (the two highest levels in RaR-Medicine's weight set {3,7,9,10}\{3,7,9,10\}) that are unmet in at least one SC-satisfying training rollout, excluding all suppressed criteria (SC). After deduplication, we analyze 117 SC and 230 conflict high-weight criteria. We follow the setting of Q1, Experiment 2 (One-step mitigation) which can evaluate the per-criterion satisfaction rate. Δ\Delta denotes the mean pre-to-post satisfaction-rate change in percentage points, and ↑\uparrow and ↓\downarrow denote the fractions of criteria whose satisfaction rates increase and decrease, respectively.

MethodSC Δ\DeltaSC ↑\uparrowSC ↓\downarrowConflict high-weight criteria Δ\DeltaConflict high-weight criteria ↑\uparrowConflict high-weight criteria ↓\downarrow
CriPO w/o AC-6.2832.48%60.68%+8.1850.00%18.70%
CriPO-4.7340.17%55.56%+7.9648.70%21.74%

As shown in the table, AC mitigates the intended SC suppression, where the mean SC change improves by 1.55 percentage points over CriPO w/o AC, the improved fraction rises from 32.48% to 40.17%, and the regressed fraction falls from 60.68% to 55.56%. Conflict high-weight criteria retain a mean satisfaction-rate improvement of 7.96 percentage points, only 0.22 percentage points below CriPO w/o AC (+8.18). Together, quantifying results of the trade-off shows that AC mitigates SC suppression while retaining most of the average satisfaction-rate improvement on conflict high-weight criteria.

We have added this analysis to Appendix C.13 in our revised paper.

Reviewer_ftz8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CriPO, a rubric-based RL method that addresses two problems: unexplored criteria and suppressed criteria. It uses self-distillation to introduce missing behaviors and corrects token-level advantages to preserve useful behaviors. Experiments on medicine and science benchmarks show improvements over several existing methods.

Critical strengths
  1. The paper identifies an interesting problem with scalar reward aggregation in rubric-based RL. The distinction between unexplored and suppressed criteria is meaningful.

  2. The method is well motivated, with two components targeting different problems.

  3. The experiments are fairly comprehensive, including multiple models, ablations, and efficiency analysis.

Critical weaknesses
  1. The contribution of advantage correction is not fully clear. Table 11 shows that adding advantage correction only improves the average score by 0.6 and 1.5 points on Qwen3-1.7B and Qwen3-4B, respectively. It is unclear whether these relatively small gains are consistent across random seeds. The ablation also lacks a GRPO + AC baseline, making it difficult to assess the independent contribution of this component.

  2. The reliability of token localization is not fully evaluated. The counterfactual teacher selects tokens based on their absolute probabilities, but low probability does not necessarily indicate relevance to a suppressed criterion. Although the random-selection ablation is helpful, a comparison with other localization strategies, such as teacher-student probability differences, would strengthen the method.

  3. The impact of response length is not fully analyzed. CriPO produces longer responses than GRPO, and Tables 12 and 13 show that some configurations achieve higher scores with substantially longer outputs. It is unclear how much of the improvement comes from better responses rather than simply covering more rubric criteria.

Questions for authors
  1. Can the authors provide results for the additional gains from advantage correction, along with an AC-only ablation?

  2. How does the proposed token localization compare with alternative approaches, such as using teacher-student probability differences?

  3. Do the performance improvements still hold when controlling for response length?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ftz8 (1/2)

Thank you for acknowledging our work and providing the constructive comments! We hope the following responses can address your remaining concerns.

Q1: Ablation results with AC-only variant on multiple seeds

To address your concern, we have added the GRPO+AC baseline and conducted a multi-seed ablation (using seed 40, 41, and 42; values of mean±std are reported) at both model scales on the medicine task, where Qwen3-32B is used as the LLM judge. Results in the table below shows that AC consistently improves the average score both as a standalone addition to GRPO (+0.7/+1.1 on 1.7B/4B model) and when combined with behavior injection (+0.6/+1.5 on 1.7B/4B model).

ModelMethodRaR-MedicineHealthBenchLLMEval-MedAvg.
Qwen3-1.7BGRPO36.7 ± 0.753.4 ± 0.825.1 ± 1.038.4 ± 0.1
GRPO+AC37.7 ± 0.853.3 ± 0.726.1 ± 1.439.1 ± 0.7
GRPO+BI41.0 ± 1.356.1 ± 0.727.5 ± 1.441.5 ± 0.7
GRPO+BI+AC (CriPO)41.6 ± 0.457.5 ± 0.327.2 ± 1.542.1 ± 0.5
Qwen3-4BGRPO57.6 ± 0.762.3 ± 0.438.7 ± 0.652.9 ± 0.4
GRPO+AC59.1 ± 0.363.0 ± 0.539.9 ± 1.054.0 ± 0.4
GRPO+BI61.3 ± 0.564.5 ± 0.238.8 ± 1.454.8 ± 0.4
GRPO+BI+AC (CriPO)63.0 ± 0.665.9 ± 0.339.9 ± 0.556.3 ± 0.3

Q2: Effect of alternative SC token localization strategy (pt−psp_t-p_s)

To address your concern, we compare CriPO's absolute teacher-probability (pt) threshold with the proposed teacher-student probability difference (pt-ps). Our main findings and rationale are as follows.

  1. Empirically, CriPO's absolute ptp_t thresholding is more effective at mitigating suppressed criteria and achieves higher end-to-end performance on both benchmarks.

  2. More importantly, subtracting psp_s can introduce task-related differences unrelated to SC localization, while directly using the absolute ptp_t is better aligned with the SC localization task. As the table below shows, ptp_t edits an existing response to remove SC-satisfying content given the response and the criteria, while psp_s answers the original query given the user query. Therefore, using pt−psp_t-p_s as a localization metric may instead introduce ambiguous semantics.

DistributionPurpose of the conditioning promptConditioning variables
ptp_t (counterfactual teacher)Edit the existing response to remove or modify SC-satisfying content(yi,Csi,yi,<t)(y_i,C_s^i,y_{i,<t})
psp_s (student)Answer the original user query(x,yi,<t)(x,y_{i,<t})

We have corrected the notation in line 254 to reflect that the counterfactual teacher is not conditioned on the query xx. For consistency with the paper, ptp_t denotes the SC counterfactual-teacher distribution pi,tTp^T_{i,t} in Section 3.2, and psp_s denotes the student distribution ptSp^S_t in Section 3.1, evaluated on the same rollout yiy_i. Both probabilities are evaluated at the observed token yi,ty_{i,t}.

Experimental Setup

We compare CriPO w/o AC with three AC variants that use different SC-token localization rules.

  • CriPO selects tokens with absolute counterfactual-teacher probability pt(yt)<0.1p_t(y_t)<0.1, retaining up to ζ\zeta tokens (default ζ=10\zeta=10) with the lowest probabilities per rollout.
  • CriPO-Δ\Delta selects tokens satisfying pt(yt)<ps(yt)p_t(y_t)<p_s(y_t), ranks them by δt=pt(yt)−ps(yt)\delta_t=p_t(y_t)-p_s(y_t) in ascending order, and retains up to ζ\zeta tokens per rollout.
  • CriPO-Random randomly selects the same number of tokens as CriPO in each rollout.

Both experiments use Qwen3-4B as the policy model on RaR-Medicine and Qwen3-32B as the rubric judge. We adopt the following two evaluation setups. On the one hand, the one-step experiment measures how the different AC variants affect the satisfaction of the same diagnosed SCs after one update. We use the first five training batches (64 queries per batch, 320 queries in total). For each batch, we (i) sample 32 responses per query before the update to estimate each criterion's pre-update satisfaction rate; (ii) use the original eight training rollouts per query to diagnose SCs and perform one update for each variant; (iii) independently sample 32 responses per query from each updated variant to estimate post-update satisfaction rates. We use a relatively large one-step learning rate of 10−510^{-5} to make the effect of the update measurable. On the other hand, the end-to-end experiment evaluates the effects of the different variants.

AuthorsOpenReview ↗

Response to Reviewer ftz8 (2/2)

Results of One-Step Experiment

Across the 117 diagnosed SCs, CriPO provides the largest mitigation gain over CriPO w/o AC of +1.55 percentage points, compared with +0.53 for CriPO-Δ\Delta and +0.67 for CriPO-Random. CriPO-Δ\Delta selects more tokens per active rollout (8.88 vs. 5.24) and already includes 1,162 of CriPO's 1,431 selected tokens (81.2%). This substantial overlap and larger correction budget accompany a smaller mitigation gain, consistent with the difference rule introducing additional prompt-induced disagreements that are less useful for preserving SCs.

MethodSelected Tokens per rolloutSC satisfaction rate changeMitigation gain
CriPO w/o AC---6.28--
CriPO5.24-4.73+1.55
CriPO-Δ\Delta8.88-5.74+0.53
CriPO-Random5.24-5.61+0.67

Results of End-to-End Experiment

The completed full-training comparison shows that CriPO outperforms CriPO-Δ\Delta by 1.8 percentage points on RaR-Medicine and 0.6 points on HealthBench. CriPO also outperforms random selection on both benchmarks. Scores are percentages.

MethodRaR-MedicineHealthBench
CriPO w/o AC61.464.2
CriPO63.165.6
CriPO-Δ\Delta61.365.0
CriPO-Random61.665.1

We have added these analyses in our revised paper in Appendix C.13 (where we also compare a type-and-position-matched selection strategy).

Q3: Comparison under Length Control

To assess the performance gains under explicit length control, we compare GRPO, CriPO, and two length-penalized variants, CriPO-LP1280 and CriPO-LP2560, using Qwen3-4B trained on RaR-Medicine for 400 steps. Following DAPO [1], we add a soft overlong penalty to the scalar rubric reward before computing group-relative advantages. The penalty starts at 1280 or 2560 tokens, respectively, decreases linearly to -1 at 5120 tokens (the default max response length), and has coefficient 1.

The training dynamics are shown in our revised paper in Appendix C.10 Figure 11, and we only report these metrics on the last training step here. The results indicates that, as the length penalty becomes stronger through a lower onset threshold, responses become shorter while evaluation scores decrease only modestly. Specifically, CriPO-LP2560 reduces the response length by approximately 210 tokens, while essentially preserving the evaluation performance of CriPO. CriPO-LP1280 further reduces the response length that approaches the GRPO length, with only a decrease of 1.3 percentage points from CriPO while remaining 4.0 points above GRPO. Both length-penalized variants also have lower final policy entropy than CriPO, while retaining higher entropy than GRPO. Overall, these results show that CriPO retains strong performance under explicit training-time length control.

MethodEval Score (%)EntropyResponse length (k)
GRPO57.80.120.78
CriPO-LP128061.80.170.83
CriPO-LP256063.10.171.08
CriPO63.10.191.29

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS 2025

Reviewer_kcVmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CriPO, a method that combining on-policy self-distillation (OPSD) with GRPO to address unexplored and suppressed criteria in rubric-based RL. CriPO selectively distills behaviors associated with unmet criteria and uses a counterfactual teacher to localize tokens for negative-advantage correction. Experiments on several benchmarks covering medicine and science domains show improvements over the evaluated baselines with modest additional training overhead.

Critical strengths
  • Clear and practically relevant problem formulation: The distinction between unexplored criterion-satisfying behaviors and observed behaviors receiving non-positive advantages provides a practical perspective on information loss in scalar rubric-reward aggregation.
  • Targeted and computationally efficient design: CriPO applies selective distillation and advantage correction to student-generated trajectories without requiring additional teacher rollouts. The reported average training-time overhead is less than GRPO, with improved convergence efficiency.
  • Broad empirical evaluation: Experiments cover multiple benchmarks, model scales and families, alternative judges, repeat runs, and detailed ablations, providing substantial evidence of improvements over GRPO.
Critical weaknesses
  • Missing comparisons with existing hybrid OPD–RL methods: Although this paper has cited SRPO, TRACE, SDAR, and RLSD, none is included as an experimental baseline. Comparisons against standalone OPSD and GRPO, together with partial ablations of CriPO, do not establish whether its UC/SC-specific selection and correction rules outperform existing routing or gating mechanisms, or whether the gains primarily reflect the general benefits of combining OPD with RL.
  • The suppressed-criterion diagnostic lacks direct validation against actual learning suppression: Equation 3 identifies suppressed criteria using the summed advantages of satisfying responses. However, this scalar statistic does not account for gradient directions or parameter sharing and therefore does not directly determine whether criterion satisfaction decreases after an update. The reported prevalence establishes that the diagnostic frequently triggers, rather than validating the claimed optimization mechanism.

Typos:

  • p2, Figure 1: 1 pts
  • p3, L131: criteria with no rollout managed to satisfy
Questions for authors

See the above Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kcVm (1/3)

Thank you for providing the constructive comments! We hope the following responses can address your concerns.

Q1: Comparison with existing hybrid OPD+RL methods

We have added comparisons with rubric-based adaptations of SRPO, TRACE, SDAR, and RLSD, using Qwen3-4B trained on RaR-Medicine and evaluated on RaR-Medicine and HealthBench. Overall, CriPO achieves the highest scores of 63.1% and 65.6%, exceeding the strongest adapted baseline by 2.8 and 1.9 percentage points, respectively, supporting the effectiveness of our components designed specifically for the challenges of rubric-based RL, namely UC/SC-specific selection and correction.

Specifically, these methods were originally developed for other training settings, so we adapt them to rubric-based RL as follows. The common RL sampling and training hyperparameters follow CriPO.

  • SRPO routes failed rollouts to OPSD to avoid ambiguous self-distillation on already-correct rollouts. We treat rollouts with rubric reward ≥0.8\geq 0.8 as successful and optimize them with GRPO, while routing the remaining rollouts to entropy-weighted OPSD with the full rubric as teacher context. We also retain SRPO's loss aggregation and entropy weighting mechanisms.
  • TRACE selects critical token segments for localized OPSD. We use Qwen3-32B to identify segments of the highest-advantage rollout that could be revised to satisfy UC, with selected tokens capped at 40% of the response length. This replaces CriPO's cumulative-KL filtering for UC token localization, with the remaining components, including advantage correction, retained.
  • SDAR gates teacher supervision to control noisy distillation signals. We provide the full rubric as teacher context in place of retrieved skills and apply gap gating on every rollout, using gate sharpness β=5.0\beta=5.0 and auxiliary-loss coefficient λSDAR=0.01\lambda_{\mathrm{SDAR}}=0.01.
  • RLSD uses self-distillation signals for token-level credit assignment within GRPO. We provide the full rubric as teacher context, set ϵw=0.2\epsilon_w=0.2, and linearly decay the mixing coefficient from λ=0.5\lambda=0.5 to zero over the first 100 steps.

The table below shows the comparison results on medicine tasks using Qwen3-4B, and Qwen3-32B is adopted as the LLM judge. Overall, CriPO achieves the highest score on both benchmarks, exceeding TRACE by 2.8 percentage points on RaR-Medicine and 1.9 points on HealthBench. SRPO underperforms GRPO on both benchmarks, which may reflect its reliance on OPSD-only updates for lower-reward rollouts. The adapted TRACE variant differs from CriPO in UC token localization and improves over GRPO by 2.5 and 1.0 points on the two benchmarks, while CriPO's cumulative-KL filtering yields further gains. SDAR performs similarly to GRPO on RaR-Medicine and scores 2.7 points lower on HealthBench, suggesting limited additional benefit from teacher-reliability gating in this setting. RLSD scores 1.1 and 2.4 points below GRPO on the two benchmarks, suggesting that its credit-assignment mechanism requires further adaptation to rubric-based RL.

MethodRaR-Medicine (gain)HealthBench (gain)
Qwen3-4B49.7 (0.0)59.5 (0.0)
+ GRPO57.8 (+8.1)62.7 (+3.2)
+ SRPO53.8 (+4.1)50.4 (-9.1)
+ TRACE60.3 (+10.6)63.7 (+4.2)
+ SDAR57.5 (+7.8)60.0 (+0.5)
+ RLSD56.7 (+7.0)60.3 (+0.8)
+ CriPO63.1 (+13.4)65.6 (+6.1)

We have added these comparison results in Appendix C.7 in our revised paper.

AuthorsOpenReview ↗

Response to Reviewer kcVm (2/3)

Q2: Whether the rollout-level SC diagnostic reflects actual learning suppression?

Thank you for raising this insightful question. To address your concern, we designed a new controlled experiment. Specifically, we measure the per-criterion satisfaction rates on a training batch before and after one GRPO update, to test whether the criteria identified by CriPO undergo actual learning suppression. The experiment yields two main findings.

  1. The satisfying-rollout advantage sum is positively associated with the post-update satisfaction change. This supports the rationale behind our SC diagnostic method.
  2. At comparable selection counts, CriPO identifies a higher fraction of empirically suppressed criteria than the alternative diagnostic methods, e.g., CriPO 66.4% v.s. random 44.2%. As precisely predicting how a high-dimensional parameter update affects an individual criterion is non-trivial, these results support CriPO as an efficient rollout-level proxy for prioritizing likely suppression.

The detailed experimental setup and results are as follows.

Experimental setup

We conduct this experiment on the first five training batches of RaR-Medicine (64 queries per batch), using Qwen3-4B as the policy model. For each batch, we

  1. sample 32 responses per query before the update to estimate each criterion's pre-update satisfaction rate;
  2. perform one standard GRPO update using the original eight training rollouts per query; and
  3. independently sample 32 responses per query after the update to estimate each criterion's post-update satisfaction rate.

The additional evaluation rollouts are used exclusively to measure behavioral change. SC diagnosis and the policy update use only the eight training rollouts per query. We use a relatively large one-step learning rate of 10−510^{-5} to make the effect of the update measurable. Let NjpreN_j^{\mathrm{pre}} and NjpostN_j^{\mathrm{post}} denote the numbers of evaluation responses satisfying criterion jj before and after the update, respectively. The corresponding satisfaction rates are Rjpre=Njpre/32R_j^{\mathrm{pre}}=N_j^{\mathrm{pre}}/32 and Rjpost=Njpost/32R_j^{\mathrm{post}}=N_j^{\mathrm{post}}/32. We analyze 1,250 criteria that are observed but not saturated in the eight training rollouts (0<∣Sj∣<80<|\mathcal{S}_j|<8) and have non-boundary pre-update satisfaction rates in the 32 evaluation rollouts (0<Njpre<320<N_j^{\mathrm{pre}}<32).

Finding 1: The satisfying-rollout advantage sum is positively associated with the post-update satisfaction change

To test whether the satisfying-rollout advantage sum, the core signal in CriPO's SC diagnostic, tracks actual learning suppression, we group all 1,250 eligible criteria by the summed GRPO advantage of their satisfying training rollouts, ∑i∈SjAi\sum_{i\in\mathcal{S}_j}A_i. The table below shows that more negative satisfying-rollout advantage sums are associated with stronger empirical suppression. Specifically, the average satisfaction-rate change decreases as the advantage sum becomes more negative, and both negative-sum bins exhibit an average decline. Also, as the advantage sum decreases across bins, the fraction of criteria whose satisfaction increases falls monotonically, while the fraction whose satisfaction decreases rises monotonically. These results support the premise behind our SC diagnostic that the aggregate advantage received by criterion-satisfying rollouts is informative about whether a subsequent GRPO update encourages or suppresses that criterion.

Satisfying-rollout advantage sum#CriteriaMean sumAvg. RjpreR_j^{\mathrm{pre}}Avg. RjpostR_j^{\mathrm{post}}Avg. Rjpost−RjpreR_j^{\mathrm{post}}-R_j^{\mathrm{pre}}IncreasedDecreased
[2,+∞)[2,{+\infty})7042.847.3%57.6%+10.365.6%29.1%
[1,2)[1,2)3031.650.9%55.9%+5.057.4%35.0%
[0,1)[0,1)1300.546.7%49.5%+2.950.8%38.5%
[−1,0)[-1,0)75-0.557.1%50.8%-6.340.0%53.3%
(−∞,−1)({-\infty},-1)38-2.051.6%34.1%-17.621.1%73.7%
AuthorsOpenReview ↗

Response to Reviewer kcVm (3/3)

Finding 2: At comparable selection counts, CriPO identifies a higher fraction of empirically suppressed criteria than the alternative diagnostic methods

CriPO selects a criterion when the satisfying-rollout advantage sum is negative, or when it is zero and the criterion is satisfied by at most half of the group. We compare it with four diagnostic families.

  • Random matched randomly selects the same number of criteria as CriPO for each query.
  • Negative-any-kk selects a criterion when at least kk of its satisfying training rollouts have negative advantage (k=3,4k=3,4).
  • Low-frequency-kk selects a criterion when at most kk training rollouts satisfy it (∣Sj∣≤k|\mathcal{S}_j|\leq k, k=1,2k=1,2).
  • Sign-majority selects a criterion when negative-advantage satisfying rollouts form the majority, using only the signs of the advantages.

As shown in the table below, at similar selection counts, CriPO identifies a higher fraction of empirically suppressed criteria than the alternative diagnostics. Here, we use a broad operational definition of empirical suppression that includes both declining and stagnant satisfaction counts after the update (Njpost≤NjpreN_j^{\mathrm{post}}\leq N_j^{\mathrm{pre}}). Moreover, the remaining 33.6% of CriPO-selected criteria improve substantially less than empirically increasing criteria overall. The "Non-suppressed increase" column reports the mean satisfaction-rate increase, in percentage points, among selected criteria whose satisfaction increases after the update. For CriPO, this increase is 11.5 percentage points, well below the 19.7-percentage-point average across all empirically increasing criteria. This suggests that the diagnostic also highlights relatively slow-improving criteria that may benefit from further optimization.

SC diagnostic#Selected CriteriaSuppressed ratioNon-suppressed increase
Random matched to CriPO11344.2%20.1
Negative-any-319553.8%14.3
Negative-any-49661.5%15.7
Low-frequency-121549.3%23.1
Low-frequency-238143.3%24.5
Sign-majority16758.7%15.5
CriPO11366.4%11.5

We have added this analysis to Appendix C.1 of the revised paper and also clarify that the SC diagnostic serves as a rollout-level proxy for likely learning suppression in Section 2.

p.s. We have also corrected the typos accordingly in the revised paper, thank you for pointing out.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of author response - thank you for the valuable and constructive feedback

We sincerely thank all the reviewers for their great efforts in providing constructive and valuable feedback. We appreciate their recognition of the meaningful distinction between unexplored and suppressed criteria (R1, R2, R3), the targeted design of the two complementary modules (R1, R2, R3), and the extensive empirical evaluation (R1, R2, R3).

We have addressed the reviewers' comments and concerns in individual responses to each reviewer. The reviews allowed us to strengthen our manuscript and the changes made are summarized below:

  • [R1] Added comparisons with four hybrid OPD-RL methods (Appendix C.7, Table 10)
  • [R1] Added an empirical analysis of the suppressed-criterion diagnostic (Appendix C.1, Tables 5-6)
  • [R2, R3] Added comparisons with two alternative SC token selection strategies, i.e., the teacher–student probability difference and the type-and-position matched variant (Appendices C.12-C.13, Tables 15, 17-18)
  • [R2] Added the performance of CriPO with length control (Appendix C.10, Figure 11)
  • [R3] Added an analysis of the trade-off between mitigating SC suppression and improving conflict high-weight criteria (Appendix C.13, Table 19)

We welcome the reviewers' assessments of these additional results and would be glad to clarify any remaining concerns during the discussion.

(*As abbreviations, we refer to reviewers kcVm as R1, ftz8 as R2, and 3TFC as R3, respectively.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.