- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CriPO, which combines GRPO with on-policy self-distillation for rubric-based training. It distinguishes criteria that no sampled response satisfies from criteria whose satisfying responses receive non-positive aggregate advantages. A rubric-conditioned self-teacher provides auxiliary supervision for the former, while a counterfactual teacher identifies tokens for advantage correction in the latter. Experiments evaluate the method on medicine and science QA tasks.
Critical strengths
- Useful problem formulation: Distinguishing unexplored and suppressed criteria makes the limitations of aggregate rubric rewards more concrete. The two targeted modules provide a clear connection between the diagnosis and the method.
- Substantial empirical support: The experiments include in-domain and cross-domain tasks, multiple model sizes, component ablations, different judges, seed variation, and a random-token control. Together, these provide useful evidence for the overall improvement.
Critical weaknesses
- Token localization needs more direct validation: Low probability under the counterfactual teacher may reflect rewriting or formatting changes rather than criterion-specific content. The random-token control is helpful, but a control matched by token type and position would better establish the specificity of the selected tokens (Section 4.3; Appendix D.2).
- Criterion trade-offs need clearer analysis: A criterion-satisfying response can have a negative advantage because it performs poorly on more important criteria. It is unclear when local advantage correction preserves useful behavior without weakening these intended trade-offs. Per-criterion gains and regressions would help clarify this.
Questions for authors
- Does editing the selected tokens affect the intended criterion more than editing control tokens matched by type and position?
- When rubric criteria conflict, how does advantage correction affect the higher-weighted criteria? Please report both criterion-level improvements and regressions.
2 comments
Response to Reviewer 3TFC (2/2)
Experiment 3. End-to-end training
The completed end-to-end comparison shows that CriPO achieves the highest score among these variants on both benchmarks, outperforming CriPO-Type/Position by 1.1 percentage points on RaR-Medicine and 0.3 points on HealthBench.
| Method | RaR-Medicine | HealthBench |
|---|---|---|
| CriPO w/o AC | 61.4 | 64.2 |
| CriPO | 63.1 | 65.6 |
| CriPO-Type/Position | 62.0 | 65.3 |
| CriPO-Random | 61.6 | 65.1 |
We have added this analysis in our revised paper in Appendix C.13 (where we also compares a teacher-student probability-difference selection strategy).
Q2: Trade-offs with conflict high-weight criteria
Thank you for raising this insightful question. Our additional experiments reveal a trade-off between optimizing suppressed criteria and improving conflict high-weight criteria. Even so, the reduction in the latter's satisfaction-rate gain is small compared with the improvement on suppressed criteria.
Specifically, we define conflict high-weight criteria as criteria with weight 9 or 10 (the two highest levels in RaR-Medicine's weight set ) that are unmet in at least one SC-satisfying training rollout, excluding all suppressed criteria (SC). After deduplication, we analyze 117 SC and 230 conflict high-weight criteria. We follow the setting of Q1, Experiment 2 (One-step mitigation) which can evaluate the per-criterion satisfaction rate. denotes the mean pre-to-post satisfaction-rate change in percentage points, and and denote the fractions of criteria whose satisfaction rates increase and decrease, respectively.
| Method | SC | SC | SC | Conflict high-weight criteria | Conflict high-weight criteria | Conflict high-weight criteria |
|---|---|---|---|---|---|---|
| CriPO w/o AC | -6.28 | 32.48% | 60.68% | +8.18 | 50.00% | 18.70% |
| CriPO | -4.73 | 40.17% | 55.56% | +7.96 | 48.70% | 21.74% |
As shown in the table, AC mitigates the intended SC suppression, where the mean SC change improves by 1.55 percentage points over CriPO w/o AC, the improved fraction rises from 32.48% to 40.17%, and the regressed fraction falls from 60.68% to 55.56%. Conflict high-weight criteria retain a mean satisfaction-rate improvement of 7.96 percentage points, only 0.22 percentage points below CriPO w/o AC (+8.18). Together, quantifying results of the trade-off shows that AC mitigates SC suppression while retaining most of the average satisfaction-rate improvement on conflict high-weight criteria.
We have added this analysis to Appendix C.13 in our revised paper.
Response to Reviewer 3TFC (1/2)
Thank you for acknowledging our work and providing the constructive comments! We hope the following responses can address your remaining concerns.
Q1: Effect of alternative SC token localization strategy (controlled type and position)
To address your concern, we compare CriPO's SC token selection strategy with a type-and-position-matched control, and evaluate them along three dimensions: (i) whether deleting the selected tokens changes the rubric judge's decision, (ii) how much the advantage correction (AC) module mitigates the suppressed criteria after one update training step, and (iii) end-to-end performance after full training. These experiments show that CriPO's localization method is more closely associated with SC-related behaviors and enables more effective mitigation of their suppression when used with AC.
The compared variant: Type-and-position-matched localization. For each token selected by CriPO, we choose the nearest eligible token in the same response that is not selected by CriPO, has not already been selected yet, and belongs to the same one of the five token types used in the paper (in Table 3): formatting/structural boundaries, knowledge/semantic content, logical/discourse connectors, grammatical function words, and numbers/option labels. Categories are assigned by deterministic lexical rules over decoded token pieces.
All experiments use Qwen3-4B as the policy model on RaR-Medicine and Qwen3-32B as the LLM judge. Experiments 1 and 2 use the first five training batches (64 queries per batch, 320 queries in total). In addition to type-and-position-matched localization, we report random token selection with the same token budget as CriPO. We refer to these two variants as CriPO-Type/Position and CriPO-Random, respectively.
Experiment 1: Direct deletion. Do the selected tokens affect the judge's decision?
We edit the original negative-advantage responses at positions selected by CriPO, CriPO-Type/Position, or CriPO-Random and ask the rubric judge to re-evaluate the target SC. We use two editing modes.
Token onlydeletes only the selected token pieces.Through delimiterdeletes each selected token and the subsequent local segment up to punctuation, a newline, or a Markdown boundary. We include the latter because deleting a single token may not fully capture its influence on the subsequent content of the response.Loss rateis the fraction of originally satisfied target criteria that become unmet after editing. The pooled evaluation contains 273 responses and 194 target criteria judged satisfied before editing. As shown in the table below, deleting tokens selected by CriPO causes the greatest loss of SC satisfaction under both editing modes, suggesting that CriPO's localized tokens are more closely associated with the target SCs.However, the methods compared in Experiment 1 directly delete tokens from the original rollouts, which may leave incomplete sentences and affect the LLM judge's accuracy. To evaluate CriPO's token localization more reliably, we conduct the following two additional experiments.
Experiment 2: One-update mitigation of suppressed criteria
We measure how the different AC variants affect the satisfaction of the same 117 diagnosed SCs after one update. For each training batch, we (i) sample 32 responses per query before the update to estimate each criterion's pre-update satisfaction rate, (ii) use the original eight training rollouts per query to diagnose SCs and perform one update for each variant, and (iii) independently sample 32 responses per query from each updated variant to estimate post-update satisfaction rates. We use a relatively large one-step learning rate of 10−5 to make the effect of the update measurable. As shown in the table below, the average satisfaction rate of the suppressed criteria decreases by 6.28 percentage points when training with CriPO w/o AC. All variants alleviate this suppression to some extent after adding the advantage correction module, with CriPO achieving the most pronounced mitigation.