acceptodds
Under review as a conference paper at ICLR 2027

Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers

Abstract

Rubrics have emerged as an alternative to RLVR in open-ended domains where a single ground-truth final answer is not available. Existing rubric-based training methods rely on an LLM verifier that scores each rollout against rubrics. This introduces substantial training-time overhead, exposes optimization to verifier-specific biases, and reduces rubric feedback to a sparse end-of-trajectory signal. We propose Rubric-Guided Self-Distillation (RGSD), a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student. RGSD distills the rubric-conditioned teacher distribution into the student token-by-token, replacing sparse trajectory-level rewards with dense per-token learning signals and removing the LLM judge from the training loop entirely. Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves rubric satisfaction comparable to judge-based GRPO while using one on-policy rollout per prompt and no training-time verifier calls. Ablations show that raw rubrics provide a stronger teacher enrichment signal than self-generated reference responses, while a stronger GRPO judge can outperform RGSD in some settings, positioning RGSD as a complementary verifier-free alternative when verifier cost or reliability is the bottleneck.

Primary area:
reinforcement learning
Keywords:
rubrics, opsd, post-training, grpo

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qENqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces RGSD, a method that uses per-prompt rubrics as privileged context for a frozen teacher and performs self-distillation with a clipped Jensen–Shannon divergence. The authors position the method as an alternative that achieves comparable performance to rubric-based RL while removing the expensive verifier or LLM judge from the training loop and generating only one sample per prompt. Across Qwen2.5 models, RGSD achieves comparable rubric satisfaction with substantially shorter responses than GRPO. Experiments on Qwen3-Thinking models also show competitive performance, although the response-length advantage is not consistent. The evaluation covers both in-domain and out-of-distribution benchmarks in medical and science domains.

Critical strengths
  1. The paper proposes a different way of using prompt-specific rubrics in model training as an alternative to the prevalent rubric-based RL approach. Using rubrics as privileged teacher context removes the need for an LLM verifier during training.

  2. The paper is well presented and easy to follow.

Critical weaknesses
  1. Limited methodological novelty. I view RGSD as a specific instance of existing self-distillation methods such as SDPO and OPSD, with the main distinction being the use of per-prompt rubrics as privileged context. The methodological contribution beyond this choice appears limited.

  2. Narrow domain coverage. The experiments focus only on medical and science domains, which seems insufficient to establish RGSD as a broadly comparable alternative to rubric-based GRPO.

  3. Limited comparative evaluation across model families. Although Appendix C.3 includes OLMo-2-1124-7B-Instruct and Mistral-7B-Instruct-v0.3, these are older checkpoints, and neither is compared with a matching GRPO baseline. Comparisons using more recent models from other families would provide stronger support for the broader applicability of RGSD.

  4. Limited evaluation of overall response quality. Rubric satisfaction alone may not fully capture the quality of the resulting models. It would be helpful to also report hard or soft win rates against reference responses from newer models, the starting policies, and the corresponding GRPO-trained models.

  5. Limited evidence of broader generalization. Although the paper includes OOD evaluation within medical and science, it would be more informative to train on prompts spanning a broader range of domains and evaluate on benchmarks covering different held-out subdomains.

Questions for authors
  1. Could the authors clarify why a general-purpose LLM judge was used rather than a generative reward model for the GRPO baseline?

  2. Is the 4096-token completion budget sufficient for the thinking models? Could the authors report how often responses are truncated?

  3. How was the learning rate of 4.2 * 10^-6 selected?

  4. Could the authors clarify why the entire thinking trace is masked rather than only tokens that explicitly refer to the rubrics?

  5. Could the authors explain the choice to focus only on the medical and science domains of RubricHub?

  6. Could the authors evaluate the trained models using additional performance metrics beyond rubric satisfaction?

Reviewer_MwhfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Rubric-Guided Self-Distillation, a post-training method without needing a external LLM verifier for open-ended tasks. Building on On-Policy Self-Distillation, RGSD use prompt-specific rubrics as privileged information for the teacher model, thus enables direct token level supervision on the rubric criteria. RGSD reports comparable rubric satisfaction to GRPO on open-ended science and medicine tasks, from in-distribution RubricHub to out-of-distribution HealthBench and ResearchQA.

Critical strengths
  1. This paper addresses a limitation of rubric-based RL, where the reward is obtained from an additional LLM verifier, which add extra cost, and is unreliable due to the verifier quality. RGSD turns rubrics into dense training signals and is particularlly suitable for open-ended tasks.
  2. Their results show that RGSD improves RubricHub-medical by 7.9 points compared with 4.0 points for GRPO, and RubricHub-science by 4.3 points compared with +1.7 poinrts for GRPO, suggesting that rubric provides dense supervision without additional llm-as-a-judge reward.
Critical weaknesses
  1. The algorithmic novelty over existing works is limited, where the main difference is replacing reference answers with rubrics as the teacher's privileged information over on-policy self-distillation.
  2. The generalization of performance remains insufficiently established. RGSD’s gain mainly comes from in-domain scores. The paper only evaluates RGSD on two out-of-domain datasets: HealthBench and ResearchQA, which is insufficient. Both benchmarks remain within the same medical and science domains used during training. It is unclear whether the proposed approach generalizes to all open-ended tasks or different rubric distributions.
  3. The paper positions RGSD as a cheaper alternative to rubric-based GRPO. Yet RGSD requires a frozen teacher model to compute token distributions across student-generated sequences. The paper demonstrates verifier-call efficiency and rollout efficiency, but not necessarily total computational efficiency.
  4. In RGSD, the teacher and student observe different conditioning contexts, potentially introducing mismatched supervision. The authors address rubric leakage in reasoning traces through masking. This raises an extra concern, since the reasoning tokens are masked, it is unclear whether the method improves reasoning or primarily final-answer in rubric style.
Questions for authors
  1. What is the main advantage of RGSD over existing on-policy self-distillation methods? Could the authors compare against a simpler verifier-free baseline, such as SFT on rubric-conditioned responses, or other on-policy distillation methods to clarify their main advantage?
  2. RGSD underperforms GRPO on HealthBench, despite showing stronger in-domain results. Can the authors provide evidence of generalization to other open-ended domains or different rubric distributions?
  3. What is the full computational comparison of RGSD with GRPO?
  4. In Figure 4, why do the rubric-conditioned and self-golden teachers use different transition instructions? Would the performance difference remain if both methods used the same instructions, changing only the conditioning information?
  5. For Qwen3-Thinking models, the authors mask thinking tokens to avoid rubric leakage. Since these tokens account for a large proportion of the generated response, how does this affect the model's reasoning ability? Is the observed improvement primarily from better rubric style generation rather than improved reasoning? Could the authors provides more evaluation on response quality?
Reviewer_qJm7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose RGSD for open-ended tasks, using rubrics as privileged context for a teacher model and distilling its token-level distributions along student-generated trajectories. The teacher and student are initialized from the same base model, with the teacher kept frozen during training. On RubricHub, HealthBench, and ResearchQA, RGSD performs comparably to GRPO, with slightly higher scores overall, while eliminating the time spent on verifier scoring during training.

Critical strengths

The paper is well organized, and the training details are clearly documented in the appendices.

Critical weaknesses
  • The core method is highly similar to the published RCSD paper: both provide rubrics as context to the teacher and perform self-distillation along student-generated trajectories. The authors should clarify the novelty and advantages of their approach relative to this work. If this concern is adequately addressed, I would be willing to raise my score.
  • Although the method eliminates reward-scoring overhead, OPD itself introduces additional computational overhead (K = 128). The paper does not compare the actual runtime of the two methods, leaving the practical training-efficiency advantage unclear.
  • In Equation (1), the denominator should sum only over weights satisfying wi>0w_i > 0. Furthermore, if negative weights are allowed, sJs_J can be negative, so its stated range needs to be clarified or corrected.
Questions for authors

Please see the questions and clarification requests in the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.