acceptodds
Under review as a conference paper at ICLR 2027

OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation

Abstract

Many useful language-model tasks cannot be evaluated by exact outcome verification. Rubric-based reinforcement learning (RL) addresses this issue by scoring open-ended responses against explicit criteria. However, because the reward is assigned after the complete response, the training signal does not directly identify which individual decisions contributed to the final score. We propose a two-stage training framework that uses rubrics first as privileged teacher context for dense token-level supervision, then as rewards for further RL. In the first stage, rubric-privileged on-policy distillation (RP-OPD), a student without access to the rubric matches a rubric-aware teacher's next-token distributions at student-generated prefixes. In the second stage, RL directly optimizes the rubric reward and improves beyond the observed distillation plateau. We evaluate the framework on health and science tasks using open-weight models. Across HealthBench, ResearchQA, and RubricHub Science, we compare post-training methods and vary the amount of SFT or RP-OPD training before RL, finding that our two-stage framework achieves the highest scores among the methods evaluated. RP-OPD + RL shows limited signs of reward hacking on RubricHub Science, whereas the SFT + RL baseline increasingly receives high rewards for claims of rubric compliance without providing the required content. These findings support using rubrics to guide on-policy distillation before applying rubric-based RL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Rubric-based Reinforcement Learning, On-policy Distillation, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GbmcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper explores a method that combines on-policy distillation (OPD) using rubrics as privileged context with rubric-based RL for open-ended tasks. The authors show that this approach outperforms rubric-based RL, OPD with rubrics, and other baselines such as SFT on HealthBench, ResearchQA, and RubricHub Science.

Critical strengths
  • The method is simple and straightforward, which makes it easy to implement.
  • The experiments cover multiple benchmarks and include a sufficient number of baselines to show that warm-starting RL with OPD produces better results than using either method alone.
Critical weaknesses
  • My main concern is that prior works have already explored combining OPD and RL and showed the benefits compared to using either method alone [1, 2, 3]. This paper provides little to no new intuition or analysis explaining why combining the two approaches outperforms each individual method, or a new method for combining them.
  • Most main results are conducted on Qwen2.5 models. This makes the findings less convincing, as earlier work showed that we can achieve performance gains on Qwen2.5 even from spurious rewards [4].

[1] Yang, Chenxu, et al. "Self-distilled rlvr." arXiv preprint arXiv:2604.03128 (2026).

[2] Li, Gengsheng, et al. "Unifying group-relative and self-distillation policy optimization via sample routing." arXiv preprint arXiv:2604.02288 (2026).

[3] Ding, Ken. "Hdpo: Hybrid distillation policy optimization via privileged self-distillation." arXiv preprint arXiv:2603.23871 (2026).

[4] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." ICML 2026.

Questions for authors

In Section 5.4, beyond a standard SFT + RL baseline, did the authors explore existing methods to mitigate reward hacking for rubric-based RL?

5 comments

Thank you for your review and for noting that our method is easy to implement and that our experiments include sufficient baselines. We address your concerns below:

  1. My main concern is that prior works have already explored combining OPD and RL and showed the benefits compared to using either method alone [1, 2, 3]. This paper provides little to no new intuition or analysis explaining why combining the two approaches outperforms each individual method, or a new method for combining them.

We agree that [1–3] show clear benefits from combining self-distillation and RL on tasks with verifiable rewards. However, we respectfully disagree that our paper offers no new intuition. We believe the findings of [1–3] do not directly carry over to rubric-based RL, for the following reasons.

  • Rubric-based RL has no ground truth. In RLVR, the reward compares the final answer with a known solution, so it is exact and directly measures the goal. In rubric-based RL, the task is open-ended and many different responses can be good. The reward is a proxy made of two parts, a rubric that describes only some properties of a good response (7.5 to 26.9 criteria per prompt on average in our data), and an LLM judge that applies the rubric imperfectly [5].
  • [1–3] rely on the reward being exact and binary. [1] uses the reward to set the direction of each update because it is reliable, [2] routes correct and failed rollouts to different objectives, and [3] targets prompts where all rollouts fail. Rubric scores are continuous rather than 0 or 1, and they are only a proxy. One could threshold them to get a correct/failed split, but it is unclear where to set the threshold and how well this would work, so we do not think results from RLVR can simply be assumed to hold here.
  • Because the reward is a proxy, the policy can learn to exploit the judge. This is a known challenge in rubric-based RL, documented in several recent studies [5, 6, 7]. Together with the cost of a judge call for every rollout, it requires extra care that RLVR largely does not need. In our experiments, we track exploitation during training, regrade affected responses, and evaluate all main results with a separate shortcut-aware judge.

For these reasons, we do not think the combination methods in [1–3] suit rubric-based RL. Our method is designed for this setting, using the same rubric both as privileged information for the teacher and as the reward for RL. Our findings, such as how the first stage affects whether RL exploits the judge (§5.4), are specific to rubric-based RL and do not follow from results in RLVR.

  1. Most main results are conducted on Qwen2.5 models. This makes the findings less convincing, as earlier work showed that we can achieve performance gains on Qwen2.5 even from spurious rewards [4].

[4] shows that spurious rewards improve math scores mainly for Qwen2.5-Math, by amplifying behaviors the model already prefers from pretraining, and that these gains often do not carry over to other model families such as Llama3. Our results are unlikely to come from this effect:

  • Benchmarks released after Qwen2.5. Qwen2.5 was released in September 2024, while HealthBench (May 2025), ResearchQA (August 2025), and RubricHub (January 2026) came later. Their rubrics, which define both the reward and the evaluation score, could not have been seen during pretraining.
  • Different model family. We also evaluate Llama-3.1-8B with a Llama-3.1-70B teacher, a model family for which [4] finds that spurious rewards often fail. On HealthBench, RP-OPD+RL reaches 0.634, compared with 0.602 for direct RL and 0.526 for SFT+RL (Table 1).
  1. In Section 5.4, beyond a standard SFT + RL baseline, did the authors explore existing methods to mitigate reward hacking for rubric-based RL?

We see our work as complementary to existing mitigation methods, since they act at different stages of training. Existing methods act on the reward, for example by using a stronger judge [5] or by randomly dropping rubric criteria when computing the reward [8]. Our results show that reward hacking can also be reduced upstream, at the warm-start stage, before RL begins. The warm start shapes which behaviors the policy brings into RL, and therefore what RL can later amplify. In our experiments, changing the SFT data only changed the form of hacking, from compliance claims to answer outlines (Appendix D.2), while RP-OPD+RL showed little hacking in both settings (§5.4, Appendix D.2). Reward-side methods can be applied on top of our warm start, and we will add this discussion to the revision.

Thank you again for your thoughtful feedback, which has helped us strengthen the paper, and we would be truly grateful if you would consider raising your score if our responses have addressed your concerns.

[1] Yang, Chenxu, et al. "Self-Distilled RLVR." arXiv:2604.03128, 2026.
[2] Li, Gengsheng, et al. "Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing." arXiv:2604.02288, 2026.
[3] Ding, Ken. "HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation." arXiv:2603.23871, 2026.
[4] Shao, Rulin, et al. "Spurious Rewards: Rethinking Training Signals in RLVR." ICML 2026.
[5] Mahmoud, Anas, et al. "Reward Hacking in Rubric-Based Reinforcement Learning." arXiv:2605.12474, 2026.
[6] Rezaei, MohammadHossein, et al. "Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers." arXiv:2606.12507, 2026.
[7] He, Yun, et al. "AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following." ACL 2026.
[8] Yang, Minglai, et al. "Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL." arXiv:2608.11669, 2026.

Dear Reviewer Gbmc,

Thank you again for your review. As the discussion period ends on November 18, we wanted to kindly follow up on our response. We have tried to address your concerns about the relation to methods combining distillation with RLVR, the choice of Qwen2.5 models, and how our approach relates to reward-hacking mitigation. If anything remains unclear, we would be glad to clarify before the deadline. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score.

Reviewer_C2ExOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a two-stage training framework for rubric-based RL. First, the teacher model sees the rubric as privileged context and the student model matches the teacher’s next-token distribution at student’s prefixes without access to the rubric. The second stage is Rubric-RL with GRPO. The paper compares this method with rubric-conditioned SFT, direct rubric RL, etc. on HealthBench, ResearchQA, and RubricHub Science, and claims rubric access improves distillation, but distillation alone does not replace rubric-based RL; RP-OPD is a better warm start than SFT; and that SFT+RL learns to score well by claiming rubric compliance without the required content.

Critical strengths

This paper provides consistent experimental evidence for the effectiveness of RP-OPD as a warm-start strategy for rubric-based RL. Across three benchmarks and multiple student models, RP-OPD + RL consistently outperforms both SFT + RL and direct RL. The evaluation also uses held-out test sets and an independent judge, which strengthens the empirical findings.

The reward-hacking analysis is particularly interesting. The authors identify a failure mode in which SFT-based models increasingly receive high rewards for claims of rubric compliance without providing the required content. The comparison with RP-OPD, together with concrete examples and shortcut-aware re-grading, offers useful insights into how warm-start strategies may influence undesirable behaviors during RL.

Critical weaknesses

Insufficient ablation to isolate the contribution of rubric-privileged distillation. While the paper compares OPD with and without rubric access in stage 1, it does not systematically evaluate their downstream RL performance. In particular, a comparison between vanilla OPD + RL and RP-OPD + RL is missing. Without this experiment, it is difficult to determine whether the final performance improvements primarily come from on-policy distillation itself or from the proposed rubric-privileged teacher supervision. This weakens the evidence for the specific contribution of RP-OPD.

Methodological novelty is not enough. The proposed framework combines existing components, including on-policy distillation, rubric-based teacher supervision, and GRPO-based RL. Although the empirical findings are valuable, the methodological contribution appears relatively incremental. The paper could more clearly distinguish its contributions from existing studies.

Questions for authors

Q1. The paper compares OPD with and without rubric, but only during stage 1. Could the authors provide a comparison between vanilla OPD + RL and RP-OPD + RL to clarify the contribution of rubric-privileged supervision?

Q2. Could the authors compare the training costs of RP-OPD + RL and baselines? Would RP-OPD still perform better under the same computational budget?

Q3. The reward-hacking analysis mainly focuses on RubricHub Science. Are similar behaviors observed on HealthBench and ResearchQA? Does RP-OPD + RL consistently exhibit less reward hacking across different benchmarks?

3 comments

Thank you for your careful review, and for finding our results consistent across benchmarks and models and the reward-hacking analysis interesting. We address your concerns below:

  1. Insufficient ablation to isolate the contribution of rubric-privileged distillation. [...] a comparison between vanilla OPD + RL and RP-OPD + RL is missing. Without this experiment, it is difficult to determine whether the final performance improvements primarily come from on-policy distillation itself or from the proposed rubric-privileged teacher supervision. (also Q1)

We agree that an end-to-end comparison would make this clearer. We believe the rubric is what makes the warm start useful, and that vanilla OPD + RL would behave much like direct RL, which we compare against in every setting. Without the rubric, the teacher does not know which criteria a response will be graded on. Its own score drops to about 0.5 on all three benchmarks, compared with 0.83 to 0.98 with the rubric (Figure 2), which is already below what direct RL reaches. Distilling from such a teacher gives the student little that helps later RL. In stage 1, OPD and SFT without the rubric indeed stay close to the base model (Figure 2), so vanilla OPD + RL would start from about where direct RL starts, and direct RL ends below RP-OPD + RL in every setting of Table 1.

On-policy training mainly leaves more headroom for the subsequent RL. Our SFT baseline also learns from the rubric-aware teacher, but on the teacher's responses rather than the student's own. Before RL, the two warm starts reach similar scores, yet RL improves much more from RP-OPD (Figure 4). With longer SFT, RL plateaus earlier, while later RP-OPD checkpoints continue to improve (§5.2). RP-OPD also keeps higher token entropy, which may leave more room for exploration during RL.

We will add the teacher's no-rubric scores to Figure 2 and make this argument explicit in the revision.

  1. Methodological novelty is not enough. The proposed framework combines existing components, including on-policy distillation, rubric-based teacher supervision, and GRPO-based RL. [...] The paper could more clearly distinguish its contributions from existing studies.

We agree that each component has been studied before. We believe the contribution lies in how they are adapted to rubric-based RL, which differs in important ways from the verifiable-reward setting where distillation and RL have been combined so far [1–4].

  • OPD with rubrics has been studied, but not well combined with RL. Existing work uses rubric-privileged OPD as a replacement for RL [5, 6]. We find that it falls short of RL on its own (§5.1), and instead use it as a warm start, studying how it shapes the RL that follows.
  • Rubric-based RL has no ground truth. In RLVR, the reward checks the answer against a known solution. In rubric-based RL, many responses can be good, and the reward comes from an LLM judge applying a rubric that covers only part of what makes a response good [7]. Methods that combine distillation with RLVR depend on an exact, binary reward, for example to tell which rollouts failed [2, 3], and it is not clear how they would carry over.
  • The warm start affects whether RL exploits the judge. Because the reward is a proxy, RL can learn to exploit the judge, a known problem in rubric-based RL [6, 7, 8]. In our experiments, SFT on rubric-aware teacher responses leads the student to claim that it meets the rubric, and RL amplifies these claims, while RP-OPD + RL, which learns from the student's own responses, rarely does this (§5.4).

We will state these differences more clearly in the related work.

  1. Could the authors compare the training costs of RP-OPD + RL and baselines? Would RP-OPD still perform better under the same computational budget?

Both pipelines share the same RL stage and differ only in the first stage, where the two methods cost about the same. SFT needs the teacher to generate all training targets before training can begin, while RP-OPD only has the teacher score the student's own responses during training. On RubricHub Science, we estimate that generating the 115,672 SFT targets (about 96M tokens) alone takes up to about a day, similar to the roughly 21 hours of the RP-OPD stage, and both are shorter than the roughly two days of RL that follows.

We expect RP-OPD + RL to remain better under a matched budget, because extra compute in either stage does not close the gap for SFT + RL. Longer SFT leads to earlier plateaus during RL and lower final scores (§5.2, Appendix B.1), while later RP-OPD checkpoints continue to support RL gains. Longer RL does not help much either. SFT + RL stays below RP-OPD + RL even in long RL runs (Figure 3), and on RubricHub Science its later reward gains come increasingly from claims of compliance, with regrading removing about half of its reward at RL step 600 (0.830 to 0.404, §5.4).

For direct RL, Figures 3 and 8 plot all runs against stage-one steps plus RL steps. Since an RP-OPD step costs less than an RL step, this comparison already gives direct RL more compute. We will add a full cost breakdown in GPU-hours to the revision.

  1. The reward-hacking analysis mainly focuses on RubricHub Science. Are similar behaviors observed on HealthBench and ResearchQA? Does RP-OPD + RL consistently exhibit less reward hacking across different benchmarks?

Yes, similar reward hacking has been reported in other rubric-based RL settings. [8] observes self-certification in instruction-following RL, and [7] finds that LLM judges often fail to verify whether rubric criteria are met. In our experiments, how strongly it appears depends on how easy the criteria are to check. In our experiments, whether this happens depends on how easy the criteria are to check. In RubricHub Science, 12.5% of criteria ask for a derivation, proof, or calculation. Checking these takes careful work, so the judge often trusts a claim that the work has been done. In HealthBench and ResearchQA, only about 1% of criteria are of this kind. Most simply ask whether the response mentions some specific content, such as a fact or a recommendation, which the judge can check directly. Claiming compliance without that content earns no credit, so RL has nothing to reinforce.

RP-OPD + RL is less prone to this behavior because of how the student is trained in stage 1, so we expect the benefit to hold wherever the judge can be fooled. With SFT, the student imitates teacher responses written with the rubric in view, which sometimes state that a criterion is met, and RL then reinforces this pattern once the judge rewards it. With RP-OPD, the student learns from its own responses and never imitates such statements. We will add this analysis to the revision.

Thank you again for your positive assessment and thoughtful feedback, which has helped us strengthen the paper. We hope our responses have clarified the remaining concerns, and we are happy to discuss any further questions.

[1] Yang, Chenxu, et al. "Self-Distilled RLVR." arXiv:2604.03128, 2026.
[2] Li, Gengsheng, et al. "Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing." arXiv:2604.02288, 2026.
[3] Ding, Ken. "HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation." arXiv:2603.23871, 2026.
[4] Li, Boyan, et al. "Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR." arXiv:2609.04108, 2026.
[5] Bablani, Deepika, et al. "Rubrics as Privileged Information for Open-Ended Generation." arXiv:2608.02948, 2026.
[6] Rezaei, MohammadHossein, et al. "Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers." arXiv:2606.12507, 2026.
[7] Mahmoud, Anas, et al. "Reward Hacking in Rubric-Based Reinforcement Learning." arXiv:2605.12474, 2026.
[8] He, Yun, et al. "AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following." ACL 2026.

Dear Reviewer C2Ex,

Thank you again for your positive assessment. As the discussion period ends on November 18, we wanted to kindly follow up on our response regarding the role of the rubric versus on-policy training, the training costs, and reward hacking across benchmarks. We would be glad to clarify anything further before the deadline, and we would appreciate it if you could let us know whether our response has addressed your concerns.

Reviewer_NwSuOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors improve performance on open-ended datasets by combining OPD with rubric-based RL. They further improve upon standard OPD by providing rubrics to the teacher model as privileged contextual information.

Critical strengths
  • The paper is well written, with details clearly documented in the appendices.
  • Providing rubrics to the teacher model is a novel aspect of the paper and leads to performance improvements.
  • The experiments are comprehensive, and the results appear strong. The approach is promising and could be widely adopted by the community.
Critical weaknesses
  • This paper is highly similar to this work in its methodology, datasets, and models. The relationship between the two papers needs clarification. If this concern is adequately addressed, I would raise my score.
  • OPD has previously been combined with RLVR (reference), which limits the novelty of this paper.
  • Regarding the training details in Table 2, why is the learning rate for RP-OPD on RubricHubScience 3×10−53 \times 10^{-5}, while all the others use 1×10−51 \times 10^{-5}? Why do SFT and RP-OPD use different batch sizes?
  • Is the ability of RP-OPD to mitigate reward hacking a general finding, or has it only been observed on RubricHubScience?
  • Are the regrading rules for the reward function used solely to correct the evaluation? Why not retrain using the revised prompt directly?
Questions for authors
  • In line 53, what does “prefix” refer to in “only the prompt and prefix”?
  • What is the equation in line 140 intended to convey?
5 comments

Thank you for your careful review and for your openness to raising the score. We address your concerns below:

  1. This paper is highly similar to this work in its methodology, datasets, and models. The relationship between the two papers needs clarification. If this concern is adequately addressed, I would raise my score.

We thank the reviewer for pointing us to [1]. It was posted on arXiv on August 3, 2026, less than two months before the ICLR deadline, and was developed independently of ours. Under the ICLR reviewer guidelines, it is considered as concurrent work. We will discuss it in the revision. While both papers use rubrics as privileged information for on-policy distillation, there are several key differences:

  • Warm start vs. replacement for RL. [1] proposes distillation as an alternative to RL. We use it as a warm start for RL and study how the two stages interact, e.g., longer SFT leads to earlier RL plateaus, while longer RP-OPD continues to support RL gains (§5.2).

  • Different conclusions about distillation vs. RL. [1] finds that distillation alone outperforms RL on HealthBench. In our experiments, distillation alone does not match RL on any of the three benchmarks (§5.1, Figure 2), and RGSD [2] also finds that self-distillation does not clearly outperform RL. We believe the difference comes from how much the RL baseline is trained: [1] trains RL for only 50 steps with batch size 256, while we train for more than 400 steps with batch size 512, and RGSD for 490–775 steps with batch size 128. With a longer-trained RL baseline, the advantage of OPSD reported in [1] may not hold.

  • Self-distillation is a special case of our setup. [1] uses the student itself as the teacher. We treat the teacher as a design variable: in Appendix C.1, using the 3B student as its own teacher corresponds to self-distillation, and stronger external teachers (14B, 32B) clearly outperform it. We also compare distillation objectives (Appendix C.4).

  • Reward hacking. We show that SFT+RL learns to exploit the rubric judge while RP-OPD+RL largely does not (§5.4). [1] does not study this.

We hope this clarifies the relationship, and we would be grateful if the reviewer would reconsider the score.

  1. OPD has previously been combined with RLVR (reference), which limits the novelty of this paper.

We acknowledge that OPD has been combined with RL in recent work, including [3]. It was posted on September 4, 2026, less than a month before the deadline, and is concurrent with ours. It also addresses a different setting:

  • Verifiable vs. rubric-based rewards. [3] studies math and logic tasks with exact-answer verifiers. We study open-ended tasks scored by an LLM judge against rubrics.
  • Rubric-privileged teacher. The teacher in [3] has no privileged information. Ours sees the rubric, which improves distillation (Figure 2).
  • Reward hacking as an engineering challenge. Unlike an exact verifier, an LLM judge may give credit to a response that only claims to meet the rubric, as also reported in [2, 4, 5]. The training reward alone cannot be trusted, so we track suspicious patterns during training, regrade affected responses, and evaluate all main results with a separate, shortcut-aware judge. Our RP-OPD warm start also helps mitigate this problem (§5.4).
  1. Regarding the training details in Table 2, why is the learning rate for RP-OPD on RubricHubScience 3e-5, while all the others use 1e-5? Why do SFT and RP-OPD use different batch sizes?

Thank you for the careful reading.

Learning rate: RubricHub Science has both the largest training set (28,918 prompts, vs. 4,500 for HealthBench and 16,961 for ResearchQA) and the most detailed rubrics (26.9 criteria per prompt on average, vs. 11.4 and 7.5). This gives the student much more to learn from the rubric-aware teacher. With 1e-5, RP-OPD on this dataset was still improving at the end of training, so we raised the learning rate to 3e-5 for faster convergence. All other learning rates are the same across datasets.

Batch size: SFT and RP-OPD learn from different data, so matching their batch sizes would not make them more comparable. SFT trains offline on teacher responses, while RP-OPD trains on rollouts the student generates at each step. Teacher responses also tend to be longer, so an SFT batch of 256 already contains a comparable number of training tokens. We set the batch size separately for each method and kept it fixed across datasets.

  1. Is the ability of RP-OPD to mitigate reward hacking a general finding, or has it only been observed on RubricHubScience?

Similar reward hacking has been reported in other rubric-RL work: [4] observes self-certification in instruction-following RL, and [5] finds that judges often fail to verify whether rubric criteria are met. In our experiments, SFT+RL shows this behavior strongly on RubricHub, but much less on HealthBench and ResearchQA. On HealthBench, models sometimes claimed to satisfy a criterion, but the judge still gave no credit if the content was missing. On RubricHub, the judge often gave credit even when the derivation was missing or wrong. We think this comes down to how easy the criteria are to check: a multi-step derivation is hard for an LLM judge to verify, so it tends to trust the claim, while most HealthBench and ResearchQA criteria only ask whether some content is present.

RP-OPD+RL is likely less affected because of how the student is trained in stage 1. With SFT, the student imitates teacher responses written with the rubric visible, which sometimes state that a criterion is met, and RL then reinforces this pattern. With RP-OPD, the student is trained on its own rollouts and never imitates such statements, so it rarely learns to make them. We will add this discussion to the revision.

  1. Are the regrading rules for the reward function used solely to correct the evaluation? Why not retrain using the revised prompt directly?

Yes, regrading is only used for evaluation. We did not retrain with the revised prompt because it only fixes the specific hack we found, and the policy can still find other ways to exploit the judge. In fact, when we changed how the SFT data was generated (Appendix D.2), SFT+RL stopped claiming compliance but learned a different hack: it wrote outlines of an answer (e.g., "Provide a quantitative example…") without the content, and still received full credit. Our goal is not to build an unhackable reward, but to compare warm starts under the same imperfect reward, as in practice. Regrading measures this difference: at RL step 600, it lowers SFT+RL from 0.830 to 0.404, while RP-OPD+RL changes only from 0.894 to 0.889 (§5.4).

  1. In line 53, what does "prefix" refer to in "only the prompt and prefix"?

"Prefix" means the part of the response the student has generated so far. The teacher and student see the same prefix and both predict the next token, but only the teacher also sees the rubric. We will define the term in the introduction.

  1. What is the equation in line 140 intended to convey?

Thank you for catching this. The equation was a typo. It was meant to contrast the teacher and student distributions: the teacher conditions on the rubric, πteacher(yt∣x,r,y<t)\pi_{\text{teacher}}(y_t \mid x, r, y_{<t}), while the student does not, πθ(yt∣x,y<t)\pi_\theta(y_t \mid x, y_{<t}). We will correct this in the revised version.

Thank you again for your thoughtful feedback, which has helped us strengthen the paper, and we would be truly grateful if you would consider raising your score if our responses have addressed your concerns.

[1] Bablani, Deepika, et al. "Rubrics as Privileged Information for Open-Ended Generation." arXiv:2608.02948, 2026.
[2] Rezaei, MohammadHossein, et al. "Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers." arXiv:2606.12507, 2026.
[3] Li, Boyan, et al. "Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR." arXiv:2609.04108, 2026.
[4] He, Yun, et al. "AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following." ACL 2026.
[5] Mahmoud, Anas, et al. "Reward Hacking in Rubric-Based Reinforcement Learning." arXiv:2605.12474, 2026.

Dear Reviewer NwSu,

Thank you again for your review. As the discussion period ends on November 18, we wanted to kindly follow up on our response. We have tried to address each of your concerns, including the relation to concurrent work, the training configurations, and the reward-hacking analysis. If anything remains unclear, we would be glad to clarify before the deadline. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

our core contribution

We thank all reviewers for their feedback. We would like to briefly restate our core contribution, since we believe the remaining concerns about novelty come from viewing rubric-based RL through the lens of RLVR.

Rubric-based RL is not RLVR. In RLVR, the reward is an exact verifier, and recent work has found that on-policy self-distillation works best combined with RL rather than as a replacement for it [1–4]. In rubric-based RL, the reward comes from an LLM judge scoring an open-ended response against a rubric. This is much more expensive to compute, so existing work often trains RL with a smaller judge, small batches, or few steps.

As a result, rubric-based RL has drawn the opposite conclusion. Recent papers propose rubric-privileged self-distillation as a replacement for RL and report that it matches or beats RL [5, 6]. Their RL baselines, however, are light, for example 50 RL steps [5], or 32 prompts with 4 rollouts per step and a 14B judge [6].

We challenge this conclusion. We train RL with 512 prompts and 8 rollouts per step, a 32B judge, and up to about 1,000 steps. Against this stronger baseline, OPD alone improves quickly at first but plateaus below RL on all three benchmarks (§5.1). OPD does not replace RL in rubric-based RL either.

We then propose a more reasonable way to use OPD, as a warm start for RL. Combination methods from RLVR rely on a verifiable reward and do not carry over directly. We instead use OPD as a warm start and then run RL, with the same rubric guiding the teacher in the first stage and serving as the reward in the second. This combination achieves the best results in every setting we evaluated (Table 1).

To our knowledge, this is the first work to question the replacement view in rubric-based RL and to show how OPD and RL should be combined in this setting.

[1] Yang et al. "Self-Distilled RLVR." arXiv:2604.03128, 2026.
[2] Li et al. "Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing." arXiv:2604.02288, 2026.
[3] Ding. "HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation." arXiv:2603.23871, 2026.
[4] Li et al. "Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR." arXiv:2609.04108, 2026.
[5] Bablani et al. "Rubrics as Privileged Information for Open-Ended Generation." arXiv:2608.02948, 2026.
[6] Gu et al. "Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation." arXiv:2606.19327, 2026.

Dear AC and Reviewers,

Thank you again for the constructive discussion. We have uploaded a revised version of the paper. Following the reviewers' feedback, we added a discussion of how our work relates to concurrent rubric-privileged self-distillation methods and to methods that combine self-distillation with RLVR, in both the introduction (Section 1) and the related work (Section 7). We also added a discussion of reward hacking in rubric-based RL and its existing mitigations (Section 7). All changes are highlighted in blue.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.