- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper explores a method that combines on-policy distillation (OPD) using rubrics as privileged context with rubric-based RL for open-ended tasks. The authors show that this approach outperforms rubric-based RL, OPD with rubrics, and other baselines such as SFT on HealthBench, ResearchQA, and RubricHub Science.
Critical strengths
- The method is simple and straightforward, which makes it easy to implement.
- The experiments cover multiple benchmarks and include a sufficient number of baselines to show that warm-starting RL with OPD produces better results than using either method alone.
Critical weaknesses
- My main concern is that prior works have already explored combining OPD and RL and showed the benefits compared to using either method alone [1, 2, 3]. This paper provides little to no new intuition or analysis explaining why combining the two approaches outperforms each individual method, or a new method for combining them.
- Most main results are conducted on Qwen2.5 models. This makes the findings less convincing, as earlier work showed that we can achieve performance gains on Qwen2.5 even from spurious rewards [4].
[1] Yang, Chenxu, et al. "Self-distilled rlvr." arXiv preprint arXiv:2604.03128 (2026).
[2] Li, Gengsheng, et al. "Unifying group-relative and self-distillation policy optimization via sample routing." arXiv preprint arXiv:2604.02288 (2026).
[3] Ding, Ken. "Hdpo: Hybrid distillation policy optimization via privileged self-distillation." arXiv preprint arXiv:2603.23871 (2026).
[4] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." ICML 2026.
Questions for authors
In Section 5.4, beyond a standard SFT + RL baseline, did the authors explore existing methods to mitigate reward hacking for rubric-based RL?
5 comments
-
Most main results are conducted on Qwen2.5 models. This makes the findings less convincing, as earlier work showed that we can achieve performance gains on Qwen2.5 even from spurious rewards [4].
[4] shows that spurious rewards improve math scores mainly for Qwen2.5-Math, by amplifying behaviors the model already prefers from pretraining, and that these gains often do not carry over to other model families such as Llama3. Our results are unlikely to come from this effect:
- Benchmarks released after Qwen2.5. Qwen2.5 was released in September 2024, while HealthBench (May 2025), ResearchQA (August 2025), and RubricHub (January 2026) came later. Their rubrics, which define both the reward and the evaluation score, could not have been seen during pretraining.
- Different model family. We also evaluate Llama-3.1-8B with a Llama-3.1-70B teacher, a model family for which [4] finds that spurious rewards often fail. On HealthBench, RP-OPD+RL reaches 0.634, compared with 0.602 for direct RL and 0.526 for SFT+RL (Table 1).
-
In Section 5.4, beyond a standard SFT + RL baseline, did the authors explore existing methods to mitigate reward hacking for rubric-based RL?
We see our work as complementary to existing mitigation methods, since they act at different stages of training. Existing methods act on the reward, for example by using a stronger judge [5] or by randomly dropping rubric criteria when computing the reward [8]. Our results show that reward hacking can also be reduced upstream, at the warm-start stage, before RL begins. The warm start shapes which behaviors the policy brings into RL, and therefore what RL can later amplify. In our experiments, changing the SFT data only changed the form of hacking, from compliance claims to answer outlines (Appendix D.2), while RP-OPD+RL showed little hacking in both settings (§5.4, Appendix D.2). Reward-side methods can be applied on top of our warm start, and we will add this discussion to the revision.
Thank you again for your thoughtful feedback, which has helped us strengthen the paper, and we would be truly grateful if you would consider raising your score if our responses have addressed your concerns.
[1] Yang, Chenxu, et al. "Self-Distilled RLVR." arXiv:2604.03128, 2026.
[2] Li, Gengsheng, et al. "Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing." arXiv:2604.02288, 2026.
[3] Ding, Ken. "HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation." arXiv:2603.23871, 2026.
[4] Shao, Rulin, et al. "Spurious Rewards: Rethinking Training Signals in RLVR." ICML 2026.
[5] Mahmoud, Anas, et al. "Reward Hacking in Rubric-Based Reinforcement Learning." arXiv:2605.12474, 2026.
[6] Rezaei, MohammadHossein, et al. "Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers." arXiv:2606.12507, 2026.
[7] He, Yun, et al. "AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following." ACL 2026.
[8] Yang, Minglai, et al. "Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL." arXiv:2608.11669, 2026.
Dear Reviewer Gbmc,
Thank you again for your review. As the discussion period ends on November 18, we wanted to kindly follow up on our response. We have tried to address your concerns about the relation to methods combining distillation with RLVR, the choice of Qwen2.5 models, and how our approach relates to reward-hacking mitigation. If anything remains unclear, we would be glad to clarify before the deadline. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score.
Thank you for your review and for noting that our method is easy to implement and that our experiments include sufficient baselines. We address your concerns below:
We agree that [1–3] show clear benefits from combining self-distillation and RL on tasks with verifiable rewards. However, we respectfully disagree that our paper offers no new intuition. We believe the findings of [1–3] do not directly carry over to rubric-based RL, for the following reasons.
For these reasons, we do not think the combination methods in [1–3] suit rubric-based RL. Our method is designed for this setting, using the same rubric both as privileged information for the teacher and as the reward for RL. Our findings, such as how the first stage affects whether RL exploits the judge (§5.4), are specific to rubric-based RL and do not follow from results in RLVR.