- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ROPD, a black-box on-policy distillation method that replaces teacher logits with rubric-based feedback. For each prompt, a rubric is generated by contrasting multiple teacher responses with the student’s current rollouts. A verifier evaluates all student responses using the same rubric, and the resulting aggregated scores are used as rewards for GRPO. The paper reports strong gains over both black-box baselines and logit-based OPD, especially on math benchmarks, and provides analysis of reward quality and training dynamics.
Critical strengths
- The paper studies a timely and interesting problem related to black-box OPD, and provides a controlled comparison against logit-based OPD. The experiments results are informative and consistent.
- The design of the method is well motivated and practical. The combination of teacher-grounded, student-conditioned, shared rubrics and blind verification is intuitive and makes the reward easier to inspect than a single opaque judge score.
Critical weaknesses
-
Generic and Teacher-only rubrics are generated once before training, whereas student-conditioned rubrics depend on current rollouts, whereas Student-only and full ROPD regenerate rubrics every step, so instead of student conditioning matters more than teacher grounding, it could be the case that dynamic rubrics is more effective than static ones. In addition, giving the Rubricator a single teacher response leads to 47.08, while giving it no teacher response at all (Generic) leads to 60.79. If teacher responses were the main driver, removing them entirely should hurt more than keeping one. This weakens the statement that multi-teacher coverage is the primary driver (+17.9).
-
The related-work discussion should more clearly distinguish ROPD from recent online/contrastive rubric-generation methods. For example, [1] already derives criteria online by comparing current-policy and control-policy responses, aggregates the criteria, and uses rubric-based rewards for policy optimization. [2] studies eliciting rubrics from high-quality, diverse responses, including examples from stronger models. The paper should clarify technical differences and empirical findings that remain distinctive after these comparisons.
[1] Online Rubrics Elicitation from Pairwise Comparisons. ICML 2026.
[2] Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training. ICLR 2026.
Questions for authors
-
For the reward signal ablation on AIME24, student-only achieves 62.25, one teacher response drops to 47.08, while four teacher responses reach 65.02. What could be the reason that one teacher response performs much worse than no teacher response?
-
How’s the ROPD performance compared to the baseline of purely using GRPO with verifiable reward?
4 comments
2 · Distinctions from recent online rubric methods [1, 2] (C2)
We have expanded Section 5 and Appendix B to discuss [1] (OnlineRubrics) and [2] (Chasing the Tail). While all three methods share the item-level binary verification and weighted-sum GRPO framework, they differ in problem formulation, rubric construction, and empirical findings:
- Technical distinctions: [1] targets reward hacking in online RL by sampling pairs of responses from the current policy and a peer control policy () at each step to extract and merge new criteria into an initial rubric, without conditioning on external stronger teachers. [2] improves reward discrimination in the high-reward tail by iteratively contrasting top-2 responses from an offline candidate pool (spanning multiple frontier models) prior to RL training, rather than reconstructing rubrics at every RL step around the student's live error distribution. In contrast,
ROPDaddresses black-box On-Policy Distillation (replacing white-box teacher logits with text-only supervision) by jointly feeding strong teacher solutions and the current group of student rollouts at each training step (Equation 1), simultaneously anchoring on multi-path teacher principles and targeting the student's current failure modes (raising AIME24 from 63.43 under dynamicTeacher-onlyto 65.02 under fullROPDin Table 1-1). - Unique empirical findings: Our study establishes three controlled results absent from [1, 2]—(i) Outperforming white-box logit distillation under matched models: with the exact same open-weight teacher (Qwen3-30B-A3B) and student (Qwen3-4B),
ROPDachieves a 4-math-benchmark average of 45.87, outperforming white-boxLOPD(32.82) andExOPD(35.25) with 9.6× higher sample efficiency to reachLOPD's peak accuracy (Table 2, Figure 5); (ii) Mechanistic diagnosis of why teacher likelihood decouples from correctness: on a fixed 3,120-response pool (Table 10), white-box teacher token likelihood favors superficially fluent but logically flawed derivations, yielding lower correctness-ranking AUCs (0.658 / 0.524 / 0.351 on Base / LOPD / ROPD rollouts) than both the same teacher's holistic scalar scoreT-Judge(0.792 / 0.781 / 0.834) and item-levelRubricrewards (0.861 / 0.882 / 0.898); and (iii) Quantifying cross-capability distillation interface components: we isolate single-path binding at and its recovery at (47.08 63.27), group-shared rubrics (+3.77), and source-blind verification (+3.27) (Table 5).
3 · Comparison with pure GRPO + verifiable reward (Q2)
Under the exact training recipe of Tables 1 and 2 (non-thinking Qwen3-4B student, DAPO-Math-17K, and identical GRPO hyperparameters and step budgets), we replaced the reward with binary 0/1 matching against dataset ground-truth answers to evaluate the pure GRPO + verifiable reward (RLVR) baseline (revised Section 3.1 and Appendix E).
Across the four competition math benchmarks (120 problems in total; Table 3-1), RLVR achieves an average accuracy of 41.23, outperforming outcome-unverified white-box LOPD (32.82) and ExOPD (35.25), yet remaining substantially below answer-key-free ROPD—which reaches 45.87 (+4.64) with the matched Qwen3-30B-A3B teacher and 49.28 (+8.05) with the GPT-5.2-chat teacher. On hard AIME/HMMT problems, all student rollouts frequently fail to reach the correct final answer, causing binary RLVR rewards to collapse to all zeros with zero within-group advantage; ROPD preserves informative within-group gradients via teacher-guided intermediate milestone criteria (Observable Quality) for rollouts that establish valid equations or transformations, while maintaining high-alignment final verification (AUC 0.898) once correct solutions are reached.
Table 3-1: Math performance comparison between pure GRPO + verifiable reward (RLVR) and distillation methods (Qwen3-4B student, DAPO-Math-17K)
| Training Method | Teacher / Signal Source | Uses Answer Key | Supervision Interface | 4-Math Avg. (%) | vs. RLVR | vs. Matched ROPD |
|---|---|---|---|---|---|---|
| Qwen3-4B (Base) / SFT | — / Qwen3-30B-A3B | No | No RL / Offline static text | 15.63 / 17.29 | −25.60 / −23.94 | −30.24 / −28.58 |
| LOPD / ExOPD (Table 2) | Qwen3-30B-A3B | No | Online white-box teacher logits | 32.82 / 35.25 | −8.41 / −5.98 | −13.05 / −10.62 |
| GRPO + Verifiable Reward (RLVR, new) | Dataset Ground-Truth | Yes | No teacher (0/1 answer match) | 41.23 | 0.00 | −4.64 |
| ROPD (matched open teacher, Table 2) | Qwen3-30B-A3B | No | Black-box teacher text (contrastive rubric) | 45.87 | +4.64 | 0.00 |
| ROPD (strong black-box teacher, Table 1) | GPT-5.2-chat | No | Black-box teacher text (contrastive rubric) | 49.28 | +8.05 | +3.41 |
Dear Reviewer Q6JB,
As the discussion deadline is fast approaching, we wanted to gently follow up and check if our previous responses and revisions have adequately addressed your concerns. Please let us know if anything remains unclear or if you need any further clarification—we would be more than happy to discuss further before the window closes. Thank you again for your time and constructive feedback!
Best regards, The Authors
As the discussion period is nearing its end, we would like to kindly follow up and check whether our rebuttal has addressed all of your concerns. We hope our clarifications were helpful and would sincerely appreciate it if you could consider adjusting your score accordingly.
Response to Reviewer Q6JB
We thank the reviewer for the constructive feedback and for recognizing our method design, experimental breadth, and presentation clarity. We group the main weaknesses (C1, C2) and questions (Q1, Q2) by theme into the 3 responses below.
1 · Confounded ablations & m=1 degradation (C1, Q1)
Under the exact setup of Table 5 (GPT-5.2-chat teacher, Qwen3-4B student, and identical training budgets), we added three controlled experiments: dynamic per-step
Genericrubrics (61.24), dynamic per-stepTeacher-onlyrubrics (63.43), and fullROPDwith m=2 teacher responses (63.27).Genericfrom one-off static generation to dynamic per-step regeneration changes AIME24 accuracy only from 60.79 to 61.24 (+0.45), showing that higher generation frequency alone yields negligible gain without teacher or student responses. Holding dynamic per-step generation fixed, conditioning on student rollouts reaches 62.25 (Student-only), conditioning on m=4 teacher responses reaches 63.43 (dynamicTeacher-only, surpassing the scalar judgeT-Judgeat 62.50), and jointly conditioning on both in fullROPDreaches 65.02 (+3.78 over dynamicGeneric, +2.77 overStudent-only, and +1.59 over dynamicTeacher-only). Both dynamic controls have been added to the revised Table 5a and Section 4.3.Generic(60.79) and m=1 (47.08) in the original manuscript differ simultaneously across three variables—generation frequency, student conditioning, and teacher count m; the revised Section 4.3 removes the direct cross-table comparison. Holding dynamic per-step generation and student conditioning fixed, the single-variable trajectory along m is m=0 (62.25) →m=1 (47.08) →m=2 (63.27) →m=4 (65.02). At m=1, contrasting student rollouts against a single teacher solution causes the Rubricator to hard-code path-specific choices from that solution as mandatory criteria, penalizing alternative valid student derivations. Increasing to m=2 introduces cross-solution diversity that prompts the Rubricator to extract path-agnostic shared criteria, raising AIME24 to 63.27 and recovering 90.2% (+16.19 points) of the 17.94-point gap between m=1 and m=4 in a single step before saturating at m=4 (65.02).Table 1-1: Controlled ablation of rubric sources and teacher response count m under GPT-5.2-chat after removing the static-vs.-dynamic confound (revised Tables 5a / 5b)