acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis

Abstract

On-policy distillation (OPD) is a powerful paradigm for LLM post-training, yet its reliance on teacher logits restricts it to white-box teachers. In this paper, we study whether rubric-based supervision can provide a text-only feedback interface for black-box OPD, and what makes it effective. Our instantiation, Rubric-based On-Policy Distillation (ROPD), induces rubrics by contrasting teacher responses with the student's current rollouts and uses them to score these rollouts for on-policy optimization. We find that, under matched settings, text-only ROPD outperforms logit-based OPD methods on four competition-math benchmarks, reaching the best AIME24 accuracy of standard logit-based OPD with 9.6 fewer training samples. Our analysis yields three findings: rubric rewards track correctness far better than teacher log-likelihood (AUC 0.90 vs. 0.35); training with them repairs more failed criteria and regresses fewer passed ones than logit-based supervision; and contrasting teacher responses with student rollouts is central to rubric quality. These results position rubric-based OPD as a flexible, black-box-compatible alternative to logit-based OPD. Our implementation is available at https://anonymous.4open.science/r/ROPD-8257.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, rubric-based reward, black-box distillation, LLM reasoning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Q6JBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ROPD, a black-box on-policy distillation method that replaces teacher logits with rubric-based feedback. For each prompt, a rubric is generated by contrasting multiple teacher responses with the student’s current rollouts. A verifier evaluates all student responses using the same rubric, and the resulting aggregated scores are used as rewards for GRPO. The paper reports strong gains over both black-box baselines and logit-based OPD, especially on math benchmarks, and provides analysis of reward quality and training dynamics.

Critical strengths
  • The paper studies a timely and interesting problem related to black-box OPD, and provides a controlled comparison against logit-based OPD. The experiments results are informative and consistent.
  • The design of the method is well motivated and practical. The combination of teacher-grounded, student-conditioned, shared rubrics and blind verification is intuitive and makes the reward easier to inspect than a single opaque judge score.
Critical weaknesses
  • Generic and Teacher-only rubrics are generated once before training, whereas student-conditioned rubrics depend on current rollouts, whereas Student-only and full ROPD regenerate rubrics every step, so instead of student conditioning matters more than teacher grounding, it could be the case that dynamic rubrics is more effective than static ones. In addition, giving the Rubricator a single teacher response leads to 47.08, while giving it no teacher response at all (Generic) leads to 60.79. If teacher responses were the main driver, removing them entirely should hurt more than keeping one. This weakens the statement that multi-teacher coverage is the primary driver (+17.9).

  • The related-work discussion should more clearly distinguish ROPD from recent online/contrastive rubric-generation methods. For example, [1] already derives criteria online by comparing current-policy and control-policy responses, aggregates the criteria, and uses rubric-based rewards for policy optimization. [2] studies eliciting rubrics from high-quality, diverse responses, including examples from stronger models. The paper should clarify technical differences and empirical findings that remain distinctive after these comparisons.

[1] Online Rubrics Elicitation from Pairwise Comparisons. ICML 2026.

[2] Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training. ICLR 2026.

Questions for authors
  • For the reward signal ablation on AIME24, student-only achieves 62.25, one teacher response drops to 47.08, while four teacher responses reach 65.02. What could be the reason that one teacher response performs much worse than no teacher response?

  • How’s the ROPD performance compared to the baseline of purely using GRPO with verifiable reward?

4 comments

Response to Reviewer Q6JB

We thank the reviewer for the constructive feedback and for recognizing our method design, experimental breadth, and presentation clarity. We group the main weaknesses (C1, C2) and questions (Q1, Q2) by theme into the 3 responses below.


1 · Confounded ablations & m=1m=1 degradation (C1, Q1)

Under the exact setup of Table 5 (GPT-5.2-chat teacher, Qwen3-4B student, and identical training budgets), we added three controlled experiments: dynamic per-step Generic rubrics (61.24), dynamic per-step Teacher-only rubrics (63.43), and full ROPD with m=2m=2 teacher responses (63.27).

  1. Information-source ablation without the static-vs.-dynamic confound (Table 1-1): Switching problem-only Generic from one-off static generation to dynamic per-step regeneration changes AIME24 accuracy only from 60.79 to 61.24 (+0.45), showing that higher generation frequency alone yields negligible gain without teacher or student responses. Holding dynamic per-step generation fixed, conditioning on student rollouts reaches 62.25 (Student-only), conditioning on m=4m=4 teacher responses reaches 63.43 (dynamic Teacher-only, surpassing the scalar judge T-Judge at 62.50), and jointly conditioning on both in full ROPD reaches 65.02 (+3.78 over dynamic Generic, +2.77 over Student-only, and +1.59 over dynamic Teacher-only). Both dynamic controls have been added to the revised Table 5a and Section 4.3.
  2. Single-path binding at m=1m=1 and recovery at m=2m=2 (Table 1-1): Static Generic (60.79) and m=1m=1 (47.08) in the original manuscript differ simultaneously across three variables—generation frequency, student conditioning, and teacher count mm; the revised Section 4.3 removes the direct cross-table comparison. Holding dynamic per-step generation and student conditioning fixed, the single-variable trajectory along mm is m=0m=0 (62.25) →m=1\to m=1 (47.08) →m=2\to m=2 (63.27) →m=4\to m=4 (65.02). At m=1m=1, contrasting student rollouts against a single teacher solution causes the Rubricator to hard-code path-specific choices from that solution as mandatory criteria, penalizing alternative valid student derivations. Increasing to m=2m=2 introduces cross-solution diversity that prompts the Rubricator to extract path-agnostic shared criteria, raising AIME24 to 63.27 and recovering 90.2% (+16.19 points) of the 17.94-point gap between m=1m=1 and m=4m=4 in a single step before saturating at m=4m=4 (65.02).

Table 1-1: Controlled ablation of rubric sources and teacher response count mm under GPT-5.2-chat after removing the static-vs.-dynamic confound (revised Tables 5a / 5b)

VariantGeneration Freq.Teachers mmStudent-ConditionedAIME24 (%)vs. Dynamic Genericvs. m=0m=0 (Student-only)
Generic / Teacher-only (static, original)One-off0 / 4No60.79 / 61.39−0.45 / +0.15−1.46 / −0.86
Generic (dynamic, new)Per-step0No61.240.00−1.01
T-Judge (holistic scalar judge, original)Per-step——62.50+1.26+0.25
Teacher-only (dynamic, new)Per-step4No63.43+2.19+1.18
Student-only (dynamic, original, m=0m=0 baseline)Per-step0Yes62.25+1.010.00
w/o multi-teacher (original, single-path binding)Per-step1Yes47.08−14.16−15.17
Two teacher responses (new, 90.2% gap recovered)Per-step2Yes63.27+2.03+1.02 (+16.19 vs. m=1m=1)
ROPD Full (dynamic, original)Per-step4Yes65.02+3.78+2.77 (+17.94 vs. m=1m=1)

2 · Distinctions from recent online rubric methods [1, 2] (C2)

We have expanded Section 5 and Appendix B to discuss [1] (OnlineRubrics) and [2] (Chasing the Tail). While all three methods share the item-level binary verification and weighted-sum GRPO framework, they differ in problem formulation, rubric construction, and empirical findings:

  1. Technical distinctions: [1] targets reward hacking in online RL by sampling pairs of responses from the current policy and a peer control policy (πref/πold\pi_{\text{ref}}/\pi_{\text{old}}) at each step to extract and merge new criteria into an initial rubric, without conditioning on external stronger teachers. [2] improves reward discrimination in the high-reward tail by iteratively contrasting top-2 responses from an offline candidate pool (spanning multiple frontier models) prior to RL training, rather than reconstructing rubrics at every RL step around the student's live error distribution. In contrast, ROPD addresses black-box On-Policy Distillation (replacing white-box teacher logits with text-only supervision) by jointly feeding mm strong teacher solutions and the current group of GG student rollouts at each training step (Equation 1), simultaneously anchoring on multi-path teacher principles and targeting the student's current failure modes (raising AIME24 from 63.43 under dynamic Teacher-only to 65.02 under full ROPD in Table 1-1).
  2. Unique empirical findings: Our study establishes three controlled results absent from [1, 2]—(i) Outperforming white-box logit distillation under matched models: with the exact same open-weight teacher (Qwen3-30B-A3B) and student (Qwen3-4B), ROPD achieves a 4-math-benchmark average of 45.87, outperforming white-box LOPD (32.82) and ExOPD (35.25) with 9.6× higher sample efficiency to reach LOPD's peak accuracy (Table 2, Figure 5); (ii) Mechanistic diagnosis of why teacher likelihood decouples from correctness: on a fixed 3,120-response pool (Table 10), white-box teacher token likelihood favors superficially fluent but logically flawed derivations, yielding lower correctness-ranking AUCs (0.658 / 0.524 / 0.351 on Base / LOPD / ROPD rollouts) than both the same teacher's holistic scalar score T-Judge (0.792 / 0.781 / 0.834) and item-level Rubric rewards (0.861 / 0.882 / 0.898); and (iii) Quantifying cross-capability distillation interface components: we isolate single-path binding at m=1m=1 and its recovery at m=2m=2 (47.08 →\to 63.27), group-shared rubrics (+3.77), and source-blind verification (+3.27) (Table 5).

3 · Comparison with pure GRPO + verifiable reward (Q2)

Under the exact training recipe of Tables 1 and 2 (non-thinking Qwen3-4B student, DAPO-Math-17K, and identical GRPO hyperparameters and step budgets), we replaced the reward with binary 0/1 matching against dataset ground-truth answers to evaluate the pure GRPO + verifiable reward (RLVR) baseline (revised Section 3.1 and Appendix E).

Across the four competition math benchmarks (120 problems in total; Table 3-1), RLVR achieves an average accuracy of 41.23, outperforming outcome-unverified white-box LOPD (32.82) and ExOPD (35.25), yet remaining substantially below answer-key-free ROPD—which reaches 45.87 (+4.64) with the matched Qwen3-30B-A3B teacher and 49.28 (+8.05) with the GPT-5.2-chat teacher. On hard AIME/HMMT problems, all G=8G=8 student rollouts frequently fail to reach the correct final answer, causing binary RLVR rewards to collapse to all zeros with zero within-group advantage; ROPD preserves informative within-group gradients via teacher-guided intermediate milestone criteria (Observable Quality) for rollouts that establish valid equations or transformations, while maintaining high-alignment final verification (AUC 0.898) once correct solutions are reached.

Table 3-1: Math performance comparison between pure GRPO + verifiable reward (RLVR) and distillation methods (Qwen3-4B student, DAPO-Math-17K)

Training MethodTeacher / Signal SourceUses Answer KeySupervision Interface4-Math Avg. (%)vs. RLVRvs. Matched ROPD
Qwen3-4B (Base) / SFT— / Qwen3-30B-A3BNoNo RL / Offline static text15.63 / 17.29−25.60 / −23.94−30.24 / −28.58
LOPD / ExOPD (Table 2)Qwen3-30B-A3BNoOnline white-box teacher logits32.82 / 35.25−8.41 / −5.98−13.05 / −10.62
GRPO + Verifiable Reward (RLVR, new)Dataset Ground-TruthYesNo teacher (0/1 answer match)41.230.00−4.64
ROPD (matched open teacher, Table 2)Qwen3-30B-A3BNoBlack-box teacher text (contrastive rubric)45.87+4.640.00
ROPD (strong black-box teacher, Table 1)GPT-5.2-chatNoBlack-box teacher text (contrastive rubric)49.28+8.05+3.41

Dear Reviewer Q6JB,

As the discussion deadline is fast approaching, we wanted to gently follow up and check if our previous responses and revisions have adequately addressed your concerns. Please let us know if anything remains unclear or if you need any further clarification—we would be more than happy to discuss further before the window closes. Thank you again for your time and constructive feedback!

Best regards, The Authors

As the discussion period is nearing its end, we would like to kindly follow up and check whether our rebuttal has addressed all of your concerns. We hope our clarifications were helpful and would sincerely appreciate it if you could consider adjusting your score accordingly.

Reviewer_WrDZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ROPD distills from black-box teachers through rubric feedback: several sampled teacher answers are contrasted with the current student's rollouts to induce weighted, prompt-level criteria, and a provenance-blind verifier scores rollouts against them to produce a scalar GRPO reward. With a Qwen3-4B student, experiments compare black-box methods under a GPT-5.2 teacher and, separately, text- versus logit-based supervision under a common Qwen3-30B-A3B teacher; ROPD is reported to win on four math benchmarks and to reach LOPD's best AIME24 accuracy with 9.6× fewer samples. The contribution is the feedback interface and its analysis rather than a new optimizer.

Critical strengths
  1. A practical interface with large matched-teacher gains. With Qwen3-30B-A3B as teacher, the four-benchmark math mean is 45.87 for ROPD versus 32.82 (LOPD) and 35.25 (ExOPD) (Table 2). The method (pp. 3–4), exact prompts (pp. 19–22) and pseudocode (p. 29) make the pipeline clear to implement for API-only teachers; this drives my positive leaning.
  2. Informative reward diagnostics. Rubric rewards predict correctness better than mean teacher likelihood for Base, LOPD and ROPD checkpoints alike (Table 10), adding insight into OPD reward signals beyond benchmark gains; the headline AUC gap (0.90 vs. 0.35) is ROPD-specific (LOPD 0.88/0.52, Base 0.86/0.66).
Critical weaknesses
  1. The source of the gain is not isolated, which weakens the paper's mechanism claims. In the GPT/AIME24 ablation (Table 5), a scalar teacher-judge (T-Judge, 62.50) and generic rubrics (60.79, still built and verified by the strong model) already obtain most of full ROPD's gain over the base student (65.02 vs. 24.17); no partial-information variant exceeds T-Judge, and these single-run differences come without uncertainty estimates. Table 2 lacks T-Judge/Generic controls under the Qwen teacher and a ground-truth-reward GRPO reference, so its margin compares judge-based RL with KL distillation as complete pipelines. Likewise, m=1 (47.08) falls below Generic and Student-only (62.25); this may reflect single-answer anchoring, which p. 2 itself anticipates, so the 17.9-point gap to m=1 need not demonstrate the claimed coverage benefit.
  2. The headline 9.6× efficiency figure is not yet a stable estimate. At 1.6k prompts (step 50 at batch 32), ROPD reaches 48.33 on AIME24, only 0.41 above LOPD's 47.92 threshold, while the paper's independent sample gives 43.75 (p. 24). At 15.4k prompts (≈ step 480), LOPD shows 42.1 in the analysis pool (Table 11), although p. 24 states that converged LOPD checkpoints (step ≥ 240) agree within ≤0.1%; Fig. 5a also plots a four-task mean for an AIME24 claim. Run/checkpoint mappings are needed.
Questions for authors
  1. Are T-Judge/Generic controls under the Qwen3-30B-A3B teacher, or a ground-truth-reward GRPO reference, available? Could a scalar-judge AUC be reported on the existing analysis pool as a cheaper check? Otherwise, please bound the attribution claims accordingly.
  2. Which run and teacher/Rubricator/Verifier underlie Sec. 4.2 and Tables 9–11? Page 24 mentions a GPT-5.2 training Verifier but compares ROPD step 50 with Fig. 5's Qwen-teacher run; if the teachers differ from LOPD's, the teacher-likelihood findings are confounded. Does the 9.6× crossing survive the paper's own resampling?
7 comments

Response to Reviewer WrDZ

We thank the reviewer for the thorough examination of our attribution boundaries and mechanistic analysis. We group the 2 main weaknesses (C1, C2) and 2 questions (Q1, Q2) by theme into the 4 responses below.


1 · Missing RLVR baselines & gain attribution (C1)

Under the exact training recipe of the main paper—using non-thinking Qwen3-4B on DAPO-Math-17K with identical GRPO hyperparameters and step budgets—we added the noise-free ground-truth 0/1 correctness reward baseline (RLVR), along with dynamic Generic, dynamic Teacher-only, and the m=2m=2 teacher-response control under Table 5 to remove the static-versus-dynamic confound.

  1. Table 2 boundary and the RLVR reference (Table 1-1): RLVR achieves a 4-math-benchmark average of 41.23, outperforming LOPD (32.82) and ExOPD (35.25), neither of which verifies answer correctness, yet remaining substantially below answer-key-free ROPD: ROPD reaches 45.87 (+4.64 over RLVR) with the matched Qwen3-30B-A3B teacher and 49.28 (+8.05) with the GPT-5.2-chat teacher. The revised Abstract and Section 3.3 explicitly state that Table 2 compares full distillation pipelines against RLVR: on hard competition problems where all rollouts in a group fail to reach the correct final answer, a binary 0/1 reward yields zero within-group variance, whereas ROPD's process criteria (Table 4) still provide non-zero intermediate guidance.
  2. Gain attribution beyond the strong judge and cross-benchmark validation (Tables 1-2, 1-3): We agree that the jump from the base model (24.17) to 60.79–62.50 stems primarily from driving GRPO with a strong LLM judge, which means the distillation gain must be attributed above this strong-judge baseline. First, Table 1-2 fixes generation frequency to dynamic per-step regeneration to remove the static-versus-dynamic confound: problem-only dynamic Generic reaches 61.24, only 0.45 above its static counterpart (60.79), indicating that regenerating generic criteria per step yields little benefit on its own. Conditioning on teacher solutions raises dynamic Teacher-only to 63.43, surpassing the scalar judge T-Judge (62.50) using the same teacher. Incorporating both teacher solutions and student rollouts in full ROPD further reaches 65.02, yielding net gains of +3.78 over dynamic Generic and +2.52 over T-Judge. Second, to rule out 30-problem single-benchmark sampling noise (SE ≈\approx 2.2 points; ablations are interpreted by direction and magnitude), Table 1-3 fixes the same GPT-5.2-chat judge across all four math competition benchmarks (120 problems in total, SE ≈\approx 1.1 points): ROPD consistently outperforms same-teacher T-Judge on every benchmark (+2.11 to +2.92), achieving a 4-benchmark average of 49.28 versus 46.71 (+2.57) for T-Judge and 46.08 (+3.20) for OVD.
  3. Why m=1m=1 falls below Generic: single-path binding and the m=2m=2 recovery experiment (Table 1-2): Static Generic (60.79) and m=1m=1 (47.08) in the original manuscript differ across three variables (generation frequency, student conditioning, and teacher count mm); the revised Section 4.3 removes the direct cross-table subtraction. Holding dynamic per-step generation and student conditioning fixed, the single-variable zero-teacher baseline is m=0m=0 (Student-only, 62.25). At m=1m=1, contrasting student rollouts against a single teacher response hard-codes path-specific choices from that response as mandatory criteria, penalizing valid alternative derivations. Adding m=2m=2 under the same setup raises AIME24 to 63.27, recovering 90.2% (+16.19 points) of the 17.94-point gap between m=1m=1 and m=4m=4 in one step and surpassing the zero-teacher baselines (61.24 and 62.25), before saturating at m=4m=4 (+1.75 points; updated in Section 4.3 and Insight 3).

Table 1-1: Comparison of ground-truth RLVR and distillation methods under the matched setting (Table 2 protocol, Qwen3-30B-A3B teacher)

Training MethodUses Answer KeySupervision Interface4-Math Avg. (120 probs, %)vs. RLVRvs. Matched ROPD
Qwen3-4B (Base) / SFTNoNo RL / Offline static text15.63 / 17.29−25.60 / −23.94−30.24 / −28.58
LOPD / ExOPD (Table 2)NoOnline white-box token logits32.82 / 35.25−8.41 / −5.98−13.05 / −10.62
GRPO + Ground-Truth Reward (RLVR, new)YesNo teacher, 0/1 final-answer match41.230.00−4.64
ROPD (Qwen3-30B-A3B matched teacher)NoBlack-box teacher text only, contrastive rubric45.87+4.640.00

Table 1-2: Rubric-source and teacher-count mm ablations under GPT-5.2-chat after removing the static-vs.-dynamic confound (revised Table 5)

VariantGeneration Freq.Teachers mmStudent-ConditionedAIME24 (%)vs. Dynamic Genericvs. T-Judge
Generic / Teacher-only (static, original)One-off0 / 4No60.79 / 61.39−0.45 / +0.15−1.71 / −1.11
Generic (dynamic, new)Per-step0No61.240.00−1.26
Student-only (dynamic, original, m=0m=0 baseline)Per-step0Yes62.25+1.01−0.25
T-Judge (holistic scalar judge, original)Per-step——62.50+1.260.00
Teacher-only (dynamic, new)Per-step4No63.43+2.19+0.93
w/o multi-teacher (original, single-path binding)Per-step1Yes47.08−14.16−15.42
Two teacher responses (new, recovers 90.2% gap)Per-step2Yes63.27+2.03+0.77
ROPD Full (dynamic, original)Per-step4Yes65.02+3.78+2.52

Table 1-3: Per-benchmark comparison between full ROPD and scalar baselines across four math benchmarks (120 problems) under GPT-5.2-chat (Table 1)

Method (Same GPT-5.2-chat Teacher)AIME24AIME25HMMT25 Feb.HMMT25 Nov.4-Benchmark Avg. (%)vs. ROPD
OVD / T-Judge (Table 1)61.56 / 62.5056.85 / 56.6427.92 / 28.9437.98 / 38.7546.08 / 46.71−3.20 / −2.57
ROPD (Full configuration, m=4m=4)65.0258.7531.6941.6749.280.00
Net gain of ROPD over T-Judge+2.52+2.11+2.75+2.92+2.57—

2 · 9.6× sample efficiency & evaluation protocol consistency (C2)

  1. The 9.6× sample efficiency holds across additional benchmarks and the 4-benchmark average (Table 2-1): The 0.41-point difference serves only to locate the training sample count required to match LOPD's peak performance (1.6k vs. 15.4k prompts, i.e., 9.6× sample efficiency; 5.5h vs. 34.4h, i.e., 6.3× wall-clock speedup). Crucially, the 9.6× ratio does not rest on a single AIME24 point: at the exact same 1.6k sample checkpoint (step 50), ROPD also surpasses LOPD's 15.4k-sample peak (step 480) on AIME25 (40.50 vs. 38.75, +1.75), HMMT25 Nov. (27.50 vs. 24.17, +3.33), and Figure 5a's 4-benchmark average (33.53 vs. 32.82, +0.71). Whereas the original manuscript plotted the 4-benchmark average in Figure 5a while citing only AIME24 in the main text, the revision reports both metrics side by side in the text and figure caption.
  2. Within-protocol consistency across main evaluation and analysis pool, and scope of the ≤0.1% statement (Table 2-2): All experiments comparing against white-box baselines originate from the exact same Qwen3-30B-A3B teacher run. The numerical differences noted by the reviewer—48.33% vs. 43.75% (ROPD step 50) and 47.92% vs. 42.1% (LOPD step 480)—stem from normal binomial sampling variance between the main evaluation protocol (k=16k=16 samples per problem, 480 responses) and the independent offline analysis pool (Table 11, k=8k=8 samples per problem, 240 responses). When compared under the same protocol using the same yardstick, the 9.6× crossing holds consistently in both: 48.33% > 47.92% (+0.41) under k=16k=16 main evaluation, and 43.75% > 42.08% (+1.67) under the k=8k=8 analysis pool. Furthermore, the "≤0.1%\le 0.1\% difference" statement on page 24 of Appendix G applies specifically to ROPD's converged phase (steps 150/200/250 read 55.00/55.0, 62.50/62.5, and 63.33/63.3% under k=16k=16 and k=8k=8, all agreeing within ≤0.1%\le 0.1\%); the revision narrows this statement accordingly and includes the full dual-protocol checkpoint mapping table.

Table 2-1: Per-benchmark and average comparison between ROPD at 1.6k samples and LOPD peak at 15.4k samples (Qwen3-30B-A3B matched teacher, k=16k=16)

BenchmarkROPD @ 1.6k (step 50, Pass@1 %)LOPD Peak @ 15.4k (step 480, Pass@1 %)DifferenceSample Efficiency Ratio
AIME24 (Figure 5b metric)48.3347.92+0.419.6×
AIME2540.5038.75+1.759.6×
HMMT25 Feb.17.8020.42−2.62—
HMMT25 Nov.27.5024.17+3.339.6×
4-Benchmark Avg. (Figure 5a metric)33.5332.82+0.719.6×

Table 2-2: AIME24 accuracy trajectories (Pass@1 %) of ROPD and LOPD under main evaluation (k=16k=16) and the analysis pool (k=8k=8, Table 11) with the same Qwen3-30B-A3B teacher

MethodCheckpoint Steps (Samples)Main Evaluation k=16k=16 (Fig. 5 / Table 2)Analysis Pool k=8k=8 (Table 11)
ROPDstep 50 (1.6k) / 100 / 150 / 200 / 250 (8.0k)48.33 / 56.25 / 55.00 / 62.50 / 63.3343.75 / 52.50 / 55.00 / 62.50 / 63.33
LOPDstep 80 / 160 / 240 / 320 / 400 / 480 (15.4k) / 54327.50 / 31.25 / 40.83 / 39.58 / 42.92 / 47.92 / 44.5827.50 / 31.25 / 36.25 / 38.75 / 41.67 / 42.08 / 45.42
Reviewer_La74OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper asks whether rubrics can stand in for teacher logits in on-policy distillation (OPD) when the teacher is black-box, i.e. text-only. It proposes ROPD:

Rubricator. For each prompt, a Rubricator (the teacher LLM) reads m=4m = 4 teacher responses alongside the student’s current group of n=8n = 8 rollouts. It writes one shared set of K∈[4,12]K \in [4, 12] weighted binary criteria aimed at gaps between teacher and student. Verifier. A blind Verifier (also the teacher) checks each rollout against every criterion. Reward. The weighted pass rate is used as the GRPO reward.

Results:

Black-box teacher (GPT-5.2-chat). With a Qwen3-4B student trained on DAPO-Math-17K, ROPD beats T-Judge, OVD and GAD on AIME24/25 and HMMT25. Gains on GPQA-D, HealthBench and IFEval are small. Matched white-box teacher (Qwen3-30B-A3B). Text-only ROPD (45.87 average) beats logit-based LOPD (32.82) and ExOPD (35.25). It reaches LOPD’s best AIME24 accuracy with 9.6× fewer training samples.

The analysis offers three findings:

Rubric reward tracks correctness better than teacher log-likelihood (AUC 0.90 vs 0.35). ROPD fixes more failed criteria and breaks fewer passed ones than LOPD. Building rubrics from teacher–student contrast matters, with multiple teacher answers mattering most.

Critical strengths
  • A clean, practical recipe for a real constraint. Frontier teachers increasingly expose only text. Group-shared rubrics conditioned on both teacher responses and student rollouts are a sensible way to get feedback on the student’s own rollouts without logits. The method is simple, well specified (full prompts, hyperparameters, pseudocode, code link), and easy to reuse.
  • Strong benchmark gains, plus a useful ablation design. The non-thinking math gains are large (AIME24 24.2 → 65.0). The paper separates rubric source ablations from component ablations, and the T-Judge baseline controls for judge strength. Together these make the reward-design space easy to see, even though I read the results differently from the authors (below).
Critical weaknesses
  • The gains can’t be attributed to distillation or to rubrics, because correctness reward is a confound and RLVR is missing. DAPO-Math-17K has ground-truth answers, yet there is no GRPO-with-correctness-reward (RLVR) baseline. RLVR is the obvious reference point for a non-thinking Qwen3-4B on this dataset. ROPD’s reward does include correctness: Appendix C says every rubric contains “at least one high-weight criterion that checks final-answer correctness”. This contradicts the Rubricator prompt, which forbids criteria that match the teacher’s final answer. The Verifier is the strong teacher itself. The case-study criterion C2 effectively encodes the answer (“→ no solution”). LOPD and ExOPD get no correctness signal at all. So “text-only beats logits” largely compares a correctness-aware reward with a pure imitation signal. The paper’s own ablation (Table 5) points the same way: Generic rubrics reach 60.79 on AIME24. They see no teacher responses and no student rollouts and are generated once before training. Full ROPD reaches 65.02, and T-Judge reaches 62.50. So roughly 90% of the gain over the base model (24.17) needs neither distillation nor contrastive induction. It looks like what you’d expect from RL with a strong LLM checking correctness. With one teacher answer (m=1m = 1), the score falls to 47.08, below Generic. So one teacher answer is much worse than none, which is hard to square with “teacher-grounded” being the key ingredient. Baselines needed: RLVR with ground-truth answers; a reward for matching the teacher’s majority answer; LOPD plus a correctness reward; Generic rubrics in the white-box setting. Without these, the headline comparison and Insight 3 are not supported.
  1. The reward-signal analysis is constructed in ROPD’s favor, and the reported differences have no error bars. AUC. The 0.35 log-likelihood AUC in the abstract is measured only on ROPD’s own rollouts. On Base rollouts it is 0.66 and on LOPD rollouts 0.52 (Table 10). So teacher likelihood is positively associated with correctness on the untrained policy, contrary to the claim that it “favors fluent but flawed reasoning”. Rubric reward is scored by the judge ROPD trained against. It is computed with a Qwen3-30B-A3B Verifier. Appendix G calls this “distinct from the GPT-5.2 Verifier used during training”. But the ROPD checkpoints analyzed are from the matched run (step 250, accuracy 0.633, which matches Table 2), where Table 7 says Qwen3-30B-A3B was the Rubricator and Verifier. ROPD is then being scored by the judge it was optimized against. Criterion-level comparison. Tables 4 and 9 measure progress on rubric criteria, which ROPD optimizes directly and LOPD does not. They also use different checkpoint windows (50→250 vs 80→543). The comparison favors ROPD by design. Case study. The single case study (Table 3) uses the same problem and criteria as the few-shot example in the Rubricator prompt (Appendix F), so it may not be representative. Noise. All results are single runs on 30-problem benchmarks. Most ablation gaps (0.6–3.8 points), and the gains on GPQA, HealthBench and IFEval (under 2 points), are probably within seed and sampling noise.
Questions for authors
  • What is GRPO with a ground-truth correctness reward on DAPO-Math-17K under your recipe (non-thinking Qwen3-4B, same steps)? And with a reward for matching the teacher’s majority answer? If either comes close to 65, the contribution is a strong correctness judge, not rubric-based distillation.
  • How do rubric criteria check final-answer correctness if the Rubricator may not encode the teacher’s answer? What fraction of the total rubric weight goes to answer-correctness criteria?
  • Which model was the Rubricator and Verifier for the ROPD checkpoints in Tables 10–11? Which rubrics were used to score LOPD rollouts?
  • Why does m=1m = 1 (47.08) fall so far below Generic rubrics with no teacher input (60.79)? Can you report multiple seeds with standard errors for Table 5?
  • Were LOPD and ExOPD tuned separately (learning rate, KL, steps), or run with the GRPO recipe? Was the Qwen3-30B-A3B teacher run in thinking mode while the student was non-thinking? If so, LOPD is imitating a mismatched distribution.
  • What is the cost per training sample once all calls are counted (mm teacher generations, one Rubricator call, and nn Verifier calls)? Is the 9.6× sample-efficiency figure still meaningful at that cost?
  • The prompt example says “K=8 for illustration”, but its JSON shows K=6 with three items.
  • The 0.35 AUC is presented as a general property of teacher likelihood, though it holds only on ROPD’s own rollouts.
7 comments

Response to Reviewer La74

We thank the reviewer for the thorough evaluation and for recognizing the practicality of our method and the value of our ablation design. We group the 2 main weaknesses (C1, C2) and 8 questions (Q1–Q8) by theme into the 8 responses below.


1 · Missing RLVR baselines & gain attribution (C1, Q1)

Under the exact training recipe of the main paper—using non-thinking Qwen3-4B on DAPO-Math-17K with identical GRPO hyperparameters and step budgets—we added the noise-free ground-truth 0/1 correctness reward baseline (RLVR), along with dynamic Generic and dynamic Teacher-only under Table 5 to remove the static-versus-dynamic confound.

  1. RLVR and matching the teacher's majority answer (Table 1-1): RLVR achieves a 4-math-benchmark average of 41.23, outperforming LOPD (32.82) and ExOPD (35.25), neither of which verifies answer correctness, yet remaining substantially below answer-key-free ROPD: ROPD reaches 45.87 (+4.64 over RLVR) with the matched Qwen3-30B-A3B teacher and 49.28 (+8.05) with the GPT-5.2-chat teacher. The revised Abstract and Section 3.3 now state "text-only beats logits" precisely as a comparison between a text-only interface carrying process and outcome criteria and token-level logit distillation. On hard competition problems where all rollouts in a group fail to reach the correct final answer, a binary 0/1 reward yields zero within-group variance, whereas ROPD's process criteria still provide non-zero gradients. Rewarding agreement with the teacher's majority answer approximates the ground-truth 0/1 reward using a noisy teacher consensus; noise-free RLVR (41.23) already provides the performance reference for such final-answer matching signals, and majority voting cannot be applied to open-ended long-form tasks such as HealthBench.
  2. Gain attribution beyond the strong judge and cross-benchmark validation (Tables 1-2, 1-3): We agree that the jump from the base model (24.17) to 60.79–62.50 stems primarily from driving GRPO with a strong LLM judge, which means the distillation gain must be attributed above this strong-judge baseline. First, Table 1-2 fixes generation frequency to dynamic per-step regeneration to remove the static-versus-dynamic confound: problem-only dynamic Generic reaches 61.24, only 0.45 above its static counterpart (60.79), indicating that regenerating generic criteria per step yields little benefit on its own. Conditioning on teacher solutions raises dynamic Teacher-only to 63.43, surpassing the scalar judge T-Judge (62.50) using the same teacher. Incorporating both teacher solutions and student rollouts in full ROPD further reaches 65.02, yielding net gains of +3.78 over dynamic Generic and +2.52 over T-Judge. Second, to rule out 30-problem single-benchmark sampling noise and verify that this net gain is not specific to AIME24, Table 1-3 fixes the same GPT-5.2-chat judge across all four math competition benchmarks (120 problems in total): ROPD consistently outperforms same-teacher T-Judge on every benchmark, with per-benchmark gains ranging from +2.11 to +2.92 and a 4-benchmark average of 49.28 versus 46.71 (+2.57) for T-Judge and 46.08 (+3.20) for OVD. Controlling for the strong judge, structured teacher–student contrastive rubrics provide a consistent net distillation gain over holistic scalar scoring across benchmarks. The drop to 47.08 at m=1m=1 stems from single-path binding degeneracy and recovers to 63.27 at m=2m=2, as detailed in Section 5.

Table 1-1: Comparison of ground-truth RLVR and distillation methods under the matched setting (Table 2 protocol, Qwen3-30B-A3B teacher)

Training MethodUses Answer KeySupervision Interface4-Math Avg. (120 probs, %)vs. RLVRvs. Matched ROPD
Qwen3-4B (Base) / SFTNoNo RL / Offline static text15.63 / 17.29−25.60 / −23.94−30.24 / −28.58
LOPD / ExOPD (Table 2)NoOnline white-box token logits32.82 / 35.25−8.41 / −5.98−13.05 / −10.62
GRPO + Ground-Truth Reward (RLVR, new)YesNo teacher, 0/1 final-answer match41.230.00−4.64
ROPD (Qwen3-30B-A3B matched teacher)NoBlack-box teacher text only, contrastive rubric45.87+4.640.00
GRPO + Ground-Truth Reward (RLVR, new)YesNo teacher, 0/1 final-answer match41.230.00−4.64
ROPD (Qwen3-30B-A3B matched teacher)NoBlack-box teacher text only, contrastive rubric45.87+4.640.00

Table 1-2: Rubric-source ablation under GPT-5.2-chat after removing the static-vs.-dynamic confound (revised Table 5a)

VariantGeneration Freq.Teachers mmStudent-ConditionedAIME24 (%)vs. Dynamic Genericvs. T-Judge
Generic / Teacher-only (static, original)One-off0 / 4No60.79 / 61.39−0.45 / +0.15−1.71 / −1.11
Generic (dynamic, new)Per-step0No61.240.00−1.26
Student-only (dynamic, original)Per-step0Yes62.25+1.01−0.25
T-Judge (holistic scalar judge, original)Per-step——62.50+1.260.00
Teacher-only (dynamic, new)Per-step4No63.43+2.19+0.93
ROPD Full (dynamic, original)Per-step4Yes65.02+3.78+2.52

Table 1-3: Per-benchmark comparison between full ROPD and scalar baselines across four math benchmarks (120 problems) under GPT-5.2-chat (Table 1)

Method (Same GPT-5.2-chat Teacher)AIME24AIME25HMMT25 Feb.HMMT25 Nov.4-Benchmark Avg. (%)vs. ROPD
OVD / T-Judge (Table 1)61.56 / 62.5056.85 / 56.6427.92 / 28.9437.98 / 38.7546.08 / 46.71−3.20 / −2.57
ROPD (Full configuration, m=4m=4)65.0258.7531.6941.6749.280.00
Net gain of ROPD over T-Judge+2.52+2.11+2.75+2.92+2.57—
ROPD (Full configuration, m=4m=4)65.0258.7531.6941.6749.280.00
Net gain of ROPD over T-Judge+2.52+2.11+2.75+2.92+2.57—

2 · Reward-signal analysis design, 0.35 AUC scope, and sampling noise (C2, Q8)

  1. Scope of the 0.35 AUC and non-circularity of the judge (Table 2-1): The 0.351 AUC is measured solely on ROPD rollouts; the revised Abstract and Section 4.2 remove phrasing that presents it as a general constant of teacher likelihood. To rule out circularity from scoring ROPD with the same judge it was trained against, we added the holistic scalar T-Judge AUC from the exact same Qwen3-30B-A3B model on the same 3,120 responses. On Base and LOPD, neither of which was trained against the judge, item-level Rubric AUCs reach 0.861 and 0.882, consistently exceeding both teacher likelihood (0.658 and 0.524) and the same judge's scalar scores (0.792 and 0.781) by +0.069 and +0.101. On LOPD in particular, both likelihood and scalar scores degrade due to superficially fluent but flawed derivations, whereas the item-level Rubric rises to 0.882.
  2. Analytical role of Tables 4 and 9: As the reviewer notes, ROPD directly optimizes rubrics whereas LOPD optimizes logits, and the two trajectories are evaluated over their respective convergence windows (steps 50→250 for ROPD vs. steps 80→543 for LOPD); using rubric pass rates to rank the two methods would therefore favor ROPD by construction. We agree with this boundary and clarify two points in the revision: first, the conclusion that ROPD outperforms LOPD does not rely on rubric judge scores, but is independently established by ground-truth competition accuracy in Table 2 (4-benchmark average 45.87 vs. 32.82, +13.05 points); second, Tables 4 and 9 are not intended as a cross-method performance leaderboard, but as a fine-grained attribution analysis explaining why LOPD improves slowly on ground-truth accuracy—namely, within LOPD's own training window (steps 80→543), token-likelihood imitation repairs some reasoning steps while simultaneously inducing frequent regressions (pass→fail) on previously mastered steps. The revision explicitly notes the step windows in the captions of Tables 4 and 9 and tightens Section 4.2 and Insight 2 to frame these results as a qualitative characterization of how the two training signals evolve, rather than direct proof of overall method superiority.
  3. Representativeness of the Table 3 case study: Section 4.1 serves solely as a single-problem mechanistic illustration, while the statistical claim is supported by Rubric outperforming scalar scoring across the 3,120-response analysis pool in Table 2-1; the revised Appendix C adds a complete case study on a problem that does not appear in the prompt.
  4. Single-run sampling noise and non-math domain boundaries: On a 30-problem benchmark (3.33 points per problem, SE ≈\approx 2.2 points), single-benchmark ablations are interpreted by direction and magnitude; our main mathematical claims rest on consistent positive gains across all four benchmarks totaling 120 problems (SE ≈\approx 1.1 points), where all four subtasks improve in Table 1-3 (+2.57 average) and matched ROPD exceeds RLVR by +4.64. Gains on GPQA-D (+0.84), HealthBench (+1.60), and IFEval (+0.07) serve only as evidence of no out-of-domain degradation, which is now stated explicitly in the Table 1 caption.

Table 2-1: Correctness-ranking AUC of three signals from the same Qwen3-30B-A3B on the fixed analysis pool (3,120 AIME24 responses, revised Table 10)

Rollout Policy FamilySample SizeTrained Against Judge?Teacher Likelihood AUCT-Judge Scalar AUC (new)Rubric Reward AUCRubric − ScalarRubric − Likelihood
Base (untrained)240No0.6580.7920.861+0.069+0.203
LOPD (logit distillation)1,680No0.5240.7810.882+0.101+0.358
ROPD (rubric distillation)1,200Yes0.3510.8340.898+0.064+0.547
LOPD (logit distillation)1,680No0.5240.7810.882+0.101+0.358
ROPD (rubric distillation)1,200Yes0.3510.8340.898+0.064+0.547

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.