acceptodds
Under review as a conference paper at ICLR 2027

Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers

Abstract

Reinforcement learning can substantially improve a reasoning teacher, but it is unclear which of those improvements survive when the teacher supervises a smaller on-policy student. We study this question in mathematical reasoning by comparing teacher lineages before and after GRPO, multiple student scales, direct GRPO, and several on-policy distillation objectives. The central finding is that transfer is structured rather than scalar: teacher strength alone does not make dense distillation competitive, while an RL-improved teacher creates useful but metric-dependent student gains. This motivates Transfer-Stratified On-Policy Distillation (TS-OPD), which screens training problems by the joint sampled success of the student and teacher, routes acquisition problems to gated forward KL, routes consolidation problems to gated reverse KL, and adds an entropy brake to protect sampled coverage. Across the main comparison, TS-OPD is the strongest student objective for macro average correctness with the GRPO-improved teacher, while pass@K remains more mixed. Ablations show that the gains come from routing and token gating rather than skipping problems. These results support a transfer-aware view of OPD: stronger teachers help when the supervision direction and token budget match the student's observed ability, not merely because the teacher endpoint is stronger. We submit our code as supplementary material and will publicly release it upon acceptance.

Primary area:
generative models
Keywords:
LLM, OPD, GRPO, Post train

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GtEZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how improvements from GRPO-trained reasoning teachers transfer to smaller students through on-policy distillation. It proposes TS-OPD, which categorizes problems by teacher–student success rates, applies forward KL to knowledge gaps and reverse KL to precision gaps, and introduces token-level gating. Experiments across three student sizes and eight mathematical reasoning benchmarks show improvements in average correctness, although the effectiveness of the proposed routing mechanism is not fully isolated.

Critical strengths
  1. This paper introduces a useful idea to use joint teacher–student success to distinguish knowledge acquisition from precision improvement. The KL-direction analysis in Eq. 4 provides a reasonable motivation for the proposed routing.
  2. The evaluation covers three student sizes, two teacher checkpoints, and multiple benchmarks. In particular, Table 3 compares routing against uniform KL objectives on the same selected problems, providing evidence that the improvements are not simply caused by filtering.
Critical weaknesses
  1. Table 3 shows that routing outperforms uniform forward or reverse KL, but this does not establish that assigning forward KL specifically to knowledge gaps and reverse KL to precision gaps is necessary. The improvement could arise from combining both objectives rather than matching them to the proposed strata. A swapped-routing control or a fixed forward/reverse KL mixture on the same training problems would distinguish these explanations.
  2. The A/C/P/N labels are determined from only eight samples per model and remain fixed throughout training. Such estimates may misclassify low-success problems, particularly those assigned to the knowledge-gap stratum. Also, Section 5.3 assumes that Direct GRPO receives no reward on A problems, although zero success during calibration does not imply zero reward during training. Moreover, most results use a single training seed, while several improvements over competing objectives are relatively small (Table 2). This makes it difficult to assess whether the reported advantages are robust to sampling and training variability.
Questions for authors
  1. Would TS-OPD still outperform a fixed forward/reverse KL mixture or a reversed A/C routing strategy under the same training budget?
  2. How stable are the problem assignments under different calibration samples or larger values of KcalK_{\mathrm{cal}}? Does this affect the final performance?
  3. Could the authors report multiple training seeds for the main TS-OPD versus routing-only and uniform-KL comparisons, particularly where the improvements are small?
1 comment

Thank you for the suggestions on routing assignment, calibration stability, and multi-seed comparisons. We address each point below.

1. Is matched routing better than simply mixing the two KL objectives?

Matched routing outperforms both random and reversed routing in avg@8 and pass@8, supporting the value of matching the KL direction to each problem's state.

We use the Qwen3-4B student and GRPO-Qwen3-8B teacher. The three routing variants share the same retained training problems and training budget, with token selection and the entropy brake disabled. Random routing preserves the number of problems assigned to each KL objective while shuffling the problem-to-objective assignment; reversed routing swaps the KL directions assigned to A and C.

Results are below. All values are avg@8 and pass@8 macro averages over seven benchmarks from the same single-run ablation, excluding MATH-test (owing to the limited rebuttal time, and because MATH-test scores are already high and have little effect on the conclusions), in percent.

Methodavg@8pass@8
OPD-r47.3764.02
OPD-f47.6466.47
TS-OPD routing-only47.9267.01
TS-OPD full48.8765.36
Random router47.5463.47
Reverse router46.9262.73

2. Few calibration samples, fixed labels, and GRPO rewards on A problems

Most A/C labels remain unchanged with more calibration samples, while a small fraction of A problems move to C; zero successes in eight calibration samples also does not imply that no reward is ever received during GRPO training.

We fix the initial models and decoding settings and compare nested prefixes of the same set of responses at K=8, 16, and 32. Problems that move from A to C remain in training, but their KL assignment changes. We will correct the statement that Direct GRPO receives no reward on A problems.

Details are in the two tables below: the first gives each stratum's share of all training problems, and the second gives per-problem retention rates from K=8 to K=32.

Stratum proportions under different calibration budgets (%):

Student / Calibration KACPN
0.6B / 823.2453.923.5519.29
0.6B / 1622.1554.523.8319.50
0.6B / 3221.2656.274.2718.20
1.7B / 87.3936.137.6848.80
1.7B / 166.7436.188.3548.73
1.7B / 326.4838.479.1545.89

Label transitions from K=8 to K=32:

StudentA retentionC retention
0.6B91.10%99.01%
1.7B86.46%97.56%

3. Multiple training seeds for full, routing-only, and uniform KL

We have expanded the experiments to three training seeds: with the post-GRPO teacher, full achieves the highest mean avg@8 at every student size.

The complete table below covers both teacher checkpoints, all methods, and all three student sizes. Values are mean ± standard deviation. avg@8 and pass@8 are equally weighted macro averages over seven benchmarks, excluding MATH-test (owing to the limited rebuttal time, and because MATH-test scores are already high and have little effect on the conclusions), in percent; transfer gap is in percentage points. Base and GRPO denote the teacher's pre- and post-GRPO checkpoints.

Uniform OPD in the main table uses all training problems, whereas TS uses the retained training pool, so the table compares overall training recipes.

StudentMetricBase-8B OPD-rBase-8B OPD-fBase-8B TS-OPD routing onlyBase-8B TS-OPD fullGRPO-8B OPD-rGRPO-8B OPD-fGRPO-8B TS-OPD routing onlyGRPO-8B TS-OPD fullDirect GRPO
0.6Bavg@8 ↑14.57 ± 0.5914.03 ± 0.8615.80 ± 1.1616.46 ± 0.5222.04 ± 0.6321.17 ± 0.4722.88 ± 0.9223.77 ± 0.5821.07 ± 0.95
pass@8 ↑27.95 ± 0.7628.53 ± 1.6830.54 ± 1.6531.33 ± 1.8139.28 ± 1.4237.98 ± 1.2940.38 ± 1.4940.72 ± 0.7736.42 ± 0.88
Transfer gap ↓14.25 ± 0.5914.79 ± 0.8613.02 ± 1.1612.36 ± 0.526.78 ± 0.637.66 ± 0.475.94 ± 0.925.05 ± 0.587.75 ± 0.95
1.7Bavg@8 ↑23.32 ± 0.6321.24 ± 0.2123.35 ± 0.2523.30 ± 0.5433.14 ± 0.6433.62 ± 0.4233.87 ± 0.3334.72 ± 0.4933.41 ± 0.05
pass@8 ↑40.08 ± 1.3439.33 ± 0.4344.04 ± 1.6140.85 ± 1.0952.44 ± 0.6752.81 ± 0.9455.67 ± 0.8455.56 ± 0.9552.41 ± 0.77
Transfer gap ↓17.00 ± 0.6319.08 ± 0.2116.97 ± 0.2517.02 ± 0.547.18 ± 0.646.70 ± 0.426.45 ± 0.335.60 ± 0.496.91 ± 0.05
4Bavg@8 ↑29.37 ± 0.4029.54 ± 0.4130.07 ± 0.2230.59 ± 0.2047.28 ± 0.3447.62 ± 0.0247.75 ± 0.2249.07 ± 0.2046.32 ± 0.30
pass@8 ↑47.29 ± 1.7348.83 ± 1.4147.20 ± 1.5747.15 ± 0.8465.74 ± 1.5867.45 ± 0.9268.41 ± 1.6166.35 ± 0.9763.36 ± 0.98
Reviewer_xJ9XOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a “transfer gap”, that is, why a GRPO-improved teacher’s gains only partly reach a smaller student under on-policy distillation (OPD). It proposes TS-OPD, a simple method that labels each training problem by the sampled success of the teacher and the raw student. It then applies gated forward KL to problems where the student never succeeds and gated reverse KL with an entropy brake to problems where the student sometimes succeeds, and skips the rest. Experiments show improvements on eight maths benchmarks.

Critical strengths

The proposed question is interesting, and the core idea of choosing the KL direction per problem from joint teacher–student success is simple and easy to implement. The experiments are comprehensive, including a component-by-component ablation and detailed per-problem outcome analyses.

Critical weaknesses
  1. I am very sceptical about the conceptual basis of the paper. For instance, based on the math and explanation from the paper, the transfer gap adds nothing beyond accuracy: for a fixed teacher pair and student, ΔT\Delta T and a(S0)a(S_{0}) are constants, so GtransferG_{transfer} is simply a constant minus student accuracy. “Smallest transfer gap” therefore restates “highest accuracy”, yet presented as a separate finding. Further explanation would be helpful.

  2. The figures are very small, especially the main figure. The text should be larger, given the amount of white space available.

  3. According to the paper, forward KL uses the dense teacher top-512 distribution, while reverse KL uses a single sampled token. Routing effects may therefore reflect estimator variance rather than KL direction; that is, KL direction is confounded with the estimator.

  4. The study uses one teacher size, one model family, one training set and thinking mode disabled, with a single epoch at a learning rate of 1e-6. With the original Qwen3-8B teacher, every OPD variant leaves the 4B student below its starting accuracy, which raises questions about the OPD setup. Moreover, evaluating only Qwen models is not sufficient to demonstrate the generality of the method [1].

  5. AIME and HMMT each contain only 30 problems, yet the authors report results from a single training seed. Since the results are very close and the dataset is very small, the authors should report standard deviations.

[1] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arXiv preprint arXiv:2506.10947 (2025).

Questions for authors

See above.

3 comments

Thank you for the comments on the concept definition, KL implementation, and experimental reliability. We address them point by point below.

1. Transfer gap duplicates accuracy

We agree: with the teacher pair, student initialization, and evaluation set fixed, transfer gap provides no information beyond accuracy. It describes an observed phenomenon and motivates the research question; it is not a new evaluation metric that we propose.

We observed a gap between the teacher's improvement from GRPO and the student's distillation gain, and we aim to study how to close this gap and whether different student states call for different distillation objectives. We will clarify the role of the concept and remove wording that presents the smallest transfer gap as a separate performance advantage.

2. Figures and text are too small

We will enlarge the figures and text and explicitly label Figure 1 as a conceptual illustration.

3. KL direction is confounded with the estimator

With sampled forward KL, both TS variants still outperform uniform forward KL in avg@8.

We agree that the original implementation changes both the KL direction and the estimator: forward KL uses the teacher's top-512-plus-tail distribution, whereas reverse KL uses student-sampled tokens. To examine the dependence on dense forward KL, we replace the forward-KL branch with a sampled-token implementation for the Qwen3-4B student and GRPO-Qwen3-8B teacher, keeping all other components of each method unchanged. We will specify the sampling distribution and the gradient estimator.

Results are below. All values are @8 macro averages over seven benchmarks from the same ablation, excluding MATH-test (owing to the limited rebuttal time, and because MATH-test scores are already high and have little effect on the conclusions). Scores are percentages; differences are percentage points.

MethodDense avg@8Sampled avg@8Δ avgDense pass@8Sampled pass@8Δ pass
OPD-f47.6447.32-0.3266.4767.17+0.70
TS-OPD routing only47.9248.22+0.3067.0167.60+0.59
TS-OPD full48.8748.50-0.3765.3665.96+0.60

4. 4B degradation with the base teacher and limited model-family scope

We confirm the 4B degradation with the base teacher; the added Gemma comparison extends the model-family coverage.

In the original results, the initial Qwen3-4B student and the base teacher have similar aggregate scores, indicating that the base teacher offers limited additional advantage. With the post-GRPO Gemma teacher, both TS variants achieve higher mean accuracy and coverage than all other listed student baselines.

Results are in the two tables below. The first retains the macro avg@K under the original paper's sampling budgets; the second summarizes Gemma results at K=8 over seven benchmarks, with trained methods reported as mean ± standard deviation. The first three rows are reference checkpoints, and Direct GRPO requires no teacher.

Original degradation with the base teacher (%):

Model / MethodOriginal macro avg@K
Initial Qwen3-4B student37.22
Pre-GRPO Qwen3-8B teacher38.03
Base OPD-r36.07
Base OPD-f36.12
Base TS routing-only36.53
Base TS full36.79

Added Gemma model-family comparison (%):

Model / MethodTeacheravg@8pass@8
Original Gemma 2B—16.9128.54
Original Gemma 9B—28.2643.18
GRPO Gemma 9B—37.9453.73
Direct GRPO—23.18 ± 0.8436.54 ± 1.56
OPD-rGRPO Gemma 9B22.83 ± 0.3736.47 ± 1.28
OPD-fGRPO Gemma 9B23.50 ± 0.4936.69 ± 1.46
TS-OPD routing onlyGRPO Gemma 9B23.61 ± 0.6338.01 ± 0.88
TS-OPD fullGRPO Gemma 9B23.58 ± 0.8138.04 ± 0.48

5. Single seed and small evaluation sets

We have expanded the experiments to three training seeds. With the post-GRPO teacher, full achieves the highest mean avg@8 at all three student sizes, while the best pass@8 method varies with student size.

The complete table below covers both teacher checkpoints, all methods, and all three student sizes. Values are mean ± standard deviation. avg@8 and pass@8 are equally weighted macro averages over seven benchmarks, excluding MATH-test (owing to the limited rebuttal time, and because MATH-test scores are already high and have little effect on the conclusions), in percent; transfer gap is in percentage points. Base and GRPO denote the pre- and post-GRPO checkpoints of the same teacher. Bold and italic entries mark the best and second-best means in each row, respectively.

StudentMetricBase-8B OPD-rBase-8B OPD-fBase-8B TS-OPD routing onlyBase-8B TS-OPD fullGRPO-8B OPD-rGRPO-8B OPD-fGRPO-8B TS-OPD routing onlyGRPO-8B TS-OPD fullDirect GRPO
0.6Bavg@8 ↑14.57 ± 0.5914.03 ± 0.8615.80 ± 1.1616.46 ± 0.5222.04 ± 0.6321.17 ± 0.4722.88 ± 0.9223.77 ± 0.5821.07 ± 0.95
pass@8 ↑27.95 ± 0.7628.53 ± 1.6830.54 ± 1.6531.33 ± 1.8139.28 ± 1.4237.98 ± 1.2940.38 ± 1.4940.72 ± 0.7736.42 ± 0.88
Transfer gap ↓14.25 ± 0.5914.79 ± 0.8613.02 ± 1.1612.36 ± 0.526.78 ± 0.637.66 ± 0.475.94 ± 0.925.05 ± 0.587.75 ± 0.95
1.7Bavg@8 ↑23.32 ± 0.6321.24 ± 0.2123.35 ± 0.2523.30 ± 0.5433.14 ± 0.6433.62 ± 0.4233.87 ± 0.3334.72 ± 0.4933.41 ± 0.05
pass@8 ↑40.08 ± 1.3439.33 ± 0.4344.04 ± 1.6140.85 ± 1.0952.44 ± 0.6752.81 ± 0.9455.67 ± 0.8455.56 ± 0.9552.41 ± 0.77
Transfer gap ↓17.00 ± 0.6319.08 ± 0.2116.97 ± 0.2517.02 ± 0.547.18 ± 0.646.70 ± 0.426.45 ± 0.335.60 ± 0.496.91 ± 0.05
4Bavg@8 ↑29.37 ± 0.4029.54 ± 0.4130.07 ± 0.2230.59 ± 0.2047.28 ± 0.3447.62 ± 0.0247.75 ± 0.2249.07 ± 0.2046.32 ± 0.30
pass@8 ↑47.29 ± 1.7348.83 ± 1.4147.20 ± 1.5747.15 ± 0.8465.74 ± 1.5867.45 ± 0.9268.41 ± 1.6166.35 ± 0.9763.36 ± 0.98

Thank you for the additional experiments, and for the effort that went into running them. Having gone through the new results, I’m still not fully convinced that the performance gains are strong enough. In several cases, the improvements fall within the standard deviation, and in some cases the method performs slightly below the baselines.

I appreciate the work put into addressing my concerns, and in recognition of that, I have raised my score to 2. That said, I still feel the contribution isn’t yet strong enough for acceptance, so I continue to lean towards rejection.

Reviewer_JcrvOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper aims to study how reasoning improvements from RL -- resulting in a specialist teacher model -- can then transfer to student models via on-policy distillation. The paper studies a potential gap, wherein gains from RL do not fully transfer via OPD, and proposes TS-OPD, which applies different OPD objectives to different problems based on teacher/student performance.

Critical strengths
  • The research question -- understanding if/when gains from RL transfer when then doing OPD -- is interesting and timely.
  • The proposed training recipe seems to outperform OPD baselines.
Critical weaknesses

The paper has several methodological and measurement flaws, and some of the conclusions do not seem to be supported by the evidence

  • Most training runs use a single seed and do not quantify error.
  • The "transfer gap" metric does not measure transfer of gains from RL. It is defined as the difference between teacher improvement over the initial teacher from RL and student improvement over the initial student. But this does not isolate the transfer of abilities acquired during RL. The student OPD improvement could be from knowledge acquired from the base teacher. Thus the paper does not isolate the quantity (transfer from RL) that it claims to study.
  • Why is the number of sampled responses K different across benchmarks? This is implicitly measuring a different task for each benchmark and makes the use of macro-average pass@k across benchmarks quite flawed (this is one of the main metrics in the paper). With different sample budgets per benchmark, the pass@k averaged over different k for each benchmark is an unequal mixture; this is not a justified metric to report.

In addition, several aspects of the paper's writing/presentation make it hard to understand. Some examples:

  • The intro points to Figure 1 as having problem-level analysis on MATH, but it reads as more of a teaser figure (no dataset-specific analysis or quantitative results).
  • The A/C/P/N labels are quite confusing and unintuitive; if shorter labels for the metrics are required then why not label them based on the actual metric names? The labels are also overloaded: they refer to the teacher v. student in section 3 and then the accuracy before/after training in Section 5. This makes it very difficult to parse and interpret the results.
  • "Two teacher lineages" is used to refer to a single model before and after RL; this is confusing as it implies different model families.

The conclusions of this paper are unsupported by the evidence as presented.

Questions for authors
  1. Can the authors provide multi-seed results that quantify error?
  2. Can the authors provide evidence that isolates the transfer from RL specifically, since this is the stated question of the paper?
  3. Why is the number of sampled responses different across benchmarks? Can the authors provide equal-budget metrics for pass@k if they are averaging across benchmarks?
  4. Can the authors explain why full TS-OPD appears to be inconsistent in whether it improves pass@k over routing-only?
3 comments

Thank you for pointing out issues in metric interpretation, attribution of teacher gains, and the evaluation setup. We present the comparisons directly below, together with the planned revisions.

1. Multiple seeds and uncertainty quantification

We have expanded the experiments to three training seeds and report standard deviations. With the post-GRPO teacher, full achieves the highest mean avg@8 at every student size, while the coverage advantage varies with student size.

The table below lists both teacher checkpoints, all methods, and all three student sizes in full, rather than only a subset of favorable comparisons. Values are mean ± standard deviation. avg@8 and pass@8 are equally weighted macro averages over seven benchmarks, excluding MATH-test (owing to the limited rebuttal time, and because MATH-test scores are already high and have little effect on the conclusions), in percent; transfer gap is in percentage points. Base and GRPO denote the teacher's pre- and post-GRPO checkpoints.

StudentMetricBase-8B OPD-rBase-8B OPD-fBase-8B TS-OPD routing onlyBase-8B TS-OPD fullGRPO-8B OPD-rGRPO-8B OPD-fGRPO-8B TS-OPD routing onlyGRPO-8B TS-OPD fullDirect GRPO
0.6Bavg@8 ↑14.57 ± 0.5914.03 ± 0.8615.80 ± 1.1616.46 ± 0.5222.04 ± 0.6321.17 ± 0.4722.88 ± 0.9223.77 ± 0.5821.07 ± 0.95
pass@8 ↑27.95 ± 0.7628.53 ± 1.6830.54 ± 1.6531.33 ± 1.8139.28 ± 1.4237.98 ± 1.2940.38 ± 1.4940.72 ± 0.7736.42 ± 0.88
Transfer gap ↓14.25 ± 0.5914.79 ± 0.8613.02 ± 1.1612.36 ± 0.526.78 ± 0.637.66 ± 0.475.94 ± 0.925.05 ± 0.587.75 ± 0.95
1.7Bavg@8 ↑23.32 ± 0.6321.24 ± 0.2123.35 ± 0.2523.30 ± 0.5433.14 ± 0.6433.62 ± 0.4233.87 ± 0.3334.72 ± 0.4933.41 ± 0.05
pass@8 ↑40.08 ± 1.3439.33 ± 0.4344.04 ± 1.6140.85 ± 1.0952.44 ± 0.6752.81 ± 0.9455.67 ± 0.8455.56 ± 0.9552.41 ± 0.77
Transfer gap ↓17.00 ± 0.6319.08 ± 0.2116.97 ± 0.2517.02 ± 0.547.18 ± 0.646.70 ± 0.426.45 ± 0.335.60 ± 0.496.91 ± 0.05
4Bavg@8 ↑29.37 ± 0.4029.54 ± 0.4130.07 ± 0.2230.59 ± 0.2047.28 ± 0.3447.62 ± 0.0247.75 ± 0.2249.07 ± 0.2046.32 ± 0.30
pass@8 ↑47.29 ± 1.7348.83 ± 1.4147.20 ± 1.5747.15 ± 0.8465.74 ± 1.5867.45 ± 0.9268.41 ± 1.6166.35 ± 0.9763.36 ± 0.98

2. How much of the student's improvement comes from the post-GRPO teacher

We answer this from two angles: first, how much the post-GRPO teacher improves the student's OPD results compared with the pre-GRPO teacher; second, how much of the teacher's GRPO improvement on specific problems is passed on to the student through distillation.

The first table holds the student initialization, training data, and uniform OPD recipe fixed, and changes only the teacher checkpoint. The additional benefit is the difference between the avg@8 scores of the two distilled students, so the gain that the same recipe already obtains from the base teacher is subtracted. This measures the student benefit brought by the change of teacher checkpoint. Values are mean ± standard deviation of the seven-benchmark avg@8; differences are percentage points.

The second table comes from the original paper and uses its original sampling budgets. On the same test problems where GRPO improves the teacher, it reports the fraction on which the student also improves over its initial model, describing the overlap of improved problems. Each percentage is the sum of the rounded proportions of the original "newly solved" and "more reliable" categories. OPD-r/f use all training problems, while Direct GRPO (A∪C) and TS use the same retained training pool. Details are in the two tables below.

Fixed distillation recipe, different teacher checkpoints:

Student / Fixed objectivePre-GRPO teacherPost-GRPO teacherAdditional benefit (pp)
0.6B / OPD-r14.57 ± 0.5922.04 ± 0.63+7.47
0.6B / OPD-f14.03 ± 0.8621.17 ± 0.47+7.14
1.7B / OPD-r23.32 ± 0.6333.14 ± 0.64+9.82
1.7B / OPD-f21.24 ± 0.2133.62 ± 0.42+12.38
4B / OPD-r29.37 ± 0.4047.28 ± 0.34+17.91
4B / OPD-f29.54 ± 0.4147.62 ± 0.02+18.08

Per-problem improvement overlap in the original paper:

StudentTeacher-improved problemsOPD-r also improvesOPD-f also improvesTS full also improvesDirect GRPO (A∪C) also improves
0.6B56741.2%49.4%50.1%37.1%
1.7B56760.6%58.0%62.1%59.5%
4B56766.3%65.1%68.4%58.1%

3. Averaging pass@K across different sampling budgets

We have unified the revised main table to avg@8 and pass@8 on seven benchmarks.

The seven benchmarks are MATH500, Minerva, OlympiadBench, AIME24, AIME25, HMMT-Feb25, and HMMT-Nov25. Each problem is sampled 8 times, and avg@8 and pass@8 are macro-averaged with equal weights (1/7 each). MATH-test is excluded from this main table (owing to the limited rebuttal time, and because MATH-test scores are already high and have little effect on the conclusions). The revised results are shown in the main table under point 1 above.

4. Figure 1 and presentation

Figure 1 is a conceptual illustration. We will correct the wording in the introduction that describes it as a quantitative per-problem analysis.

5. Overloaded A/C/P/N labels

We will no longer use A/C/P/N for both training strata and evaluation outcomes, and will use descriptive names instead.

Training strata compare the teacher with the initial student; evaluation outcomes compare the same student's success rates before and after training. The names are listed below.

Original labelTraining stratumEvaluation outcome
AZero student successNewly solved
CPartial student successMore reliable
PStudent aheadRegressed
NTiedUnchanged

6. "Two teacher lineages" is misleading

We will remove "two teacher lineages," which could suggest different model families, and use "pre- and post-GRPO teacher" instead.

7. Mixed pass@k changes from routing-only to full

Full improves avg@8 for all three Qwen students but does not necessarily improve pass@8 at the same time; on Gemma, the two variants have similar pass@8, indicating that the change in coverage depends on the setting.

We hypothesize that full makes the student answer already-covered problems correctly more often, while some low-success problems may lose coverage. When coverage losses exceed newly covered problems, avg@8 rises while pass@8 falls. We will test this explanation through additional repeated runs.

Results are in the two tables below, both using post-GRPO teachers. Values are mean ± standard deviation; differences are percentage points.

Full versus routing-only for Qwen students:

StudentRouting avg@8Full avg@8Δ avgRouting pass@8Full pass@8Δ pass
0.6B22.88 ± 0.9223.77 ± 0.58+0.8940.38 ± 1.4940.72 ± 0.77+0.34
1.7B33.87 ± 0.3334.72 ± 0.49+0.8555.67 ± 0.8455.56 ± 0.95-0.11
4B47.75 ± 0.2249.07 ± 0.20+1.3268.41 ± 1.6166.35 ± 0.97-2.06

Corresponding results for the Gemma 2B student:

Gemma 2B methodavg@8pass@8
TS-OPD routing only23.61 ± 0.6338.01 ± 0.88
TS-OPD full23.58 ± 0.8138.04 ± 0.48

Thank you to the authors for their response and for the effort that they put into the new experiments. I appreciate the addition of the pre/post-GRPO teacher, and will raise my score to a 2 accordingly. However, after reading through the new results and the other reviews, I concur with reviewer xJ9X that it is unclear if the proposed method consistently improves over the baseline, given that the gaps fall within the noise range. As a result I still recommend rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.