- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds on on-policy self-distillation (OPSD), where standard OPSD uses one fixed teacher. The authors argue that small random Gaussian perturbations of the teacher's weights produce experts that raise the probability of different reference tokens, so a pool of them covers more reference positions than the unperturbed teacher. The proposed N-OPSD builds a pool of experts offline from random perturbations, using greedy selection that credits each expert only for gains in reference-token probability beyond the current pool, after filtering out high-confidence positions and sharp peaks. During training, at each student position it takes the token with the highest probability under any expert as the anchor, then picks among experts that agree on that token, and distills that expert's full distribution with OPSD's clipped forward KL. On Qwen3-1.7B, 4B and 8B trained on OpenThoughts, N-OPSD improves over OPSD, and ranks first in all model-benchmark cells.
Critical strengths
The idea is new for OPD and simple to implement. Looking for extra supervision in the teacher's own weight neighborhood, rather than in more teachers or more context, needs no extra models or data, and the cost stays in training. The gains are consistent: N-OPSD beats OPSD, EOPD and PW-OPSD at all three scales, the three main methods are rerun under the same data, budget and step checkpoint with three runs each, and the gains over OPSD are several times the run-to-run standard deviation.
The ablations are broad and mostly well matched. Selection rules are compared on the same candidates and router, and routing rules on the same pool. Two results are informative beyond this method: a random pool with the full router does worse than the fixed teacher, and random routing on the selected pool only matches it, so neither selection nor routing alone explains the gain. The paper also reports training cost honestly and shows that higher reference-side coverage at larger radii does not translate into better students.
Critical weaknesses
Some hyperparameters appear to be chosen on the test benchmarks. Expert selection uses a split of the training data disjoint from AIME and HMMT, and the 8B radius is calibrated on 300 training-distribution prefixes, which is good practice. For the pool size, routing quantile and gate threshold, however, the paper describes no held-out downstream selection, and each default matches the best test score in its sweep: 25 experts, quantile 0.75 and gate threshold 0.99.
The evaluation is also narrow: three math benchmarks of 30 problems each and one model family, so a two-point average gain corresponds to about two problems, and transfer beyond math is untested.
The rerun baselines use their default settings, so the finetuning budget is not matched, and the procedure behind the 1.7B and 4B radii is described only for 8B. Separately, all ablations except the random pool are single runs, while the full method's run-to-run standard deviation is 0.46 and up to 0.89 on one benchmark.
The evidence for "complementary corrections" needs controls. Coverage counts a position if at least one expert raises the reference-token probability, so a pool of 25 noisy copies covers more positions than one teacher almost by construction, even if the perturbations carry no useful structure. A fair comparison is coverage from 25 random perturbations, or from the teacher sampled at a matched temperature or with dropout. The newly covered tokens are mostly "the", "we", backslashes and numerals (Table 16), which reads more like lexical noise than reasoning corrections.
On the training side, routing toward the highest-peak token and an upper quantile of support likely gives a sharper target than the fixed teacher, so part of the gain may come from target sharpness rather than new information. A control that sharpens the fixed teacher's distribution, for example by temperature, to match the routed target's peak would separate that effect.
The paper also does not test whether weight-space neighbors are needed at all. Because the teacher only scores student prefixes, its target can be diversified more cheaply, for example by varying the privileged context (alternative reference solutions or prompt formats), and no such pool is compared. Eq. 3 adds isotropic noise with one absolute radius to the full parameter vector, without saying whether embeddings, normalization gains or the output head are included, since weight scales differ widely across tensor types, the same radius perturbs some tensors far more than others in relative terms.
Questions for authors
Are all parameters perturbed, including embeddings, normalization gains and the output head? Did you try scaling the noise to each tensor's magnitude, or perturbing only some layers, and how sensitive are the results to this choice?
How were the pool size, quantile and gate threshold chosen? If on AIME and HMMT, can you report results with these settings chosen on a held-out split of the training distribution (as was done for the radius), and give the baselines a comparable search?
How does selection-feasible coverage of the PTA pool compare with 25 random perturbations at the same radius, or 25 samples of the fixed teacher with dropout or temperature? This would show whether the coverage gain reflects selection or multiplicity.
Can you compare against a fixed teacher whose distribution is sharpened (for example by temperature) to match the peak probability of the routed target? If this closes much of the gap, the gain is mainly a target-sharpening effect.
Can you report the key ablations (selection rule, quantile, pool size) over three runs, given that most gaps are below one point?
Related prior work on distilling from stochastic copies of one teacher, such as dropout-generated teachers filtered by the student (Aslam et al., ECML/PKDD 2025, "Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation"), should be discussed.
6 comments
Response to Reviewer Zt6z (Part 2/4)
Q3.
How does selection-feasible coverage of the PTA pool compare with 25 random perturbations at the same radius, or 25 samples of the fixed teacher with dropout or temperature? This would show whether the coverage gain reflects selection or multiplicity.
A3. We tested the random-perturbation control in your question. At the same size and radius, the PTA-selected pool has higher selection-feasible coverage than all three random pools, exceeding their mean by 1.55 percentage points. This identifies a coverage benefit beyond pool size.
All perturbation pools use K = 25 and σ = 0.002, with the same analysis data and filtering. The denominator is 118,917 reference positions retained by SCGate. A position is counted once if any teacher in the pool provides positive filtered PTA.
| Teacher pool | Selection-feasible coverage (%) |
|---|---|
| Original teacher M₀ | 15.69 |
| Random pools (mean ± SD) | 24.98 ± 0.07 |
| PTA-selected pool, K = 25 | 26.53 |
Eq. 8 selects experts for the filtered PTA they add beyond the current pool. With the same router, this selection raises average student accuracy from 64.29 for Random Top-K to 66.57 for N-OPSD (original Tables 4 and 1). These student results support the value of selection for problem solving.
Revised paper: Table 23 and Appendix E.2 report random-pool coverage; §3.3 separates the benefit of selection from pool size.
Q4.
Can you compare against a fixed teacher whose distribution is sharpened (for example by temperature) to match the peak probability of the routed target? If this closes much of the gap, the gain is mainly a target-sharpening effect.
A4. Thank you for pointing out target sharpening as an important alternative explanation for the gains. Your proposed control helps separate this effect from the benefit of the selected experts. We lowered the original teacher's temperature to match the routed target's average peak probability. N-OPSD still has higher mean accuracy on all three benchmarks, so this fixed-temperature control does not close the gap. The differences are 1.01, 1.76, and 2.59 percentage points.
| Benchmark | OPSD | OPSD + SCGate | OPSD + SCGate, T = 0.8176330566 | N-OPSD |
|---|---|---|---|---|
| AIME 2024 | 77.41 ± 0.32 | 77.87 ± 0.70 | 77.69 ± 0.85 | 78.70 ± 0.69 |
| AIME 2025 | 71.02 ± 0.32 | 71.11 ± 0.48 | 71.02 ± 0.89 | 72.78 ± 0.28 |
| HMMT February 2025 | 45.46 ± 0.42 | 45.74 ± 0.42 | 45.65 ± 0.98 | 48.24 ± 0.89 |
We prepared 500 independent OpenThoughts questions for temperature calibration, with 400 for fitting T and 100 for checking the match. Exact and near-duplicate filtering excludes training, expert-selection, analysis, and benchmark questions.
Both teachers use the same fixed prefixes generated by the initial student and the same positions retained by SCGate. We first obtain N-OPSD's teacher distributions using the complete q = 0.75 router used in training. We take the maximum probability over the full vocabulary at each position and average across positions. We then adjust the original teacher's temperature to match this average, obtaining T = 0.8176330566.
The routed and sharpened peaks are 76.4869% and 76.4963% on the fit split, and 76.6543% and 76.5223% on the check split. T is chosen using only the fit split and held fixed throughout all three training runs. Online SCGate, clipped forward KL, and student sampling and evaluation temperatures are unchanged.
Revised paper: Table 4 and §3.4 report the sharpening control; Appendix B.7 gives the independent calibration protocol.
Response to Reviewer Zt6z (Part 3/4)
Q5.
Can you report the key ablations (selection rule, quantile, pool size) over three runs, given that most gaps are below one point?
A5. We completed three-run ablations of selection, quantile, and pool size. PTA selection retains higher mean accuracy than answer-accuracy selection. At K = 25, q = 0.75 gives higher means than q = 1. At q = 0.75, K = 25 gives higher means than K = 50. Some nearby settings remain close relative to run variability, as shown by the means ± sample SD below.
Selection rule
Independent PTA Top-K gives higher means than Quality Top-K on all three benchmarks. The PTA selector improves them further by accounting for gains already supplied by the current pool. This supports selecting experts for their additional contribution. Its AIME 2024 gap from Greedy sample coverage remains small relative to run variability.
| Benchmark | Random Top-K | Quality Top-K | Greedy sample coverage | Independent PTA Top-K | PTA selector (Ours) |
|---|---|---|---|---|---|
| AIME 2024 | 77.04 ± 0.85 | 76.94 ± 0.73 | 78.43 ± 0.89 | 77.59 ± 0.85 | 78.70 ± 0.69 |
| AIME 2025 | 70.28 ± 0.48 | 70.09 ± 1.05 | 72.31 ± 0.89 | 71.57 ± 0.58 | 72.78 ± 0.28 |
| HMMT February 2025 | 45.56 ± 0.73 | 45.83 ± 0.83 | 46.85 ± 1.12 | 47.04 ± 0.70 | 48.24 ± 0.89 |
Quantile, K = 25
With the pool and anchor rule fixed, q = 0.75 gives higher means than q = 1 on all three benchmarks. This favors quantile selection over always choosing the expert with the strongest support for the anchor token. The q = 0.50 and q = 0.75 results are close on both AIME benchmarks.
| Benchmark | q = 0 | q = 0.25 | q = 0.50 | q = 0.75 | q = 1 |
|---|---|---|---|---|---|
| AIME 2024 | 76.94 ± 1.00 | 77.04 ± 0.58 | 78.61 ± 0.73 | 78.70 ± 0.69 | 76.76 ± 1.31 |
| AIME 2025 | 70.19 ± 0.89 | 71.76 ± 0.89 | 72.69 ± 0.58 | 72.78 ± 0.28 | 71.85 ± 1.12 |
| HMMT February 2025 | 46.02 ± 0.98 | 46.67 ± 1.00 | 47.22 ± 0.73 | 48.24 ± 0.89 | 46.94 ± 1.11 |
Pool size, q = 0.75
At q = 0.75, means improve from K = 1 to K = 25 and fall at K = 50. K = 25 gives the highest means among these tested pool sizes.
| Benchmark | K = 1 | K = 5 | K = 10 | K = 25 | K = 50 |
|---|---|---|---|---|---|
| AIME 2024 | 76.20 ± 1.05 | 77.22 ± 0.73 | 78.06 ± 0.56 | 78.70 ± 0.69 | 78.33 ± 0.73 |
| AIME 2025 | 69.63 ± 0.89 | 71.39 ± 0.73 | 71.85 ± 0.58 | 72.78 ± 0.28 | 70.56 ± 1.00 |
| HMMT February 2025 | 45.56 ± 0.83 | 47.04 ± 1.12 | 47.22 ± 0.56 | 48.24 ± 0.89 | 44.35 ± 0.98 |
Joint settings
At K = 50, lowering q from 0.75 to 0.50 raises HMMT performance but reduces AIME 2025 performance. Pool size and routing strength therefore interact, with different effects across benchmarks.
| Benchmark | K = 50, q = 0.75 | K = 50, q = 0.50 | K = 50, q = 0.25 |
|---|---|---|---|
| AIME 2024 | 78.33 ± 0.73 | 79.54 ± 0.70 | 76.67 ± 1.21 |
| AIME 2025 | 70.56 ± 1.00 | 70.00 ± 0.56 | 72.59 ± 0.70 |
| HMMT February 2025 | 44.35 ± 0.98 | 48.43 ± 1.05 | 45.65 ± 0.85 |
For configurations originally tested once, we retained that run and added two. Default N-OPSD and Random Top-K three-run summaries are reused from the submitted paper.
Revised paper: §§3.3–3.6 explain the updated findings. Tables 9, 11, and 17 give three-run summaries; Appendix G gives individual runs. Original single-run comparisons are retained separately.
Q6.
Related prior work on distilling from stochastic copies of one teacher, such as dropout-generated teachers filtered by the student (Aslam et al., ECML/PKDD 2025, "Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation"), should be discussed.
A6. We added Aslam et al. and clarified the shared idea: useful teacher diversity can come from one model. Their method generates representations with dropout, then uses the student to filter and weight them for feature distillation. Aslam et al., ECML/PKDD 2025.
N-OPSD studies this idea in OPSD with shared privileged context. It selects local parameter perturbations by marginal filtered PTA and routes one expert's full next-token distribution at each student state.
Revised paper: §4.3 cites Aslam et al. and explains the shared idea and the distinction in OPSD.
Response to Reviewer Zt6z (Part 1/4)
Thank you for the detailed questions and controls.
Math results are Qwen3-8B Average@12 (%), means ± sample SD over three runs. Each benchmark has 30 questions and 12 samples per question. Table numbers refer to the revised paper unless labeled original.
Q1.
A1. The default perturbs all parameters, including embeddings, normalization gains, and the output head. Across three runs, scaled-noise means stay within 0.19 percentage points of the default. Perturbing only the last 12 blocks lowers the means by 0.65–0.83 points, but they remain above OPSD. One-dimensional parameters and Q/K normalization are included in the default.
Scaled noise applies W′=W+α∥W∥FZ/∥Z∥F to each parameter tensor separately, with Gaussian Z and α≈0.04376. All parameters participate, including embeddings, normalization gains, and the output head. Each normalization vector is scaled separately.
Partial-layer noise applies W′=W+σZ, with σ≈0.00532, only to zero-indexed blocks 24–35, including their attention, MLP, and normalization parameters. Earlier blocks, embeddings, final normalization, and the output head stay unchanged.
We calibrate the strengths to make the changes in teacher predictions comparable to those from the default σ = 0.002. Both controls use the same independent 400-fit/100-check split and eight strength probes. For each strength, we average DKL(qperturbed∥qM0) over eight pilot experts and retained positions, then choose the fit KL closest to the default. The match is approximate: 0.0539 for the default, 0.0471 for scaled noise, and 0.0596 for partial-layer noise. Strength selection uses calibration data only; Appendix B.10 lists the exact strengths and probes.
Each variant uses the same 500 candidate seeds plus M₀ and reselects K = 25 by filtered PTA on the same 1,000 questions. Routing, loss, training data, batch size, and the step-100 rule are shared. Each selected pool is fixed across three training runs. Default results are reused from the submitted paper.
This comparison evaluates the last 12 blocks at their own calibrated strength against the default full-parameter setting. It changes both parameter scope and noise strength.
Revised paper: §2.1 specifies parameter scope; §3.5 summarizes the control results. Table 8 and Appendix B.10 give the results, formulas, strength probes, and matching error.
Q2.
A2. We appreciate your close attention to parameter selection. The original manuscript omitted the validation procedure. K, q, and τ were chosen on held-out OpenThoughts data, without using AIME or HMMT scores. We have now added this procedure. Baselines use their published parameters, without an additional search.
We select parameters by Average@12 at the fixed step-100 checkpoint, using 1,000 distinct held-out math questions with checkable answers from the training version of OpenThoughts. Question-level exact and near-duplicate filtering excludes training, expert-selection, analysis, temperature/perturbation-calibration, and test questions. Students answer from the problem alone.
The tested values are K ∈ {1, 5, 10, 25, 50}, q ∈ {0, 0.25, 0.50, 0.75, 1}, and τ ∈ {0.95, 0.98, 0.99, 0.995}. We tested these values without evaluating every combination. The selected defaults are K = 25, q = 0.75, and τ = 0.99.
Validation also selects the radius, searching from 8B to 4B to 1.7B. Each smaller model starts from the preceding selected radius:
Baselines share initialization, training data, optimizer-step count, the fixed step-100 checkpoint, and evaluation. Hyperparameter search budgets differ because baselines retain their published settings.
Revised paper: §3.1 and Appendix B.8 now state the validation split, actual search values, fixed checkpoint rule, and published-parameter baseline scope.