acceptodds
Under review as a conference paper at ICLR 2027

Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) trains mathematical reasoning models using a privileged teacher that sees a reference solution and supervises student-sampled prefixes. Standard OPSD uses one fixed parameter setting at every state, but nearby settings may offer additional supervision. We find that local parameter perturbations reveal complementary reference-aligned corrections under the same reference context. Different experts supply these corrections at different reference positions. Their pool covers more such positions than the unperturbed privileged teacher. We introduce Neighborhood OPSD (N-OPSD) to turn these corrections into supervision at student-visited states. Offline, greedy selection builds a compact pool of frozen experts by rewarding filtered reference-token gains beyond the pool's current best at each position. The highest-peak expert need not provide the best training target. Online routing therefore separates the anchor direction from its level of support. MaxPeak selects the anchor token, and quantile selection chooses among experts whose top token matches it. The student learns from the chosen expert's full next-token distribution through the clipped forward-KL objective inherited from OPSD. We evaluate on AIME 2024, AIME 2025, and HMMT February 2025. Across three independent runs per method, Neighborhood OPSD improves the three-benchmark Average@12 over OPSD by 2.75, 1.67, and 1.94 points on Qwen3-1.7B, 4B, and 8B, respectively. Student-prefix continuations support using the pool beyond the reference trajectories used for selection. Matched ablations support filtered reference-token gains as a selection criterion. Accounting for overlap within the pool and routing by state further improve student accuracy. Inference uses only the distilled student.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, On-policy distillation, On-policy self-distillation, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Zt6zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on on-policy self-distillation (OPSD), where standard OPSD uses one fixed teacher. The authors argue that small random Gaussian perturbations of the teacher's weights produce experts that raise the probability of different reference tokens, so a pool of them covers more reference positions than the unperturbed teacher. The proposed N-OPSD builds a pool of experts offline from random perturbations, using greedy selection that credits each expert only for gains in reference-token probability beyond the current pool, after filtering out high-confidence positions and sharp peaks. During training, at each student position it takes the token with the highest probability under any expert as the anchor, then picks among experts that agree on that token, and distills that expert's full distribution with OPSD's clipped forward KL. On Qwen3-1.7B, 4B and 8B trained on OpenThoughts, N-OPSD improves over OPSD, and ranks first in all model-benchmark cells.

Critical strengths

The idea is new for OPD and simple to implement. Looking for extra supervision in the teacher's own weight neighborhood, rather than in more teachers or more context, needs no extra models or data, and the cost stays in training. The gains are consistent: N-OPSD beats OPSD, EOPD and PW-OPSD at all three scales, the three main methods are rerun under the same data, budget and step checkpoint with three runs each, and the gains over OPSD are several times the run-to-run standard deviation.

The ablations are broad and mostly well matched. Selection rules are compared on the same candidates and router, and routing rules on the same pool. Two results are informative beyond this method: a random pool with the full router does worse than the fixed teacher, and random routing on the selected pool only matches it, so neither selection nor routing alone explains the gain. The paper also reports training cost honestly and shows that higher reference-side coverage at larger radii does not translate into better students.

Critical weaknesses

Some hyperparameters appear to be chosen on the test benchmarks. Expert selection uses a split of the training data disjoint from AIME and HMMT, and the 8B radius is calibrated on 300 training-distribution prefixes, which is good practice. For the pool size, routing quantile and gate threshold, however, the paper describes no held-out downstream selection, and each default matches the best test score in its sweep: 25 experts, quantile 0.75 and gate threshold 0.99.

The evaluation is also narrow: three math benchmarks of 30 problems each and one model family, so a two-point average gain corresponds to about two problems, and transfer beyond math is untested.

The rerun baselines use their default settings, so the finetuning budget is not matched, and the procedure behind the 1.7B and 4B radii is described only for 8B. Separately, all ablations except the random pool are single runs, while the full method's run-to-run standard deviation is 0.46 and up to 0.89 on one benchmark.

The evidence for "complementary corrections" needs controls. Coverage counts a position if at least one expert raises the reference-token probability, so a pool of 25 noisy copies covers more positions than one teacher almost by construction, even if the perturbations carry no useful structure. A fair comparison is coverage from 25 random perturbations, or from the teacher sampled at a matched temperature or with dropout. The newly covered tokens are mostly "the", "we", backslashes and numerals (Table 16), which reads more like lexical noise than reasoning corrections.

On the training side, routing toward the highest-peak token and an upper quantile of support likely gives a sharper target than the fixed teacher, so part of the gain may come from target sharpness rather than new information. A control that sharpens the fixed teacher's distribution, for example by temperature, to match the routed target's peak would separate that effect.

The paper also does not test whether weight-space neighbors are needed at all. Because the teacher only scores student prefixes, its target can be diversified more cheaply, for example by varying the privileged context (alternative reference solutions or prompt formats), and no such pool is compared. Eq. 3 adds isotropic noise with one absolute radius to the full parameter vector, without saying whether embeddings, normalization gains or the output head are included, since weight scales differ widely across tensor types, the same radius perturbs some tensors far more than others in relative terms.

Questions for authors

Are all parameters perturbed, including embeddings, normalization gains and the output head? Did you try scaling the noise to each tensor's magnitude, or perturbing only some layers, and how sensitive are the results to this choice?

How were the pool size, quantile and gate threshold chosen? If on AIME and HMMT, can you report results with these settings chosen on a held-out split of the training distribution (as was done for the radius), and give the baselines a comparable search?

How does selection-feasible coverage of the PTA pool compare with 25 random perturbations at the same radius, or 25 samples of the fixed teacher with dropout or temperature? This would show whether the coverage gain reflects selection or multiplicity.

Can you compare against a fixed teacher whose distribution is sharpened (for example by temperature) to match the peak probability of the routed target? If this closes much of the gap, the gain is mainly a target-sharpening effect.

Can you report the key ablations (selection rule, quantile, pool size) over three runs, given that most gaps are below one point?

Related prior work on distilling from stochastic copies of one teacher, such as dropout-generated teachers filtered by the student (Aslam et al., ECML/PKDD 2025, "Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation"), should be discussed.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Zt6z (Part 1/4)

Thank you for the detailed questions and controls.

Math results are Qwen3-8B Average@12 (%), means ± sample SD over three runs. Each benchmark has 30 questions and 12 samples per question. Table numbers refer to the revised paper unless labeled original.

Q1.

Are all parameters perturbed, including embeddings, normalization gains and the output head? Did you try scaling the noise to each tensor's magnitude, or perturbing only some layers, and how sensitive are the results to this choice?

A1. The default perturbs all parameters, including embeddings, normalization gains, and the output head. Across three runs, scaled-noise means stay within 0.19 percentage points of the default. Perturbing only the last 12 blocks lowers the means by 0.65–0.83 points, but they remain above OPSD. One-dimensional parameters and Q/K normalization are included in the default.

BenchmarkDefault N-OPSDNoise scaled to tensor magnitudePerturbing only some layers
AIME 202478.70 ± 0.6978.52 ± 0.8577.87 ± 0.98
AIME 202572.78 ± 0.2872.96 ± 0.9872.13 ± 0.85
HMMT February 202548.24 ± 0.8948.43 ± 1.2547.50 ± 1.00

Scaled noise applies W′=W+α∥W∥FZ/∥Z∥FW'=W+\alpha\|W\|_F Z/\|Z\|_F to each parameter tensor separately, with Gaussian ZZ and α≈0.04376\alpha\approx0.04376. All parameters participate, including embeddings, normalization gains, and the output head. Each normalization vector is scaled separately.

Partial-layer noise applies W′=W+σZW'=W+\sigma Z, with σ≈0.00532\sigma\approx0.00532, only to zero-indexed blocks 24–35, including their attention, MLP, and normalization parameters. Earlier blocks, embeddings, final normalization, and the output head stay unchanged.

We calibrate the strengths to make the changes in teacher predictions comparable to those from the default σ = 0.002. Both controls use the same independent 400-fit/100-check split and eight strength probes. For each strength, we average DKL(qperturbed∥qM0)D_{\mathrm{KL}}(q_{\mathrm{perturbed}}\|q_{M_0}) over eight pilot experts and retained positions, then choose the fit KL closest to the default. The match is approximate: 0.0539 for the default, 0.0471 for scaled noise, and 0.0596 for partial-layer noise. Strength selection uses calibration data only; Appendix B.10 lists the exact strengths and probes.

Each variant uses the same 500 candidate seeds plus M₀ and reselects K = 25 by filtered PTA on the same 1,000 questions. Routing, loss, training data, batch size, and the step-100 rule are shared. Each selected pool is fixed across three training runs. Default results are reused from the submitted paper.

This comparison evaluates the last 12 blocks at their own calibrated strength against the default full-parameter setting. It changes both parameter scope and noise strength.

Revised paper: §2.1 specifies parameter scope; §3.5 summarizes the control results. Table 8 and Appendix B.10 give the results, formulas, strength probes, and matching error.

Q2.

How were the pool size, quantile and gate threshold chosen? If on AIME and HMMT, can you report results with these settings chosen on a held-out split of the training distribution (as was done for the radius), and give the baselines a comparable search?

A2. We appreciate your close attention to parameter selection. The original manuscript omitted the validation procedure. K, q, and τ were chosen on held-out OpenThoughts data, without using AIME or HMMT scores. We have now added this procedure. Baselines use their published parameters, without an additional search.

We select parameters by Average@12 at the fixed step-100 checkpoint, using 1,000 distinct held-out math questions with checkable answers from the training version of OpenThoughts. Question-level exact and near-duplicate filtering excludes training, expert-selection, analysis, temperature/perturbation-calibration, and test questions. Students answer from the problem alone.

The tested values are K ∈ {1, 5, 10, 25, 50}, q ∈ {0, 0.25, 0.50, 0.75, 1}, and τ ∈ {0.95, 0.98, 0.99, 0.995}. We tested these values without evaluating every combination. The selected defaults are K = 25, q = 0.75, and τ = 0.99.

Validation also selects the radius, searching from 8B to 4B to 1.7B. Each smaller model starts from the preceding selected radius:

ModelTested σ valuesSelected σ
Qwen3-8B0.006, 0.004, 0.002, 0.0010.002
Qwen3-4B0.002, 0.0012, 0.00060.0012
Qwen3-1.7B0.0012, 0.0006, 0.00020.0006

Baselines share initialization, training data, optimizer-step count, the fixed step-100 checkpoint, and evaluation. Hyperparameter search budgets differ because baselines retain their published settings.

Revised paper: §3.1 and Appendix B.8 now state the validation split, actual search values, fixed checkpoint rule, and published-parameter baseline scope.

AuthorsOpenReview ↗

Response to Reviewer Zt6z (Part 2/4)

Q3.

How does selection-feasible coverage of the PTA pool compare with 25 random perturbations at the same radius, or 25 samples of the fixed teacher with dropout or temperature? This would show whether the coverage gain reflects selection or multiplicity.

A3. We tested the random-perturbation control in your question. At the same size and radius, the PTA-selected pool has higher selection-feasible coverage than all three random pools, exceeding their mean by 1.55 percentage points. This identifies a coverage benefit beyond pool size.

All perturbation pools use K = 25 and σ = 0.002, with the same analysis data and filtering. The denominator is 118,917 reference positions retained by SCGate. A position is counted once if any teacher in the pool provides positive filtered PTA.

Teacher poolSelection-feasible coverage (%)
Original teacher M₀15.69
Random pools (mean ± SD)24.98 ± 0.07
PTA-selected pool, K = 2526.53

Eq. 8 selects experts for the filtered PTA they add beyond the current pool. With the same router, this selection raises average student accuracy from 64.29 for Random Top-K to 66.57 for N-OPSD (original Tables 4 and 1). These student results support the value of selection for problem solving.

Revised paper: Table 23 and Appendix E.2 report random-pool coverage; §3.3 separates the benefit of selection from pool size.

Q4.

Can you compare against a fixed teacher whose distribution is sharpened (for example by temperature) to match the peak probability of the routed target? If this closes much of the gap, the gain is mainly a target-sharpening effect.

A4. Thank you for pointing out target sharpening as an important alternative explanation for the gains. Your proposed control helps separate this effect from the benefit of the selected experts. We lowered the original teacher's temperature to match the routed target's average peak probability. N-OPSD still has higher mean accuracy on all three benchmarks, so this fixed-temperature control does not close the gap. The differences are 1.01, 1.76, and 2.59 percentage points.

BenchmarkOPSDOPSD + SCGateOPSD + SCGate, T = 0.8176330566N-OPSD
AIME 202477.41 ± 0.3277.87 ± 0.7077.69 ± 0.8578.70 ± 0.69
AIME 202571.02 ± 0.3271.11 ± 0.4871.02 ± 0.8972.78 ± 0.28
HMMT February 202545.46 ± 0.4245.74 ± 0.4245.65 ± 0.9848.24 ± 0.89

We prepared 500 independent OpenThoughts questions for temperature calibration, with 400 for fitting T and 100 for checking the match. Exact and near-duplicate filtering excludes training, expert-selection, analysis, and benchmark questions.

Both teachers use the same fixed prefixes generated by the initial student and the same positions retained by SCGate. We first obtain N-OPSD's teacher distributions using the complete q = 0.75 router used in training. We take the maximum probability over the full vocabulary at each position and average across positions. We then adjust the original teacher's temperature to match this average, obtaining T = 0.8176330566.

The routed and sharpened peaks are 76.4869% and 76.4963% on the fit split, and 76.6543% and 76.5223% on the check split. T is chosen using only the fit split and held fixed throughout all three training runs. Online SCGate, clipped forward KL, and student sampling and evaluation temperatures are unchanged.

Revised paper: Table 4 and §3.4 report the sharpening control; Appendix B.7 gives the independent calibration protocol.

AuthorsOpenReview ↗

Response to Reviewer Zt6z (Part 3/4)

Q5.

Can you report the key ablations (selection rule, quantile, pool size) over three runs, given that most gaps are below one point?

A5. We completed three-run ablations of selection, quantile, and pool size. PTA selection retains higher mean accuracy than answer-accuracy selection. At K = 25, q = 0.75 gives higher means than q = 1. At q = 0.75, K = 25 gives higher means than K = 50. Some nearby settings remain close relative to run variability, as shown by the means ± sample SD below.

Selection rule

Independent PTA Top-K gives higher means than Quality Top-K on all three benchmarks. The PTA selector improves them further by accounting for gains already supplied by the current pool. This supports selecting experts for their additional contribution. Its AIME 2024 gap from Greedy sample coverage remains small relative to run variability.

BenchmarkRandom Top-KQuality Top-KGreedy sample coverageIndependent PTA Top-KPTA selector (Ours)
AIME 202477.04 ± 0.8576.94 ± 0.7378.43 ± 0.8977.59 ± 0.8578.70 ± 0.69
AIME 202570.28 ± 0.4870.09 ± 1.0572.31 ± 0.8971.57 ± 0.5872.78 ± 0.28
HMMT February 202545.56 ± 0.7345.83 ± 0.8346.85 ± 1.1247.04 ± 0.7048.24 ± 0.89

Quantile, K = 25

With the pool and anchor rule fixed, q = 0.75 gives higher means than q = 1 on all three benchmarks. This favors quantile selection over always choosing the expert with the strongest support for the anchor token. The q = 0.50 and q = 0.75 results are close on both AIME benchmarks.

Benchmarkq = 0q = 0.25q = 0.50q = 0.75q = 1
AIME 202476.94 ± 1.0077.04 ± 0.5878.61 ± 0.7378.70 ± 0.6976.76 ± 1.31
AIME 202570.19 ± 0.8971.76 ± 0.8972.69 ± 0.5872.78 ± 0.2871.85 ± 1.12
HMMT February 202546.02 ± 0.9846.67 ± 1.0047.22 ± 0.7348.24 ± 0.8946.94 ± 1.11

Pool size, q = 0.75

At q = 0.75, means improve from K = 1 to K = 25 and fall at K = 50. K = 25 gives the highest means among these tested pool sizes.

BenchmarkK = 1K = 5K = 10K = 25K = 50
AIME 202476.20 ± 1.0577.22 ± 0.7378.06 ± 0.5678.70 ± 0.6978.33 ± 0.73
AIME 202569.63 ± 0.8971.39 ± 0.7371.85 ± 0.5872.78 ± 0.2870.56 ± 1.00
HMMT February 202545.56 ± 0.8347.04 ± 1.1247.22 ± 0.5648.24 ± 0.8944.35 ± 0.98

Joint settings

At K = 50, lowering q from 0.75 to 0.50 raises HMMT performance but reduces AIME 2025 performance. Pool size and routing strength therefore interact, with different effects across benchmarks.

BenchmarkK = 50, q = 0.75K = 50, q = 0.50K = 50, q = 0.25
AIME 202478.33 ± 0.7379.54 ± 0.7076.67 ± 1.21
AIME 202570.56 ± 1.0070.00 ± 0.5672.59 ± 0.70
HMMT February 202544.35 ± 0.9848.43 ± 1.0545.65 ± 0.85

For configurations originally tested once, we retained that run and added two. Default N-OPSD and Random Top-K three-run summaries are reused from the submitted paper.

Revised paper: §§3.3–3.6 explain the updated findings. Tables 9, 11, and 17 give three-run summaries; Appendix G gives individual runs. Original single-run comparisons are retained separately.

Q6.

Related prior work on distilling from stochastic copies of one teacher, such as dropout-generated teachers filtered by the student (Aslam et al., ECML/PKDD 2025, "Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation"), should be discussed.

A6. We added Aslam et al. and clarified the shared idea: useful teacher diversity can come from one model. Their method generates representations with dropout, then uses the student to filter and weight them for feature distillation. Aslam et al., ECML/PKDD 2025.

N-OPSD studies this idea in OPSD with shared privileged context. It selects local parameter perturbations by marginal filtered PTA and routes one expert's full next-token distribution at each student state.

Revised paper: §4.3 cites Aslam et al. and explains the shared idea and the distinction in OPSD.

Reviewer_A4DsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on OPSD by adding small random parameter noise to the teacher that sees the reference solution, which gives a set of "neighbor teachers." The authors first pick 25 teachers that complement each other in raising the probability of reference tokens. During training, they choose one fairly confident teacher at each position and distill the student toward its distribution. On Qwen3-1.7B, 4B and 8B, the three-benchmark math average improves over OPSD by 1.67 to 2.75 points, and inference uses only the student.

Critical strengths
  1. The idea is new and simple. It needs no extra teacher or data, only small perturbations around the original teacher, and inference cost is unchanged.
  2. The experiments are fairly complete. The method beats OPSD, EOPD and PW-OPSD at all three model scales, with ablations on expert selection, routing, perturbation size and pool size.
Critical weaknesses
  1. The gains may come from a sharper target rather than complementary information from the neighbors. The neighbor teachers are nearly identical to the original one (JS divergence around 0.004). Averaging their distributions or picking one at random gives no gain; only picking a more confident teacher helps, and picking the most confident one makes results worse. This looks more like a mild sharpening of the teacher's distribution. The evidence for complementarity is also weak. The coverage increase comes from taking the max over 25 random perturbations, which would rise even if the noise carried no information. The newly covered tokens are mostly words like "the" and "we", commas and numbers, which do not look like meaningful reasoning corrections.
  2. Each training step takes about 1.42 times as long as OPSD, but there is no comparison with OPSD under the same compute.
Questions for authors
  1. How does N-OPSD compare with simply lowering the temperature of the original teacher's distribution?
  2. How were q, K and τ chosen, and did the choice use test-set results? Also, did you try layer-normalized noise or perturbing only some layers?
6 comments
AuthorsOpenReview ↗

Response to Reviewer A4Ds (Part 1/3)

Thank you for the careful review.

Math results are Qwen3-8B Average@12 (%), means ± sample SD over three runs. Each benchmark has 30 questions and 12 samples per question. Table numbers refer to the revised paper unless labeled original.

Q1.

How does N-OPSD compare with simply lowering the temperature of the original teacher's distribution?

A1. Thank you for suggesting this control to test whether lowering the original teacher's temperature can explain N-OPSD's gains.

We lowered the original teacher's temperature to match the routed target's average peak probability. N-OPSD still has higher mean accuracy on all three benchmarks, so this fixed-temperature control does not close the gap. The differences are 1.01, 1.76, and 2.59 percentage points.

BenchmarkOPSDOPSD + SCGateOPSD + SCGate, T = 0.8176330566N-OPSD
AIME 202477.41 ± 0.3277.87 ± 0.7077.69 ± 0.8578.70 ± 0.69
AIME 202571.02 ± 0.3271.11 ± 0.4871.02 ± 0.8972.78 ± 0.28
HMMT February 202545.46 ± 0.4245.74 ± 0.4245.65 ± 0.9848.24 ± 0.89

We prepared 500 independent OpenThoughts questions for temperature calibration, with 400 for fitting T and 100 for checking the match. Exact and near-duplicate filtering excludes training, expert-selection, analysis, and benchmark questions.

Both teachers use the same fixed prefixes generated by the initial student and the same positions retained by SCGate. We first obtain N-OPSD's teacher distributions using the complete q = 0.75 router used in training. We take the maximum probability over the full vocabulary at each position and average across positions. We then adjust the original teacher's temperature to match this average, obtaining T = 0.8176330566.

The routed and sharpened peaks are 76.4869% and 76.4963% on the fit split, and 76.6543% and 76.5223% on the check split. T is chosen using only the fit split and held fixed throughout all three training runs. Online SCGate, clipped forward KL, and student sampling and evaluation temperatures are unchanged.

Revised paper: Table 4 and §3.4 report the control and its scope; Appendix B.7 gives the calibration protocol.

AuthorsOpenReview ↗

Response to Reviewer A4Ds (Part 2/3)

Q2.

How were q, K and τ chosen, and did the choice use test-set results? Also, did you try layer-normalized noise or perturbing only some layers?

A2. Thank you for pointing this out. We omitted the validation procedure from the original manuscript, which left the basis for parameter selection unclear. We selected q, K, and τ on independent validation data, without using AIME or HMMT test scores.

We select parameters by Average@12 at the fixed step-100 checkpoint, using 1,000 distinct held-out math questions with checkable answers from the training version of OpenThoughts. Question-level exact and near-duplicate filtering excludes training, expert-selection, analysis, temperature/perturbation-calibration, and test questions. Students answer from the problem alone.

The tested values are K ∈ {1, 5, 10, 25, 50}, q ∈ {0, 0.25, 0.50, 0.75, 1}, and τ ∈ {0.95, 0.98, 0.99, 0.995}. We tested these values without evaluating every combination. The selected defaults are K = 25, q = 0.75, and τ = 0.99.

Thank you also for suggesting noise scaling and perturbing only some layers. These controls help test whether the gains depend on using one absolute noise scale across tensors of different magnitudes or on perturbing all layers. We tested noise scaled to each tensor and perturbations limited to the last 12 blocks. Both variants achieve higher mean accuracy than OPSD on all three benchmarks. Scaled-noise means differ from the default by at most 0.19 percentage points. Partial-layer means are lower by 0.65–0.83 points.

BenchmarkDefault N-OPSDNoise scaled to tensor magnitudePerturbing only some layers
AIME 202478.70 ± 0.6978.52 ± 0.8577.87 ± 0.98
AIME 202572.78 ± 0.2872.96 ± 0.9872.13 ± 0.85
HMMT February 202548.24 ± 0.8948.43 ± 1.2547.50 ± 1.00

The partial-layer control uses its own calibrated strength, so this comparison changes both the perturbed layers and the noise strength.

Scaled noise applies W′=W+α∥W∥FZ/∥Z∥FW'=W+\alpha\|W\|_F Z/\|Z\|_F to each parameter tensor separately, with Gaussian ZZ and α≈0.04376\alpha\approx0.04376. All parameters participate, including embeddings, normalization gains, and the output head. Each normalization vector is scaled separately.

Partial-layer noise applies W′=W+σZW'=W+\sigma Z, with σ≈0.00532\sigma\approx0.00532, only to zero-indexed blocks 24–35, including their attention, MLP, and normalization parameters. Earlier blocks, embeddings, final normalization, and the output head stay unchanged.

We calibrate the strengths to make the changes in teacher predictions comparable to those from the default σ = 0.002. Both controls use the same independent 400-fit/100-check split and eight strength probes. For each strength, we average DKL(qperturbed∥qM0)D_{\mathrm{KL}}(q_{\mathrm{perturbed}}\|q_{M_0}) over eight pilot experts and retained positions, then choose the fit KL closest to the default. The match is approximate: 0.0539 for the default, 0.0471 for scaled noise, and 0.0596 for partial-layer noise. Strength selection uses calibration data only; Appendix B.10 lists the exact strengths and probes.

Each variant uses the same 500 candidate seeds plus M₀ and reselects K = 25 by filtered PTA on the same 1,000 questions. Routing, loss, training data, batch size, and the step-100 rule are shared. Each selected pool is fixed across three training runs. Default results are reused from the submitted paper.

Our partial-layer experiment covers only the last 12 blocks. We have added comparisons of different positions and numbers of perturbed blocks as future work in Appendix F (p. 26), to understand which choices provide useful teacher supervision.

Revised paper: §2.1 states the default parameter scope; Appendix B.8 gives validation and search details; §3.5 and Table 8 report the perturbation controls, with protocols in Appendix B.10.

AuthorsOpenReview ↗

Response to Reviewer A4Ds (Part 3/3)

Additional concern from Critical Weaknesses: complementarity

The evidence for complementarity is also weak. The coverage increase comes from taking the max over 25 random perturbations, which would rise even if the noise carried no information. The newly covered tokens are mostly words like "the" and "we", commas and numbers, which do not look like meaningful reasoning corrections.

We compared the selected pool with three random pools of the same size and radius. Random pools account for much of the coverage increase, while PTA selection adds a further 1.55 percentage points. All perturbation pools use K = 25 and σ = 0.002 on the same analysis data.

The denominator is 118,917 reference positions retained by SCGate. Under the same filtering rules, a position is counted once if any teacher in the pool provides positive filtered PTA.

Teacher poolSelection-feasible coverage (%)
Original teacher M₀15.69
Random pools (mean ± SD)24.98 ± 0.07
PTA-selected pool, K = 2526.53

The average JS divergence describes how similar the teachers are overall. PTA selection checks each position for filtered PTA beyond what the current pool already provides (Eq. 8). Teachers that are close on average can therefore still contribute at different positions.

To test whether selection helps problem solving, we compare students trained with the same router: Random Top-K reaches 64.29 average accuracy, versus 66.57 for N-OPSD (original Tables 4 and 1).

The original prediction and continuation tests show that neighbor teachers make more accurate predictions on both reference prefixes and prefixes generated by the student. At σ = 0.002, the MaxPeak anchor raises token accuracy on the same reference prefixes from 91.33% to 95.58% (Table 22). On 300 incorrect or unfinished prefixes from the initial student, Raw MaxPeak raises continuation accuracy from 83.33% to 86.67% and completion from 90.00% to 94.67% (Table 24; Appendix E.3).

The routing ablations test how to use these teachers. Uniform averaging and Random routing give lower student accuracy than N-OPSD on the same pool (§3.4). Taking the strongest support at q = 1 also gives lower means than q = 0.75. The temperature control in A1 further shows that matching the average peak probability leaves a performance gap. Together, these results support selecting complementary experts and routing their predictions according to the student's current prefix.

Revised paper: Table 23 and Appendix E.2 report the random-pool coverage control; §3.3 explains selection beyond pool size.

Additional concern from Critical Weaknesses: compute

Each training step takes about 1.42 times as long as OPSD, but there is no comparison with OPSD under the same compute.

We checked actual cumulative elapsed times and evaluated two N-OPSD checkpoints within the OPSD 100-step budget. Counting online training only, 60-step N-OPSD has higher means on all three benchmarks. Including one full offline construction, 45-step N-OPSD remains close to OPSD overall, with a mean 0.15 percentage points lower.

BenchmarkOPSD, 100 stepsN-OPSD, 45 stepsN-OPSD, 60 steps
AIME 202477.41 ± 0.3277.59 ± 0.9878.24 ± 0.85
AIME 202571.02 ± 0.3271.30 ± 0.1671.94 ± 0.73
HMMT February 202545.46 ± 0.4244.54 ± 0.9845.93 ± 1.12
Average64.63 ± 0.1664.48 ± 0.6265.37 ± 0.73

All timings use eight H200 GPUs. The budget is the measured OPSD 100-step duration, 4 h 10 min 36 s. The 45-step comparison includes the full 1 h 05 min 33 s for offline generation, scoring, and selection; the 60-step comparison counts online training only. Both satisfy their stated budget based on cumulative elapsed times.

The online-only mean improves by 0.74 percentage points. With all offline work included, the two AIME means are slightly higher, while HMMT is 0.92 percentage points lower.

The original 1.423× figure estimates step time relative to OPSD + SCGate and excludes offline work. The budget comparisons above use the measured elapsed time of standard OPSD.

Revised paper: §3.6 and Table 5 report both budget comparisons. Appendix D.5 and Table 21 give measured full-run costs and accounting rules.

Reviewer_aJssOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Neighborhood OPSD (N-OPSD), which improves on-policy self-distillation by selecting complementary teachers from local parameter perturbations. The method uses greedy expert selection and state-wise routing to provide better token-level supervision. Experiments on three mathematical reasoning benchmarks demonstrate consistent improvements across Qwen3-1.7B, 4B, and 8B models.

Critical strengths
  1. The idea of exploiting local parameter perturbations for complementary supervision is interesting and well-motivated.
  2. Experiments demonstrate consistent improvements over several OPSD baselines across different model scales.
  3. The ablation studies provide useful evidence for the effectiveness of expert selection and state-wise routing.
Critical weaknesses
  1. Computational overhead. The method requires evaluating hundreds of candidate experts during offline selection and maintaining 25 experts during training. The additional cost may limit its practicality, especially for larger models.
  2. Limited generalization. Experiments focus exclusively on Qwen3 models and mathematical reasoning benchmarks. It remains unclear whether the approach generalizes to other architectures and tasks.
  3. Limited understanding of expert selection. The method relies on random parameter perturbations and several heuristics. Further analysis is needed to explain why the selected experts provide reliable supervision beyond the reference trajectories.
Questions for authors
  1. What is the total computational overhead, including offline expert construction and selection, compared with standard OPSD?
  2. How sensitive is the method to the expert pool selected offline as the student policy evolves during training?
2 comments
AuthorsOpenReview ↗

Response to Reviewer aJss (Part 1/2)

Thank you for the review.

Math results are Qwen3-8B Average@12 (%), means ± sample SD over three runs. Each benchmark has 30 questions and 12 samples per question. Table numbers refer to the revised paper unless labeled original.

Q1.

What is the total computational overhead, including offline expert construction and selection, compared with standard OPSD?

A1. Including one full offline pool construction, 100-step N-OPSD takes 7 h 38 min 01 s, or 1.828× the elapsed time of 100-step OPSD. Online training alone takes 1.566× as long. All timings use eight H200 GPUs. Measured training times include initialization and intermediate and final checkpoint saving.

Cost componentElapsed time
OPSD, 100 steps4 h 10 min 36 s
OPSD + SCGate, 100 steps4 h 42 min 58 s
N-OPSD, 100 steps, online training6 h 32 min 28 s
Offline expert generation, scoring, and selection1 h 05 min 33 s
N-OPSD, one offline construction plus 100 steps7 h 38 min 01 s

The total includes one full pool construction and one student training run. The original 1.423× figure estimates online step time relative to OPSD + SCGate and excludes offline work.

We also verified checkpoints within the OPSD 100-step time budget. Including offline construction, the 45-step mean is close to OPSD (64.48% vs. 64.63%). The two AIME means are slightly higher, while HMMT is 0.92 points lower. Counting online training only, 60-step N-OPSD reaches 65.37%, improving the mean by 0.74 points (Table 5).

Offline, we score 500 perturbed candidates plus M₀ on 1,000 questions. During training, we regenerate perturbations from seeds and restore shared base weights, so all 25 teachers need not stay in memory. Two passes also avoid storing every expert's full-vocabulary distribution at once. These choices reduce memory use. The measured training time includes the repeated teacher forward passes. Our timings cover Qwen3-8B, and larger-model costs remain unmeasured. Inference uses only the trained student.

Revised paper: §3.6 and Table 5 report the budget comparisons; Appendix D.5 and Table 21 report the measured costs.

Q2.

How sensitive is the method to the expert pool selected offline as the student policy evolves during training?

A2. Thank you for pointing out that the usefulness of a fixed expert pool may change as the student learns. This is an important question for understanding when expert reselection is needed.

Our experiments keep the pool fixed throughout training. We have not compared this with reselection during training, so its potential benefit remains an open question.

We would like to study this further, comparing student accuracy and the added cost of periodic reselection. Appendix F outlines this comparison, which we could not complete within the rebuttal time and compute budget.

AuthorsOpenReview ↗

Response to Reviewer aJss (Part 2/2)

Additional concern from Critical Weaknesses: generalization

Limited generalization. Experiments focus exclusively on Qwen3 models and mathematical reasoning benchmarks. It remains unclear whether the approach generalizes to other architectures and tasks.

We added HumanEval evaluation, where N-OPSD improves mean avg@8 by 1.45 percentage points over OPSD. This extends the evidence to code generation with Qwen3-8B.

BenchmarkBaseOPSDN-OPSD
HumanEval (avg@8)88.0388.39 ± 0.9189.84 ± 0.84

Both methods train Qwen3-8B on 30,000 coding examples from OpenThoughts-114k for 100 optimizer steps, with effective batch size 64 and evaluation at step 100. Trained methods report means over three independent training runs. Base is evaluated once.

HumanEval has 164 problems. We generate eight solutions per problem in thinking mode and report avg@8, the fraction of outputs passing its unit tests.

With the time available for rebuttal, we evaluated code generation on HumanEval using Qwen3-8B. We agree that broader tests are important and would like to extend this study to more code benchmarks, including LiveCodeBench v6, and models with different architectures.

Revised paper: §3.2 reports the code result. Appendix B.9 gives the protocol and distinguishes one Base evaluation from three trained runs. Appendix F retains the model-family limit.

Additional concern from Critical Weaknesses: supervision beyond reference trajectories

Limited understanding of expert selection. The method relies on random parameter perturbations and several heuristics. Further analysis is needed to explain why the selected experts provide reliable supervision beyond the reference trajectories.

Thank you for highlighting the need to evaluate selected experts on the student's own reasoning. Our continuation test shows that the selected pool can improve accuracy on prefixes generated by the student.

Offline selection uses reference solutions to find experts that provide complementary corrections. During training, all selected experts receive the same reference context and the student's current prefix. Each expert predicts the next token from this input, and routing is recomputed at every position. Thus, supervision is computed on the trajectory the student actually follows.

Our continuation test evaluates the selected teachers on 300 prefixes from incorrect or unfinished trajectories generated by the initial student. These prefixes may already depart from a correct solution, allowing us to examine how the teachers help the student continue from its own reasoning. Both Raw MaxPeak and M₀ receive the same prefixes and reference solutions, with the same budget of 4,096 new tokens. Raw MaxPeak reaches 86.67% continuation accuracy, compared with 83.33% for M₀ (Table 24; Appendix E.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.