acceptodds
Under review as a conference paper at ICLR 2027

LEAP: Learning from Experts Acquired Proficiency in Multi-Teacher On-Policy Distillation

Abstract

Multi-teacher on-policy distillation offers a promising approach to integrating specialized capabilities into a single language model. However, matching each teacher’s output distribution may introduce competing supervision and interfere with capabilities acquired from other experts. We introduce LEAP, a framework that focuses on transferring the proficiency experts acquire during post-training. LEAP uses differences between each expert and its pre-specialization reference model to represent acquired proficiency, and integrates these signals on student-generated trajectories. We investigate this approach through supervision analysis, sequential expert distillation, and domain-routed training with varying query mixtures. On matched student rollouts, LEAP reduces conflicting supervision compared with conventional on-policy distillation. In sequential training, it improves retention of previously acquired capabilities while maintaining the acquisition of new skills. Across different query mixtures, LEAP achieves a more favorable trade-off between improving emphasized domains and preserving capabilities in underrepresented domains. These findings support transferring experts’ post-training gains as an effective principle for capability integration, enabling more reliable accumulation of specialized expertise in a single student.

Primary area:
reinforcement learning
Keywords:
multi-teacher on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wFdVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LEAP for multi-teacher on-policy distillation. Instead of matching each expert's full output distribution, LEAP transfers the expert's post-training change relative to its pre-specialization base model. It constructs a student-centered advantage proxy from the resulting log-probability differences and applies a direction-consistency gate. Same-rollout analysis shows a reduction in token-level sign conflicts, while sequential and mixed-domain experiments with math, science, and coding experts report stronger capability acquisition and retention than standard OPD across several domain ratios.

Critical strengths

S1: Meaningful Problem and Insight. The paper identifies an important source of interference in multi-teacher OPD: using each expert's full distribution can cause the shared base-student discrepancy to make otherwise compatible expert updates appear conflicting. Focusing instead on the proficiency acquired during post-training offers an intuitive and useful perspective on multi-expert knowledge transfer.

S2: Clear, Well-Motivated, and Effective Method. LEAP instantiates this insight through an expert-minus-base signal, a student-centered advantage proxy, and a direction-consistency gate. The design is clear and coherent, and it consistently outperforms standard OPD in the reported sequential and mixed-domain experiments.

Critical weaknesses

W1: Limited Reliability of the Mathematics Evaluation. AIME24 and AIME25 contain only 30 problems each, so a 3.3-point change corresponds to one question. Some apparently large gains may therefore reflect only a few items. No larger complementary benchmark or repeated-run uncertainty is reported, limiting confidence in the mathematics results.

W2: Restricted Multi-Teacher Setting. Although LEAP is presented as multi-teacher distillation, each sample is hard-routed to one domain teacher and the signal is defined independently for that teacher. The experiments therefore do not address joint supervision when several experts are relevant to the same sample. How conflicting deltas from three or more experts should be aggregated, especially across heterogeneous reference models, remains undefined.

Questions for authors

Q1: Given the small size of AIME24 and AIME25, do the mathematics gains persist on a larger benchmark such as MATH-500 or GSM8K, and how variable are the results across repeated runs?

Q2: Since LEAP hard-routes each sample to one teacher and defines the supervision signal independently per teacher, how should the method combine signals when multiple experts are relevant to the same sample or produce conflicting post-training deltas? For three or more experts, what aggregation or conflict-resolution rule determines the final student update?

4 comments
AuthorsOpenReview ↗

1. Mathematics gains persist on larger benchmarks, and results variable across repeated runs

1. Mathematics gains persist on larger benchmarks, and results variable across repeated runs

We have extended the evaluation to three teachers and 200 training steps, including MATH500 and GSM8K, and report results across repeated runs below.

1.1 Three teachers for 100 steps

BenchmarkOPDExOPDLEAP
MATH50077.07 ± 0.3178.73 ± 0.6183.73 ± 1.27
GSM8K82.36 ± 0.2483.35 ± 0.6484.94 ± 0.49
AIME2422.22 ± 5.0924.44 ± 6.9434.44 ± 5.09
AIME2515.56 ± 5.0915.56 ± 3.8525.56 ± 3.85
GPQA-D33.33 ± 2.6735.35 ± 2.2036.20 ± 1.77
SciBench31.93 ± 0.8833.91 ± 0.8036.63 ± 0.58
LCBv58.13 ± 1.5110.48 ± 1.4915.63 ± 0.41

The mathematics gains persist on both larger benchmarks. LEAP improves over OPD by 6.66 points on MATH500 and 2.85 points on GSM8K, and over ExOPD by 5.00 and 1.59 points, respectively.

Thus, while the AIME results exhibit appreciable run-to-run variation, the larger-benchmark results provide additional evidence for the mathematics gains. LEAP also achieves the highest mean score on all seven benchmarks.

1.2 Longer training steps bring extra benefits

Longer training also yields further gains. The table reports mean scores across three runs at 100 and 200 training steps. The mean score at 200 steps is higher on all seven benchmarks, indicating that more training steps bring benefits.

BenchmarkLEAP 100 steps200 steps
MATH50083.7386.60
GSM8K84.9486.86
AIME2434.4442.22
AIME2525.5632.22
GPQA-D36.2040.35
SciBench36.6339.05
LCBv515.6319.33
AuthorsOpenReview ↗

2. Same-query supervision and conflict handling with multiple teachers

2. Same-query supervision and conflict handling with multiple teachers

Thank you for this question. We first analyze how our methods reduces conflicts between two teachers supervising the same student-generated trajectory. In actual training, each sample is routed to the most relevant teacher to transfer its specialization gains. We then examine cross-domain gradients to assess whether supervision from different teachers produces conflicting student updates under this routing scheme, and how can this conflict be reduce to help learning.

2.1 Same-trajectory analysis: how compatible are different teachers' signals?

To examine this directly, we select 100 SciBench prompts covering scientific knowledge and mathematical reasoning from seven textbook sources. At each student checkpoint, we generate one trajectory per prompt and score it with both M2RL-RL_Math and M2RL-RL_Science, using their shared Qwen3-4B-Base reference. All supervision variants use identical trajectories within each checkpoint.

A signal is active when its absolute magnitude is at least 10−410^{-4}. We report conflict over all valid completion tokens and, separately, over tokens where both teachers are active. The random-mask control matches LEAP's active coverage and uses five random seeds.

All values are percentages. “All” and “Joint” denote conflict rates over all tokens and jointly active tokens, respectively. Active coverage is reported as Math / Science.

StepMethodAllJointCoverage
0OPD0.8752.72433.51 / 33.94
0Random mask0.5415.72513.59 / 13.18
0LEAP0.1931.61713.59 / 13.18
20OPD2.2985.09046.38 / 46.91
20Random mask1.0966.06026.47 / 25.78
20LEAP0.4111.76926.47 / 25.78
50OPD4.0757.08558.53 / 59.28
50Random mask1.8176.94838.78 / 37.67
50LEAP0.6952.04538.78 / 37.67

The results support these observations:

  • LEAP improves compatibility among retained signals. Its jointly active conflict rate is lower than OPD's at all three checkpoints.
  • The selected positions matter beyond coverage. Random masking lowers conditional conflict rates, whereas LEAP substantially reduces them at matched coverage.

These results demonstrate improved pairwise signal compatibility.

AuthorsOpenReview ↗

2.2 Routed training: the resulting domain gradients are more compatible

2.2 Routed training: the resulting domain gradients are more compatible

Same-trajectory sign agreement does not directly establish compatibility between updates on different samples. We therefore analyze the actual routed setting using 64 Math and 64 Science examples arranged into 16 balanced batches, each containing four examples per domain.

We compute cosine similarity between the two domains' distillation gradients. The negative-cosine proportion is the fraction of batches with opposing domain gradients.

Each entry reports OPD → LEAP. Negative (%) denotes the percentage of batches with negative gradient cosine.

CheckpointGrad. cosineNegative (%)
Initial0.03777 → 0.0306725.00 → 37.50
Step 200.07547 → 0.1682012.50 → 0.00
Step 500.08763 → 0.159470.00 → 0.00

LEAP does not improve overall gradient alignment at initialization, but achieves higher alignment at steps 20 and 50. At step 20, it also reduces negative-cosine batches from 12.50% to zero.

Layer-group analysis further identifies where these changes occur. We divide the student's 28 layers into shallow (0–9), middle (10–18), and deep (19–27) groups. Each entry below reports OPD → LEAP.

CheckpointLayer groupGradient cosineNegative-cosine batches
InitialShallow overall0.0439 → 0.036431.25% → 37.50%
InitialShallow attention0.0307 → 0.010631.25% → 50.00%
InitialShallow MLP0.0456 → 0.039937.50% → 37.50%
InitialMiddle overall0.0285 → 0.021525.00% → 25.00%
InitialMiddle attention0.0312 → 0.022525.00% → 31.25%
InitialMiddle MLP0.0210 → 0.018425.00% → 31.25%
InitialDeep overall−0.0061 → 0.016937.50% → 12.50%
InitialDeep attention−0.0049 → 0.029337.50% → 12.50%
InitialDeep MLP−0.0089 → 0.007650.00% → 31.25%
Step 20Shallow overall0.0954 → 0.210312.50% → 0.00%
Step 20Shallow attention0.0699 → 0.151725.00% → 6.25%
Step 20Shallow MLP0.0989 → 0.217812.50% → 0.00%
Step 20Middle overall0.0486 → 0.114912.50% → 0.00%
Step 20Middle attention0.0553 → 0.12896.25% → 0.00%
Step 20Middle MLP0.0294 → 0.071518.75% → 6.25%
Step 20Deep overall0.0285 → 0.043418.75% → 6.25%
Step 20Deep attention0.0541 → 0.057312.50% → 0.00%
Step 20Deep MLP0.0159 → 0.036750.00% → 12.50%
Step 50Shallow overall0.1027 → 0.18260.00% → 0.00%
Step 50Shallow attention0.0562 → 0.13110.00% → 0.00%
Step 50Shallow MLP0.1087 → 0.18920.00% → 0.00%
Step 50Middle overall0.0711 → 0.11590.00% → 0.00%
Step 50Middle attention0.0784 → 0.12580.00% → 0.00%
Step 50Middle MLP0.0491 → 0.084312.50% → 0.00%
Step 50Deep overall0.0671 → 0.12016.25% → 0.00%
Step 50Deep attention0.0806 → 0.07910.00% → 6.25%
Step 50Deep MLP0.0606 → 0.140612.50% → 0.00%

LEAP’s alignment gains expand from deep layers at initialization to all layer groups by step 20 and persist at step 50. The attention/MLP breakdown shows that deep-layer MLPs show fewer negative-cosine batches (50% → 12.5% at step 20; 12.5% → 0% at step 50) and higher cosine at step 50 (0.0606 → 0.1406).

These results suggest that LEAP’s teacher-specific supervision improves compatibility of routed domain updates.

Thank you for the additional experiments and analyses. The new results largely address my concerns regarding the reliability of the mathematics evaluation. However, my main concern about multi-teacher supervision remains unresolved. In particular, the authors analyze pairwise signal compatibility and gradient alignment, but do not clarify how conflicting signals from multiple relevant experts are aggregated into the final student update, especially with three or more heterogeneous teachers. While I appreciate the additional experimental efforts, this remains an important limitation of the method. I will maintain my current rating, although I am now leaning more toward 2.5.

Reviewer_cRstOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LEAP, a multi-teacher on-policy distillation method that transfers only each expert’s post-training gains, measured against pre-RL base model instead of the full teacher distribution. Experiments with math, science and coding teachers illustrate fewer conflicting signals and better retention than standard OPD.

Critical strengths
  1. The paper is well written and easy to follow, and the spurious teacher conflict idea is clearly illustrated
  2. The proposed method LEAP outperforms standard OPD in both sequential and mixed-domain distillation.
Critical weaknesses
  1. The technical contribution is unclear. Transferring post-training policy changes was proposed in OPD$^2$ and Direct-OPD as illustrated in the paper L136-137. It is unclear what is the difference between the proposed method and other related works.
  2. The connection between fewer conflicts and better performance is not well-established. The analysis shows that removing shared base-student offset reduces sign conflicts. However, the consistency gate zeroes out about 75% of each teacher signals (L284-285), so part of conflict reduction may come from abstaining rather than from resolving conflicts. An ablation study on each component is needed to validate this.
  3. The baselines are limited since standard OPD is the only baseline. It is better to compare against related OPD method such as OPD$^2$, Direct-OPD (mentioned in related works).
  4. The computational overhead is not discussed. Please report the extra training cost compared with other OPD methods.
Questions for authors
  1. How much does each component such as base centering, consistency gate contribute to the performance of proposed method?
  2. How would LEAP work when teachers come from different base models or when the base checkpoint is not available?
9 comments
AuthorsOpenReview ↗

1. Novelty and relationship to prior methods

1. Novelty and relationship to prior methods

Thank you for raising this concern. We agree that the relationship to prior methods and the scope of our contribution should be stated more precisely. Our contribution centers on understanding and mitigating supervision interference when doing multi-teacher on-policy distillation, consolidating multiple specialized teachers into a single student.

Our motivation comes from a practical training pipeline: a base model is post-trained into several domain experts, whose acquired capabilities are subsequently distilled into one student. Conventional OPD transfers each teacher's full teacher–student discrepancy, mixing specialization-induced changes with the shared base–student discrepancy:

log⁡pTi−log⁡πθ=(log⁡pTi−log⁡pB)+(log⁡pB−log⁡πθ).\log p_{T_i}-\log \pi_\theta=(\log p_{T_i}-\log p_B)+(\log p_B-\log \pi_\theta).

We argue that the specializaiton delta, ΔTi=log⁡pTi−log⁡pB\Delta_{T_i } = \log p_{T_i}-\log p_B measures what teacher learned in post-training. The shared offset OTi=(log⁡pB−log⁡πθ)O_{T_i} = (\log p_B-\log \pi_\theta), however, can make two teachers produce opposite-signed OPD signals even when their specialization deltas agree. LEAP investigates this source of apparent teacher conflict and transfers base-referenced specialization signals to support capability integration.

Our contribution has three connected aspects:

  • Diagnosing multi-teacher interference. We analyze how the base–student discrepancy can introduce gradient conflicts between experts on student-generated trajectories. Quantitatively we study how can gradient conflict be reduce through our methods (seen in Official Comment: Relationship between token conflicts and performance gains).
  • Designing supervision around interference diagnosis. We utilize specialization deltas for multi-teacher OPD, and then use action-value/state-value formulation to further calibrate this signal. Ablation study further shows each component's effect.(seen in Official Comment: Component ablations)
  • Evaluating capability consolidation. We examine signal compatibility, capability acquisition and retention during sequential distillation, and cross-domain performance under different training mixtures, supplemented by the two-teacher and three-teacher experiments.
AuthorsOpenReview ↗

2. Component contributions and the relationship between conflict reduction and performance

2. Component contributions and the relationship between conflict reduction and performance

Thank you for highlighting the need to separate delta extraction, centering, and gating. We have added component ablations and a matched-coverage random-mask control.

2.1 The approximately 75% figure describes final signal inactivity, not the signals removed by the gate.

We define a signal as inactive when its absolute magnitude is below 10−410^{-4}. Approximately 66% of signals are already inactive before gating; gating increases this proportion to approximately 75%.

StatisticMathScience
Already inactive before gating66.22%66.35%
Inactive after gating75.01%75.91%
Newly inactive because of gating8.79%9.55%

Relative to initially active signals, gating removes approximately 26.0% on Math and 28.4% on Science, retaining approximately 74.0% and 71.6%, respectively. We will therefore correct the manuscript's imprecise statement that approximately 75% of signals are “zeroed by gating”.

AuthorsOpenReview ↗

2.2 Delta supervision provides substantial gains, while centering and gating interact.

2.2 Delta supervision provides substantial gains, while centering and gating interact.

We rerun and compare standard OPD, pure teacher–base delta supervision, LEAP without the gate, LEAP without expectation centering, and full LEAP. (The two-teacher Full LEAP row reports a fresh run conducted for this ablation study)

Two-teacher setting

MethodAIME24AIME25GPQA-DSciBenchGSM8KMATH500
Standard OPD26.7013.3033.3034.9080.8084.38
Only delta53.3326.6741.41‾\underline{\text{41.41}}38.8189.2087.04
Without gate23.3316.7023.2330.0986.0086.00
Without expectation centering36.6736.6737.8839.24‾\underline{\text{39.24}}87.20‾\underline{\text{87.20}}88.63
Full LEAP43.30‾\underline{\text{43.30}}33.30‾\underline{\text{33.30}}42.4240.1787.20‾\underline{\text{87.20}}87.19‾\underline{\text{87.19}}

Three-teacher setting

MethodAIME24AIME25GPQA-DSciBenchLCBv5GSM8KMATH500
Standard OPD23.3313.3332.3233.437.8682.5676.60
Only delta26.6723.33‾\underline{\text{23.33}}37.3740.1119.7885.7483.00‾\underline{\text{83.00}}
Without gate23.3320.0034.8432.418.1383.0977.40
Without expectation centering33.33‾\underline{\text{33.33}}20.0039.3936.3317.34‾\underline{\text{17.34}}86.2783.00‾\underline{\text{83.00}}
Full LEAP36.6726.6737.87‾\underline{\text{37.87}}37.35‾\underline{\text{37.35}}15.7185.90‾\underline{\text{85.90}}84.20

These results support three observations:

  • Delta supervision is a major contributor. Using delta outperforms standard OPD on every listed benchmark in both settings and reaches SOTA at some time, demonstrating substantial gains without centering or gating.
  • Gating is consequential within the centered formulation. Centering is estimation under state-value and action-value formulation, but may introduce noise under student's distribution. Full LEAP therefore introduces gating and outperforms the ungated version on every listed benchmark in both settings.
  • Full LEAP consistently improves over standard OPD. This holds across all listed benchmarks in both settings. We further compare LEAP with ExOPD, in which experiment LEAP achieves best performance consistantly.

Although centering without gating yields suboptimal performance, we observe that the L2L_2 norms of teacher-specific gradient signals become increasingly balanced during training with the full combination of delta supervision, centering, and gating. We quantify gradient-norm imbalance, where i and j represents science and math teacher:

R=∥gTi∥2∥gTj∥2R = \frac{\lVert g_{T_i}\rVert_2}{\lVert g_{T_j}\rVert_2}

Student checkpointStandard OPDOnly deltaWithout gateFull LEAP
Initial39.1332.3722.8221.01
Step 205.184.093.052.78
Step 503.603.053.012.69

Across all three checkpoints, full LEAP yields the gradient-norm ratio closest to 1 among the compared methods, indicating more balanced gradient magnitudes between the science and math teacher.

Reviewer_qpC2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores the problem in multi-teacher OPD to reduce conflicts among teacher. Instead of directly using the teacher-student gap, the proposed method, LEAP uses teacher-base delta as a proxy for post-training gains, along with advantage centering and consistency gain. Experiments show consistent improvement over standard OPD in sequential and mixed-domain distillation settings.

Critical strengths
  1. The paper tackles an interesting and practical problem.
  2. The proposed method is straightforward and easy to understand, with a clear motivation.
  3. The experiments show consistently substantial performance gains over standard OPD.
Critical weaknesses
  1. My main concern is about the novelty of the method. The approach is basically considered combining existing techniques, such as delta distillation and advantage centering, with a consistency gate. Similar ideas have already been explored as authors noted, but the paper lacks direct comparisons with these methods.
  2. Moreover, the paper lacks an important ablation study around the main three components - delta distillation, advantage centering, and consistency gate. Especially given 75% signals are filtered out by the gate, it is unclear which component leads to the gain.
  3. The overall experiments appears to be quite limited to Qwen3 models, two teachers at a time, and relatively short training runs. This raises a concern about the generalizability.
  4. The main motivation of this paper is to reduce conflicts among teachers. However, it seems that the original conflict rate is only 4.44%, which is surprisingly lower than I expected. I'm not fully convinced that reducing this already row rate can explain such a huge performance gain.
Questions for authors

Please see weaknesses section above.

10 comments
AuthorsOpenReview ↗

1. Novelty and relationship to prior methods

1. Novelty and relationship to prior methods

Thank you for raising this concern. We agree that the relationship to prior methods and the scope of our contribution should be stated more precisely. Our contribution centers on understanding and mitigating supervision interference when doing multi-teacher on-policy distillation, consolidating multiple specialized teachers into a single student.

Our motivation comes from a practical training pipeline: a base model is post-trained into several domain experts, whose acquired capabilities are subsequently distilled into one student. Conventional OPD transfers each teacher's full teacher–student discrepancy, mixing specialization-induced changes with the shared base–student discrepancy:

log⁡pTi−log⁡πθ=(log⁡pTi−log⁡pB)+(log⁡pB−log⁡πθ).\log p_{T_i}-\log \pi_\theta=(\log p_{T_i}-\log p_B)+(\log p_B-\log \pi_\theta).

We argue that the specializaiton delta, ΔTi=log⁡pTi−log⁡pB\Delta_{T_i } = \log p_{T_i}-\log p_B measures what teacher learned in post-training. The shared offset OTi=(log⁡pB−log⁡πθ)O_{T_i} = (\log p_B-\log \pi_\theta), however, can make two teachers produce opposite-signed OPD signals even when their specialization deltas agree. LEAP investigates this source of apparent teacher conflict and transfers base-referenced specialization signals to support capability integration.

Our contribution has three connected aspects:

  • Diagnosing multi-teacher interference. We analyze how the base–student discrepancy can introduce gradient conflicts between experts on student-generated trajectories. Quantitatively we study how can gradient conflict be reduce through our methods (seen in Official Comment: Relationship between token conflicts and performance gains).
  • Designing supervision around interference diagnosis. We utilize specialization deltas for multi-teacher OPD, and then use action-value/state-value formulation to further calibrate this signal. Ablation study further shows each component's effect.(seen in Official Comment: Component ablations)
  • Evaluating capability consolidation. We examine signal compatibility, capability acquisition and retention during sequential distillation, and cross-domain performance under different training mixtures, supplemented by the two-teacher and three-teacher experiments.
AuthorsOpenReview ↗

2. Component ablations and the effect of gating

2. Component ablations and the effect of gating

Thank you for pointing out the missing ablations and the ambiguity in the reported 75% figure. We have conducted additional two-teacher and three-teacher experiments, emprically proving that LEAP achieves best learning effort by comparison with representative opd methods. A matched-coverage random-mask analysis further shows that conflict reduction is not explained solely by retaining fewer signals, but by detecting what kind of signals should be retained.

2.1 Delta supervision provides substantial gains, while centering and gating interact.**

We compare standard OPD, pure teacher–base delta supervision, LEAP without the gate, LEAP without expectation centering, and full LEAP. (The two-teacher Full LEAP row reports a fresh run conducted for this ablation study)

The result shows that delta supervision provides substantial gains over standard OPD. Gating improves the centered variant on all reported benchmarks.

Two-teacher setting

MethodAIME24AIME25GPQA-DSciBenchGSM8KMATH500
Standard OPD26.7013.3033.3034.9080.8084.38
Only delta53.3326.6741.41‾\underline{\text{41.41}}38.8189.2087.04
Without gate23.3316.7023.2330.0986.0086.00
Without expectation centering36.6736.6737.8839.24‾\underline{\text{39.24}}87.20‾\underline{\text{87.20}}88.63
Full LEAP43.30‾\underline{\text{43.30}}33.30‾\underline{\text{33.30}}42.4240.1787.20‾\underline{\text{87.20}}87.19‾\underline{\text{87.19}}

Three-teacher setting

MethodAIME24AIME25GPQA-DSciBenchLCBv5GSM8KMATH500
Standard OPD23.3313.3332.3233.437.8682.5676.60
Only delta26.6723.33‾\underline{\text{23.33}}37.3740.1119.7885.7483.00‾\underline{\text{83.00}}
Without gate23.3320.0034.8432.418.1383.0977.40
Without expectation centering33.33‾\underline{\text{33.33}}20.0039.3936.3317.34‾\underline{\text{17.34}}86.2783.00‾\underline{\text{83.00}}
Full LEAP36.6726.6737.87‾\underline{\text{37.87}}37.35‾\underline{\text{37.35}}15.7185.90‾\underline{\text{85.90}}84.20

These results support three observations:

  • Delta supervision is a major contributor. Using delta outperforms standard OPD on every listed benchmark in both settings and reaches SOTA at some time, demonstrating substantial gains without centering or gating.
  • Gating is consequential within the centered formulation. Centering is estimation under state-value and action-value formulation, but may introduce noise under student's distribution. Full LEAP therefore introduces gating and outperforms the ungated version on every listed benchmark in both settings.
  • Full LEAP consistently improves over standard OPD. This holds across all listed benchmarks in both settings. We further compare LEAP with ExOPD, in which experiment LEAP achieves best performance consistantly.

Although centering without gating yields suboptimal performance, we observe that the L2L_2 norms of teacher-specific gradient signals become increasingly balanced during training with the full combination of delta supervision, centering, and gating. We quantify gradient-norm imbalance, where i and j represents science and math teacher:

R=∥gTi∥2∥gTj∥2R = \frac{\lVert g_{T_i}\rVert_2}{\lVert g_{T_j}\rVert_2}

Student checkpointStandard OPDOnly deltaWithout gateFull LEAP
Initial39.1332.3722.8221.01
Step 205.184.093.052.78
Step 503.603.053.012.69

Across all three checkpoints, full LEAP yields the gradient-norm ratio closest to 1 among the compared methods, indicating more balanced gradient magnitudes between the science and math teacher.

2.2 Matched-coverage random masking shows that conflict reduction is not explained solely by retaining fewer signals. Selective abstention is important.

We perform a diagnostic on 100 SciBench prompts covering scientific knowledge and mathematical reasoning from seven textbook sources. The random-mask control retains the same number of signals as LEAP but selects their positions randomly.

CheckpointWithout gateRandom maskLEAP
Initial5.715%5.725%1.617%
Step 206.156%6.060%1.769%
Step 506.947%6.948%2.045%

The random mask is matched to LEAP's coverage.

Random masking leaves the conditional conflict rate (defined as the fraction of tokens receiving opposite-signed teacher signals) close to that of the ungated signal, whereas LEAP reduces it substantially across all three checkpoints. This supports the importance of which signals are retained, beyond their number.

Together, these experiments identify delta supervision as a major contributor to performance gains and provide evidence for the gate's selective conflict-reduction mechanism. We will incorporate these results into the revision.

2.3 The approximately 75% figure describes final signal inactivity, not the fraction of active signals removed by the gate.

We define a signal as inactive when its absolute magnitude is below 10−410^{-4}. Approximately 66% of signals are already inactive before gating; gating increases this proportion to approximately 75%.

StatisticMathScience
Already inactive before gating66.22%66.35%
Inactive after gating75.01%75.91%
Newly inactive because of gating8.79%9.55%

Relative to initially active signals, gating removes approximately 26.0% on Math and 28.4% on Science, retaining approximately 74.0% and 71.6%, respectively. We will therefore correct the manuscript's imprecise statement that approximately 75% of signals are “zeroed by gating”.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.