- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LEAP for multi-teacher on-policy distillation. Instead of matching each expert's full output distribution, LEAP transfers the expert's post-training change relative to its pre-specialization base model. It constructs a student-centered advantage proxy from the resulting log-probability differences and applies a direction-consistency gate. Same-rollout analysis shows a reduction in token-level sign conflicts, while sequential and mixed-domain experiments with math, science, and coding experts report stronger capability acquisition and retention than standard OPD across several domain ratios.
Critical strengths
S1: Meaningful Problem and Insight. The paper identifies an important source of interference in multi-teacher OPD: using each expert's full distribution can cause the shared base-student discrepancy to make otherwise compatible expert updates appear conflicting. Focusing instead on the proficiency acquired during post-training offers an intuitive and useful perspective on multi-expert knowledge transfer.
S2: Clear, Well-Motivated, and Effective Method. LEAP instantiates this insight through an expert-minus-base signal, a student-centered advantage proxy, and a direction-consistency gate. The design is clear and coherent, and it consistently outperforms standard OPD in the reported sequential and mixed-domain experiments.
Critical weaknesses
W1: Limited Reliability of the Mathematics Evaluation. AIME24 and AIME25 contain only 30 problems each, so a 3.3-point change corresponds to one question. Some apparently large gains may therefore reflect only a few items. No larger complementary benchmark or repeated-run uncertainty is reported, limiting confidence in the mathematics results.
W2: Restricted Multi-Teacher Setting. Although LEAP is presented as multi-teacher distillation, each sample is hard-routed to one domain teacher and the signal is defined independently for that teacher. The experiments therefore do not address joint supervision when several experts are relevant to the same sample. How conflicting deltas from three or more experts should be aggregated, especially across heterogeneous reference models, remains undefined.
Questions for authors
Q1: Given the small size of AIME24 and AIME25, do the mathematics gains persist on a larger benchmark such as MATH-500 or GSM8K, and how variable are the results across repeated runs?
Q2: Since LEAP hard-routes each sample to one teacher and defines the supervision signal independently per teacher, how should the method combine signals when multiple experts are relevant to the same sample or produce conflicting post-training deltas? For three or more experts, what aggregation or conflict-resolution rule determines the final student update?
4 comments
2. Same-query supervision and conflict handling with multiple teachers
2. Same-query supervision and conflict handling with multiple teachers
Thank you for this question. We first analyze how our methods reduces conflicts between two teachers supervising the same student-generated trajectory. In actual training, each sample is routed to the most relevant teacher to transfer its specialization gains. We then examine cross-domain gradients to assess whether supervision from different teachers produces conflicting student updates under this routing scheme, and how can this conflict be reduce to help learning.
2.1 Same-trajectory analysis: how compatible are different teachers' signals?
To examine this directly, we select 100 SciBench prompts covering scientific knowledge and mathematical reasoning from seven textbook sources. At each student checkpoint, we generate one trajectory per prompt and score it with both M2RL-RL_Math and M2RL-RL_Science, using their shared Qwen3-4B-Base reference. All supervision variants use identical trajectories within each checkpoint.
A signal is active when its absolute magnitude is at least . We report conflict over all valid completion tokens and, separately, over tokens where both teachers are active. The random-mask control matches LEAP's active coverage and uses five random seeds.
All values are percentages. “All” and “Joint” denote conflict rates over all tokens and jointly active tokens, respectively. Active coverage is reported as Math / Science.
| Step | Method | All | Joint | Coverage |
|---|---|---|---|---|
| 0 | OPD | 0.875 | 2.724 | 33.51 / 33.94 |
| 0 | Random mask | 0.541 | 5.725 | 13.59 / 13.18 |
| 0 | LEAP | 0.193 | 1.617 | 13.59 / 13.18 |
| 20 | OPD | 2.298 | 5.090 | 46.38 / 46.91 |
| 20 | Random mask | 1.096 | 6.060 | 26.47 / 25.78 |
| 20 | LEAP | 0.411 | 1.769 | 26.47 / 25.78 |
| 50 | OPD | 4.075 | 7.085 | 58.53 / 59.28 |
| 50 | Random mask | 1.817 | 6.948 | 38.78 / 37.67 |
| 50 | LEAP | 0.695 | 2.045 | 38.78 / 37.67 |
The results support these observations:
- LEAP improves compatibility among retained signals. Its jointly active conflict rate is lower than OPD's at all three checkpoints.
- The selected positions matter beyond coverage. Random masking lowers conditional conflict rates, whereas LEAP substantially reduces them at matched coverage.
These results demonstrate improved pairwise signal compatibility.
2.2 Routed training: the resulting domain gradients are more compatible
2.2 Routed training: the resulting domain gradients are more compatible
Same-trajectory sign agreement does not directly establish compatibility between updates on different samples. We therefore analyze the actual routed setting using 64 Math and 64 Science examples arranged into 16 balanced batches, each containing four examples per domain.
We compute cosine similarity between the two domains' distillation gradients. The negative-cosine proportion is the fraction of batches with opposing domain gradients.
Each entry reports OPD → LEAP. Negative (%) denotes the percentage of batches with negative gradient cosine.
| Checkpoint | Grad. cosine | Negative (%) |
|---|---|---|
| Initial | 0.03777 → 0.03067 | 25.00 → 37.50 |
| Step 20 | 0.07547 → 0.16820 | 12.50 → 0.00 |
| Step 50 | 0.08763 → 0.15947 | 0.00 → 0.00 |
LEAP does not improve overall gradient alignment at initialization, but achieves higher alignment at steps 20 and 50. At step 20, it also reduces negative-cosine batches from 12.50% to zero.
Layer-group analysis further identifies where these changes occur. We divide the student's 28 layers into shallow (0–9), middle (10–18), and deep (19–27) groups. Each entry below reports OPD → LEAP.
| Checkpoint | Layer group | Gradient cosine | Negative-cosine batches |
|---|---|---|---|
| Initial | Shallow overall | 0.0439 → 0.0364 | 31.25% → 37.50% |
| Initial | Shallow attention | 0.0307 → 0.0106 | 31.25% → 50.00% |
| Initial | Shallow MLP | 0.0456 → 0.0399 | 37.50% → 37.50% |
| Initial | Middle overall | 0.0285 → 0.0215 | 25.00% → 25.00% |
| Initial | Middle attention | 0.0312 → 0.0225 | 25.00% → 31.25% |
| Initial | Middle MLP | 0.0210 → 0.0184 | 25.00% → 31.25% |
| Initial | Deep overall | −0.0061 → 0.0169 | 37.50% → 12.50% |
| Initial | Deep attention | −0.0049 → 0.0293 | 37.50% → 12.50% |
| Initial | Deep MLP | −0.0089 → 0.0076 | 50.00% → 31.25% |
| Step 20 | Shallow overall | 0.0954 → 0.2103 | 12.50% → 0.00% |
| Step 20 | Shallow attention | 0.0699 → 0.1517 | 25.00% → 6.25% |
| Step 20 | Shallow MLP | 0.0989 → 0.2178 | 12.50% → 0.00% |
| Step 20 | Middle overall | 0.0486 → 0.1149 | 12.50% → 0.00% |
| Step 20 | Middle attention | 0.0553 → 0.1289 | 6.25% → 0.00% |
| Step 20 | Middle MLP | 0.0294 → 0.0715 | 18.75% → 6.25% |
| Step 20 | Deep overall | 0.0285 → 0.0434 | 18.75% → 6.25% |
| Step 20 | Deep attention | 0.0541 → 0.0573 | 12.50% → 0.00% |
| Step 20 | Deep MLP | 0.0159 → 0.0367 | 50.00% → 12.50% |
| Step 50 | Shallow overall | 0.1027 → 0.1826 | 0.00% → 0.00% |
| Step 50 | Shallow attention | 0.0562 → 0.1311 | 0.00% → 0.00% |
| Step 50 | Shallow MLP | 0.1087 → 0.1892 | 0.00% → 0.00% |
| Step 50 | Middle overall | 0.0711 → 0.1159 | 0.00% → 0.00% |
| Step 50 | Middle attention | 0.0784 → 0.1258 | 0.00% → 0.00% |
| Step 50 | Middle MLP | 0.0491 → 0.0843 | 12.50% → 0.00% |
| Step 50 | Deep overall | 0.0671 → 0.1201 | 6.25% → 0.00% |
| Step 50 | Deep attention | 0.0806 → 0.0791 | 0.00% → 6.25% |
| Step 50 | Deep MLP | 0.0606 → 0.1406 | 12.50% → 0.00% |
LEAP’s alignment gains expand from deep layers at initialization to all layer groups by step 20 and persist at step 50. The attention/MLP breakdown shows that deep-layer MLPs show fewer negative-cosine batches (50% → 12.5% at step 20; 12.5% → 0% at step 50) and higher cosine at step 50 (0.0606 → 0.1406).
These results suggest that LEAP’s teacher-specific supervision improves compatibility of routed domain updates.
Thank you for the additional experiments and analyses. The new results largely address my concerns regarding the reliability of the mathematics evaluation. However, my main concern about multi-teacher supervision remains unresolved. In particular, the authors analyze pairwise signal compatibility and gradient alignment, but do not clarify how conflicting signals from multiple relevant experts are aggregated into the final student update, especially with three or more heterogeneous teachers. While I appreciate the additional experimental efforts, this remains an important limitation of the method. I will maintain my current rating, although I am now leaning more toward 2.5.
1. Mathematics gains persist on larger benchmarks, and results variable across repeated runs
1. Mathematics gains persist on larger benchmarks, and results variable across repeated runs
We have extended the evaluation to three teachers and 200 training steps, including MATH500 and GSM8K, and report results across repeated runs below.
1.1 Three teachers for 100 steps
The mathematics gains persist on both larger benchmarks. LEAP improves over OPD by 6.66 points on MATH500 and 2.85 points on GSM8K, and over ExOPD by 5.00 and 1.59 points, respectively.
Thus, while the AIME results exhibit appreciable run-to-run variation, the larger-benchmark results provide additional evidence for the mathematics gains. LEAP also achieves the highest mean score on all seven benchmarks.
1.2 Longer training steps bring extra benefits
Longer training also yields further gains. The table reports mean scores across three runs at 100 and 200 training steps. The mean score at 200 steps is higher on all seven benchmarks, indicating that more training steps bring benefits.