acceptodds
Under review as a conference paper at ICLR 2027

The Joint-Distillation Gap: Which Teacher Corrections Fit Together Under Shared Student Updates?

Abstract

Why can a shared student learn less from several specialized teachers than from each teacher separately, even when routing is correct and domain exposure is matched? We study this joint-distillation gap through the updates needed to learn teacher corrections. The cost of adding a correction depends on what the student is already learning, so a correction favored in isolation may be a poor addition to the selected supervision. This observation motivates Jointly Feasible Mode Distillation (JFMD). It decomposes teacher–student residuals into modes and selects corrections by their reliability-adjusted value and the projected cost of adding them to the current set. Across three model families and eight benchmarks, JFMD improves aggregate transfer over the evaluated student baselines. Controlled training comparisons across candidate pairs and matched-count ablations provide empirical support for conditional cost, while held-out analyses connect calibration reliability to stability on new examples. The gains persist across the tested student sizes and under teacher-routing errors. Together, the results suggest that learning requirements shared across corrections provide a useful explanation for the gap and a guide for selecting supervision. Our code is available at https://anonymous.4open.science/r/JFMD-392B.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multi-Domain Teacher Distillation, Set-Conditional Supervision Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QufDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the joint-distillation gap: a shared student may learn less from multiple specialized teachers than separate students, even under correct teacher routing and matched domain exposure. The proposed method, Jointly Feasible Mode Distillation (JFMD), decomposes teacher–student residuals into modes and selects corrections by their reliability-adjusted value and the projected cost of adding them to the current set. The method is evaluated across three model families and eight benchmarks, with additional analyses of student capacity, routing errors, and held-out stability.

Critical strengths
  1. The paper addresses a meaningful problem by examining the relationships among teacher correction signals. It extends the consideration of teacher strength to the question of which corrections are suitable for joint learning.
  2. This work accounts for both cooperation and conflict. Similar update requirements can share updates, unrelated requirements incur separate costs, and opposing requirements may not be simultaneously satisfiable. This provides a richer perspective than examining negative gradient alignment alone.
Critical weaknesses
  1. The paper specifies the purposes and sizes of the calibration and probe datasets, but does not clearly explain their relationships to the training, validation, and test sets. These data determine the correction modes and their selection, and therefore influence the final model even when they do not directly update the student parameters.
  2. A teacher’s systematic biases may persist across different calibration samples. A stable residual direction therefore indicates repeatability, but does not by itself establish that the corresponding correction is correct or beneficial.
  3. As the student changes, the filtered residuals and distillation targets are recomputed. Although this mechanism has a reasonable feedback rationale, it does not guarantee that every update reduces the errors along all selected directions.
Questions for authors
  1. Each domain uses only a small number of probe examples to estimate response gradients. Is this sample size sufficient? How much does the selected mode set change when a different probe sample is used? As a potential extension, would periodically reselecting the directions improve the method?
  2. Could stable but incorrect teacher biases be retained by the selection procedure? As a potential extension, should different domains use different reliability thresholds?
3 comments
AuthorsOpenReview ↗

Response to Reviewer QufD

Thank you for recognizing the importance of the problem and the value of considering cooperation and conflict among teacher corrections.

W1. Calibration, probe, and data splits

Calibration and probe are part of model fitting. We sample 256 training examples per domain for calibration to construct modes and estimate reliability. By default, four examples per domain are drawn from this pool as probes to estimate response gradients and select modes. Thus, probe is a subset of calibration, which is a subset of training. These examples remain in the training sampling pool.

Per domain, calibration uses up to 2,048 valid response-token contexts and probe up to 256, excluding prompt and padding positions. Validation supports checkpoint selection and training diagnostics; test data support task evaluation. Neither split is used to construct or select modes.

W2/Q2. Stable teacher biases and domain-specific thresholds

Reliability measures agreement across calibration splits. We agree that stable but incorrect biases can score highly and be retained. Before student training, we assessed target quality on 1,500 examples from the mathematics training set, grouped by teacher-answer correctness. We report mean reference-answer NLL per response token.

Teacher answerCountStudent NLLTeacher NLLFiltered NLL
Correct1,2422.8020.66670.5463
Incorrect2582.5330.67370.6347

Filtered targets have lower mean NLL than both the student and original teacher in both groups. On teacher-incorrect examples, filtering lowers NLL relative to the teacher by 0.0390 per token, indicating higher mean reference-answer log probability.

We compared the global threshold rmin⁡=0.5r_{\min}=0.5 with domain-specific quantile thresholds:

rmin⁡,d=Q1−p({ρd,k}k=1Kd).r_{\min,d}=Q_{1-p}\left(\{\rho_{d,k}\}_{k=1}^{K_d}\right).

Here, pp is the fraction of all candidates passing the global threshold, and Q1−pQ_{1-p} is the empirical quantile of the KdK_d reliability scores in domain dd. This approximately matches overall candidate pass rates. Other settings are fixed.

ThresholdAvg. accuracyAvg. generation score
Global70.2629.85
Domain-specific70.3529.83

Domain-specific thresholds raise accuracy by 0.09 pp and lower generation score by 0.02 points. Results are similar, so we retain the global threshold as the default.

W3. Dynamic targets and training behavior

Selected directions remain fixed by default, while their residual coefficients are recomputed as the student changes. We agree that each update need not reduce residuals along every selected direction. On Qwen3, we measured aggregate selected-direction residual energy on the same calibration data at initialization and after training.

DomainFinal/initial residual energy (range across seeds)
Math0.038–0.051
Science0.0482–0.0529
Summarization0.0204–0.0228
Translation0.0183–0.0190

Final energy falls to 1.8%–5.3% of its initial value across the four domains, showing reduced teacher–student prediction differences in the selected subspace on these data.

Q1. Probe size, sampling sensitivity, and periodic reselection

We fix the calibration bank, student initialization, projection seed, and selection settings and compare 4, 8, and 16 probe examples per domain. We report the mean pairwise Jaccard similarity between selected sets obtained from independent probe draws, alongside downstream task performance.

Probes/domainMean JaccardAvg. accuracyAvg. generation score
40.79070.2629.85
80.81570.5529.89
160.81170.8230.11

Four probes yield a mean Jaccard of 0.790 under a fixed calibration bank. Increasing to 16 improves accuracy by 0.56 pp and generation score by 0.26 points. Four probes are a practical default in this setting, while larger probe sets offer further gains.

We also compared one-time selection with periodic reselection on Qwen3.

Mode selection scheduleAvg. accuracyAvg. generation scoreTime (h)
Once before training70.2629.854.07
Reselect every 20% of training70.5529.985.13
Reselect every 10% of training70.7530.126.53

Reselecting every 10% improves accuracy by 0.49 pp and generation score by 0.27 points. Runtime rises from 4.07 to 6.53 hours (60%). Given this trade-off, we retain one-time selection as the default and report the gains and runtime of reselection.

The revision clarifies data splits and target updates and includes these experimental settings and results. We also provide plotting scripts and data for the main-text figures and source files for the model diagram to support verification and redrawing.

The authors’ response has addressed my concerns. I will maintain my current score.

AuthorsOpenReview ↗

Thank You for Your Feedback

Dear Reviewer QufD,

Thank you for your follow-up and for confirming that our response has addressed your concerns. We appreciate your time and thoughtful review.

Best regards,

The Authors of Submission 21527

Reviewer_nJHCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies multi-teacher distillation with domain-specialized teachers. The authors propose a new multi-teacher distillation strategy that decomposes teacher-student prediction different with SVD, takes the main directions to filter teacher supervision during training. Experiments show slight improvement over baselines.

Critical strengths
  1. The proposed method is overall better than the compared methods.
  2. The paper includes an ablation study.
Critical weaknesses
  1. The presentation is poor. Most terms are introduced before being defined; the wording is overly complex and inaccurate, and in my opinion, many, if not most, sentences are not understandable. It seems heavily AI-assisted if not directly generated. For example:
  • l.048 joint and separate SoftKD are not defined.
  • Figure 1 is extremely unclear. Is it separate SoftKD? Then what is single? What is oracle routing here? The fact that each input is given the right teacher?
  • l141. What is a correction mode? Not defined at this stage
  • l147. “Euclidean update cost in LoRA parameter space”. What is that supposed to mean? And we use LoRA now? That was never mentioned in the paper before.
  • l148. What is a residual mode?
  • l149. What is a mode standalone cost, let alone a mode?
  • Domain is also never defined, same as the tackled problem at hand or the way the distillation is done.
  • What is a calibration context?
  • Residual is used in the manuscript many times before but is defined as late as Section 4.
  1. While the method shows improvement, I do not see any motivation for the proposed strategy.
  2. The compared methods in the experimental section are mostly not multi-teacher distillation methods. Therefore, the comparison looks inadequate. More than that, the performance of the proposed methods are barely above softKD.
Questions for authors
  1. Why are most baselines generic distillation methods rather than methods designed for multi-teacher distillation?
  2. Similarly, why are the multi-teacher methods discussed in the related work missing from the experiments?
  3. What are the motivations behind using residual mode?
  4. How much do the results depend on domain order and composition?
  5. What are the limitations of the approach?
4 comments
AuthorsOpenReview ↗

Response to Reviewer nJHC

Thank you for your feedback on the presentation.

W1. Presentation and terminology. The original AI Use Statement discloses AI-assisted language polishing and editing. We clarified Fig. 1 and introduced SoftKD settings and modes before the cost formulation.

  • Joint/separate SoftKD, oracle routing, and Fig. 1. Joint SoftKD trains one shared student on two domains; separate SoftKD trains one per domain. Original L48–50 describe matched initialization and exposure, with teachers assigned by known domain labels. Prior multi-teacher work uses this setup and the term oracle routing [1, 2]. Fig. 1(a) now identifies separate SoftKD trajectories; Fig. 1(b) marks separate SoftKD as the zero-line reference. We will clarify routing in the caption.
  • Residual modes. Teacher–student residuals represent prediction differences [3]. Original L135–137 define modes as residual directions over calibration contexts and refer to Sec. 4 for construction. L141 and L148 follow this definition.
  • Update cost and LoRA. Original L143–146 and Eq. (1) define CC as the minimum update cost for meeting selected targets. L149 gives standalone cost C({i})C(\{i\}); L147–148 explain MM. For trainable LoRA parameters θ\theta and M=IM=I, the cost is 12∥Δθ∥22\tfrac12\|\Delta\theta\|_2^2. LoRA connects this definition to the training parameter space.
  • Domains. Domain is established in domain adaptation [4]. Original L32–44 identify mathematics, science, translation, and summarization and the goal of integrating their teachers' capabilities in one student.
  • Calibration contexts. Each contains the input and answer prefix before the predicted token [5]. We use them to construct modes and estimate statistics.

W2/Q3. Motivation. The original abstract (L13–18) asks why joint distillation can trail separate distillation with correct routing and matched exposure. L48–55 and Fig. 1 show the gap. L57–70 and Sec. 3 analyze update reuse and interference, motivating selection by additional update cost given retained corrections.

Teacher corrections can include changes that are costly to learn together. Residual modes split prediction differences into selectable parts. JFMD builds targets from useful parts with lower added cost. Equal standalone costs can yield different joint costs depending on update reuse.

W3/Q1–2. Baselines. Baselines share student, teacher, routing, and task settings, retaining their own objectives and sampling. We added MOPD [1], an on-policy multi-teacher method using student-generated trajectories, on LLaMA.

MethodAvg. accuracyAvg. generation score
SoftKD30.8425.18
MOPD32.1725.33
JFMD34.6026.82

JFMD gains 2.43 pp in accuracy and 1.49 points in generation score over MOPD (Appendix D.1, Table 8). Gains over SoftKD span all three families:

Gain over SoftKDQwenLLaMAGemma
Accuracy (pp)+3.00+3.76+1.32
Generation score (points)+1.28+1.64+0.78

Q4/Q5. Domain order, composition, and limitations. Training uses mixed-domain data; mode-selection coverage processes domains in a fixed order. Teacher–student gap closure is

G=sdistilled−sinitialsteacher−sinitial,G=\frac{s_{\mathrm{distilled}}-s_{\mathrm{initial}}}{s_{\mathrm{teacher}}-s_{\mathrm{initial}}},

The scores denote the distilled student, initial student, and teacher. We report GG as percentages for each domain pair. JFMD improves closure over Joint SoftKD by 1.02–5.16 pp across all six pairs.

Domain pairJoint SoftKD GG (%)JFMD GG (%)Gain (pp)
Math + Science45.4547.06+1.61
Science + Summarization72.7176.52+3.81
Science + Translation70.4373.49+3.06
Math + Summarization52.5654.40+1.84
Math + Translation52.2853.30+1.02
Summarization + Translation79.4384.59+5.16

JFMD adds calibration and response-gradient computation before training. Initial cost rankings can drift; on Qwen3, we compared them with rankings re-estimated during training.

Training progress20%40%60%80%
Spearman correlation with initial ranking0.910.830.800.78

We provide Python plotting scripts and corresponding data for the main-text figures, plus editable source files for the model diagram, for verification and redrawing. We simplified organization and wording to address your concerns.

[1] Ma et al. MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. 2026.

[2] Gao et al. Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation. 2026.

[3] Li et al. ResKD: Residual-Guided Knowledge Distillation. IEEE TIP, 2021.

[4] Daumé III. Frustratingly Easy Domain Adaptation. ACL, 2007.

[5] Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR, 2024.

AuthorsOpenReview ↗

We hope to receive your feedback.

Dear Reviewer nJHC,

Thank you for your feedback on our submission. As the discussion period draws to a close, we would like to follow up on our response and revisions. We have clarified the presentation and motivation, added a comparison with MOPD, and provided further analysis of domain composition and the limitations of our approach. We have also provided the data and plotting code for all figures in the paper, along with the original editable source files for the model diagram.

We would welcome your feedback on these updates, particularly on any concerns that remain unresolved. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly. Thank you for your hard work and support.

Best regards,

The Authors of Submission 21527

Reviewer_nJHCOpenReview ↗

Thank you for your rebuttal

I thank the authors for their reply and for addressing my comments.

However, some of my concerns still remain.

W1.

"Original L135–137 define modes as residual directions over calibration contexts". That is true, and at this exact time calibration contexts are also not defined. Therefore, residual modes AND calibration contexts are not defined at this stage of the paper. These are not standard concepts that the reader is supposed to know.

Update cost and LoRA. The problem is not whether LoRA makes sense. The problem is that we never either 1. introduces LoRA in the paper or 2. specify why we are using LoRA. This implies that we are doing parameter-efficient fine tuning. Why? This is certainly not the standard way to do distillation; so the paper should justify the setting. Is this method tied to LoRA, can it be used in more generic distillation settings? This question is unclear and ignored.

W2/Q3.

While the identified problem makes sense; my problem is the lack of intuitive motivation behind the proposed mechanisms. It is not clear where this sits in the current literature, what ideas it build on, what are the assumptions. These are the actual valuable takeaways for the future potential readers.

I still do not see any discussions on the limitations, future work, etc. Something to put the work in perspective. I think the work is valuable experimentally, but, personally, this work in this current stage is not understandable by a human. It needs serious re-writing with clear intuitive motivations behind the proposed mechanism, as well as contextualisation when comparing with previous work and finally limitations.

Therefore, despite other reviews, I will maintain my score of 1.

AuthorsOpenReview ↗

Response to Remaining Concerns and Clarification of Revisions

Thank you for your follow-up and for recognizing the work’s experimental value.

W1. Residual modes and calibration contexts.

Original L135–137 state: “Our units of analysis are residual modes, directions in the teacher–student residual over calibration contexts. Their construction is given in Section 4”. This introduces residual modes and explicitly points to their construction. Both precede the questioned L141 and L148. The referenced Section 4, L198–199, introduces the calibration-context set used in that construction. The issue is therefore whether this initial explanation is sufficient, rather than whether the concept has been introduced.

Our initial W1 response defines a calibration context as the input and answer prefix preceding the predicted token. The revision addresses presentation order: Section 3.1, L131–133, introduces modes through prediction differences without relying on “calibration contexts”; Section 4, L204–210, defines these contexts before constructing modes. These revisions preceded your follow-up. If these passages remain insufficient, please identify the specific missing explanation.

W1. LoRA: introduction, rationale, and generality.

These are three distinct questions.

First, LoRA was specified in the original manuscript. Appendix C.4 provides its configuration. An earlier main-text explanation would help, but this setting was not absent from the paper.

Second, LoRA is an established parameter-efficient choice in LLM distillation. ToDi uses LoRA for its TinyLLaMA student [1], while DistiLLM-2 explicitly uses LoRA throughout its experiments for training efficiency [2]. These precedents do not support characterizing LoRA-based distillation as an unusual setting. We likewise use LoRA to reduce trainable parameters and training-resource requirements.

Third, JFMD’s conditional-cost formulation does not require LoRA-specific parameterization. In Eq. (1), the response gradient is

$ g_i=\nabla_{\theta}J_i(\theta_0). $

The formulation evaluates the update needed to meet correction targets in the chosen trainable parameter space. In principle, it also applies to other adapters or full-parameter fine-tuning by recomputing response gradients and costs in that space. Parameterization affects the costs but does not restrict the selection principle to LoRA.

W2/Q3. Mechanism motivation, prior work, and assumptions.

The original manuscript provides mechanism-level intuition, beyond motivating the problem. Original Fig. 2(a) illustrates shared, independent, and opposing update requirements; L149–151 explain why equal standalone costs can yield different joint costs. Our initial W2/Q3 response also explains this dependence on update reuse.

Revised Section 3.1, L120–129, develops this into a two-dimensional example. Each correction individually costs 0.5. Two corrections jointly cost 0.5 if one update satisfies both, or 1 if they require independent updates. This directly motivates evaluating additional cost given the corrections already selected.

The mechanism follows this logic: residual modes provide selectable correction directions; energy and cross-sample reliability assess their value; response gradients describe their update requirements; conditional cost accounts for interactions with retained corrections.

The original introduction and related work connect this approach to multi-teacher distillation, task interactions, and gradient conflicts. Revised Section 2, L95–107, clarifies the distinction between modifying updates during training and selecting correction directions before distillation. Section 3.1 explicitly states the local first-order approximation.

Your follow-up does not identify why the original illustration or revised example is insufficient, or which assumption or connection remains unclear. Identifying this specific gap would enable a targeted clarification.

Limitations and future work.

“I still do not see any discussions on the limitations” does not account for our initial Q4/Q5 response. That response explicitly identifies additional calibration and response-gradient computation and possible ranking drift, and reports an analysis of rankings re-estimated during training.

Revised Appendices D.5–D.6 examine ranking changes, periodic reselection and its computational cost, and probe-size and sampling sensitivity. Fixed selection may become less suitable during training, while reselection adds computation. The local approximation also limits conclusions about nonlinear training trajectories. These limitations motivate more efficient adaptive reselection.

We appreciate your feedback. We respectfully ask that you assess the specific revised passages and analyses above and distinguish unresolved concerns from those already addressed.

[1] ToDi: Token-wise Distillation via Fine-Grained Divergence Control. EMNLP 2025. Oral

[2] DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML 2025. Oral

Reviewer_DS57OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why jointly distilling several domain-specific teachers into one student can yield worse performance than distilling them into separate students, despite correct routing and matched per-domain exposure. JFMD decomposes teacher–student output residuals into modes and selects corrections using their reliability-adjusted energy and estimated additional update cost given the selected set. The selected modes filter teacher supervision during training. Experiments cover three model families and eight benchmarks. The main contribution is to make interactions among teacher corrections an explicit consideration in supervision selection.

Critical strengths
  • The problem is practically relevant, and the motivation is clear. Comparing joint and separate distillation under correct routing and matched exposure helps establish that the observed gap deserves further study.

  • JFMD evaluates each correction's additional cost given the selected supervision. This provides a reasonable way to account for both update sharing and interference when choosing what to distill.

  • The evaluation is fairly comprehensive, and the reported gains across three model families are encouraging. Controlled training and matched-count ablations make the case for conditional over standalone-cost selection more convincing.

Critical weaknesses
  • The cost estimates rely on gradients at initialization, while selection remains fixed during training. It is less clear how well these estimates describe mode interactions as the student changes.

  • The stability analysis supports the reliability score, but its separate contribution to task performance remains less clear. An ablation removing reliability weighting while retaining conditional cost would help clarify this.

  • The proposed explanation seems plausible, but the association across six domain pairs does not establish how much of the joint-distillation gap is explained by correction interactions. The evidence for this mechanism remains suggestive.

Questions for authors

Refer to Weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer DS57

Thank you for recognizing our motivation, conditional-cost selection, and evaluation. We address your three concerns below.

W1. Initial cost estimates during training

On Qwen3-1.7B, we compared the initial cost ranking with rankings re-estimated during training.

Training progress20%40%60%80%
Spearman correlation with initial ranking0.910.830.800.78

Agreement declines but remains substantial at 80% of training. We also evaluated periodic mode reselection. Parentheses show changes from one-time selection.

Mode selection scheduleAvg. accuracyAvg. generation scoreTotal time (h)
Once before training70.2629.854.07
Reselect every 20% of training70.55 (+0.29 pp)29.98 (+0.13)5.13 (+26%)
Reselect every 10% of training70.75 (+0.49 pp)30.12 (+0.27)6.53 (+60%)

The initial estimates retain useful ranking information, while periodic reselection improves both task metrics at additional computational cost. We retain one-time selection as the default given this trade-off.

W2. Contribution of reliability weighting

On Qwen3-1.7B, we replace candidate value ρiei\rho_i e_i with eie_i, keeping the reliability threshold, conditional cost calculation, and retained mode count per domain fixed.

Candidate value (conditional cost retained)Avg. accuracyAvg. generation score
Energy eie_i68.6028.77
Reliability-weighted energy ρiei\rho_i e_i70.2629.85
Gain from weighting+1.66 pp+1.08 points

This comparison isolates the contribution of continuous reliability weighting under the same candidate eligibility rules, conditional-cost mechanism, and supervision size.

W3. Correction interactions and the joint-distillation gap

Across six domain pairs, we compare standalone-cost selection (Singleton) with conditional-cost selection, holding candidate values, per-domain mode counts, and training settings fixed. Standalone cost evaluates each candidate in isolation; conditional cost evaluates its additional cost given the selected set. Teacher–student gap closure is defined as:

G=sdistilled−sinitialsteacher−sinitial,G=\frac{s_{\mathrm{distilled}}-s_{\mathrm{initial}}}{s_{\mathrm{teacher}}-s_{\mathrm{initial}}},

where the three scores correspond to the distilled student, initial student, and teacher. Conditional-cost gain is the increase in gap closure over Singleton. We normalize this gain and the gain of full JFMD over Joint SoftKD by the original joint-distillation gap:

Rcond=GJFMD−GSingletonGSeparate−GJoint×100%.R_{\mathrm{cond}}=\frac{G_{\mathrm{JFMD}}-G_{\mathrm{Singleton}}}{G_{\mathrm{Separate}}-G_{\mathrm{Joint}}}\times100\%.

Rtotal=GJFMD−GJointGSeparate−GJoint×100%.R_{\mathrm{total}}=\frac{G_{\mathrm{JFMD}}-G_{\mathrm{Joint}}}{G_{\mathrm{Separate}}-G_{\mathrm{Joint}}}\times100\%.

The table reports GG as percentages for each domain pair. Separate SoftKD is the separate-distillation reference. Conditional-cost gains are reported in percentage points.

Domain pairSeparate SoftKD GG (%)Joint SoftKD GG (%)Singleton GG (%)JFMD GG (%)Cond. gain (pp)RcondR_{\mathrm{cond}}RtotalR_{\mathrm{total}}
Math + Science53.9145.4546.2447.06+0.829.7%19.10%
Science + Summ.80.8672.7174.3276.52+2.2027.0%46.80%
Science + Transl.78.2370.4370.6873.49+2.8136.0%39.19%
Math + Summ.57.5252.5653.0154.40+1.3928.0%37.20%
Math + Transl.54.8952.2852.5653.30+0.7428.4%39.14%
Summ. + Transl.81.8479.4380.8884.59+3.71153.9%213.71%

Conditional-cost selection improves gap closure by 0.74–3.71 pp across all six pairs. These gains correspond to 9.7%–36.0% of the original joint-distillation gap in five pairs and 153.9% for summarization plus translation. This controlled comparison quantifies how much of the gap can be closed by using correction interactions to select supervision.

We added these settings and results and streamlined the revised manuscript. We also provide plotting scripts and data for the main-text figures and source files for the model diagram to support verification and redrawing. We hope these additions address your concerns.

Thank you for the detailed response. The new stability analysis and reselection experiments address W1, and the reliability-weighting ablation with conditional cost retained addresses W2. Regarding W3, the new controls support conditional selection. However, it remains unclear how much the gains reflect within-domain benefits versus alleviating the additional difficulties of joint training. Conditional cost captures both within-domain and cross-domain interactions. Applying JFMD to separate distillation would help distinguish these effects. I maintain my rating of 3 (Weak acceptance).

AuthorsOpenReview ↗

Clarifying Within-Domain and Joint-Training Gains

Thank you for your detailed suggestion. We agree that applying JFMD to separate distillation helps clarify the contributions of within-domain learning and cross-domain interactions to its gains. Following your suggestion, we evaluated SoftKD and JFMD separately in each domain. The separate and joint runs use the same student initialization, domain teachers, and per-domain training exposure, as well as the same evaluation and checkpoint-selection protocol.

DomainSoftKDJFMDGain
Math14.5014.73+0.23
Science64.3064.80+0.50
Summarization35.1035.53+0.43
Translation36.6936.85+0.16

Scores use the original task metrics on a 0–100 scale; gains are task-score points.

JFMD improves over SoftKD in all four domains, indicating that within-domain benefits are present. To compare these improvements with the gains under joint training, we use the teacher-gap closure metric G defined in our previous response, keeping the initial-student and teacher references and domain-pair aggregation unchanged. For each domain pair, JFMD’s gain over SoftKD is:

Δr=100(GJFMDr−GSoftKDr),r∈{Separate,Joint}.\Delta^{r}=100\left(G_{\mathrm{JFMD}}^{r}-G_{\mathrm{SoftKD}}^{r}\right),\qquad r\in\{\mathrm{Separate},\mathrm{Joint}\}.

We then subtract the separate-distillation gain from the joint-distillation gain:

E=ΔJoint−ΔSeparate.E=\Delta^{\mathrm{Joint}}-\Delta^{\mathrm{Separate}}.

A positive E means that JFMD improves more under joint training, reducing the separate–joint performance gap relative to SoftKD.

Domain pairSeparate gainJoint gainExtra gain
Math + Science+0.75+1.61+0.86
Science + Summ.+1.17+3.81+2.64
Science + Transl.+0.61+3.06+2.45
Math + Summ.+1.04+1.84+0.80
Math + Transl.+0.48+1.02+0.54
Summ. + Transl.+0.90+5.16+4.26
Mean+0.83+2.75+1.92

All values are teacher-gap closure percentage points. Separate and Joint gains are measured relative to their respective SoftKD baselines; Extra gain is Joint gain minus Separate gain. Means are over the six domain pairs.

The gains are larger under joint training for all six domain pairs, averaging 2.75 pp, compared with 0.83 pp under separate distillation. This corresponds to an average 1.92 pp reduction in the separate–joint gap.

These results suggest that JFMD improves within-domain learning, with greater benefits associated with addressing cross-domain interactions during joint training. We have added these results and the corresponding analysis to Appendix D.3. Thank you for suggesting this comparison, which helps clarify the sources of the gains.

Comments on the paper

2 comments
Area_Chair_MJM9OpenReview ↗

Discussion!

Dear reviewers,

It seems that the authors have carefully responded to your reviews.

Please take your time to go over them and let authors know what you think, to your earliest convenience.

Best,
AC.

AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

Dear AC and Reviewers,

Thank you for the constructive discussion. We summarize the main outcomes below.

Recognized contributions. DS57 and QufD recognize the importance of the joint-distillation problem and the value of considering cooperation and interference among teacher corrections. DS57 highlights the evaluation across three model families, while nJHC explicitly acknowledges the work’s experimental value.

Reviewer feedback.

  • QufD confirms that our response has addressed their concerns and maintains Weak acceptance.
  • DS57 confirms that W1 and W2 are addressed, recognizes that the additional controls support conditional selection, and maintains Weak acceptance. We subsequently provide the requested comparison separating within-domain benefits from joint-training gains.
  • nJHC maintains rejection, with concerns about presentation, motivation, scope, and limitations. Our follow-up provides specific manuscript locations and corresponding clarifications.

Main additions. We strengthen the evidence with:

  • Separate versus joint distillation: JFMD’s larger gains under joint training reduce the separate–joint gap by an average of 1.92 percentage points across six domain pairs.
  • Additional multi-teacher baseline: JFMD exceeds MOPD on LLaMA by 2.43 pp in accuracy and 1.49 points in generation score.
  • Controlled analyses: reliability-weighting ablation, ranking stability and reselection costs, probe sensitivity, and target-quality analysis.

Manuscript revisions. We reorganize the presentation, expand the original intuition into a concrete two-dimensional example, improve definition order, clarify connections to prior work and data separation, and explain LoRA’s role and the method’s limitations. Supporting analyses and reproducibility materials are provided.

We respectfully ask that the final assessment consider these contributions, reviewer-confirmed resolutions, and additional evidence.

The Authors of Submission 21527

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.