- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the joint-distillation gap: a shared student may learn less from multiple specialized teachers than separate students, even under correct teacher routing and matched domain exposure. The proposed method, Jointly Feasible Mode Distillation (JFMD), decomposes teacher–student residuals into modes and selects corrections by their reliability-adjusted value and the projected cost of adding them to the current set. The method is evaluated across three model families and eight benchmarks, with additional analyses of student capacity, routing errors, and held-out stability.
Critical strengths
- The paper addresses a meaningful problem by examining the relationships among teacher correction signals. It extends the consideration of teacher strength to the question of which corrections are suitable for joint learning.
- This work accounts for both cooperation and conflict. Similar update requirements can share updates, unrelated requirements incur separate costs, and opposing requirements may not be simultaneously satisfiable. This provides a richer perspective than examining negative gradient alignment alone.
Critical weaknesses
- The paper specifies the purposes and sizes of the calibration and probe datasets, but does not clearly explain their relationships to the training, validation, and test sets. These data determine the correction modes and their selection, and therefore influence the final model even when they do not directly update the student parameters.
- A teacher’s systematic biases may persist across different calibration samples. A stable residual direction therefore indicates repeatability, but does not by itself establish that the corresponding correction is correct or beneficial.
- As the student changes, the filtered residuals and distillation targets are recomputed. Although this mechanism has a reasonable feedback rationale, it does not guarantee that every update reduces the errors along all selected directions.
Questions for authors
- Each domain uses only a small number of probe examples to estimate response gradients. Is this sample size sufficient? How much does the selected mode set change when a different probe sample is used? As a potential extension, would periodically reselecting the directions improve the method?
- Could stable but incorrect teacher biases be retained by the selection procedure? As a potential extension, should different domains use different reliability thresholds?
3 comments
The authors’ response has addressed my concerns. I will maintain my current score.
Thank You for Your Feedback
Dear Reviewer QufD,
Thank you for your follow-up and for confirming that our response has addressed your concerns. We appreciate your time and thoughtful review.
Best regards,
The Authors of Submission 21527
Response to Reviewer QufD
Thank you for recognizing the importance of the problem and the value of considering cooperation and conflict among teacher corrections.
W1. Calibration, probe, and data splits
Calibration and probe are part of model fitting. We sample 256 training examples per domain for calibration to construct modes and estimate reliability. By default, four examples per domain are drawn from this pool as probes to estimate response gradients and select modes. Thus, probe is a subset of calibration, which is a subset of training. These examples remain in the training sampling pool.
Per domain, calibration uses up to 2,048 valid response-token contexts and probe up to 256, excluding prompt and padding positions. Validation supports checkpoint selection and training diagnostics; test data support task evaluation. Neither split is used to construct or select modes.
W2/Q2. Stable teacher biases and domain-specific thresholds
Reliability measures agreement across calibration splits. We agree that stable but incorrect biases can score highly and be retained. Before student training, we assessed target quality on 1,500 examples from the mathematics training set, grouped by teacher-answer correctness. We report mean reference-answer NLL per response token.
Filtered targets have lower mean NLL than both the student and original teacher in both groups. On teacher-incorrect examples, filtering lowers NLL relative to the teacher by 0.0390 per token, indicating higher mean reference-answer log probability.
We compared the global threshold rmin=0.5 with domain-specific quantile thresholds:
rmin,d=Q1−p({ρd,k}k=1Kd).
Here, p is the fraction of all candidates passing the global threshold, and Q1−p is the empirical quantile of the Kd reliability scores in domain d. This approximately matches overall candidate pass rates. Other settings are fixed.
Domain-specific thresholds raise accuracy by 0.09 pp and lower generation score by 0.02 points. Results are similar, so we retain the global threshold as the default.
W3. Dynamic targets and training behavior
Selected directions remain fixed by default, while their residual coefficients are recomputed as the student changes. We agree that each update need not reduce residuals along every selected direction. On Qwen3, we measured aggregate selected-direction residual energy on the same calibration data at initialization and after training.
Final energy falls to 1.8%–5.3% of its initial value across the four domains, showing reduced teacher–student prediction differences in the selected subspace on these data.
Q1. Probe size, sampling sensitivity, and periodic reselection
We fix the calibration bank, student initialization, projection seed, and selection settings and compare 4, 8, and 16 probe examples per domain. We report the mean pairwise Jaccard similarity between selected sets obtained from independent probe draws, alongside downstream task performance.
Four probes yield a mean Jaccard of 0.790 under a fixed calibration bank. Increasing to 16 improves accuracy by 0.56 pp and generation score by 0.26 points. Four probes are a practical default in this setting, while larger probe sets offer further gains.
We also compared one-time selection with periodic reselection on Qwen3.
Reselecting every 10% improves accuracy by 0.49 pp and generation score by 0.27 points. Runtime rises from 4.07 to 6.53 hours (60%). Given this trade-off, we retain one-time selection as the default and report the gains and runtime of reselection.
The revision clarifies data splits and target updates and includes these experimental settings and results. We also provide plotting scripts and data for the main-text figures and source files for the model diagram to support verification and redrawing.