- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Latent-MOPD consolidates multiple domain specialists into a single student by augmenting domain-routed, on-policy token distillation with late-layer hidden-state supervision from the same teacher. A crossfade schedule gradually shifts supervision from representations to token predictions. The teachers remain frozen, and inference uses only the student. The paper evaluates three settings: same-lineage 1.5B teachers distilled into a 1.5B student, larger Qwen-based 7B teachers distilled into the same student through a learned projector, and distillation from a parameter-merged initialization. The default same-lineage configuration outperforms token-only distillation on all nine reported benchmarks and exceeds the per-benchmark best teacher on five. The paper also presents layer and schedule comparisons and a batching control showing that, in an all-layer representation-only configuration, domain-pure updates are substantially more stable than interleaved updates.
Critical strengths
1.Broad and concrete multi-domain gains. Under the shared 62-step same-lineage protocol, Latent-MOPD outperforms token-only distillation on all nine benchmarks, including AIME24 (46.0→50.8), LiveCodeBench-easy (68.3→73.1), and MuSR (50.7→52.4). The 1.5B student also exceeds the best individual 1.5B teacher on five benchmarks. This is consequential because it directly supports the paper's central goal of consolidating complementary domain capabilities into one deployable student, with no teacher required at inference.
2.Useful evidence beyond a single main-table comparison. The gains extend to larger teachers and a parameter-merged initialization. In addition, the controlled batching experiment shows a large stability difference when only domain grouping within an update is changed: domain-pure training reaches 91.7/56.7 on MATH-500/GYM-200 after 62 steps, versus 43.2/12.8 for interleaved training. This provides a practically useful finding about organizing representation-level multi-teacher supervision.
Critical weaknesses
1.The complete recipe is supported more clearly than its individual design choices. Crossfade simultaneously raises the token-loss weight and lowers the representation-loss weight, so its additional benefit cannot currently be attributed to either change or their interaction. Likewise, the last-1/last-3 comparisons do not distinguish the effect of supervising more layers from the effect of selecting those particular depths. This matters because the paper presents layer selection and supervision scheduling as central elements of the method, but their specific rationale is not yet fully isolated. 2.The incremental value of multiple teachers is not directly established in the larger-teacher setting. AceReason is the strongest teacher on five of the six Table 2 benchmarks, yet there is no matched result using AceReason alone with the same joint-supervision recipe. Therefore, the experiments establish that the proposed recipe improves over the tested multi-teacher baselines more directly than they establish that complementary supervision from multiple teachers is necessary. This limits the interpretation of the paper's central multi-teacher claim.
Questions for authors
1.How were the supervised layers selected? Can the authors clarify whether the last-1/last-3 gains reflect the number of supervised layers or the suitability of those particular depths? 2.Given that constant-weight joint supervision already helps, what explains the additional crossfade gain: gradually removing the representation constraint, increasing token supervision, or their interaction? Can the current results distinguish these possibilities? 3.In Table 2, what additional benefit do the other two 7B teachers provide beyond AceReason? Is a result using AceReason alone under the same prompts, budget, and joint-supervision setup available? 4.During domain rotation, did improvements in one domain coincide with temporary degradation in another? The math and logic curves show overall gains; observations of math, code, and logic at matched checkpoints would help clarify cross-domain interference.