- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies cross-layer Meta Expert sharing combined with layer-private Local Experts, varying sharing ratios, activation allocation, boundary layers, and shared sub-pools. It reports improved training loss across model scales and six-task downstream gains for MR75 + Loop2. Reduced-expert configurations match the baseline's training loss with approximately 70–80% of its routed-expert budget.
Critical strengths
- Systematic architectural exploration. The study separates sharing proportions, activation allocation, boundary-layer treatment, and sub-pool organization across four training scales, providing useful evidence about sharing topology.
- Positive downstream results. After 200B training tokens, MR75 + Loop2 improves all six reported tasks over Vanilla; MMLU rises from 40.87% to 41.75%, and C-Eval from 43.54% to 44.87% (Table 1).
Critical weaknesses
- Generalization and efficiency claims exceed the evaluation. The reported loss averages the final ten training steps, without held-out language-modeling loss or perplexity. Table 1 uses equal expert budgets and therefore does not validate downstream equivalence after expert reduction. Equal top-8 activation also does not establish equal total FLOPs when routing candidate pools differ. Missing benchmarks alone do not establish selective reporting.
- The mechanism analysis needs validation. Appendix A.2 computes covariance similarity rather than the stated linear CKA, undermining the claimed metric properties. Equations (7)–(8) describe training-update interactions, not direct evidence of improved retrieval efficiency; the SGD-based update interpretation also requires reconciliation with the Muon experiments.
Questions for authors
- Beyond the stated capability coverage, how was the six-task suite selected? Does Table 1 include all completed downstream evaluations? If additional tasks were evaluated, please report their complete results.
- Are held-out cross-entropy or perplexity and reduced-expert downstream results available? Please specify the evaluation data and protocol, or limit the 70–80% budget claim to training-loss equivalence.
- Which similarity formula generated the redundancy curves, and do the conclusions survive correction or justification? What evidence supports improved retrieval beyond shared-parameter training interactions, particularly under Muon?
- Do the FLOP estimates include router scoring and selection for each configuration? Please provide full accounting or qualify the comparison as matching executed-expert computation.
9 comments
2. Meta Expert's parameter savings preserve downstream performance
Table 1 uses equal expert budgets and therefore does not validate downstream equivalence after expert reduction.
We have supplemented long-training experiments with reduced expert counts: Meta Expert matches the baseline with 37.5% fewer experts in downstream evaluation, while the Vanilla MoE degrades significantly under the same reduction. Please refer to the second point of our common response to all reviewers, or to Appendix A.6 for details.
3. The redundancy curves come from a valid PSD kernel, the mislabeling does not affect the conclusion
Appendix A.2 computes covariance similarity rather than the stated linear CKA. Which similarity formula generated the redundancy curves, and do the conclusions survive correction?
We thank the reviewer for this careful observation. The kernel that generated all redundancy curves is the cosine similarity between vectorized centered second-moment matrices, Kij=cos(vec(Yic⊤Yic),vec(Yjc⊤Yjc)) , as defined in Eqs. (9)–(10).
We acknowledge that we incorrectly referred to this kernel as "linear CKA". The confusion arose from an equivalent formulation of linear CKA as the cosine similarity between vectorized Gram matrices: in that equivalence, the Gram matrices are the sample-space Grams YY⊤ , whereas our feature map uses the feature-space second moment Y⊤Y . Since both are second-moment statistics with identical vec-cosine forms, and since our kernel happens to share linear CKA's invariance to orthogonal transformations and isotropic scaling, we mistakenly identified it as linear CKA.
Importantly, this is purely a naming error rather than a computational one. The kernel is a valid positive semi-definite similarity measure, all reported curves and redundancy statistics are exactly the outputs of Eqs. (9)–(10), and therefore no result or conclusion is affected. In the revision, we have renamed the kernel accordingly and removed the incorrect CKA reference. We have additionally verified that recomputing the redundancy curves with standard linear CKA (Kornblith et al., 2019) yields the same qualitative conclusions.
1. The reported loss is effectively a held-out estimate and remains stable across random conditions
We clarify that our pretraining corpus is streamed without data repetition and contains abundant unique samples, so each token is seen exactly once during training. The loss over the final training steps is therefore computed on tokens the model has never fitted, making it effectively equivalent to a held-out estimate — a standard practice in LLM pretraining. We would add this clarification to the future revision if possible.
Regarding the stability of the comparison, we have supplemented multi-seed validation showing that the relative variance across seeds is an order of magnitude smaller than our improvement (see Appendix A.8 and the second point of our common response). The downstream results under reduced expert counts are provided in Appendix A.6.