- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work proposes representing experts in MoE models as sparse linear combinations of a shared dictionary of matrices with per-layer transformations. Experiments on synthetically generated weight matrices report improved weight reconstruction for the proposed method over generic independent expert compression techniques.
Critical strengths
- The specific proposed inter-layer expert weight sharing method appears to be novel, although not to the extent claimed by the authors.
Critical weaknesses
- Unrealistic synthetic evaluation: The proposed method is only tested on synthetically generated weight matrices. It does not appear that the authors apply AtomShare to an actual pretrained model. Each expert consists of a 14x14 matrix; whereas, the vast majority of experts in MoE models today use SwiGLU modules. This is not a realistic setting. Further, Appendix A.4 appears to describe constructing the synthetic experts from shared latent directions that would inherently benefit the proposed compression method:
“A stack-wide set of twenty-four latent directions is shared across every depth and carried into each layer by an orthogonal frame; each expert is a short combination of them plus a per-expert residual, and the fraction of expert energy the combination carries is the shared-energy fraction, 0.548 for the primary, second and third stacks.”
- Weak baselines and poor experimental design: The baselines are generic compression methods rather than SOTA prior work. 2:4 sparsity for example should be applied via a reasonable algorithm such as SparseGPT, Wanda etc. The authors also fail to cite or compare against other weight-sharing compression algorithms such as MoBE (https://arxiv.org/abs/2508.05257). The main evaluation metric used is not a language modelling nor downstream evaluation task but rather based on the sum of squared weight reconstruction errors. This is not a particularly informative measure that supports the claims being made.
- Materialization cost: Requiring 9212 MACs to materialize a single 14x14 expert weight is very impractical.
Questions for authors
- How does AtomShare compare with existing MoE / LLM compression techniques on real pretrained models and benchmarked on downstream tasks?