acceptodds
Under review as a conference paper at ICLR 2027

AtomShare: One Shared Atom Set Across a Mixture-of-Experts Stack for Budget-Constrained Weight Compression

Abstract

A sparse mixture-of-experts stack stores one weight matrix per expert at every depth, and its stored size therefore grows with the expert count while only a few experts run per token. However, existing compressors fit each matrix in a basis of its own and price its error by how far that matrix moves; they therefore cannot reach the redundancy that recurs across the experts of a layer and they spend the budget against a distance the router never applies. We propose AtomShare, which stores a whole stack as one atom set shared across depths, carried into each layer by a norm-preserving map, with every expert held as a sparse code against that dictionary. AtomShare achieves this by fitting one atom set against every expert of the stack in each layer's recorded input geometry, by setting each unit's code length from the deviation it delivers at the layer output on the tokens the router sends it, and by dividing one stored-size budget across the layers by the deviation each layer's injection propagates to the end of the stack. Sharing the dictionary is what makes code length a single divisible pool, and the atoms, maps and codes are therefore fit together as one alternation rather than matrix by matrix. On an eight-layer, forty-eight-expert weight stack AtomShare reaches a retained score of 0.5483 against 0.5150 for the strongest per-expert baseline while holding fewer bits per parameter, and its compression ratio rises with the expert count, from 6.149 at eight experts to 8.385 at sixteen, where every per-expert store but one holds its own ratio fixed across that range. A mechanism ablation read at the routed output puts the recorded input geometry first: removing it raises the delivered deviation by 67 to 93 per cent.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts compression, shared dictionary learning, sparse weight coding, cross-layer parameter sharing, routing-aware error metric, budget-constrained bit allocation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_DbGWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes representing experts in MoE models as sparse linear combinations of a shared dictionary of matrices with per-layer transformations. Experiments on synthetically generated weight matrices report improved weight reconstruction for the proposed method over generic independent expert compression techniques.

Critical strengths
  • The specific proposed inter-layer expert weight sharing method appears to be novel, although not to the extent claimed by the authors.
Critical weaknesses
  • Unrealistic synthetic evaluation: The proposed method is only tested on synthetically generated weight matrices. It does not appear that the authors apply AtomShare to an actual pretrained model. Each expert consists of a 14x14 matrix; whereas, the vast majority of experts in MoE models today use SwiGLU modules. This is not a realistic setting. Further, Appendix A.4 appears to describe constructing the synthetic experts from shared latent directions that would inherently benefit the proposed compression method:

“A stack-wide set of twenty-four latent directions is shared across every depth and carried into each layer by an orthogonal frame; each expert is a short combination of them plus a per-expert residual, and the fraction of expert energy the combination carries is the shared-energy fraction, 0.548 for the primary, second and third stacks.”

  • Weak baselines and poor experimental design: The baselines are generic compression methods rather than SOTA prior work. 2:4 sparsity for example should be applied via a reasonable algorithm such as SparseGPT, Wanda etc. The authors also fail to cite or compare against other weight-sharing compression algorithms such as MoBE (https://arxiv.org/abs/2508.05257). The main evaluation metric used is not a language modelling nor downstream evaluation task but rather based on the sum of squared weight reconstruction errors. This is not a particularly informative measure that supports the claims being made.
  • Materialization cost: Requiring 9212 MACs to materialize a single 14x14 expert weight is very impractical.
Questions for authors
  • How does AtomShare compare with existing MoE / LLM compression techniques on real pretrained models and benchmarked on downstream tasks?
Reviewer_ZP8qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

AtomShare has developed a method for storing a multi-head attention (MoE) weight stack using one common set of atoms that are used throughout all of the depth levels. The householder reflector along with the geometry of the input to each individual layer carry this single set of atoms forward. A sparse representation of each expert is created based upon the atom set. The length of the code is determined by the amount of variance in the outputs of the routed token; while the number of bits allocated per-layer is based on the measurement of how much error is amplified at each level. When tested on synthetic stacks created by the researchers (8 layers x 48 experts of size 14x14), AtomShare achieved a retention rate of 0.5483 compared to 0.5150 for the best performing per-expert baseline, while requiring less computational resources. And AtomShare's compression ratios increased from 6.149 (with 8 experts) to 8.385 (with 16 experts).

Critical strengths
  • It appears that sharing redundancy among experts at different levels within a single stack-wide budget will be an area for further investigation.
  • Careful storage accounting includes all common elements such as terms, codes, and indices used by each expert; it also includes the MAC's required to recreate an expert.
Critical weaknesses
  • There were no true models used in this study. Experts were 14x14 linear mappings, routing was done based on the nearest center, and stacks were created using "shared latent directions across depths + residual," which was the same type of model that was assumed for this method and therefore made the results circular. Perplexity and downstream accuracy were not reported.

  • Issues with metrics. Gamma did not increase and the no-geometry version had a smaller gamma. The score used to measure performance favored geometrically aware methods and was negatively impacted by the zero storage baseline (0.407). Several other baselines were between 300-750 percent higher than the relative residual indicating there may be issues with either the implementation or the metrics being used.

  • Ablation studies do not support the story told. At each of the five operating points studied, uniform code lengths performed better (+0.009-0.019) than other options. Stack coupled allocations produced no difference in performance. The addition of the flip term did not produce differences in how allocations occurred. The optimal number of atoms was 20 but the authors chose to use 28.

  • Comparisons of scalability and methods. Approximately 74 percent of total storage is due to common overhead. It takes 47 times the amount of MAC's to build an expert when using low rank. None of the methods compared here share information between experts. Additionally none of these methods were evaluated under the same amount of available storage.

Questions for authors

Can you report perplexity and zero-shot accuracy on at least one real MoE?

For SwiGLU experts at realistic dimensions, how is GlG_l defined, and what do the atom refit and expert rebuild cost?

Reviewer_zGZVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

AtomShare represents expert weight matrices using a dictionary shared across experts and layers, together with layer-specific orthogonal maps and an input-geometry transform. The method alternates between sparse coding, dictionary updates, and map updates, then allocates code lengths using routed reconstruction errors and empirical layer-amplification estimates.

Critical strengths
  1. I think sharing a dictionary across experts and layers is a reasonable approach to reducing the storage overhead of independently represented experts. Compact orthogonal maps provide a potentially useful way to accommodate differences between layers.

2.The proposed storage account includes the dictionary, layer maps, geometry factors, coefficients, and support indices.

Critical weaknesses
  1. All evaluations use synthetic stacks. The primary setting contains eight layers of forty-eight 14×1414\times14 experts, totaling only 75,264 expert weights. More importantly, Appendix A.4 constructs these experts from twenty-four latent directions shared across depths and transported by orthogonal frames. This closely matches the representation AtomShare is designed to recover. The two transfer settings retain the same construction and shared-energy fraction. These are useful controlled experiments, but they do not establish that learned MoE weights exhibit the required structure or that the method preserves useful model behavior.

  2. Equation (1) minimizes end-of-stack deviation Γ\Gamma, but the headline “retained score” is exp⁡(−0.9ρ)\exp(-0.9\rho), where ρ\rho is a normalized, geometry-weighted reconstruction residual. It is not a measure of retained task performance. In Table 3, removing cross-depth sharing improves Γ\Gamma at every budget; at b=0.65b=0.65, the values are 1.0425 without sharing and 1.1140 for the full method. Appendix A.11 dismisses this ordering as “saturated,” without establishing why it is uninformative. Equation (16) has no saturation bound at one.

  3. Several baseline definitions and storage figures cannot be reconciled from the provided description. Appendix A.5 describes the 2:4 baseline as retaining two of every four weights. Retaining half of the 75,264 weights as 16-bit values would require 37,632 words before metadata, whereas Table 12 reports 9,408 words at b=0.35b=0.35. Additional quantization or another encoding could explain this, but it is not specified. Likewise, the “mixed-precision low-rank residual” baseline is described only as SVD truncation in the recorded input geometry; its precision scheme and residual representation are not explained.

Questions for authors
  1. Why should the ordering of Γ\Gamma be disregarded in the evaluated regime?
  2. What are the exact numerical formats and encodings used by each baseline, including sparsity metadata, quantization parameters, and codebooks? In particular, how is the 2:4 storage in Table 12 obtained?
Reviewer_vfRFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AtomShare, a way to compress mixture-of-experts weights using one dictionary shared across layers and experts. Each expert has a sparse code, while layer-specific Householder maps adapt the dictionary. The method allocates code length using routed-token error and estimated downstream error amplification. On a synthetic eight-layer stack, it reports a retained score of 0.5483 versus 0.5150 for its strongest per-expert baseline, at 2.452 versus 2.494 bits per weight.

Critical strengths
  • The method addresses both shared expert structure and the fact that experts see different routed inputs.
  • The storage accounting includes the dictionary, layer maps, geometry factors, coefficients, and support indices.
  • Ablations show that input geometry matters: removing it increases routed-token deviation by 67–93%. The paper also reports cases where simpler variants outperform the full method.
  • It tests several budgets, expert counts, and stack depths, and reports the cost of reconstructing experts.
Critical weaknesses
  • Evaluation is limited to small synthetic stacks: the main setup uses 14×14 expert matrices constructed from shared latent directions, a structure favorable to the proposed dictionary. There are no pretrained MoE models or downstream task results.
  • The retained score is a transformed weight-reconstruction metric measured in the same input geometry used by the method. It is not a measure of language-model quality.
  • Evidence for the allocation mechanisms is mixed. Uniform code length has a better retained score at every tested ablation point, and removing stack-coupled allocation sometimes improves routed deviation. The routing-flip term changes no allocations in the reported experiment.
  • Comparisons are not always at equal storage: five of nine compressed systems exceed the stated budget at one or more settings. The paper also lacks comparisons with shared-expert MoE compression methods.
  • Reconstruction is expensive in the reported operation count: AtomShare needs 9,212 multiply-accumulates per expert, the most of the tested systems. The paper reports no measured serving latency or throughput.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.