- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
While mixture-of-experts (MoE) models reduce inference costs by only activating a subset of weights at each layer, they still require large amounts of memory for storage. An appealing method for reducing this memory cost is to merge the weights of experts that behave similarly. Existing approaches rely on calibration data, which produce brittle merges that do not maintain performance on tasks that are not included in the calibration task pool. To address this gap, the authors propose Spectrum-Aware Latent Expert Merging (SLEM). In order to measure the experts' similarity, SLEM embeds their weights through a variational auto-encoder (VAE) architecture with a hyperspherical prior in latent space. Given the geometry of the latent space, once the expert weights are embedded, they are aligned and combined through spherical interpolation; the result is later decoded into the merged weights.
Critical strengths
- SLEM performs competitively, with a large gap in average accuracy compared to baselines.
- The method does not require calibration data.
Critical weaknesses
- Several key ablations are missing:
- the choice of a hyperspherical prior is not motivated by experiments, a simple baseline could be the typical gaussian prior used for VAEs;
- generally, the and hyperparameters are very important for stability during training and performance of the VAEs, but the current discussion of their effects in this scenario is very incomplete, and it would help to better understand SLEM's sensitivity to certain tuning choices;
- hungarian matching for alignment in latent space, currently it is unknown whether this additional step is bringing any gains.
- The cost of training the VAEs is discussed in the limitations section, but there is no comparison with baselines in terms of GPU hours or FLOPs.
Questions for authors
- Is the hypershperical prior and spherical interpolation in latent space strategy actually bringing benefits over gaussian prior with linear interpolation?
- How do and interact and what is the effect of tuning them?
- If the method does not use calibration data, how should the rank be chosen?
7 comments
Response to Reviewer 2MsA (2/4)
W1-2 (Q2). The current discussion of and effects is incomplete. How do and interact and what is the effect of tuning them?
determines how tightly latent samples concentrate around the mean direction of each distribution, while controls the balance between reconstruction and KL regularization. SLEM uses the same fixed for the posterior and prior, while learning the posterior directions and the prior direction for each layer.
Let denote the latent dimension, the mean direction, and a sample. Defining the angle between the mean direction and the sample as , the root mean square (RMS) angle in the high-concentration regime, where is sufficiently large relative to , can be approximated as follows.
=\sqrt{\mathbb{E}_{z\sim\mathrm{vMF}(\mu,\kappa)}[\theta^2]} \approx \sqrt{\frac{d_z-1}{\kappa}}, \qquad \kappa\gg d_z.$$ **Accordingly, a larger $\kappa$ concentrates samples more tightly around the mean direction, while a larger $d_z$ increases the angular spread at the same $\kappa$.** Achieving a very small reconstruction error is important for a VAE that reconstructs expert weights. If $\kappa$ is too small, samples spread farther from the mean direction, potentially increasing overlap between the latent distributions of experts or subnetworks that need to remain distinguishable and making reconstruction more difficult. **We therefore first fixed $d_z$ and retained latent sampling during training, set $\beta=0$, and increased $\kappa$ through power-of-two candidates ($\kappa=2^j$), starting from smaller values and monitoring the reconstruction loss**. This allowed us to reduce the angular spread of sampling and select a $\kappa$ that enabled sufficiently accurate reconstruction of the weights. With $\kappa$ fixed, $\beta$ controls how strongly KL regularization aligns the learned posterior directions with each layer’s learned, shared prior direction. Larger $\beta$ strengthens this alignment, but excessive pressure can compromise the representational differences needed for reconstruction and increase reconstruction error. Smaller $\beta$ relaxes the constraint and favors reconstruction. **Because $\kappa$ also affects the strength of the KL penalty, the two hyperparameters are not fully independent. We therefore fixed $\kappa$ and reduced $\beta$ from larger values to balance reconstruction and regularization.** The final settings were $\kappa=2^{19}=524{,}288$ and $\beta=10^{-3}$. **The effect of KL regularization on merging can also be seen in Table 6 of Appendix E.** This experiment compares $\beta=10^{-3}$ with $\beta=0$ while keeping the selected $\kappa$ and the remaining training settings unchanged. **We also conducted an additional experiment on LFM2-8B-A1B with $\beta=10^{-4}$ to further examine the effect of regularization strength.** The following table combines the original results with this new experiment, reporting reconstruction-only performance and performance after 25% expert reduction. | Model | $\beta$ | Reconstruction only: General knowledge avg. | Reconstruction only: Mathematics & Coding avg. | After merging: General knowledge avg. | After merging: Mathematics & Coding avg. | |:---|:---|---:|---:|---:|---:| | LFM2-8B-A1B | $10^{-3}$ | $61.98\pm0.32$ | $67.97\pm0.72$ | $54.82$ | $56.74$ | | | $10^{-4}$ | $61.64\pm0.19$ | $67.29\pm0.38$ | $53.43$ | $47.40$ | | | $0$ | $61.78\pm0.19$ | $67.41\pm0.62$ | $48.97$ | $41.17$ | | OLMoE-1B-7B-0125-Instruct | $10^{-3}$ | $58.98\pm0.32$ | $38.62\pm0.42$ | $47.05$ | $11.40$ | | | $0$ | $58.98\pm0.53$ | $38.52\pm0.42$ | $43.76$ | $6.23$ | The reconstruction-only models perform similarly, but removing KL regularization reduces performance after merging on both models. In the additional LFM2-8B-A1B experiment, $\beta=10^{-4}$ yielded post-merging performance between that obtained with $\beta=0$ and $\beta=10^{-3}$, **further supporting the importance of KL regularization for organizing the latent space for interpolation.** We interpret **the alignment induced by KL regularization as helping form a latent space suitable for interpolation by mitigating the risk of passing through regions that the decoder has not sufficiently encountered during training when interpolating between different representations**. In the revised manuscript, we will add this additional ablation and clarify the roles of $\kappa$ and $\beta$, their interaction, and the sequential procedure for selecting them based on reconstruction loss.Response to Reviewer 2MsA (3/4)
W1-3. Hungarian matching for alignment in latent space, currently it is unknown whether this additional step is bringing any gains.
We conducted an additional controlled experiment that removes only Hungarian matching from the full SLEM pipeline. We kept SLEM's selected expert pairs fixed and set the subnetwork permutation to the identity, matching subnetworks with the same original indices. We then applied the same spectral weighting rule and latent-space merging procedure.
The following results are under 25% expert reduction.
| Model | Method | General knowledge avg. | Mathematics & Coding avg. |
|---|---|---|---|
| LFM2-8B-A1B | SLEM | 54.82 | 56.74 |
| SLEM w/o subnetwork matching | 53.56 | 51.41 | |
| ERNIE-4.5-21B-A3B-PT | SLEM | 58.70 | 53.82 |
| SLEM w/o subnetwork matching | 55.71 | 34.55 |
Removing subnetwork matching reduced the average performance on general knowledge and on mathematics and code generation for both models, with particularly large decreases in generation performance. These differences show that establishing appropriate correspondences between the subnetworks of the selected experts is important beyond selecting which experts to merge.
We will add this ablation to the revised manuscript to clarify the effect of subnetwork matching.
W2. The cost of training the VAEs is discussed in the limitations section, but there is no comparison with baselines in terms of GPU hours or FLOPs.
Because compression is performed entirely offline, our evaluation focused on the performance of the resulting compressed models. Accordingly, we did not include comparisons of compression time or FLOPs across baselines in the original evaluation, but reported the VAE training time for ERNIE-4.5-21B-A3B-PT as SLEM's main additional offline cost. We additionally measured the total computation time across all compression stages and the peak GPU memory of each stage for SLEM using a single NVIDIA H200 GPU on a server equipped with an Intel(R) Xeon(R) Platinum 8462Y+ CPU.
Other workloads were running concurrently on the same server, so the reported times reflect measurements in a shared-server environment. The reported times cover the computation in each stage and exclude model loading and checkpoint saving. Memory usage was measured using nvidia-smi command. All measurements in the following table were conducted under the 25% expert reduction setting. Since each stage uses a single GPU, the reported computation times in hours also correspond to GPU-hours.
| Model | Measurement | VAE training | Spectral importance | Grouping, alignment, and decoding | Total computation time |
|---|---|---|---|---|---|
| LFM2-8B-A1B | Process time (h) | 6.73 | 0.10 | 0.07 | 6.90 |
| Peak GPU memory (MiB) | 54,955 | 1,019 | 8,299 | - | |
| OLMoE-1B-7B-0125-Instruct | Process time (h) | 7.21 | 0.06 | 0.06 | 7.33 |
| Peak GPU memory (MiB) | 42,139 | 981 | 7,917 | - | |
| ERNIE-4.5-21B-A3B-PT | Process time (h) | 18.68 | 0.18 | 0.22 | 19.08 |
| Peak GPU memory (MiB) | 76,395 | 1,029 | 12,345 | - |
Most of the compression time is spent on VAE training, which also has the highest peak GPU memory usage. Computing spectral importance and performing expert grouping, subnetwork alignment, and decoding require comparatively less time and GPU memory.
All compression stages, including VAE training, are performed offline. At inference time, the model uses the merged expert weights computed in advance, so neither VAE encoding and decoding nor spectral analysis is required. Thus, the computation needed to use the learned latent space is completed during compression, with no additional VAE computation or associated latency during inference.
For HC-SMoE, the calibration forward passes and the subsequent expert merging took approximately two minutes on LFM2-8B-A1B. SLEM requires substantially more offline computation in this setting, primarily due to VAE training. This additional cost enables calibration-free merging with improved generation performance in our task-agnostic evaluation.
We will add the compression time and peak GPU memory for each stage to the Appendix in the revised manuscript.
Response to Reviewer 2MsA (1/4)
We sincerely thank the reviewer for carefully reading our paper and for providing insightful comments. We address the reviewer's comments and present additional experimental results below.
W1-1 (Q1). A simple baseline could be the typical Gaussian prior used for VAEs. Is the hyperspherical prior and spherical interpolation in latent space strategy actually bringing benefits over Gaussian prior with linear interpolation?
We additionally trained a VAE with a Gaussian prior for ERNIE-4.5-21B-A3B-PT and incorporated it into SLEM. The Gaussian variant uses linear interpolation and L2-based matching. To also examine the effect of the learned representations, we included SLEM-PS, which performs expert grouping, subnetwork matching, and merging entirely in parameter space without a VAE. SLEM-PS likewise uses L2-based matching and linear interpolation.
The following results are under 25% expert reduction.
SLEM with the vMF prior outperformed the Gaussian variant on all four mathematics and code generation tasks, while their average general knowledge performance was similar. The Gaussian variant also achieved higher average performance on general knowledge and on mathematics and code generation than SLEM-PS. These results support the importance of the latent geometry and the matching and interpolation configuration used for merging, and show that using learned Gaussian latent representations can be more effective than direct parameter-space merging, even when both use L2 matching and linear interpolation.
In particular, the Gaussian variant and SLEM-PS both showed a large performance drop on MATH500. One possible interpretation is that the Gaussian VAE learns representations useful for reconstruction while retaining some geometric properties of parameter space, leaving the functional contributions of individual subnetworks insufficiently preserved during merging. In contrast, the vMF configuration fixes the norm of each latent vector to 1, allowing direction-based matching without the influence of differences in magnitude, and keeps the representations on the unit sphere during spherical interpolation. These spherical constraints may have helped reduce the effects of changes in representation magnitude and preserve the functional contributions of individual subnetworks during merging.
We will add the comparison with the Gaussian variant and SLEM-PS to the revised manuscript to clarify both the effect of learned representations and the advantages of the configuration using a hyperspherical prior and spherical interpolation.