acceptodds
Under review as a conference paper at ICLR 2027

SLEM: Spectrum-Aware Latent Expert Merging

Abstract

Mixture-of-Experts (MoE) models scale model capacity efficiently by activating only a small subset of experts for each input, while still requiring all parameters to be stored in memory, resulting in substantial memory overhead. Expert merging techniques has recently emerged as a promising compression paradigm that reduces this overhead by consolidating multiple experts into a smaller set. However, existing approaches have two key limitations: (i) direct convex interpolation in parameter space is not generally guaranteed to yield meaningful interpolation of expert functions; and (ii) estimating expert similarity or merging coefficients often relies on calibration data, introducing dependence on the calibration distribution. To address these limitations, we propose Spectrum-Aware Latent Expert Merging (SLEM), a calibration-free expert merging framework that occurs in a structured latent representation of expert weights using a VAE. SLEM groups experts within and across layers based on latent similarities and derives merging coefficients from expert weight spectra. It then performs geometry-aware latent interpolation and decodes the resulting representations into merged expert weights. Compared with existing expert merging methods, SLEM achieves comparable or better performance on general knowledge benchmarks and higher average performance on generation benchmarks involving mathematical reasoning and code generation.

Primary area:
generative models
Keywords:
expert merging, latent space merging, spectrum-aware merging, calibration-free merging

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2MsAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

While mixture-of-experts (MoE) models reduce inference costs by only activating a subset of weights at each layer, they still require large amounts of memory for storage. An appealing method for reducing this memory cost is to merge the weights of experts that behave similarly. Existing approaches rely on calibration data, which produce brittle merges that do not maintain performance on tasks that are not included in the calibration task pool. To address this gap, the authors propose Spectrum-Aware Latent Expert Merging (SLEM). In order to measure the experts' similarity, SLEM embeds their weights through a variational auto-encoder (VAE) architecture with a hyperspherical prior in latent space. Given the geometry of the latent space, once the expert weights are embedded, they are aligned and combined through spherical interpolation; the result is later decoded into the merged weights.

Critical strengths
  • SLEM performs competitively, with a large gap in average accuracy compared to baselines.
  • The method does not require calibration data.
Critical weaknesses
  1. Several key ablations are missing:
  • the choice of a hyperspherical prior is not motivated by experiments, a simple baseline could be the typical gaussian prior used for VAEs;
  • generally, the β\beta and κ\kappa hyperparameters are very important for stability during training and performance of the VAEs, but the current discussion of their effects in this scenario is very incomplete, and it would help to better understand SLEM's sensitivity to certain tuning choices;
  • hungarian matching for alignment in latent space, currently it is unknown whether this additional step is bringing any gains.
  1. The cost of training the VAEs is discussed in the limitations section, but there is no comparison with baselines in terms of GPU hours or FLOPs.
Questions for authors
  1. Is the hypershperical prior and spherical interpolation in latent space strategy actually bringing benefits over gaussian prior with linear interpolation?
  2. How do β\beta and κ\kappa interact and what is the effect of tuning them?
  3. If the method does not use calibration data, how should the rank rr be chosen?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 2MsA (1/4)

We sincerely thank the reviewer for carefully reading our paper and for providing insightful comments. We address the reviewer's comments and present additional experimental results below.


W1-1 (Q1). A simple baseline could be the typical Gaussian prior used for VAEs. Is the hyperspherical prior and spherical interpolation in latent space strategy actually bringing benefits over Gaussian prior with linear interpolation?

We additionally trained a VAE with a Gaussian prior for ERNIE-4.5-21B-A3B-PT and incorporated it into SLEM. The Gaussian variant uses linear interpolation and L2-based matching. To also examine the effect of the learned representations, we included SLEM-PS, which performs expert grouping, subnetwork matching, and merging entirely in parameter space without a VAE. SLEM-PS likewise uses L2-based matching and linear interpolation.

The following results are under 25% expert reduction.

MethodGeneral knowledge avg.GSM8KMATH500MBPPHumanEvalMathematics & Coding avg.
SLEM (vMF prior + spherical interpolation)58.7072.2544.4046.8051.8353.82
SLEM (Gaussian prior + linear interpolation)58.9567.8515.2042.6048.7843.61
SLEM-PS (parameter space + linear interpolation)57.1868.616.4031.0028.0533.52

SLEM with the vMF prior outperformed the Gaussian variant on all four mathematics and code generation tasks, while their average general knowledge performance was similar. The Gaussian variant also achieved higher average performance on general knowledge and on mathematics and code generation than SLEM-PS. These results support the importance of the latent geometry and the matching and interpolation configuration used for merging, and show that using learned Gaussian latent representations can be more effective than direct parameter-space merging, even when both use L2 matching and linear interpolation.

In particular, the Gaussian variant and SLEM-PS both showed a large performance drop on MATH500. One possible interpretation is that the Gaussian VAE learns representations useful for reconstruction while retaining some geometric properties of parameter space, leaving the functional contributions of individual subnetworks insufficiently preserved during merging. In contrast, the vMF configuration fixes the norm of each latent vector to 1, allowing direction-based matching without the influence of differences in magnitude, and keeps the representations on the unit sphere during spherical interpolation. These spherical constraints may have helped reduce the effects of changes in representation magnitude and preserve the functional contributions of individual subnetworks during merging.

We will add the comparison with the Gaussian variant and SLEM-PS to the revised manuscript to clarify both the effect of learned representations and the advantages of the configuration using a hyperspherical prior and spherical interpolation.

AuthorsOpenReview ↗

Response to Reviewer 2MsA (2/4)

W1-2 (Q2). The current discussion of κ\kappa and β\beta effects is incomplete. How do κ\kappa and β\beta interact and what is the effect of tuning them?

κ\kappa determines how tightly latent samples concentrate around the mean direction of each distribution, while β\beta controls the balance between reconstruction and KL regularization. SLEM uses the same fixed κ\kappa for the posterior and prior, while learning the posterior directions and the prior direction for each layer.

Let dzd_z denote the latent dimension, μ\mu the mean direction, and z∼vMF(μ,κ)z\sim\mathrm{vMF}(\mu,\kappa) a sample. Defining the angle between the mean direction and the sample as θ=arccos⁡(μ⊤z)∈[0,π]\theta=\arccos(\mu^\top z)\in[0,\pi], the root mean square (RMS) angle in the high-concentration regime, where κ\kappa is sufficiently large relative to dzd_z, can be approximated as follows.

=\sqrt{\mathbb{E}_{z\sim\mathrm{vMF}(\mu,\kappa)}[\theta^2]} \approx \sqrt{\frac{d_z-1}{\kappa}}, \qquad \kappa\gg d_z.$$ **Accordingly, a larger $\kappa$ concentrates samples more tightly around the mean direction, while a larger $d_z$ increases the angular spread at the same $\kappa$.** Achieving a very small reconstruction error is important for a VAE that reconstructs expert weights. If $\kappa$ is too small, samples spread farther from the mean direction, potentially increasing overlap between the latent distributions of experts or subnetworks that need to remain distinguishable and making reconstruction more difficult. **We therefore first fixed $d_z$ and retained latent sampling during training, set $\beta=0$, and increased $\kappa$ through power-of-two candidates ($\kappa=2^j$), starting from smaller values and monitoring the reconstruction loss**. This allowed us to reduce the angular spread of sampling and select a $\kappa$ that enabled sufficiently accurate reconstruction of the weights. With $\kappa$ fixed, $\beta$ controls how strongly KL regularization aligns the learned posterior directions with each layer’s learned, shared prior direction. Larger $\beta$ strengthens this alignment, but excessive pressure can compromise the representational differences needed for reconstruction and increase reconstruction error. Smaller $\beta$ relaxes the constraint and favors reconstruction. **Because $\kappa$ also affects the strength of the KL penalty, the two hyperparameters are not fully independent. We therefore fixed $\kappa$ and reduced $\beta$ from larger values to balance reconstruction and regularization.** The final settings were $\kappa=2^{19}=524{,}288$ and $\beta=10^{-3}$. **The effect of KL regularization on merging can also be seen in Table 6 of Appendix E.** This experiment compares $\beta=10^{-3}$ with $\beta=0$ while keeping the selected $\kappa$ and the remaining training settings unchanged. **We also conducted an additional experiment on LFM2-8B-A1B with $\beta=10^{-4}$ to further examine the effect of regularization strength.** The following table combines the original results with this new experiment, reporting reconstruction-only performance and performance after 25% expert reduction. | Model | $\beta$ | Reconstruction only: General knowledge avg. | Reconstruction only: Mathematics & Coding avg. | After merging: General knowledge avg. | After merging: Mathematics & Coding avg. | |:---|:---|---:|---:|---:|---:| | LFM2-8B-A1B | $10^{-3}$ | $61.98\pm0.32$ | $67.97\pm0.72$ | $54.82$ | $56.74$ | | | $10^{-4}$ | $61.64\pm0.19$ | $67.29\pm0.38$ | $53.43$ | $47.40$ | | | $0$ | $61.78\pm0.19$ | $67.41\pm0.62$ | $48.97$ | $41.17$ | | OLMoE-1B-7B-0125-Instruct | $10^{-3}$ | $58.98\pm0.32$ | $38.62\pm0.42$ | $47.05$ | $11.40$ | | | $0$ | $58.98\pm0.53$ | $38.52\pm0.42$ | $43.76$ | $6.23$ | The reconstruction-only models perform similarly, but removing KL regularization reduces performance after merging on both models. In the additional LFM2-8B-A1B experiment, $\beta=10^{-4}$ yielded post-merging performance between that obtained with $\beta=0$ and $\beta=10^{-3}$, **further supporting the importance of KL regularization for organizing the latent space for interpolation.** We interpret **the alignment induced by KL regularization as helping form a latent space suitable for interpolation by mitigating the risk of passing through regions that the decoder has not sufficiently encountered during training when interpolating between different representations**. In the revised manuscript, we will add this additional ablation and clarify the roles of $\kappa$ and $\beta$, their interaction, and the sequential procedure for selecting them based on reconstruction loss.
AuthorsOpenReview ↗

Response to Reviewer 2MsA (3/4)

W1-3. Hungarian matching for alignment in latent space, currently it is unknown whether this additional step is bringing any gains.

We conducted an additional controlled experiment that removes only Hungarian matching from the full SLEM pipeline. We kept SLEM's selected expert pairs fixed and set the subnetwork permutation to the identity, matching subnetworks with the same original indices. We then applied the same spectral weighting rule and latent-space merging procedure.

The following results are under 25% expert reduction.

ModelMethodGeneral knowledge avg.Mathematics & Coding avg.
LFM2-8B-A1BSLEM54.8256.74
SLEM w/o subnetwork matching53.5651.41
ERNIE-4.5-21B-A3B-PTSLEM58.7053.82
SLEM w/o subnetwork matching55.7134.55

Removing subnetwork matching reduced the average performance on general knowledge and on mathematics and code generation for both models, with particularly large decreases in generation performance. These differences show that establishing appropriate correspondences between the subnetworks of the selected experts is important beyond selecting which experts to merge.

We will add this ablation to the revised manuscript to clarify the effect of subnetwork matching.


W2. The cost of training the VAEs is discussed in the limitations section, but there is no comparison with baselines in terms of GPU hours or FLOPs.

Because compression is performed entirely offline, our evaluation focused on the performance of the resulting compressed models. Accordingly, we did not include comparisons of compression time or FLOPs across baselines in the original evaluation, but reported the VAE training time for ERNIE-4.5-21B-A3B-PT as SLEM's main additional offline cost. We additionally measured the total computation time across all compression stages and the peak GPU memory of each stage for SLEM using a single NVIDIA H200 GPU on a server equipped with an Intel(R) Xeon(R) Platinum 8462Y+ CPU.

Other workloads were running concurrently on the same server, so the reported times reflect measurements in a shared-server environment. The reported times cover the computation in each stage and exclude model loading and checkpoint saving. Memory usage was measured using nvidia-smi command. All measurements in the following table were conducted under the 25% expert reduction setting. Since each stage uses a single GPU, the reported computation times in hours also correspond to GPU-hours.

ModelMeasurementVAE trainingSpectral importanceGrouping, alignment, and decodingTotal computation time
LFM2-8B-A1BProcess time (h)6.730.100.076.90
Peak GPU memory (MiB)54,9551,0198,299-
OLMoE-1B-7B-0125-InstructProcess time (h)7.210.060.067.33
Peak GPU memory (MiB)42,1399817,917-
ERNIE-4.5-21B-A3B-PTProcess time (h)18.680.180.2219.08
Peak GPU memory (MiB)76,3951,02912,345-

Most of the compression time is spent on VAE training, which also has the highest peak GPU memory usage. Computing spectral importance and performing expert grouping, subnetwork alignment, and decoding require comparatively less time and GPU memory.

All compression stages, including VAE training, are performed offline. At inference time, the model uses the merged expert weights computed in advance, so neither VAE encoding and decoding nor spectral analysis is required. Thus, the computation needed to use the learned latent space is completed during compression, with no additional VAE computation or associated latency during inference.

For HC-SMoE, the calibration forward passes and the subsequent expert merging took approximately two minutes on LFM2-8B-A1B. SLEM requires substantially more offline computation in this setting, primarily due to VAE training. This additional cost enables calibration-free merging with improved generation performance in our task-agnostic evaluation.

We will add the compression time and peak GPU memory for each stage to the Appendix in the revised manuscript.

Reviewer_oihAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SLEM is a calibration-free framework for compressing MoE models through expert merging. It trains a hyperspherical VAE on pretrained expert weights, selects and aligns expert pairs using latent similarities, derives merging coefficients from weight spectra, and decodes interpolated latent representations into merged experts. The framework permits both within-layer and cross-layer grouping. Experiments on three MoE models evaluate general knowledge, mathematical reasoning, and code generation at two expert-reduction rates.

Critical strengths
  1. The method provides a useful alternative to direct parameter-space merging without requiring input calibration data. The controlled parameter-space versus latent-space comparison supports the benefit of the learned representation. The KL ablation is also informative: similar reconstruction performance does not imply similar merging performance, highlighting the importance of organizing the latent space for interpolation.

  2. SLEM achieves substantial improvements over the evaluated merging baselines on generation tasks. It has the highest average mathematical reasoning and code generation performance across all evaluated model–reduction settings. Additional grouping, coefficient, and cross-layer analyses provide useful evidence for the design, while the paired bootstrap analysis supports the overall generation gains on LFM2.

Critical weaknesses
  1. The practical compression trade-off is incompletely quantified. Training the expert VAE takes approximately 18.7 hours on an H200 for the largest evaluated model, while the total merging cost and measured deployment memory savings are not reported. Moreover, strong relative improvements over merging baselines coexist with substantial losses from the original models, particularly for OLMoE at 25% expert reduction. General-knowledge results also favor HC-SMoE in several settings. These findings limit the breadth of the practical advantage.

  2. Some implementation and selection details require clarification. In particular, cross-layer merging needs a precise explanation of routing and parameter sharing to establish how stored expert parameters are reduced. The procedure for selecting the spectral rank and the numerical layer-wise merging constraint should also be specified, especially given the emphasis on avoiding calibration data.

Overall, I lean toward weak acceptance. The learned-space approach is interesting, and the generation-task improvements are substantial enough to support acceptance, although clearer implementation details and a fuller accounting of practical costs would strengthen the contribution.

Questions for authors
  1. How are cross-layer merged experts represented and routed in the compressed model? Please clarify whether the resulting parameters are shared across the affected layers, how the original routing entries are handled, and how layer-specific weight standardization is reversed when decoding a shared expert.

  2. How were the default spectral ranks and layer-wise merging limits selected? Please specify the selection procedure and whether it uses any downstream evaluation results, so that readers can distinguish calibration-free merging from task-dependent hyperparameter selection.

  3. Could the authors report the total compression cost, including VAE training, spectral analysis, matching, and decoding, alongside the actual stored parameter or memory reduction? This would help establish when the generation-quality advantage justifies the additional offline computation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer oihA (1/4)

We sincerely thank the reviewer for carefully reading our paper and for providing insightful comments. We address the reviewer's comments and present additional experimental results below.


W1-1 (Q3). The practical compression trade-off is incompletely quantified. Could the authors report the total compression cost, including VAE training, spectral analysis, matching, and decoding, alongside the actual stored parameter or memory reduction?

We measured the total computation time across all compression stages and the peak GPU memory of each stage using a single NVIDIA H100 GPU on a server equipped with an Intel(R) Xeon(R) Platinum 8462Y+ CPU.

Other workloads were running concurrently on the same server, so the reported times reflect measurements in a shared-server environment. The reported times cover the computation in each stage and exclude model loading and checkpoint saving. Memory usage was measured using nvidia-smi command. All measurements were conducted under the 25% expert reduction setting.

ModelMeasurementVAE trainingSpectral importanceGrouping, alignment, and decodingTotal computation time
LFM2-8B-A1BProcess time (h)6.730.100.076.90
Peak GPU memory (MiB)54,9551,0198,299-
OLMoE-1B-7B-0125-InstructProcess time (h)7.210.060.067.33
Peak GPU memory (MiB)42,1399817,917-
ERNIE-4.5-21B-A3B-PTProcess time (h)18.680.180.2219.08
Peak GPU memory (MiB)76,3951,02912,345-

Most of the computation time is spent on VAE training, while spectral importance computation, expert grouping, subnetwork alignment, and decoding require relatively less time and memory. VAE training is the main offline cost of SLEM. However, all compression stages are completed offline, and inference uses the merged expert weights, so neither VAE encoding and decoding nor spectral analysis is required.

The actual parameter counts and compression rates of the compressed models under 25% expert reduction are as follows.

ModelParameters before compressionParameters after compressionCompression rate
LFM2-8B-A1B8.34B6.40B23.23%
OLMoE-1B-7B-0125-Instruct6.92B5.31B23.28%
ERNIE-4.5-21B-A3B-PT21.95B16.86B23.21%

We will add the compression time and peak GPU memory of each stage, the parameter counts after compression, and the actual compression rates to the Appendix in the revised manuscript.


W1-2. General-knowledge results also favor HC-SMoE in several settings. Substantial losses exist after merging, particularly for OLMoE at 25% expert reduction.

Section 4.5.1 in our paper analyzes the maximum expert group size and the number of original experts preserved without merging as factors to consider when interpreting these performance differences. SLEM performs pairwise merging, limiting the group size to two, whereas HC-SMoE allows larger groups. Even at the same overall expert reduction rate, the number of original experts participating in merging can vary with group size, which also affects the number of experts that retain their original functions unchanged.

Table 2 in our paper reports results with the maximum group size of HC-SMoE restricted to the same size as SLEM. The following comparison is under 25% expert reduction.

ModelMethodMaximum group sizeGeneral knowledge avg.Mathematics & Coding avg.
LFM2-8B-A1BHC-SMoEUnrestricted49.5343.56
HC-SMoE252.1302.49
SLEM254.8256.74
ERNIE-4.5-21B-A3B-PTHC-SMoEUnrestricted65.3245.59
HC-SMoE255.5419.66
SLEM258.7053.82

Restricting the maximum group size to two reduces generation performance in both models. while its effect on general knowledge varies across models. We hypothesizes that preserving fewer original experts may contribute to this performance degradation.

Under the same pairwise merging constraint, SLEM achieved higher average performance on general knowledge and on mathematics and code generation than HC-SMoE on both LFM2-8B-A1B and ERNIE-4.5-21B-A3B-PT. This supports the interpretation that SLEM can preserve useful functional contributions from the original experts even when they are merged.

In addition, we acknowledge that substantial performance losses remain after merging. Sensitivity to merging may vary across models due to differences in architecture and training data. Nevertheless, SLEM generally achieves higher average mathematics and code generation performance than other expert merging baselines. These results support the interpretation that SLEM preserves the useful functional contributions of the original experts more effectively during merging, even for models that are particularly sensitive to compression.

AuthorsOpenReview ↗

Response to Reviewer oihA (2/4)

W2-1 (Q1). Cross-layer merging needs a precise explanation of routing and parameter sharing to establish how stored expert parameters are reduced. How are cross-layer merged experts represented and routed in the compressed model?

We keep each layer's router and routing entries unchanged and map the routing entries of the experts involved in merging to a single shared merged expert. Keeping the original router while directing inputs to the merged expert is commonly used in expert merging.

When merging two experts Eℓi,iE_{\ell_i,i} and Eℓj,jE_{\ell_j,j} from different layers, let their corresponding router vectors be rℓi,i\mathbf r_{\ell_i,i} and rℓj,j\mathbf r_{\ell_j,j}. This can be expressed as follows.

(\mathbf r_{\ell_i,i}, E_{\ell_i,i}) \longrightarrow (\mathbf r_{\ell_i,i}, \widetilde E), \qquad (\mathbf r_{\ell_j,j}, E_{\ell_j,j}) \longrightarrow (\mathbf r_{\ell_j,j}, \widetilde E). \end{aligned}$$ Here, **the two routing entries share $\widetilde E$, whose weights are stored only once**, while the top-$k$ selection and routing weight computation of each MoE layer remain unchanged. The layer-wise standardization in the Expert VAE is reversed after decoding using the statistics of each source layer. **The two reconstructed results are then combined using coefficients derived from the normalized spectral importance of the aligned subnetworks to produce a single final set of expert weights**. Let $\widetilde Z$ denote the merged joint latent representation. The result of denormalization using the statistics of source layer $\ell_i$ is as follows. $$\widehat W_{\ell_i} =\sigma_{\ell_i}D_{\theta}(\widetilde Z) +\mu_{\ell_i}.$$ $\mu_{\ell_i}$ and $\sigma_{\ell_i}$ are the standardization statistics of layer $\ell_i$, and $D_{\theta}(\widetilde Z)$ is the decoder output in the normalized space. For an aligned subnetwork pair $(k,\pi(k))$, the final weights are computed as follows. $$\alpha_k =\frac{s_{i,k}}{s_{i,k}+s_{j,\pi(k)}}, \qquad \widetilde W[k,:] =\alpha_k\widehat W_{\ell_i}[k,:] +(1-\alpha_k)\widehat W_{\ell_j}[k,:],$$ where $s_{i,k}$ and $s_{j,\pi(k)}$ are the normalized spectral importance scores of the corresponding subnetworks. **This process adjusts the contributions of the weights reconstructed from the two source layers according to the relative importance of each subnetwork**, while ultimately producing a single expert shared by all corresponding routing entries. **The denormalized results for different layers are not stored as separate experts.** We acknowledge that we did not sufficiently clarify decoding, denormalization, and shared storage after cross-layer merging. We will add the above equations and the routing-to-expert mapping to Section 3.2 in the revised manuscript to clarify this process.
AuthorsOpenReview ↗

Response to Reviewer oihA (3/4)

W2-2 (Q2). The procedure for selecting the spectral rank and the numerical layer-wise merging constraint should also be specified. How were the default spectral ranks and layer-wise merging limits selected?

Layer-wise merging limit.

We set the layer-wise merging limit so that the same number of original experts participate in merging in each layer, preventing merging from being concentrated in particular layers. A merged expert may not reproduce the functions of the two original experts exactly. Excessive merging in a particular layer may therefore concentrate representation errors in that layer. This limit is a structural constraint that distributes the total merging budget evenly across layers.

Let LL denote the number of MoE layers, EE the number of routed experts in an MoE layer, and ρ\rho the target expert reduction rate. For pairwise merging, the total number of routed experts eligible for merging is N=LEN=LE, the required number of merging pairs is M=ρNM=\rho N, and the number of experts participating in merging is 2M2M. Accordingly, the number of participating experts per layer is determined as follows.

Cℓ=2ML=2ρE.C_\ell=\frac{2M}{L}=2\rho E.

A within-layer pair adds two to the participation count of that layer, while a cross-layer pair adds one to the participation count of each source layer. The limit therefore applies to the number of original experts participating in merging. Setting each layer's upper bound to CℓC_\ell and fulfilling the total merging budget also ensures that the actual number of participating experts is the same across layers.

ModelRouted experts per layer EECℓC_\ell at 12.5% expert reductionCℓC_\ell at 25% expert reduction
LFM2-8B-A1B32816
OLMoE-1B-7B-0125-Instruct641632
ERNIE-4.5-21B-A3B-PT641632

This constraint is determined by the number of experts and the target reduction rate. Table 5 in our paper also compares SLEM with a cross-layer adaptive variant that removes the layer-wise merging limit. The following results compare the presence and absence of this limit while retaining cross-layer merging.

ModelExpert reductionMethodGeneral knowledge avg.Mathematics & Coding avg.
LFM2-8B-A1B25%SLEM54.8256.74
SLEM w/o layer-wise limit (cross-layer adaptive)55.2654.67
OLMoE-1B-7B-0125-Instruct12.5%SLEM56.3629.74
SLEM w/o layer-wise limit (cross-layer adaptive)54.9425.64
ERNIE-4.5-21B-A3B-PT25%SLEM58.7053.82
SLEM w/o layer-wise limit (cross-layer adaptive)56.7652.66

Removing the layer-wise limit reduced the average mathematics and code generation performance in all comparisons and generally reduced the average general knowledge performance. This supports the usefulness of distributing the impact of merging evenly across layers.

Reviewer_PptcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SLEM, a data-free method that compresses MoE models by merging experts. It merges experts in a learned latent space with spectrum-based coefficients. On three MoE models, it gives the best math and code scores among merging methods.

Critical strengths

(1) The gains on generation are large and need no calibration data. SLEM beats the best baseline on the math and code average by about 7 to 13 points in all six settings.

(2) The ablations are broad. They cover latent versus parameter space, spectral rank, alternative coefficients, the KL term, and group size.

Critical weaknesses

(1) The value of the VAE is not isolated in the full pipeline. The latent versus parameter comparison uses random pairs with a fixed 0.5 coefficient, not SLEM's pairs, alignment, and spectral coefficients. With uniform coefficients, the math and code average falls from 29.74 to 15.57 on OLMoE and from 53.82 to 44.91 on ERNIE, both below HC-SMoE.

(2) The baselines are weak and narrow. All baselines are calibrated on only 32 x 2048 C4 tokens, and M-SMoE scores 0.00 on all four generation tasks for OLMoE at 25% reduction. REAP reports that pruning beats merging on generative tasks, but no pruning baseline is included.

Questions for authors

(1) What is the result of parameter-space merging with SLEM's exact pairs, Hungarian alignment, and spectral coefficients? This would show what the VAE adds beyond the coefficients.

(2) How do HC-SMoE and a pruning method such as REAP perform with math and code calibration data? What is the end-to-end compression time, including VAE training?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Pptc (1/2)

We sincerely thank the reviewer for carefully reading our paper and for providing insightful comments. We address the reviewer's comments and present additional experimental results below.


W1-1 (Q1). The value of the VAE is not isolated in the full pipeline. What is the result of parameter-space merging with SLEM's exact pairs, Hungarian alignment, and spectral coefficients?

We conducted an additional controlled experiment in which only the merge was performed in the original parameter space, while SLEM's expert pairs, Hungarian alignment, and spectral coefficients were kept fixed. To examine how the learned latent representation contributes to the overall pipeline, we also evaluated SLEM-PS, which performs expert grouping, subnetwork matching, and merging entirely in parameter space without a VAE.

All experiments were conducted under 25% expert reduction.

ModelMethodGrouping and alignment spaceMerging spaceGeneral knowledge avg.Mathematics & Coding avg.
LFM2-8B-A1BSLEMLatentLatent54.8256.74
SLEM w/o latent-space mergingLatentParameter54.2955.00
SLEM-PSParameterParameter47.1837.19
ERNIE-4.5-21B-A3B-PTSLEMLatentLatent58.7053.82
SLEM w/o latent-space mergingLatentParameter57.7051.67
SLEM-PSParameterParameter57.1833.52

Even with identical expert pairs, Hungarian alignment, and spectral coefficients, latent-space merging improved the average performance for general knowledge and for math and code generation on both LFM2 and ERNIE-4.5 compared with parameter-space merging. This means that merging in latent space provides an additional performance benefit even when expert grouping and subnetwork alignment are held fixed.

The learned latent representation is also used for expert grouping and subnetwork alignment before merging. Comparing the two variants that merge in parameter space, SLEM w/o latent-space merging outperformed SLEM-PS on both models. This means that, in these settings, performing expert grouping and subnetwork alignment with learned latent representations is more effective than performing them in parameter space.

In summary, the VAE contributes not only to the merge but also to the representations of experts and their subnetworks. We will add these controlled experiments to the revised manuscript to clarify the contribution of the latent space learned by the VAE.


W1-2. With uniform coefficients, the math and code average falls from 29.74 to 15.57 on OLMoE and from 53.82 to 44.91 on ERNIE, both below HC-SMoE.

As analyzed in Section 4.5.1 of our paper, SLEM performs pairwise merging, limiting the group size to two, whereas HC-SMoE allows larger groups. Even at the same overall expert reduction rate, the number of original experts participating in merging can vary with group size, which also affects the number of experts that retain their original functions unchanged.

Tables 2 and 5 in our paper provide the following comparison on ERNIE-4.5-21B-A3B-PT under 25% expert reduction.

MethodMaximum group sizeMathematics & Coding avg.
HC-SMoEUnrestricted45.59
HC-SMoE219.66
SLEM w/ uniform coefficients244.91
SLEM253.82

Restricting the maximum group size to two substantially reduces the generation performance of HC-SMoE. We hypothesize that preserving fewer original experts may contribute to this performance degradation. Under the same pairwise merging constraint, SLEM with uniform coefficients still substantially outperforms HC-SMoE on mathematics and coding tasks. This supports the interpretation that SLEM can retain useful functional contributions from the original experts even when more experts participate in merging. Spectrum-aware coefficients further improve performance by accounting for differences in subnetwork importance.

AuthorsOpenReview ↗

Response to Reviewer Pptc (2/2)

W2 (Q2-1). All baselines are calibrated on only 32 x 2048 C4 tokens. REAP reports that pruning beats merging on generative tasks, but no pruning baseline is included. How do HC-SMoE and a pruning method such as REAP perform with math and code calibration data?

We used C4 as the calibration dataset to evaluate performance in a task-agnostic compression setting where calibration data for downstream tasks are unavailable. HC-SMoE likewise presents its method as a task-agnostic expert merging framework and uses 32 x 2,048 tokens from C4 for calibration in its main experiments. Our experiments follow this task-agnostic evaluation setting.

Since baseline performance may change when downstream domain data are available, we additionally evaluated HC-SMoE and REAP using the GSM8K (train split) for math calibration and the MBPP (train split) for code calibration. We kept the calibration buget fixed at 32 x 2,048 tokens across all conditions and used the same evaluation protocol as in our experiments to isolate the effect of calibration-domain selection.

All experiments were conducted under 25% expert reduction on LFM2-8B-A1B.

MethodCalibration datasetGeneral knowledge avg.Mathematics avg.Coding avg.
SLEMN/A54.8264.5848.90
HC-SMoEC449.5349.6737.46
GSM8K52.7070.7231.71
MBPP53.0721.8143.26
REAP (pruning)C457.7254.4913.82
GSM8K51.8371.6643.93
MBPP52.1268.2054.23

The results show that the performances of HC-SMoE and REAP can change substantially with the calibration dataset, even under an identical calibration budget. Changing the calibration domain altered the balance between mathematics and coding performance for HC-SMoE. REAP also exhibited substantial differences in performance across calibration datasets.

SLEM's advantage is that it achieves broadly competitive performance across the evaluated general knowledge, mathematics, and coding tasks without using any calibration data. It outperformed both baselines calibrated on C4 in mathematics and coding, without requiring the collection of calibration data tailored to a particular downstream domain. These results support SLEM's practical value in task-agnostic compression settings where downstream calibration data are unavailable.

In the revised manuscript, we will add the results with both general-purpose and domain-specific calibration data to clarify the effects of calibration dataset selection and SLEM's calibration-free, task-agnostic properties.


Q2-2. What is the end-to-end compression time, including VAE training?

We measured the total computation time across all compression stages and the peak GPU memory of each stage using a single NVIDIA H200 GPU on a server equipped with an Intel(R) Xeon(R) Platinum 8462Y+ CPU.

Other workloads were running concurrently on the same server, so the reported times reflect measurements in a shared-server environment. The reported times cover the computation in each stage and exclude model loading and checkpoint saving. Memory usage was measured using nvidia-smi command. All measurements were conducted under the 25% expert reduction setting.

ModelMeasurementVAE trainingSpectral importanceGrouping, alignment, and decodingTotal computation time
LFM2-8B-A1BProcess time (h)6.730.100.076.90
Peak GPU memory (MiB)54,9551,0198,299-
OLMoE-1B-7B-0125-InstructProcess time (h)7.210.060.067.33
Peak GPU memory (MiB)42,1399817,917-
ERNIE-4.5-21B-A3B-PTProcess time (h)18.680.180.2219.08
Peak GPU memory (MiB)76,3951,02912,345-

Most of the compression time is spent on VAE training, which also has the highest peak GPU memory usage. Computing spectral importance and performing expert grouping, subnetwork alignment, and decoding require comparatively less time and GPU memory.

All compression stages, including VAE training, are performed offline. At inference time, the model uses the merged expert weights computed in advance, so neither VAE encoding and decoding nor spectral analysis is required. Thus, the computation needed to use the learned latent space is completed during compression, with no additional VAE computation or associated latency during inference.

We will add the compression time and peak GPU memory for each stage to the Appendix in the revised manuscript.


We would be happy to clarify any remaining points or address further questions.

Dear Reviewer,

We sincerely thank you for the time and effort you have devoted to reviewing our paper.

We would appreciate any further feedback on our responses and additional experiments. Also, if any concerns remain, we would be happy to discuss them further and provide any clarification needed.

Best regards, The Authors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank the reviewers for carefully reading our paper and for providing insightful comments.

We have uploaded a revised manuscript incorporating additional experiments and clarifications in response to the reviewers’ feedback. We have also corrected typographical errors throughout the manuscript. Revised and newly added content is highlighted in red. The main updates and their locations are summarized below.


Experiment or clarificationSection and supporting results
Clarification of the cross-layer merging procedure.Section 3.2, Appendix B.3
Clarification of calibration-free spectral rank selection.Section 3.2, Appendix H, Figure 7
Evaluation of HC-SMoE and REAP with domain-specific calibration datasets.Appendix D, Table 7
Compression cost and actual compression rates of SLEM.Appendix C, Table 6
Ablations without subnetwork matching and with only the final merging performed in parameter space.Section 4.4, Table 2
Clarification of how κ\kappa and β\beta interact and are tuned in the vMF-based Expert VAE, and comparison with a Gaussian prior.Appendix F, Tables 9 and 10

We would be happy to clarify any remaining points or address further questions.

AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank the AC and all reviewers for their careful evaluation and constructive feedback. We summarize the recognized strengths, our responses to the concerns, and the resulting revisions below.


Strengths

  • Strong generation performance (Reviewer Pptc, oihA, and 2MsA)

    SLEM achieves substantial improvements over the evaluated merging baselines on generation tasks.

  • Calibration-free method (Reviewer Pptc, oihA, and 2MsA)

    SLEM merges experts without requiring input calibration data.

  • Evidence supporting the learned representation with broad ablations (Reviewer Pptc and oihA)

    The ablations cover latent versus parameter space, spectral rank, alternative coefficients, the KL term, and group size, with the controlled comparison supporting the benefit of the learned representation.

  • Analysis of design choices (Reviewer oihA)

    Grouping, coefficient, and cross-layer analyses provide useful evidence for the design.

  • Statistical support (Reviewer oihA)

    The paired bootstrap analysis supports the overall generation gains.


Concerns and how we addressed them

  • Clarification of implementation and rank selection (Reviewer oihA and 2MsA)

    We clarified cross-layer merging, routing, and layer-wise merging limits, and explained an entropy-based rank selection criterion requiring no calibration data.

  • Prior choice and hyperparameter tuning (Reviewer 2MsA)

    We conducted a Gaussian-prior comparison, clarified the interaction and selection of κ\kappa and β\beta, and evaluated an additional KL weight. The additional experiments support the hyperspherical configuration and regularization for interpolation.

  • Additional ablations on latent-space merging and subnetwork matching (Reviewers Pptc and 2MsA)

    We conducted additional ablations by (i) changing only the merging space while keeping SLEM’s pairs, alignment, and coefficients fixed, and (ii) removing only subnetwork matching while retaining the selected expert pairs. Both ablations reduce downstream performance, showing the importance of these steps. Comparisons with SLEM-PS further support the benefit of learned representations.

  • Baselines with domain-specific calibration (Reviewer Pptc)

    We evaluated HC-SMoE and REAP using C4, GSM8K, and MBPP as calibration data under an identical token budget, demonstrating calibration-dependent performance. We also clarified SLEM’s advantage in task-agnostic compression settings.

  • Total compression cost and actual compression rate (Reviewer Pptc, oihA, and 2MsA)

    We reported stage-wise computation time, peak GPU memory, and parameter reduction. We acknowledged SLEM’s offline cost, but clarified that inference requires no additional computation or latency.

  • Scope of performance improvements (Reviewers Pptc and oihA)

    We clarified that our paper had already analyzed how maximum group size affects the number of original experts preserved without merging, which may help explain the observed performance degradation. Under the same maximum group size of two, SLEM outperforms HC-SMoE on both general-knowledge and generation averages for the all evaluated models.

  • Table and notation readability (Reviewer 2MsA)

    We increased table font sizes and simplified repeated subscripts and superscripts in SLEM pipeline.


Revisions

We have uploaded the revised manuscript, incorporating the additional experiments and methodological clarifications into the main text and appendices. We also corrected typographical errors, increased the font size in tables, and simplified the notation and mathematical expressions to improve readability.

Please refer to our Summary of Revisions comment for further details.


The initial ratings were 2, 3, and 2 from Reviewer Pptc, oihA, and 2MsA, respectively. Reviewer 2MsA subsequently stated that the additional explanations and ablations had improved the work and that they were considering raising their score. Although we have not yet received further response from the other two reviewers, we believe the additional experiments and explanations address their main concerns.

We thank the AC and reviewers again for their time and thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.