acceptodds
Under review as a conference paper at ICLR 2027

HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts

Abstract

Sparse Mixture-of-Experts (MoE) layers route tokens through a handful of experts, and learning-free compression of these layers reduces inference cost without retraining. An obstruction limits every existing compressor in this family: three experts can each be pairwise compatible yet form an irreducible cycle when merged together, so any score that ranks experts on pairwise signals is structurally blind to which triples are jointly mergeable. We show this cycle is a precise mathematical object, the harmonic kernel of the simplicial Laplacian on a 2-complex whose vertices are experts, whose edges carry KL merge barriers, and whose faces carry triplet barriers; Hodge-decomposing the edge-barrier signal isolates it, and triplet barriers measure joint-merge cost directly. We turn the diagnostic into a selection objective: HodgeCover greedily covers the harmonic-critical edges and triplet-critical triangles, and a hybrid variant adds weight pruning on survivors. On three open-weight MoE backbones under aggressive expert reduction, HodgeCover matches state-of-the-art learning-free baselines on the expert-reduction axis, leads on the aggressive-compression frontier of the hybrid axis, and uniquely balances retained mass across all four Hodge components. These results show that exposing the harmonic kernel changes which compressor wins under aggressive compression.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
mixture-of-experts, model compression, expert pruning, Hodge decomposition, simplicial complexes, topological machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Hb1VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HodgeCover, a learning-free compression method for Sparse Mixture-of-Experts (MoE) layers. It constructs a simplicial 2-complex representing pairwise and triplet KL merge barriers between experts. By applying a discrete Hodge decomposition to this complex, the method isolates the harmonic component to identify irreducible cycles of experts that are pairwise compatible but collectively costly to merge. HodgeCover then uses a greedy submodular objective to select a survivor expert set covering these harmonic-critical edges and triplet-critical triangles. The method is evaluated on OLMoE and Qwen 3.5 architectures, demonstrating strong downstream performance in hybrid settings where expert reduction is combined with unstructured Wanda weight pruning.

Critical strengths

Novel Topological Framing: Applying the Hodge decomposition to isolate the harmonic residual of MoE mergeability is a creative, elegant approach. It effectively captures higher-order structural constraints that purely pairwise scoring methods miss, offering a fresh perspective on expert routing topology.

Strong Hybrid Compression Results: In the hybrid setting (+Wanda), the method achieves excellent downstream performance at aggressive 66% compression rates across different model scales. The preservation of model capabilities on complex reasoning tasks---such as maintaining 85.5% on GSM8K for Qwen 3.5-35B, compared to a collapse to 22.1% for STUN+Wanda---strongly supports the utility of the topological coverage objective in practical hybrid pipelines.

Critical weaknesses

Calibration coverage and robustness at high expert counts: Appendix B.10 reports sporadic zero-frequency experts on Qwen under the 2,048-token C4 calibration sample. The averaging fallback prevents undefined merges, but the paper does not quantify the prevalence of low-support experts or establish whether their estimated mergeability generalizes beyond this sample. Table 4 also shows non-monotone calibration-size sensitivity on OLMoE: DS-Avg changes from 48.57 to 45.45 to 48.90 with 2,048, 4,096, and 8,192 tokens. Corresponding calibration-size and calibration-seed experiments are not reported for the 256-expert Qwen models. This leaves uncertainty about the reliability of the calibration-derived structure at the highest expert cardinality.

Limited incremental gains over strong baselines at larger scale: On the Qwen models, HodgeCover’s pure expert-reduction DS-Avg remains within 0.3 pp of REAP (Table 1). Its 66% hybrid gains over matched REAP+Wanda and REAM+Wanda (for Qwen variants) controls are also modest, at 0.3--0.8 pp (Table 7). These results temper the large gains shown against STUN+Wanda. Because calibration-seed replicates cover only OLMoE, the robustness of these small Qwen advantages remains insufficiently characterized.

Questions for authors
  1. How does the Hodge decomposition mathematically or procedurally distinguish between genuine functional mergeability and artificial inactivity artifacts caused by the sparse 2,048-token sample on 256 experts? Could this sampling sparsity explain the non-monotone performance drop observed when scaling calibration tokens to 4k in Table 4, as well as the lack of separation from REAP on the pure expert-reduction axis for the 256-expert Qwen models?

  2. Why is the calibration size ablation (Table 4) omitted for the Qwen models? How does the method perform on Qwen 3.5-35B when the calibration set is scaled to 8k or 16k tokens to ensure all experts are adequately routed (please show the relevant stats also)? Given the O(n2∣D∣d2)O(n^2 \vert D \vert d^2) scaling, how computationally feasible is it to scale the calibration dataset to larger post-training compression sizes (e.g., 32k+ tokens) (which I believe can improve stability and improve expert coverage)?

  3. While Appendix D.7 states that the Hodge construction is metric-agnostic and could accept activation-based distributional metrics, have you empirically validated whether the harmonic kernel remains robust under local hidden-state metrics, or a data-free metric (which has fewer knobs to tune)?

5 comments

Thank you for your thoughtful review and for recognizing the motivation behind the Hodge formulation and its ability to represent higher-order structure. We have extended the calibration and statistical analyses on Qwen 3.5-35B and evaluated HodgeCover's sensitivity to alternative barrier metrics.

1. Calibration size and expert coverage on Qwen (W1, Q2). We agree that a 2,048-token calibration sequence may provide limited coverage of models with 256 experts per layer. Although the original sensitivity study showed stable survivor selection under changes to triangle sampling and other hyperparameters (App. D.2), it did not directly establish robustness to calibration size.

We therefore added a calibration-size study on Qwen 3.5-35B at 66% expert-count reduction, comparing HodgeCover and REAP [1] using 2,048, 8,192, 16,384, and 32,768 C4-train tokens. The sequences are nested prefixes, and both methods use identical calibration data at each size (revised App. D.2).

Qwen 3.5-35B, 66% expert reduction2,0488,19216,38432,768
Zero-support experts/layer (of 256)8.630.830.130.03
Experts with <8 routed slots/layer52.657.301.730.53
% of E⋆E^\star edges touching <8-slot experts (all edges)20.1 (36.3)2.7 (5.6)0.7 (1.3)0.2 (0.4)
% of survivors with <8 slots (all experts)5.98 (20.57)0.75 (2.85)0.29 (0.67)0.03 (0.21)
HodgeCover DS-Avg66.6767.2967.4067.86
REAP DS-Avg66.3865.7367.0667.34
HC plan time relative to 2,048 tokens1.0×3.0×5.8×12.3×

Support is measured in routed top-kk slots per expert on the calibration sequence. Zero-support experts are never selected as survivors at any evaluated size.

HodgeCover's DS-Avg remains at or above its 2,048-token result and exceeds REAP at every calibration size, with differences of +0.29 / +1.56 / +0.34 / +0.52 pp. Paired per-example tests give +1.57 pp [95% CI: +1.12, +2.00] at 8,192 tokens (Holm-adjusted p=0.0002p=0.0002), +0.34 pp [-0.10, +0.78] at 16,384 tokens, and +0.52 pp [+0.08, +0.97] at 32,768 tokens (p=0.008p=0.008). The estimated differences remain positive, although the 16,384-token confidence interval includes zero.

Low-support experts are also underrepresented among harmonic-critical edges and survivors. At 2,048 tokens, experts with fewer than eight routed slots constitute 20.57% of all experts but only 5.98% of survivors. Their representation decreases further as calibration coverage improves.

These results argue against low-support experts being the primary driver of the Qwen comparison at 2,048 tokens. The positive difference relative to REAP persists as coverage improves.

2. Distinguishing mergeability from sparse-routing artifacts (Q1). The Hodge decomposition itself does not distinguish functional mergeability from calibration artifacts: it decomposes the supplied pairwise-barrier signal (Thm. 1). The distinction instead depends on the barrier construction, saliency, selection procedure, and empirical coverage.

First, the layer-local barrier incorporates the router weights of the experts being merged. For an expert ii receiving no calibration tokens, wi=0w_i=0, and its pairwise barrier with an active expert jj reduces to the discrepancy between the parameter-averaged merge and jj on tokens routed to jj (revised Sec. 3.1). Thus, the barrier still evaluates the effect of the merge on the active expert's routed tokens.

Second, HodgeCover uses REAP output saliency, normalized per layer (Sec. 4.1). An expert receiving no calibration tokens has zero saliency, reflecting its lack of contribution to the layer output on that sample.

Third, the support analysis measures whether low-support experts disproportionately influence the critical-edge set or survivor selection. As shown above, they are underrepresented in both, and zero-support experts are never retained in the evaluated Qwen configurations.

Regarding the non-monotone OLMoE results in Table 4, DS-Avg changes from 48.57 to 45.45 to 48.90 at 2,048, 4,096, and 8,192 tokens. However, no OLMoE expert is unrouted at 2,048 tokens (App. B.10). The degradation at 4,096 tokens therefore cannot be explained simply by zero-support experts in the smaller sample. It is consistent with sample-to-sample variation in routing statistics and estimated barriers, and the 8,192-token result exceeds the 2,048-token result.

The narrow gap relative to REAP has a separate possible explanation. Sec. 5.3 shows that REAP's survivors retain similar proportions of the gradient, curl, harmonic, and triplet-barrier signals, despite not explicitly optimizing Hodge coverage. This is consistent with the methods' comparable performance in pure expert reduction.

The Qwen calibration study further shows that increasing calibration size neither consistently widens nor eliminates the difference relative to REAP. The evidence supports robustness of the comparison across the evaluated calibration sizes.

3. Performance margins and calibration-seed robustness (W2). We agree that the Qwen differences relative to strong baselines are modest. HodgeCover matches or exceeds REAP across all four Qwen expert-count configurations (+0.0 to +0.3 pp DS-Avg; Table 1). At 66% hybrid compression, HodgeCover+Wanda leads the evaluated hybrids on all three backbones, including the matched REAP+Wanda and REAM+Wanda controls as well as STUN+Wanda [2, 3]. Revised Sec. 5.2 reports these comparisons consistently, including the 33% results.

Paired per-example tests at 66% on Qwen 3.5-35B give +0.16 pp for pure HodgeCover versus REAP (Holm-adjusted p=0.04p=0.04), +0.51 pp for HodgeCover+Wanda versus REAP+Wanda (p=0.0091p=0.0091), and +0.57 pp versus REAM+Wanda (p=0.003p=0.003; Table 8, App. F.3). These tests characterize evaluation uncertainty for fixed compressed models.

To assess calibration variability, we added two Qwen 3.5-35B calibration seeds at 66% hybrid compression (revised App. F.3):

SeedHC+WREAP+WREAM+W
4274.60 (10.25 / 15.22)74.07 (10.57 / 15.78)73.75 (10.48 / 15.68)
4375.58 (9.41 / 15.31)74.52 (10.14 / 15.76)75.13 (9.67 / 15.62)
4473.03 (9.90 / 15.66)72.16 (10.41 / 16.30)72.58 (10.28 / 16.12)
Mean ± SD74.40 ± 1.2873.58 ± 1.2573.82 ± 1.28

Entries report DS-Avg (WikiText perplexity / C4 perplexity). HodgeCover+Wanda achieves higher DS-Avg and lower perplexity than both matched hybrids at every seed.

Paired per-example DS-Avg differences [95% CI] for HodgeCover+Wanda are:

  • Versus REAP+Wanda: +0.51 [+0.14, +0.88], +1.07 [+0.71, +1.43], and +0.88 [+0.51, +1.26].
  • Versus REAM+Wanda: +0.57 [+0.20, +0.91], +0.45 [+0.10, +0.79], and +0.46 [+0.09, +0.81].

The differences correspond to seeds 42, 43, and 44, respectively. All six intervals exclude zero.

The original OLMoE calibration-seed experiments likewise show positive margins at each evaluated seed (App. F.3). These results support consistent advantages in the evaluated hybrid settings across calibration seeds and matched evaluation examples.

4. Computational scaling with calibration size (Q2). The barrier sweep has complexity O((n2+∣T∣)∣D∣d2)O((n^2+|T|)|D|d^2), where nn is the expert count, ∣T∣|T| the number of evaluated triplets, ∣D∣|D| the calibration token count, and dd the expert hidden dimension (App. C.5).

For fixed model dimensions and triplet budget, barrier computation scales linearly with calibration size. Increasing the token count from 2,048 to 32,768 therefore multiplies the token-dependent work by 16.

On Qwen 3.5-35B, total planning time increases by 3.0× / 5.8× / 12.3× at 8,192 / 16,384 / 32,768 tokens, respectively, on the same hardware (revised App. D.2). The barrier sweep increases by 3.5× / 6.9× / 15.0×, consistent with linear scaling, while Hodge decomposition and selection remain approximately constant (1.00-1.05×).

Like REAP [1], Wanda [3], and SparseGPT [4], HodgeCover operates in a one-shot, learning-free compression setting, with planning performed offline rather than during inference. The mergeability complex is also cached across compression rates (App. C.6).

These measurements establish computational feasibility through 32,768 calibration tokens on the evaluated 256-expert model. Because cost scales linearly with calibration tokens, larger calibration sets can be processed with the same procedure.

5. Sensitivity to alternative barrier metrics (Q3). We agree that compatibility with alternative metrics does not establish empirical robustness. We therefore added two experiments evaluating the sensitivity of HodgeCover to the barrier metric (revised App. D.7).

Layer-local proxy versus full-model KL. All experiments use a layer-local output-discrepancy proxy motivated by the second-order expansion of the KL objective. The barrier is computed as the mean over calibration tokens of ∥wifi+wjfj−(wi+wj)fθˉ∥\lVert w_i f_i + w_j f_j - (w_i+w_j)f_{\bar\theta}\rVert, where θˉ\bar\theta is the parameter-averaged merge and wi,wjw_i,w_j are router weights (revised Sec. 3.1, after Eq. 2).

We compared this proxy with the full-model KL barrier on OLMoE at 33% expert reduction (layers 0, 5, 10, 15). We computed full-model KL barriers for all 2,016 expert pairs in each layer, then rebuilt the triangle sets and HodgeCover plans using the same sampling rule, saliency, and selection settings. Rank correlations and most-mergeable-pair overlaps were evaluated on 200 sampled pairs per layer.

LayerSpearman correlationTop-10% mergeable-pair overlapρharm\rho_{\mathrm{harm}}, proxy / KLSurvivor Jaccard
00.600.550.49 / 0.370.909
50.810.750.38 / 0.391.000
100.600.700.31 / 0.321.000
150.840.800.29 / 0.370.909

The proxy and full-model KL barriers exhibit moderate-to-strong rank correlations (0.60-0.84), with overlap among the lowest-barrier pairs of 0.55-0.80, compared with 0.10 by chance under the reported protocol. Harmonic energy fractions differ more in some layers, but the resulting survivor sets remain similar (Jaccard 0.909-1.000).

These results support the stability of expert selection under the two metrics in the evaluated layers.

Data-free weight-space barriers. We also replaced activation-derived barriers with parameter-space distances:

bij=∥θi−θj∥F,b_{ij}=\|\theta_i-\theta_j\|_F,

where θi\theta_i concatenates the expert's weight matrices. The triplet barrier sums the distances of the three experts to their parameter centroid. The same triangle-sampling procedure is used. No calibration activations enter the barriers, although saliency remains calibration-derived.

Mean survivor-set Jaccard indices relative to the original activation-based plans are 0.963 / 0.859 for OLMoE at 33% / 66% and 0.992 / 0.949 for Qwen 3.5-35B at the same rates (revised App. D.7).

These results show substantial agreement between survivor sets obtained with activation-based and weight-space barriers. However, changing the metric can also change the sampled triangle set and resulting Hodge decomposition. Selection is therefore not strictly metric-invariant, as reflected particularly by the OLMoE 66% result.

The experiments establish stability of the selected experts under the evaluated alternative metrics.

All revisions are marked in blue in the revised manuscript.

References
[1] Lasby et al., arXiv 2025
[2] Lee et al., ACL 2025
[3] Sun et al., arXiv 2023
[4] Frantar and Alistarh, ICML 2023

Reviewer_rJzWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HodgeCover compresses MoE models without retraining by combining expert saliency with coverage of harmonic-critical edges and high-barrier triangles. It applies Hodge decomposition to pairwise merge barriers, selects surviving experts, and redirects dropped experts to them. A hybrid variant adds Wanda pruning. Experiments on three language models show the strongest gains under hybrid compression.

Critical strengths
  1. Useful hybrid-compression results. At 66% hybrid compression, Table 7 reports gains of 3.5, 0.5, and 0.3 percentage points over the strongest other matched hybrid method on OLMoE and the two Qwen models, respectively.
  2. Substantive algorithmic and experimental support. The coverage objective has a valid greedy approximation guarantee under the reported initialization. Sensitivity studies, paired tests, and OLMoE calibration-seed repeats strengthen the evaluation.
Critical weaknesses
  1. The central interpretation exceeds the theory. The projection identity and restricted-selector results do not establish that harmonic energy is an intrinsic higher-order mergeability obstruction. Its dependence on expert numbering and the conditional loss connection in Appendix B.7 further limit this interpretation.
  2. The harmonic contribution is not isolated. Several ablations change both expert selection and weight merging/redirection. No-Triangle and the additive selector retain harmonic information. Consequently, these comparisons cannot attribute the gains specifically to Hodge decomposition (Appendices C.7 and F.1; Table 9).
Questions for authors
  1. Which result establishes the claimed higher-order loss interpretation beyond the projection identity and the restricted summaries considered in Proposition 5?
  2. With physical triangles fixed, how do consistent permutations of expert identities and router entries affect harmonic energy and survivor selection?
  3. Can you remove harmonic information while keeping saliency, triplet signals, budgets, survivor weights, redirection, and the Wanda schedule fixed?
  4. Beyond the stated scale coverage, why choose OLMoE and two Qwen models rather than include Mixtral-8×7B? What supports applicability to models with fewer experts per layer?
6 comments

Thank you for verifying the greedy approximation guarantee, for highlighting the 66% hybrid margins, and for crediting the sensitivity studies, paired tests and seed repeats. The revision states the interpretation at the scope of the theory (contribution 1 reworded) and adds the controlled ablation requested in Q3.

1. Isolating the harmonic contribution (W2 + Q3). We agree that several ablations change more than one factor. Triplet-Penalty and Greedy-Barrier are full-pipeline comparisons with merge-based survivor treatment (App. F.1, App. C.7), so they do not isolate the contribution of harmonic information.

The additive selector in App F.4 addresses a different question. It uses the same harmonic-critical and triplet-critical quantities as HodgeCover, with identical allocation, redirection, survivor treatment, and calibration. It therefore compares greedy coverage with additive scoring rather than testing the removal of harmonic information.

To isolate the harmonic contribution, we added controlled variants in revised Sec 5.4 and App F.1. Harmonic information enters HodgeCover through two components: the harmonic-critical edge set (E⋆E^\star) used for survivor selection and the Hodge-weighted redirect cost (Eq. 13). The new variants modify these components individually while holding saliency, the triangle set, per-layer budgets, survivor treatment, and the remaining compression procedure fixed.

The controlled configurations include removing harmonic weighting from redirection α=0\alpha=0, replacing harmonic-critical edges with the same number of highest-barrier edges while also setting α=0\alpha=0, removing the edge-coverage term, removing the triangle-coverage term, and retaining only saliency. Table R5 reports the results on Qwen 3.5-35B at 66% expert-count reduction:

ConfigurationDS-AvgWikiC4
HodgeCover66.6715.1318.86
Triangle term removed (λt=0\lambda_t = 0)66.2114.9618.65
Harmonic selection and redirection removed65.8314.9618.59
Edge term removed (λe=0\lambda_e = 0, α=0\alpha = 0)66.4315.0018.69
Saliency only66.4214.9918.72
Harmonic redirection removed (α=0\alpha = 0)66.5515.0718.80

Full HodgeCover achieves the highest DS-Avg in this comparison. Removing harmonic redirection reduces DS-Avg by 0.12 pp. With harmonic redirection disabled in both configurations, replacing harmonic-critical edges with raw-barrier edges reduces DS-Avg by a further 0.72 pp, isolating the contribution of harmonic-based selection relative to this alternative. Removing the triangle term reduces DS-Avg by 0.46 pp. Perplexity varies only slightly across variants, and the downstream accuracy results show the contributions of harmonic selection, harmonic redirection, and triplet coverage in this configuration.

We also corrected the description of the former "No-Triangle" ablation. Revised Secs 5.1 and 5.4 and Apps F.1 and F.7 now identify it as Hodge-MSF (pairwise complex, merge-based), accurately describing the harmonic minimum-spanning-forest merging procedure. Its results are unchanged. The contribution of the triangle-coverage term is instead evaluated by the controlled λt=0\lambda_t=0 configuration above.

These experiments provide a more direct assessment of the separate contributions of harmonic selection, harmonic redirection, and triplet coverage.

2. Theoretical interpretation and connection to compression loss (W1, Q1). We agree that the projection identity alone does not establish an unconditional relationship between harmonic energy and compression loss. We have clarified the distinction between the algebraic characterization, the exact layerwise model, and the conditional connection to calibration KL (revised Sec. 3.3, App. B.7).

Unconditional characterization. Prop. 1 establishes that ∣bharm∣2|b_{\mathrm{harm}}|^2 is exactly the squared distance between the oriented pairwise-barrier signal (b) and the subspace of vertex-potential and triangle-boundary contributions on the chosen simplicial complex. Thus, the harmonic component is the orthogonal residual of this decomposition. For the edge-additive redirect proxy, the error from omitting the harmonic component is exactly the harmonic exposure of the redirect edges, ⟨Pharmwa,bharm⟩\langle P_{\mathrm{harm}}w_a,b_{\mathrm{harm}}\rangle. This identity holds for the proxy. Cor. 1 below extends the analysis to calibration KL.

Exact connection in the layerwise model. Prop. 4 considers top-1 routing, squared output error, token-map redirection, and the stated conditions on survivor routing rates and discrepancies. Under these assumptions, the quadratic compression cost equals a routing-weighted sum of quadratic merge barriers: $ C(a)=\langle w_a,b^{(2)}\rangle. $ Prop. 5 examines realizable pairs of instances whose barrier signals differ by a harmonic perturbation but whose lower-order signals, routing information, and expert norms agree. Selectors restricted to these summaries cannot distinguish the instances, and the proposition establishes a minimax regret lower bound in terms of harmonic exposure. This result applies to the specified model and restricted information class, not arbitrary expert-selection methods.

Conditional connection to calibration KL. Cor. 1 extends the harmonic-exposure analysis to calibration KL under the edge-exposure linearization of Eq. 24. The prediction error contains both harmonic exposure and an explicit residual ξa\xi_a, bounded by ε\varepsilon. Thus, the general KL interpretation depends on the accuracy of this approximation. The zero-residual identity of Prop. 4 is specific to its quadratic layerwise model.

We revised Contribution 1 accordingly:

We define the simplicial mergeability complex of a sparse MoE layer and characterize the harmonic component of its Hodge decomposition as the part of the pairwise barrier signal that no vertex-potential or triangle-boundary explanation recovers (Prop. 1), with a loss link in a layerwise model (Props. 4–5).

The empirical analyses in Secs. 5.3–5.4 and App. F.4 provide additional evidence that these quantities are useful for expert selection. In particular, the controlled ablation shows a 0.72 pp DS-Avg advantage for harmonic-critical edge selection over raw-barrier selection when harmonic redirection is disabled. This supports the practical relevance of the harmonic signal without extending the theoretical claims beyond their assumptions.

3. Dependence on expert numbering (W1, Q2). We agree that the distinction between the topology of the complex and the orientation-dependent barrier signal should be explicit.

As clarified in Sec. 3.2 and App. B.3, the dimension and spectrum of the harmonic kernel are independent of reference orientation, up to the corresponding signed change of basis [1]. However, pairwise merge barriers are symmetric, nonnegative quantities embedded as a cochain using the lexicographic orientation induced by expert indices. Consequently, the projected harmonic coordinates, harmonic energy fraction, and critical-edge selection can depend on the reference labeling.

We added 20 seeded consistent relabelings on OLMoE and Qwen 3.5-35B (revised App. B.3). Each relabeling permutes expert identities and corresponding router entries together, preserving the uncompressed model's function. Pairwise barriers and the physical triangle set are carried through the permutation, after which HodgeCover recomputes its decomposition and survivor selection. Selected experts are mapped back to their original identities for comparison.

The results are summarized below. Harmonic-energy values are layer means, reported for the identity labeling and as mean [minimum, maximum] across relabelings. Survivor-set Jaccard values are median [minimum].

MeasureOLMoE 33%OLMoE 66%Qwen 3.5-35B 66%
Harmonic energy, identity0.3520.3520.503
Harmonic energy, relabeled0.346 [0.328, 0.363]0.346 [0.328, 0.363]0.512 [0.495, 0.526]
Survivor Jaccard1.000 [0.909]0.909 [0.750]0.977 [0.912]

We also evaluated three relabelings end-to-end on Qwen 3.5-35B at 66% expert reduction:

LabelingDS-AvgWikiC4
Original66.6715.1318.86
Relabeling 166.4914.8118.81
Relabeling 266.8814.6918.78
Relabeling 366.3014.8118.87

The harmonic-energy fractions and survivor sets show substantial agreement across relabelings, particularly on Qwen 3.5-35B. Downstream accuracy on the three evaluated Qwen relabelings remains within 0.37 pp of the original configuration. Nevertheless, selection is not exactly invariant: the minimum survivor Jaccard on OLMoE at 66% is 0.750. These results indicate limited sensitivity in the evaluated settings while confirming that the orientation convention can affect the decomposition and selected experts.

4. Model selection and applicability to fewer experts (Q4). We selected OLMoE and the two Qwen models to evaluate expert selection in regimes with many candidates per layer. OLMoE [6] has 64 experts per layer, while both Qwen models [8] have 256. This regime reflects current practice: fine-grained experts, introduced by DeepSeekMoE [4] and shown to be compute-optimal at larger budgets [5], are now standard, and DeepSeek-V3 [7] also uses 256 routed experts. Our evaluated models span model sizes from 7B to 122B total parameters (Sec. 5.1).

The number of candidate survivor sets grows combinatorially with expert count. Exhaustive subset selection [2] can be practical on an eight-expert architecture such as Mixtral [3], but becomes infeasible at 256 experts. The pairwise barrier graph grows from (82)=28\binom{8}{2}=28 edges to (2562)=32,640\binom{256}{2}=32{,}640, and the harmonic subspace has dimension $ \beta_1(K)=|E|-n+1-\operatorname{rank}(\partial_2) $ for the complete-edge construction. The 64- and 256-expert settings therefore test selection among large candidate sets.

HodgeCover itself does not require many experts. At n=8n=8, all 28 pairwise barriers can be evaluated, and there are at most (83)=56\binom{8}{3}=56 possible triplets. The triangle-sampling cap of 500 does not bind. The construction retains qualifying triplets identified by the procedure in App. B.9. Hodge decomposition and greedy coverage remain well-defined, although the harmonic component depends on the resulting complex and barrier signal.

We recognize that algorithmic applicability does not establish empirical compression performance. Our experiments demonstrate effectiveness on 64- and 256-expert layers, where the candidate space is largest. The revision states this evaluation scope.

All revisions are marked in blue in the revised manuscript.

References
[1] Lim, SIAM Review 2020
[2] Lu et al., ACL 2024
[3] Jiang et al., arXiv 2024
[4] Dai et al., ACL 2024
[5] Ludziejewski et al., ICML 2024
[6] Muennighoff et al., arXiv 2024
[7] Liu et al., arXiv 2024
[8] Qwen Team, 2026

Reviewer_s8cyOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a mathematical framework for expert selection in sparse MoEs and discuss the challenges associated with prior methods that rely on pairwise selection. Building on this framework, the authors propose HodgeCover, a learning-free compression methods that captures higher-order interactions between experts. They propose a greedy algorithm to select which experts to retain. They augment their expert selection with unstructured sparsity using Wanda.

Critical strengths

1/ Sparse MoEs is an impactful problem to work on. The authors clearly indicate the challenges associated with prior methods relying primarily on pairwise selection of experts. They provide a mathematical formulation based on Hodge decomposition, and propose a novel expert-selection strategy using greedy submodular optimization.

2/ The ablation study provided in subsection 5.4 largely strengthens the choices of the authors for the proposed algorithm.

Critical weaknesses

1/ The paper is somewhat difficult to follow. In the introduction, several technical concepts such as "merge barriers" and "harmonic kernel" are introduced without direct references to their formal definitions later introduced in the main text.

2/ The experimental results of Table1 are not very convincing compared to REAP (the method relying on pairwise expert selection). On the pure expert-reduction axis (the entire motivation of the paper), HodgeCover achieves only marginal improvements on the two Qwen models. For the hybird versions, I think it's fair if the authors also included REAP+Wanda in the main text as REAP seems to be the most competitive among "Expert-count reduction methods".

Questions for authors

1/ Is there an intuitive reason to understand why post-compression throughput of HodgeCover+Wanda is faster than REAP+Wanda’s? Are the inference speedups obtained for different datasets?

2/ Can the authors discuss what seem like minimal gains against REAP in Table 1 for "Expert-count reduction methods"?

3/ Why are the wall clock runtimes missing from the main paper and deferred to the appendix? On the hardware side, can the authors explain how come MoE selection takes place in a few seconds? Aren't we supposed to evaluate (n choose 2) merge barriers across multiple layers? Is this a fast approach and did the authors provide any novelty on that front compared to previous methods?

4 comments

Thank you for your thoughtful review and for recognizing the motivation behind higher-order mergeability, the Hodge formulation, and the greedy selection strategy. We address your questions about performance comparisons, computational tractability, throughput, and presentation below.

1. Gains relative to REAP in the expert-count setting (W2, Q2). We agree that the differences relative to REAP are small. HodgeCover matches or exceeds REAP [1] across all four Qwen configurations, with DS-Avg differences of +0.1 / +0.3 / +0.0 / +0.1 percentage points (Table 1). We have revised Section 5.2 to characterize these results as comparable performance rather than substantial superiority. At 66% expert reduction, HodgeCover also leads REAM [2] and MC-SMoE [3] by 6–16 pp.

Paired per-example testing on Qwen 3.5-35B at 66% expert reduction gives a +0.16 pp difference between HodgeCover and REAP (Holm-adjusted p=0.04p=0.04; Table 8, App. F.3). The paired tests use matched evaluation examples, so their differences can vary from the Table 1 point estimates. The result supports a statistically significant positive difference in the tested configuration.

The mechanism analysis in Section 5.3 offers a possible explanation for the narrow gap. REAP's selected experts preserve similar proportions of the gradient, curl, harmonic, and triplet-barrier signals, even without explicitly optimizing Hodge-based coverage. In contrast, Greedy-Barrier and MC-SMoE exhibit larger deviations in retained curl and triplet-barrier mass (approximately 0.10–0.25), consistent with their larger performance gaps.

The hybrid setting provides additional evidence under aggressive compression. At 66%, HodgeCover+Wanda achieves the highest DS-Avg among all four hybrids and the lowest WikiText and C4 perplexities on all three backbones (Table 7, App. F.2). On Qwen 3.5-35B, it also leads both matched hybrids at all three calibration seeds, with mean DS-Avg of 74.40±1.2874.40\pm1.28, compared with 73.58±1.2573.58\pm1.25 for REAP+Wanda and 73.82±1.2873.82\pm1.28 for REAM+Wanda (mean ±\pm SD). Per-seed paired 95% confidence intervals exclude zero for both comparisons at every seed (revised App. F.3). These results support consistent gains in the evaluated hybrid setting.

2. REAP+Wanda and REAM+Wanda comparisons (W2). REAP+Wanda and REAM+Wanda are matched controls constructed by pairing REAP and REAM with Wanda [4] (Section 5.1). Neither hybrid is reported in its source paper. The original submission included both at 33% and 66% compression on all three backbones (Table 7, App. F.2). We agree that these comparisons should also appear in the main results and have added both hybrids to Table 1 with full per-task results. The revised table shows that HodgeCover+Wanda leads both matched controls at 66% on all three backbones, and at 33% it leads on OLMoE and stays within 0.2 pp on the Qwen models.

3. Planning time and computational tractability (Q3). Like SparseGPT [5] and REAP [1], HodgeCover operates in a one-shot, learning-free compression setting, with planning performed offline rather than during inference. The mergeability complex is computed once per model and cached across compression rates. On Qwen 3.5-35B, the full first-run planning time at the 33% rate is 25.0 seconds for HodgeCover and 36.6 seconds for HodgeCover+Wanda, compared with 6.3–55.8 seconds for the evaluated baselines (Table 14, App. F.9). Subsequent compression rates reuse the cached complex and require single-digit seconds. We have added the planning times to Section 4.1.

The measured planning times are enabled by limiting triplet-barrier evaluations and reusing intermediate computations. HodgeCover evaluates all (n2)\binom{n}{2} pairwise barriers but at most 500 triplet barriers per layer, sampled from 3-cliques of the below-median-barrier graph (App. B.9). For Qwen 3.5-35B, this amounts to (2562)×40=1,305,600\binom{256}{2}\times40=1{,}305{,}600 pairwise barriers.

Each candidate barrier does not require a separate full-model forward pass. Instead, the implementation evaluates a layer-local output-discrepancy proxy for the KL objective (revised Section 3.1). Each expert is a SwiGLU gated MLP,

f(x)=Wdown(σ(Wgatex)⊙Wupx).f(x)=W_{\mathrm{down}}\bigl(\sigma(W_{\mathrm{gate}}x)\odot W_{\mathrm{up}}x\bigr).

The candidate merge averages the two experts' parameters. Because the input projections are linear, the merged expert's WgatexW_{\mathrm{gate}}x and WupxW_{\mathrm{up}}x can be obtained by averaging cached per-expert projections. The remaining nonlinear activation and output projection are then evaluated in batches using fused kernels (revised App. C.5). Hodge computations likewise reuse cached projector factors.

These implementation choices make barrier evaluations computationally feasible at the evaluated scale of 256 experts per layer. Our primary contributions are the mergeability complex and the Hodge-based selection objective. The implementation enables their practical evaluation at the studied model scales.

4. Inference throughput (Q1). HodgeCover's performance gains come with prefill throughput parity among the matched hybrids, introducing no inference cost. At 66% compression, all hybrids use the same 20% Stage-1 expert reduction (r1=0.20r_1=0.20, App. C.4), retain the same number of experts and top-kk routing, and have comparable per-token computation. Table 15 reports throughput differences within 8%, with rankings varying across backbones.

The measurements use a fixed 512-token prompt rather than separate prompts from each evaluation dataset. Revised Appendix E.4 specifies the common protocol: one prefill forward pass, batch size 1, one warm-up, and five timed runs, reporting the mean. Revised Section 4.1 and Appendix F.9 characterize the results as throughput parity rather than a systematic speed improvement. These measurements concern prefill rather than autoregressive decoding. Planning is performed offline, so the selection procedure introduces no additional per-token inference computation.

5. Forward references (W1). Thank you for identifying these omissions. We added references at first use to the merge-barrier definition (Eq. 2, Section 3.1) and the harmonic kernel ker⁡(L1)\ker(L_1) (Sections 3.2–3.3, Theorem 1). We also revised Section 3.2 to introduce the harmonic terminology alongside its definition.

All revisions are marked in blue in the revised manuscript.

References
[1] Lasby et al., arXiv 2025
[2] Jha et al., arXiv 2026
[3] Li et al., arXiv 2023
[4] Sun et al., arXiv 2023
[5] Frantar and Alistarh, ICML 2023

I appreciate the authors' efforts during the rebuttal. The writing has improved a lot. My concerns have been mostly addressed. I think the results of HodgeCover+Wanda on OLMoE 1B-7B with rate 66% do show the method's performance compared to competitors (I see this as a certificate that there is valuable insights from the approach). The remaining experimental results are not convincing. I recommend authors look at higher compression ratios, that's generally how improved compression methods cleanly outperform competitors. For the moment, I will maintain my rating given the large edits made to the initial manuscript, the other reviewers' raised concerns, and the current lack of additional impressive experimental results. I will be monitoring other reviewers' and authors' responses before giving my final assessment.

Thank you for the follow-up, for noting the improved writing, and for reading the OLMoE 66% hybrid result as evidence of the value of the approach.

A. Compression convention. We agree that higher compression is where improved methods separate most cleanly. Our rates denote the share of experts or weights removed, not retained. In the expert-count setting, 66% removes two thirds of the expert count per layer, using the same uniform per-layer convention as REAP [1] and REAM [2]. In the hybrid setting, 66% is the share of routed-expert weights removed: Stage 1 drops 20% of the experts and Wanda prunes the surviving experts' weights to reach the 66% total (App. C.4). This expert-count rate is beyond the rates at which REAP [1] and REAM [2] report their results (25% and 50% of the experts in each layer) and the 25% and 50% expert removal evaluated by NAEE [3] (6 or 4 of 8 experts kept per layer). This places our 66% setting in the high-compression regime you describe, and it is where our gains concentrate. At 66%, HodgeCover+Wanda has the best DS-Avg of all four hybrids on all three backbones, leads both matched hybrids at every Qwen 3.5-35B calibration seed, and leads STUN+Wanda [4] by up to +12.6 pp.

B. On the expert-count setting, the discussion with Reviewer ifEw adds a comparison with REAP at REAP's calibration setting of 1,024 samples × 2,048 tokens. HodgeCover matches or slightly exceeds REAP in DS-Avg at both 33% and 66% at that setting. HodgeCover's standing against REAP at high compression is therefore not dependent on our selected calibration budget.

C. The revision keeps the submitted method, protocol and numbers, and adds the studies the reviewers requested. The follow-up to Reviewer ifEw has details on the scope of the changes.

We would be glad to discuss the high-compression results or any aspect of the convention in more detail if that would be useful.

References
[1] Lasby et al., arXiv 2025
[2] Jha et al., arXiv 2026
[3] Lu et al., ACL 2024
[4] Lee et al., ACL 2025

Reviewer_ifEwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose an offline expert pruning method for sparse MoEs, HodgeCover, where they use pairwise and three-way mergeability of experts as a signal for expert importance. They describe formally the "obstruction" preventing three-way mergeability even when pairwise merging of experts is possible, and present empirical comparisons of downstream task accuracy as well as wikitext-103 and c4 perplexity across three models (Olmo 1b-7b, Qwen 3.5 35B-A3B, Qwen 3.5 122B-A10B) and three alternative expert selection methods (REAP, REAM, MC-SMoE). Additionally, they present a "hybrid" method that combines their HodgeCover method with Wanda, an unstructured pruning method; they compare this hybrid method with combinations of Wanda with REAP, REAM, and STUN, though only STUN is presented in the main body

Critical strengths

I will admit I did not read Section 3 as carefully as the other parts of the paper, so it is possible that it contains valuable ideas and contributions that stand on their own. The high level premise and motivation of the stated contributions are indeed potentially interesting, though I did not find any explicit explanation of the intuitions behind them

Critical weaknesses
  • Presentation: the writing is extremely hard to follow. While I completely understand that presentation in initial submissions can be less polished than ideal and can even lead to substantial ambiguities, in this case the presentation issues are severe and pervasive enough that it interferes with communication of contributions and claims, even with reasonable effort to read and understand. It is difficult to tell what is an intentional claim or contribution vs stylistic flair atypical of academic writing vs a standard term in the literature that I am simply unfamiliar with. (To be clear, I truly would normally tend to assume the latter, but it is also clearly not the case that all ambiguity is attributable to gaps in my understanding of the literature) Non-exhaustive list of presentation issues:
    • The first two sentences in the abstract contain multiple ambiguities, probably at least one outright incorrect statement, and arguably misrepresents the submitted work's contributions:

    Sparse Mixture-of-Experts (MoE) layers route tokens through a handful of experts, and learning-free compression of these layers reduces inference cost without retraining. An obstruction limits every existing compressor in this family: three experts can each be pairwise compatible yet form an irreducible cycle when merged together, so any score that ranks experts on pairwise signals is structurally blind to which triples are jointly mergeable. - What exactly is "this family" of "existing compressors"? Learning-free compression objectives? Software carrying out compression of experts? Only "any score that ranks experts on pairwise signals"? (The last one is maybe the most favorable reading I can think of given the surrounding context, but this is not at all a natural way to structure a sentence or present information) - "an obstruction limits every existing compressor in this family"-- is "an obstruction" used in the sense of a loss "barrier" or is it meant to be a synonym for a general "limitation"? - The first sentence does not sufficiently scope the space of related work for the second sentence to hold in soundness. MoE expert "compression" is much more general than "expert selection," which is much more general than the set of expert selection methods that use "pairwise signals," which is more general than the set of pairwise signals for expert selection that explicitly rely on pairwise mergeability, where parameter "merging" itself can mean many things. Read as is, the second sentence incorrectly implies that some general set of "every existing compressor in this family" is limited by failure to consider three-way mergeability, when the authors themselves fail to explicitly motivate expert mergeability itself as a reasonable signal for expert selection (see directly below) - If three experts are merged together, how exactly does this create an "irreduceable cycle"?

    • Please correct me if I simply missed an intuitive explanation or justification of the motivations behind the paper, but nowhere did I see any sort of explicit statement describing expert mergeability as a promising signal for expert selection. To be clear, I am not arguing against the principle-- I actually find it generally compelling-- but I was disappointed to see that it was not explained
    • The authors describe "four" hodge components in multiple spots, but to my understanding a Hodge decomposition yields three components: an exact form (gradient), a coexact form (curl), and a harmonic form
    • Non-standard use of terms like "axis," "knob," "cell," "matched," and "survivor set" throughout the manuscript. Also probably overloaded? ctrl+f "axis" / "axes" -> seems to refer at various points to lines of related work, experiments, and chart axes
    • 5.2's readability is particularly concerning
  • The paper states explicitly in multiple spots that the calibration corpus is "2,048 tokens drawn from the C4 training split". Is this a single sequence of 2,048 tokens? Multiple sequences totaling 2,048 tokens? In any case, this is a very small calibration corpus for MoEs with up to 256 experts. For comparison, in REAP, the authors note that "For models with≥ 110B parameters, we select 12,228 samples with a maximum sequence length of 16,384 tokens without truncation or packing"
  • Perhaps related to the concern above: Calculating three-way mergeability seems prohibitively expensive-- is it tractable with a reasonably sized calibration set?
  • Cherry-picked and misrepresented results:
    • the authors implicitly claim equivalency to a comparison method when their method falls short by a small margin ("HodgeCover is within ±0.3 pp DS-Avg of REAP on all four Qwen cells") but superiority when their method beats a comparison by a similarly small margin on a single seed ("the matched-control hybrids REAP+Wanda and REAM+Wanda trail HodgeCover+W at 66% by 0.3–0.8 pp DS-Avg on the two Qwen scales") -- see immediately below
    • "At 66% compression on three MoE backbones, HodgeCover+Wanda has the best perplexity on every model" -- the claim is tenuous as soon as we consider the full (single seed) results in Table 5. e.g. on Qwen 3.5-35B-A3B (66%), REAM+Wanda's wikitext ppl score eval is within 0.23 points of HodgeCover+Wanda on a single seed
    • In general, the 33% compression settings yield much more mixed results than the 66% settings, but the authors focus almost entirely on the 66% results for the "hybrid" methods without explicit explanation
    • In general, Table 1 reports only one "hybrid" method comparison (STUN + Wanda), when REAP+Wanda and REAM+Wanda were also run and are much more natural comparisons to report since REAM and REAP (but not STUN) are included in comparisons of "pure" expert selection methods. The results are much weaker on REAP+W and REAM+W
    • "HodgeCover+Wanda [...] leads STUN+Wanda (Lee et al., 2025) by up to +12.6 pp downstream accuracy" buries a much more modest (single-run) difference on the more natural comparisons: "REAP+Wanda and REAM+Wanda trail HodgeCover+W at 66% by 0.3–0.8 pp DS-Avg on the two Qwen scales and by 5–8 pp DS-Avg on OLMoE", where OLMoE is a much smaller research model than the Qwen models
    • However, even the "lead" of "0.3–0.8 pp DS-Avg" leaves out the two out of three 33% settings where HodgeCover+W does not have the highest reported number
Questions for authors

N/A

3 comments

Thank you for finding the premise and motivation interesting and for the detailed review, including the concrete presentation suggestions. We have revised the experimental comparisons, mathematical motivation, and presentation, and added calibration and computational analyses.

1. Ethics concern and hybrid presentation. We agree that the main table should include the strongest relevant comparisons, including cases where HodgeCover does not lead. The original submission reported all four hybrids at both rates on all three backbones (Table 7, App. F.2), including cases where HodgeCover+Wanda trails, and Section 5.2 discussed the matched-hybrid margins. Table 1 emphasized the published STUN+Wanda baseline. REAP+Wanda and REAM+Wanda were matched controls constructed by us, not reported in their source papers [1, 2]. We have now added both to Table 1 with full per-task results.

At 66% compression, HodgeCover+Wanda has the highest DS-Avg among evaluated hybrids, leading the strongest alternative by +3.5 / +0.5 / +0.3 percentage points (OLMoE / Qwen 3.5-35B / Qwen 3.5-122B), and the lowest WikiText and C4 perplexities on each backbone. At 33%, the corresponding margins are +0.1 / −0.2 / −0.1 pp: HodgeCover+Wanda leads on OLMoE (53.0 DS-Avg), while REAP+Wanda leads on Qwen 3.5-35B and REAM+Wanda on Qwen 3.5-122B. HodgeCover+Wanda nevertheless has the lowest C4 perplexity in all six model-rate configurations. The highlighted +12.6 pp improvement is relative to the published STUN+Wanda baseline [3, 4], not the matched controls. Revised Section 5.2 reports the direction and magnitude of both sets of comparisons.

Both hybrid rates remove 20% of experts in Stage 1, followed by Wanda sparsity of 16.25% or 57.5% for 33% or 66% total compression, respectively (App. C.4). The larger advantage at 66% is consistent with survivor selection becoming more consequential under aggressive weight pruning.

2. Interpretation of small performance differences. The original phrase "within ±0.3 pp" was imprecise. HodgeCover matches or exceeds REAP across all four Qwen expert-count configurations (+0.1 / +0.3 / +0.0 / +0.1 pp DS-Avg; Table 1). Revised Section 5.2 adopts this description. At 66% expert reduction, HodgeCover also leads REAM and MC-SMoE by 6–16 pp.

The Section 5.3 mechanism analysis offers a possible explanation for the narrow gap: REAP's survivor sets retain similar proportions of the gradient, curl, harmonic, and triplet-barrier signals without explicitly optimizing Hodge coverage. MC-SMoE and Greedy-Barrier exhibit larger deviations in curl and triplet-barrier retention, consistent with their larger performance gaps. This suggests that saliency-based selection can implicitly preserve some of the structure HodgeCover targets explicitly.

Paired per-example tests at 66% on Qwen 3.5-35B give +0.16 pp for HodgeCover versus REAP (Holm-adjusted p=0.04p=0.04), and +0.51 / +0.57 pp for HodgeCover+Wanda versus REAP+Wanda / REAM+Wanda (p=0.0091p=0.0091 / 0.0030.003; Table 8, App. F.3). All five tested contrasts favor HodgeCover and reject the null hypothesis. Because these tests use per-example logs of matched evaluations, their paired differences need not equal the Table 1 point gaps. They quantify evaluation uncertainty for fixed compressed models.

To assess calibration variability separately, we evaluated three seeds on Qwen 3.5-35B at 66% hybrid compression. HodgeCover+Wanda leads both matched hybrids at every seed: DS-Avg is 74.40±1.2874.40\pm1.28 versus 73.58±1.2573.58\pm1.25 (REAP+Wanda) and 73.82±1.2873.82\pm1.28 (REAM+Wanda), reported as mean ±\pm SD. Per-seed paired 95% confidence intervals exclude zero for both contrasts at every seed (revised App. F.3). The original OLMoE seed experiments also show positive margins at each evaluated seed. The evidence supports consistent hybrid advantages in these settings.

3. Calibration reliability. All methods use the same 2,048-token sequence constructed from C4 training data (Section 5.1, App. E.3). This ensures a controlled comparison. We therefore evaluated Qwen 3.5-35B at 66% expert-count reduction with 2,048 / 8,192 / 16,384 / 32,768 tokens (revised App. D.2). HodgeCover DS-Avg increases from 66.67 to 67.29 / 67.40 / 67.86, remaining above REAP at every size by +0.29 / +1.56 / +0.34 / +0.52 pp.

Mean zero-support experts per layer decrease from 8.63 to 0.03 across these calibration sizes, and experts receiving fewer than eight routed slots decrease from 52.65 to 0.53. Low-support experts are also underrepresented among selected survivors. Thus, the positive margin persists as routing coverage improves, providing evidence that the smallest calibration budget does not artificially favor HodgeCover in this comparison. Revised Appendix D.2 provides the full Qwen results and support statistics.

4. Computational tractability. HodgeCover evaluates all (n2)\binom{n}{2} pairwise barriers but at most ∣T∣≤500|T|\leq500 triplet barriers per layer, sampled from 3-cliques of the below-median-barrier graph. It does not evaluate every possible joint merge (App. B.9). Barrier computation is quadratic in expert count and linear in calibration tokens for fixed model dimensions (App. C.5).

Crucially, each candidate merge does not require a full-model forward pass. We compute a layer-local output-discrepancy proxy for the KL objective (revised Section 3.1). Each expert is a SwiGLU gated MLP with two linear input projections. A candidate merge averages the two experts' parameters. Because the input projections are linear, the merged expert's projected activations can be obtained by averaging cached per-expert projections. The remaining nonlinear activation and output projection are evaluated in batches with fused kernels, rather than a separate full-model pass per candidate (revised App. C.5). Hodge analysis likewise reuses cached projector factors.

On Qwen 3.5-35B, 1,305,600 pairwise barriers across 40 layers take 25.0 s for the full first-run HodgeCover plan and 36.6 s for HodgeCover+Wanda, versus 6.3–55.8 s for evaluated baselines (Table 14, App. F.9). Subsequent rates reuse the cached complex. Increasing calibration tokens 16-fold increases plan time 12.3-fold (revised App. D.2). Planning is offline, and the selection procedure adds no per-token inference cost. These measurements establish feasibility at the tested scale. Our methodological contribution is the mergeability complex and Hodge-based selection.

5. Presentation and conceptual clarity. We revised the abstract, introduction, and technical explanations to address the points raised.

Scope and motivation. The abstract now scopes its comparison to learning-free expert-selection methods based on per-expert saliency or pairwise merge costs, rather than implying that all previous MoE compression methods ignore higher-order structure. Revised Section 3.1 explains why mergeability is relevant: surviving experts must represent removed experts' functions, and merge barriers proxy the discrepancy introduced by redirection. Under the layerwise model's top-1 routing and quadratic-cost assumptions, Proposition 4 expresses compression cost as a routing-weighted sum of quadratic barriers. Proposition 6 connects the extended selection problem to a facility-location objective (Section 3.3).

Harmonic structure and joint mergeability. We agree that the description of an "irreducible cycle" blurred two different quantities. The harmonic component is the orthogonal residual of the oriented pairwise-barrier signal relative to the gradient and curl subspaces of the chosen simplicial complex (Prop. 1), not the cost of jointly merging three experts. That cost is measured separately by the triplet barrier. We redrew Fig. 1 and revised the text to distinguish them. We also corrected "four Hodge components" to three components (gradient, curl, harmonic) plus a separate triplet-barrier signal.

Terminology and readability. We replaced ambiguous uses of "axis," "cell," and "knob," clarified the survivor-set definition, added forward references, and reorganized Section 5.2 around expert-count reduction, hybrid compression, and statistical uncertainty, consistently reporting signs and magnitudes.

All revisions are marked in blue in the revised manuscript.

References
[1] Lasby et al., arXiv 2025
[2] Jha et al., arXiv 2026
[3] Lee et al., ACL 2025
[4] Sun et al., arXiv 2023

AuthorsOpenReview ↗

Response to Reviewer ifEw's follow-up

Thank you for the time you have spent on the paper across both rounds, for noting that the presentation and abstract improved, and for confirming that the tractability concern is mostly addressed. We share your view that this topic would interest many members of the ML research community, and we want to address the remaining points as concretely as we can.

A. Scope of the revision. We agree that the extent of the changes deserves careful consideration, and that a discussion period has limits on what it can verify. The method, the experimental protocol, and every number in the submitted tables are unchanged. The hybrid rows added to Table 1 (REAP+Wanda, REAM+Wanda) were reported in full in the submission's Table 7 (App. F.2). The new results are the studies the reviewers requested: calibration size with expert-support statistics (Reviewers ifEw and Hb1V), hybrid calibration seeds (Reviewer Hb1V), the controlled harmonic ablation and relabeling study (Reviewer rJzW), and barrier-metric comparisons (Reviewer Hb1V). The text changes are the clarifications the reviewers asked for: the abstract opening, Fig. 1, the component count, terminology, forward references, and the mergeability motivation. We view the revision as within the scope of a rebuttal. It clarifies the theoretical contribution, adds the requested studies within the one additional page the venue allows, and preserves every submitted result and the paper's core claims.

B. Language pass. We agree with the points on related-work density and on introducing coined terms before first use. To keep the reviewers' and the AC's task manageable, we will keep the manuscript as revised for the rest of the discussion and address these language points in the final version, with related work and first-use definitions of every coined term as the first targets.

C. Calibration.

We agree that calibration size matters, particularly for selectors such as REAP and REAM that are designed around larger calibration budgets. We see REAP's larger calibration regime as a reasonable setting as well. Our comparison holds every method to one fixed calibration budget, stated in the submission (Section 5.1, App. E.3), so differences between methods reflect the selection rule rather than the data each method sees. Our claims are made under this protocol.

On expert coverage, the calibration-size study in revised App. D.2 scales the budget to 32,768 tokens, where zero-support experts fall to 0.03 per layer (of 256). HodgeCover's DS-Avg stays at or above its 2,048-token value and ahead of REAP at every size.

To explore the calibration concern further, we ran HodgeCover and REAP on Qwen 3.5-35B in the expert-count setting at 33% and 66%, both calibrated with REAP's calibration setting of 1,024 samples × 2,048 tokens (~2M C4-train tokens total). At this budget, every expert receives at least 8 routed slots in every layer, and no expert is zero-support.

MethodRateDS-AvgWikiC4
REAP33%77.089.3713.66
HodgeCover33%77.249.4013.67
REAP66%68.3314.1719.48
HodgeCover66%68.4014.1718.47

HodgeCover's DS-Avg matches or slightly exceeds REAP at both rates. The paired per-example HodgeCover minus REAP DS-Avg difference is +0.15 pp [95% CI: −0.20, +0.50] at 33% and +0.07 pp [−0.39, +0.52] at 66%. This is the calibration regime REAP is designed around, with complete expert coverage. HodgeCover holds its position, so the comparison between the two methods does not hinge on our selected budget.

D. Points in bullet 2. To make the first-round revision easier to navigate, we list the relevant changes and section pointers for the points in bullet 2 below. The component count was corrected to three Hodge components and the triplet-barrier signal, and the underlying mathematics is unchanged. The calibration protocol is addressed in item C. The hybrid comparisons from Table 7 of the submission were placed in Table 1. The mergeability motivation is stated in revised Section 3.1. The revised abstract scopes the claim to expert-selection methods that rank experts by per-expert scores or pairwise merge costs. The broad statement in the first related-work paragraph will receive the same scoping as part of the language pass in item B. The proofs are in App. B, and the full per-task results are in App. F.

We would be glad to walk through any of these in more detail, or to discuss the new comparison at REAP's calibration setting.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of revisions

Thank you to all four reviewers for your constructive feedback and careful consideration of our work. We have revised the manuscript to address concerns regarding experimental comparisons, calibration robustness, the contribution of the Hodge decomposition, theoretical interpretation, and presentation. We summarize the principal revisions and additional results below.

Experimental comparisons (ifEw, s8cy, Hb1V). We expanded Table 1 to include REAP+Wanda and REAM+Wanda as matched hybrid baselines at both compression rates, with full per-task results. At 66% compression, HodgeCover+Wanda achieves the highest downstream average among the compared hybrids on all three models. At 33%, HodgeCover+Wanda leads on OLMoE and stays within 0.2 pp of the matched baselines on the Qwen models, with the lowest C4 perplexity in all six model-rate configurations. We revised Section 5.2 to present these comparisons consistently, including the direction and magnitude of the differences.

Calibration robustness (ifEw, Hb1V). Calibration uses a single 2,048-token C4-train sequence, identical across methods. We extended the calibration-size study to Qwen 3.5-35B at 66% expert-count reduction, evaluating 2,048–32,768 tokens and reporting per-layer expert-support statistics and computational scaling (revised App. D.2). HodgeCover's downstream accuracy remains at or above its 2,048-token result and exceeds REAP at each evaluated calibration size. We also added two calibration seeds for the Qwen 3.5-35B hybrid experiments. HodgeCover+Wanda outperforms both matched hybrids at all three calibration seeds, with per-example paired 95% confidence intervals excluding zero for both comparisons at each seed (revised App. F.3).

Contribution of the Hodge decomposition (rJzW). We added controlled ablations on Qwen 3.5-35B at 66% showing that removing harmonic selection, harmonic redirection, or the triangle term reduces downstream accuracy, with the remaining compression procedure held fixed (revised Section 5.4 and App. F.1). We also examined sensitivity to consistent permutations of expert identities and router entries, finding high agreement between the selected expert sets and similar downstream performance (revised App. B.3).

Theoretical interpretation and barrier metrics (rJzW, Hb1V). We clarified that the harmonic component is the orthogonal residual of the edge-barrier signal relative to the chosen simplicial complex, while its connection to compression loss is exact under the assumptions of the quadratic layerwise model and conditional for the general KL objective (revised Section 3.3). We also clarified the layer-local output-discrepancy proxy used to estimate merge barriers and evaluated its agreement with full-model KL barriers, along with the robustness of expert selection under a data-free weight-space metric (revised App. D.7).

Presentation and computational efficiency (ifEw, s8cy, rJzW). We rewrote the abstract, revised the introduction, and redrew Fig. 1 to clarify the motivation for mergeability-based expert selection and distinguish harmonic structure from joint triplet merge costs. We corrected the terminology concerning Hodge components, simplified terminology elsewhere, added forward references, and reorganized the experimental discussion. We also added planning times to the main text and clarified the computational and throughput protocols, finding comparable inference throughput among the matched hybrids.

All revisions are marked in blue in the revised manuscript. Detailed explanations and supporting results are provided in the individual reviewer responses.

Thank you for the detailed rebuttals. I am optimistic that a careful (human author) effort directed towards auditing experimental design and rewriting the manuscript would result in a publishable paper that many members of the ML research community would be interested in reading. Unfortunately, my fundamental concerns remain, and I strongly believe this paper should not be accepted to ICLR 2027. The changes needed go beyond what a discussion period can verify. The revision already makes substantial alternations to the paper across most sections, and fully addressing the unresolved calibration concern could require regenerating all main results. Though these changes are/would be warranted and helpful, that constitutes a new submission and should be reviewed as one. Out of respect for the authors', reviewers' and AC's time, I will not request further experiments during the review period

  1. Presentation has improved, and the abstract in particular is much more readable, but local changes addressing just the (long but non-exhaustive list of) specific issues pointed out are not enough to make the manuscript readable to a human scientist. For example, part of the reason why I did not read Section 3 as carefully was that presentation made it extremely difficult to parse as a human reader. Related work is still full of non-standard jargon and long, dense sentences. Prose throughout coins terms and uses "the" in front of them as if we should already know what "the" thing is without explanation, as readers who never saw experiments.
  2. Errors that a careful (human author) read would catch were pervasive in the original submission: in the mathematics (the Hodge component count), the experimental design (a single 2,048-token calibration sequence, still a concern and still orders of magnitude smaller than typical without justification even after additional experiments increasing the calibration set), the reporting (Table 1), the problem statement (omission of the underlying motivation of the work), and the related work (first paragraph seemingly mischaracterizes other works with broad statements about "every" related method). I spent much longer than I should have reviewing this paper, and each time I have looked more closely at a component of it, it raised additional concerns and questions. I do not have time to look closely at every component of the paper, but the known issues do not inspire confidence in the soundness of the parts of the paper I was not able to examine as closely.
  3. On the calibration set concern specifically, compared methods (REAP/REAM) are designed to use calibration sets on the order of millions of tokens. Though it is quite possible that Hodge would still outperform those methods with typically sized calibration sets, the authors provide no evidence to suggest it. Without one, it is fundamentally an incomplete and possibly unfair comparison. The authors have mostly addressed my computational tractability concern, especially given that the proposed method is meant to be run offline. But in this case, why not just provide results from a setting with calibration sets that 1. allow for more complete expert coverage and 2. are sized to give REAP/REAM a fairer shot?

Thank you for the detailed follow-up. We respond to it in full under your review thread, so that each reviewer's discussion stays in one place and is easy to follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.