- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces HodgeCover, a learning-free compression method for Sparse Mixture-of-Experts (MoE) layers. It constructs a simplicial 2-complex representing pairwise and triplet KL merge barriers between experts. By applying a discrete Hodge decomposition to this complex, the method isolates the harmonic component to identify irreducible cycles of experts that are pairwise compatible but collectively costly to merge. HodgeCover then uses a greedy submodular objective to select a survivor expert set covering these harmonic-critical edges and triplet-critical triangles. The method is evaluated on OLMoE and Qwen 3.5 architectures, demonstrating strong downstream performance in hybrid settings where expert reduction is combined with unstructured Wanda weight pruning.
Critical strengths
Novel Topological Framing: Applying the Hodge decomposition to isolate the harmonic residual of MoE mergeability is a creative, elegant approach. It effectively captures higher-order structural constraints that purely pairwise scoring methods miss, offering a fresh perspective on expert routing topology.
Strong Hybrid Compression Results: In the hybrid setting (+Wanda), the method achieves excellent downstream performance at aggressive 66% compression rates across different model scales. The preservation of model capabilities on complex reasoning tasks---such as maintaining 85.5% on GSM8K for Qwen 3.5-35B, compared to a collapse to 22.1% for STUN+Wanda---strongly supports the utility of the topological coverage objective in practical hybrid pipelines.
Critical weaknesses
Calibration coverage and robustness at high expert counts: Appendix B.10 reports sporadic zero-frequency experts on Qwen under the 2,048-token C4 calibration sample. The averaging fallback prevents undefined merges, but the paper does not quantify the prevalence of low-support experts or establish whether their estimated mergeability generalizes beyond this sample. Table 4 also shows non-monotone calibration-size sensitivity on OLMoE: DS-Avg changes from 48.57 to 45.45 to 48.90 with 2,048, 4,096, and 8,192 tokens. Corresponding calibration-size and calibration-seed experiments are not reported for the 256-expert Qwen models. This leaves uncertainty about the reliability of the calibration-derived structure at the highest expert cardinality.
Limited incremental gains over strong baselines at larger scale: On the Qwen models, HodgeCover’s pure expert-reduction DS-Avg remains within 0.3 pp of REAP (Table 1). Its 66% hybrid gains over matched REAP+Wanda and REAM+Wanda (for Qwen variants) controls are also modest, at 0.3--0.8 pp (Table 7). These results temper the large gains shown against STUN+Wanda. Because calibration-seed replicates cover only OLMoE, the robustness of these small Qwen advantages remains insufficiently characterized.
Questions for authors
-
How does the Hodge decomposition mathematically or procedurally distinguish between genuine functional mergeability and artificial inactivity artifacts caused by the sparse 2,048-token sample on 256 experts? Could this sampling sparsity explain the non-monotone performance drop observed when scaling calibration tokens to 4k in Table 4, as well as the lack of separation from REAP on the pure expert-reduction axis for the 256-expert Qwen models?
-
Why is the calibration size ablation (Table 4) omitted for the Qwen models? How does the method perform on Qwen 3.5-35B when the calibration set is scaled to 8k or 16k tokens to ensure all experts are adequately routed (please show the relevant stats also)? Given the scaling, how computationally feasible is it to scale the calibration dataset to larger post-training compression sizes (e.g., 32k+ tokens) (which I believe can improve stability and improve expert coverage)?
-
While Appendix D.7 states that the Hodge construction is metric-agnostic and could accept activation-based distributional metrics, have you empirically validated whether the harmonic kernel remains robust under local hidden-state metrics, or a data-free metric (which has fewer knobs to tune)?
5 comments
2. Distinguishing mergeability from sparse-routing artifacts (Q1). The Hodge decomposition itself does not distinguish functional mergeability from calibration artifacts: it decomposes the supplied pairwise-barrier signal (Thm. 1). The distinction instead depends on the barrier construction, saliency, selection procedure, and empirical coverage.
First, the layer-local barrier incorporates the router weights of the experts being merged. For an expert receiving no calibration tokens, , and its pairwise barrier with an active expert reduces to the discrepancy between the parameter-averaged merge and on tokens routed to (revised Sec. 3.1). Thus, the barrier still evaluates the effect of the merge on the active expert's routed tokens.
Second, HodgeCover uses REAP output saliency, normalized per layer (Sec. 4.1). An expert receiving no calibration tokens has zero saliency, reflecting its lack of contribution to the layer output on that sample.
Third, the support analysis measures whether low-support experts disproportionately influence the critical-edge set or survivor selection. As shown above, they are underrepresented in both, and zero-support experts are never retained in the evaluated Qwen configurations.
Regarding the non-monotone OLMoE results in Table 4, DS-Avg changes from 48.57 to 45.45 to 48.90 at 2,048, 4,096, and 8,192 tokens. However, no OLMoE expert is unrouted at 2,048 tokens (App. B.10). The degradation at 4,096 tokens therefore cannot be explained simply by zero-support experts in the smaller sample. It is consistent with sample-to-sample variation in routing statistics and estimated barriers, and the 8,192-token result exceeds the 2,048-token result.
The narrow gap relative to REAP has a separate possible explanation. Sec. 5.3 shows that REAP's survivors retain similar proportions of the gradient, curl, harmonic, and triplet-barrier signals, despite not explicitly optimizing Hodge coverage. This is consistent with the methods' comparable performance in pure expert reduction.
The Qwen calibration study further shows that increasing calibration size neither consistently widens nor eliminates the difference relative to REAP. The evidence supports robustness of the comparison across the evaluated calibration sizes.
3. Performance margins and calibration-seed robustness (W2). We agree that the Qwen differences relative to strong baselines are modest. HodgeCover matches or exceeds REAP across all four Qwen expert-count configurations (+0.0 to +0.3 pp DS-Avg; Table 1). At 66% hybrid compression, HodgeCover+Wanda leads the evaluated hybrids on all three backbones, including the matched REAP+Wanda and REAM+Wanda controls as well as STUN+Wanda [2, 3]. Revised Sec. 5.2 reports these comparisons consistently, including the 33% results.
Paired per-example tests at 66% on Qwen 3.5-35B give +0.16 pp for pure HodgeCover versus REAP (Holm-adjusted ), +0.51 pp for HodgeCover+Wanda versus REAP+Wanda (), and +0.57 pp versus REAM+Wanda (; Table 8, App. F.3). These tests characterize evaluation uncertainty for fixed compressed models.
To assess calibration variability, we added two Qwen 3.5-35B calibration seeds at 66% hybrid compression (revised App. F.3):
| Seed | HC+W | REAP+W | REAM+W |
|---|---|---|---|
| 42 | 74.60 (10.25 / 15.22) | 74.07 (10.57 / 15.78) | 73.75 (10.48 / 15.68) |
| 43 | 75.58 (9.41 / 15.31) | 74.52 (10.14 / 15.76) | 75.13 (9.67 / 15.62) |
| 44 | 73.03 (9.90 / 15.66) | 72.16 (10.41 / 16.30) | 72.58 (10.28 / 16.12) |
| Mean ± SD | 74.40 ± 1.28 | 73.58 ± 1.25 | 73.82 ± 1.28 |
Entries report DS-Avg (WikiText perplexity / C4 perplexity). HodgeCover+Wanda achieves higher DS-Avg and lower perplexity than both matched hybrids at every seed.
Paired per-example DS-Avg differences [95% CI] for HodgeCover+Wanda are:
- Versus REAP+Wanda: +0.51 [+0.14, +0.88], +1.07 [+0.71, +1.43], and +0.88 [+0.51, +1.26].
- Versus REAM+Wanda: +0.57 [+0.20, +0.91], +0.45 [+0.10, +0.79], and +0.46 [+0.09, +0.81].
The differences correspond to seeds 42, 43, and 44, respectively. All six intervals exclude zero.
The original OLMoE calibration-seed experiments likewise show positive margins at each evaluated seed (App. F.3). These results support consistent advantages in the evaluated hybrid settings across calibration seeds and matched evaluation examples.
4. Computational scaling with calibration size (Q2). The barrier sweep has complexity , where is the expert count, the number of evaluated triplets, the calibration token count, and the expert hidden dimension (App. C.5).
For fixed model dimensions and triplet budget, barrier computation scales linearly with calibration size. Increasing the token count from 2,048 to 32,768 therefore multiplies the token-dependent work by 16.
On Qwen 3.5-35B, total planning time increases by 3.0× / 5.8× / 12.3× at 8,192 / 16,384 / 32,768 tokens, respectively, on the same hardware (revised App. D.2). The barrier sweep increases by 3.5× / 6.9× / 15.0×, consistent with linear scaling, while Hodge decomposition and selection remain approximately constant (1.00-1.05×).
Like REAP [1], Wanda [3], and SparseGPT [4], HodgeCover operates in a one-shot, learning-free compression setting, with planning performed offline rather than during inference. The mergeability complex is also cached across compression rates (App. C.6).
These measurements establish computational feasibility through 32,768 calibration tokens on the evaluated 256-expert model. Because cost scales linearly with calibration tokens, larger calibration sets can be processed with the same procedure.
5. Sensitivity to alternative barrier metrics (Q3). We agree that compatibility with alternative metrics does not establish empirical robustness. We therefore added two experiments evaluating the sensitivity of HodgeCover to the barrier metric (revised App. D.7).
Layer-local proxy versus full-model KL. All experiments use a layer-local output-discrepancy proxy motivated by the second-order expansion of the KL objective. The barrier is computed as the mean over calibration tokens of , where is the parameter-averaged merge and are router weights (revised Sec. 3.1, after Eq. 2).
We compared this proxy with the full-model KL barrier on OLMoE at 33% expert reduction (layers 0, 5, 10, 15). We computed full-model KL barriers for all 2,016 expert pairs in each layer, then rebuilt the triangle sets and HodgeCover plans using the same sampling rule, saliency, and selection settings. Rank correlations and most-mergeable-pair overlaps were evaluated on 200 sampled pairs per layer.
| Layer | Spearman correlation | Top-10% mergeable-pair overlap | , proxy / KL | Survivor Jaccard |
|---|---|---|---|---|
| 0 | 0.60 | 0.55 | 0.49 / 0.37 | 0.909 |
| 5 | 0.81 | 0.75 | 0.38 / 0.39 | 1.000 |
| 10 | 0.60 | 0.70 | 0.31 / 0.32 | 1.000 |
| 15 | 0.84 | 0.80 | 0.29 / 0.37 | 0.909 |
The proxy and full-model KL barriers exhibit moderate-to-strong rank correlations (0.60-0.84), with overlap among the lowest-barrier pairs of 0.55-0.80, compared with 0.10 by chance under the reported protocol. Harmonic energy fractions differ more in some layers, but the resulting survivor sets remain similar (Jaccard 0.909-1.000).
These results support the stability of expert selection under the two metrics in the evaluated layers.
Data-free weight-space barriers. We also replaced activation-derived barriers with parameter-space distances:
where concatenates the expert's weight matrices. The triplet barrier sums the distances of the three experts to their parameter centroid. The same triangle-sampling procedure is used. No calibration activations enter the barriers, although saliency remains calibration-derived.
Mean survivor-set Jaccard indices relative to the original activation-based plans are 0.963 / 0.859 for OLMoE at 33% / 66% and 0.992 / 0.949 for Qwen 3.5-35B at the same rates (revised App. D.7).
These results show substantial agreement between survivor sets obtained with activation-based and weight-space barriers. However, changing the metric can also change the sampled triangle set and resulting Hodge decomposition. Selection is therefore not strictly metric-invariant, as reflected particularly by the OLMoE 66% result.
The experiments establish stability of the selected experts under the evaluated alternative metrics.
All revisions are marked in blue in the revised manuscript.
References
[1] Lasby et al., arXiv 2025
[2] Lee et al., ACL 2025
[3] Sun et al., arXiv 2023
[4] Frantar and Alistarh, ICML 2023
Thank you for your thoughtful review and for recognizing the motivation behind the Hodge formulation and its ability to represent higher-order structure. We have extended the calibration and statistical analyses on Qwen 3.5-35B and evaluated HodgeCover's sensitivity to alternative barrier metrics.
1. Calibration size and expert coverage on Qwen (W1, Q2). We agree that a 2,048-token calibration sequence may provide limited coverage of models with 256 experts per layer. Although the original sensitivity study showed stable survivor selection under changes to triangle sampling and other hyperparameters (App. D.2), it did not directly establish robustness to calibration size.
We therefore added a calibration-size study on Qwen 3.5-35B at 66% expert-count reduction, comparing HodgeCover and REAP [1] using 2,048, 8,192, 16,384, and 32,768 C4-train tokens. The sequences are nested prefixes, and both methods use identical calibration data at each size (revised App. D.2).
Support is measured in routed top-k slots per expert on the calibration sequence. Zero-support experts are never selected as survivors at any evaluated size.
HodgeCover's DS-Avg remains at or above its 2,048-token result and exceeds REAP at every calibration size, with differences of +0.29 / +1.56 / +0.34 / +0.52 pp. Paired per-example tests give +1.57 pp [95% CI: +1.12, +2.00] at 8,192 tokens (Holm-adjusted p=0.0002), +0.34 pp [-0.10, +0.78] at 16,384 tokens, and +0.52 pp [+0.08, +0.97] at 32,768 tokens (p=0.008). The estimated differences remain positive, although the 16,384-token confidence interval includes zero.
Low-support experts are also underrepresented among harmonic-critical edges and survivors. At 2,048 tokens, experts with fewer than eight routed slots constitute 20.57% of all experts but only 5.98% of survivors. Their representation decreases further as calibration coverage improves.
These results argue against low-support experts being the primary driver of the Qwen comparison at 2,048 tokens. The positive difference relative to REAP persists as coverage improves.