acceptodds
Under review as a conference paper at ICLR 2027

DyClust-MoE: One Grouping State for Expert Sharing, Routing, and Placement

Abstract

Sparse mixture-of-experts (MoE) models increase capacity, but expert representation, routing, and distributed placement are usually optimized separately. We introduce , which uses one layer-local grouping state at all three interfaces. Exact reconstruction refits admit deterministic proposals; a mass-distilled hierarchical router and bounded repair retain exactly experts; and one transaction publishes tensors, routing, and placement. At , uses 84.68% fewer stored inference-representation scalars than controlled same- Switch, reaches 438 k prefill and 171 k full-run training tokens/s on eight A100 GPUs, and yields paired prefill, training, and F+B-wire ratios of , , and . Its paired quality differences are GLUE points and WikiText-103 PPL. A budget-matched Decoupled-3State controller precompiles its independent maps and attains slightly better local objectives, yet shared grouping improves two-/four-node prefill by , training by , and reduces wire to . The method touches 1.23 bases/token and executes about the accelerator FLOPs of nominal-matmul Switch; its speed advantage comes from communication and execution organization. Controlled shifts support faster recovery, and two public MoE transfers satisfy the prespecified quality and systems margins under native routing. Code for experiments is available at https://anonymous.4open.science/r/SUBMIT-0001/README.md.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, structured compression, hierarchical routing, expert placement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hhJ8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DyClust-MoE, a framework that uses a dynamically updated expert grouping state to coordinate parameter sharing, routing, and device placement. Experts within each group share dense base matrices while retaining expert-specific low-rank residuals. Candidate groupings are generated using weight sketches, activation statistics, and co-routing information, followed by exact refitting and several admission checks. The method further includes mass-distilled group routing, a cyclic reserve with bounded assignment repair, topology-aware placement, and a bridge-and-publication mechanism for updating the grouping during training. The authors report substantial reductions in stored representation scalars together with relatively small quality changes compared with controlled Switch baselines. They also report improvements in prefill throughput, full-run training throughput, and communication volume, and evaluate transfers to OLMoE and Qwen1.5-MoE.

Critical strengths

S1. The paper studies an important MoE systems problem: parameter sharing, routing, and physical placement are coupled in practice, but are often optimized separately. Using a common grouping structure provides a plausible way to reduce both representation redundancy and communication, and the reported reductions in accessed bases, owners, and wire traffic are consistent with this motivation.

S2. The evaluation considers several system-level quantities beyond parameter count, including communication traffic, HBM usage, executed FLOPs, serving performance, and full-run training throughput. The frozen-interface experiments and the Decoupled-3State comparison are useful attempts to identify where the measured efficiency gains come from.

Critical weaknesses

W1. The method is substantially more complicated than what is justified by the current evidence. It combines deterministic sketch-based grouping, balanced assignment, repeated FP64 SVD refits, swap search, multiple reconstruction and routing admission gates, shadow router updates, a 64-slice bridge, cyclic reserve assignment, bounded repair, topology-aware placement, and transactional state publication. The paper specifies the corresponding hyperparameters and thresholds, but the rationale for many of their values is unclear. In particular, the reconstruction, cumulative-distortion, output-drift, selector, and bridge thresholds appear to be empirical design choices rather than consequences of a common optimization principle.

The existing ablations do not sufficiently establish that these mechanisms are necessary. The paper studies grouping statistics, hierarchical routing, rank, group size, bridge length, and several other components, but I could not find end-to-end experiments that remove or substantially vary the key admission gates while reporting final quality, acceptance rate, and complete training cost. The theoretical results also provide limited justification here: most establish local feasibility or implementation properties through standard results or properties of the construction, rather than guarantees on grouping quality, optimization stability, or the benefit of the complete dynamic procedure.

W2. The experiments do not cleanly isolate the paper's central claim that representation, routing, and placement should use exactly one grouping state. Decoupled-3State is a useful control, but it also changes how a fixed proposal budget is allocated across representation, routing, and placement updates. Therefore, the comparison changes both state coupling and the optimization schedule. A partially coupled baseline, such as one that shares routing and placement while keeping parameter-sharing groups independent, would provide a more direct test of the claimed advantage of a single partition. A matched comparison with the closest dynamic expert-clustering architecture would also make the incremental contribution easier to assess.

The generalization evidence is also incomplete. The primary MoE experiments start from a dense parent trained for 100B tokens, clone its FFNs into experts with small perturbations, and then continue training for 3B tokens. This initialization is favorable to a shared-base representation because the experts initially have closely related parameters. The public-model transfers partially address this concern, but OLMoE and Qwen are evaluated after an additional 4B-token continuation, retain full expert-logit computation, and are not evaluated with downstream fine-tuning after compression. The paper therefore does not show the immediate compression loss, how much continuation is required for recovery, or whether downstream adaptability and full dynamic-training efficiency are preserved on these pretrained MoE models.

Questions for authors

Q1. What is the empirical or theoretical basis for the specific reconstruction, cumulative-distortion, output-drift, selector-KL, retained-mass, recall, and bridge thresholds? How sensitive are the results to these values?

Q2. Can the authors provide end-to-end ablations that remove the cumulative distortion ledger or the interpolation bridge, while reporting final quality, regrouping acceptance rate, and full-run training cost?

Q3. Can the authors compare against a partially coupled design in which routing and placement share a grouping state but parameter-sharing groups are optimized separately? This would help determine whether one common partition is actually necessary.

Q4. How does DyClust-MoE compare under matched training, parameter, and hardware budgets with the closest dynamic expert-clustering and structured-compression methods?

Q5. How sensitive is the method to expert specialization? Can the authors report inter-expert similarity and reconstruction statistics after continuation training, or evaluate a setting in which the experts were not initialized by cloning the same dense FFN?

Q6. For OLMoE and Qwen, what is the model quality immediately after compression, before the additional 4B-token continuation? A quality-versus-continuation curve would help separate compression error from subsequent recovery.

Q7. Can the authors provide downstream fine-tuning results on at least one compressed public MoE model, together with full-run training throughput and peak regrouping-event memory? These results would clarify whether both adaptation capability and the dynamic training procedure scale beyond the primary small-model experiments.

Q8. The anonymous code repository was inaccessible to me at the time of review. Can the authors verify its availability and provide the configurations and scripts required to reproduce the grouping, routing, repair, placement, bridge, and main system results?

7 comments
AuthorsOpenReview ↗

Response to Reviewer hhJ8 (1)

We thank the reviewer for the thorough feedback regarding mechanism complexity, single-state attribution, and pretrained-model transfer evaluation. Following your suggestions, we added end-to-end gate/bridge ablations with same-state short-horizon counterfactual checks, schedule-matched and partially coupled baselines, budget-matched compression comparisons, and immediate-compression recovery and downstream fine-tuning evaluations on public MoE models.

Our refined core conclusion is strictly scoped as follows: under the evaluated parameter budgets, training horizons, and hardware topologies, coordinating parameter sharing, routing support, and physical placement through a single grouping state yields consistent system-efficiency gains within prespecified quality tolerances (while independent states regularized with explicit cross-objective access penalties are also competitive alternatives, meaning state unification is not the sole viable path); meanwhile, the roles of the cumulative distortion budget and the interpolation bridge are isolated separately through end-to-end ablations and same-state short-horizon checks.

Common Protocol for New Experiments. Unless noted otherwise, new small-model runs use an independent 3-seed paired cohort on two nodes totaling 8×A100-80GB GPUs (default E=128, top-2, 3B tokens per branch, sharing the dense parent, data order, optimizer, and evaluation setup; reported separately from the original 6-seed cohort). In Section 3, the confirmatory comparison between DyClust and the strongest decoupled baseline (Penalized-Independent) was prespecified at 5 paired seeds (its first 3 seeds enter the multi-method descriptive overview, and confirmatory inference was executed after all 5 seeds completed; sample sizes and statistics are reported separately per cohort). Quality entries report between-seed sample SDs, and contrasts report SDs of paired seed differences. Full-run training throughput divides total effective training tokens by total wall-clock time (including initial conversion, inspection of all accepted and rejected candidates, SVD refits, bridges, synchronization, and scheduled in-training validation/checkpointing; GLUE fine-tuning, standalone downstream evaluations, and system replays are accounted separately). Paired throughput intervals are computed on seed-level log-ratios.

1. Empirical-Budget Framing of Thresholds and Global Sensitivity (W1, Q1)

We agree with your critique and frame all admission thresholds explicitly as empirical quality-and-overhead budgets calibrated on disjoint pilot data, rather than constants derived from a unified optimization principle: reconstruction error bounds parameter approximation, output drift checks local function change, selector gates limit routing shift, the cumulative ledger caps repeated per-layer refactorization budgets, and bridge gates bound transient path loss fluctuations. Correspondingly, our theoretical statements are strictly limited to optimal continuous refit updates under a fixed partition (with other variables held fixed), reserve capacity feasibility under stated assumptions, and atomic epoch publication consistency.

To evaluate sensitivity to these values, we uniformly scaled all "allowable error margins" by α∈{0.5,1,2}\alpha \in \{0.5, 1, 2\} (for lower-bound thresholds τ\tau such as retained mass and recall, we scaled the error slack via 1−α(1−τ)1 - \alpha(1-\tau); bridge NLL and PPL thresholds were scaled identically, while hard capacity and consistency invariants remained unchanged):

Admission CriterionTightened (α=0.5\alpha=0.5)Default (α=1\alpha=1)Relaxed (α=2\alpha=2)
Per-event relative reconstruction RMS cap0.0150.0300.060
Per-expert reconstruction tail cap0.0600.1200.240
Per-layer cumulative distortion budget (BℓB_\ell)0.0500.1000.200
Screening relative output drift cap (DoutD_{\text{out}})0.0100.0200.040
Selector KL cap0.0100.0200.040
Retained teacher probability mass floor0.9900.9800.960
Teacher top-kk recall floor0.9950.9900.980

End-to-end sensitivity results are included in the table below: across this 4×4\times span (α∈[0.5,2.0]\alpha \in [0.5, 2.0]), mean quality shifts relative to the default remain within 0.07 GLUE points and 0.08 PPL under the tested setup.

AuthorsOpenReview ↗

Response to Reviewer hhJ8 (2)

2. End-to-End Mechanism Ablations, Short-Horizon Checks, and DyClust-Lite (W1, Q2)

In this 3-seed cohort, Full DyClust achieves 85.42±0.24 GLUE and 24.28±0.11 WikiText PPL versus 85.50±0.28 and 24.12±0.14 for paired Switch (paired difference -0.08±0.04 GLUE, +0.16±0.03 PPL), with 99 regrouping opportunities per seed. We evaluated end-to-end variants removing the interpolation bridge (retaining endpoint checks, optimizer resets, and LR ramp rules), removing the cumulative distortion ledger, removing all quality admission gates and the bridge, as well as a streamlined DyClust-Lite variant (retaining dynamic grouping, refits, per-event/cumulative reconstruction budgets, selector mass distillation, and runtime capacity repair, while removing output-drift prechecks, shadow admission gates, and the interpolation bridge):

End-to-End VariantΔGLUE (vs. Full)ΔPPL (vs. Full)Accepted Events (3 seeds)Mean Accept RateMean Full-Run TimeMax Post-Event ΔNLL
Full DyClust0025 / 28 / 3128.28%292.0 min+0.014
w/o Interpolation Bridge−0.09±0.03+0.11±0.0327 / 30 / 3330.30%291.8 min+0.083
w/o Cumulative Ledger−0.23±0.04+0.26±0.0442 / 46 / 5046.46%309.4 min+0.038
w/o Quality Gates & Bridge−0.51±0.06+0.59±0.0599 / 99 / 99100.00%326.4 min+0.310
DyClust-Lite (Streamlined)−0.04±0.03+0.05±0.0227 / 30 / 3330.30%289.8 min+0.060
Tightened Gates (α=0.5\alpha=0.5)−0.06±0.03+0.07±0.0211 / 14 / 1714.14%282.8 min+0.008
Relaxed Gates (α=2.0\alpha=2.0)−0.07±0.03+0.08±0.0238 / 42 / 4642.42%304.2 min+0.026

(Note: "Max Post-Event ΔNLL" is measured on a fixed held-out validation batch as the maximum NLL over the first 32 updates after publication minus pre-publication NLL, reporting the maximum across the three runs as an operational transition-smoothness diagnostic.)

A mutually exclusive critical-path wall-clock breakdown separates direct mechanism overhead from net runtime shifts induced by altered acceptance rates:

Training Phase BreakdownFull DyClustw/o Bridgew/o Cumulative Ledger
Ordinary training steps251.0 min250.9 min252.0 min
Proposal, pre-screen, and SVD refit32.6 min33.3 min47.6 min
Extra bridge overhead1.1 min0.0 min1.8 min
Other regrouping and publication work2.3 min2.6 min3.0 min
Initialization, in-run validation & saving5.0 min5.0 min5.0 min
Total292.0 min291.8 min309.4 min
  1. Role of the cumulative ledger & same-state short-horizon counterfactual check: Removing the cumulative ledger raises the acceptance rate to 46.46%, adding 17.4 min to full-run time (15.0 min from proposal/SVD refit and 2.4 min from ordinary, bridge, and publication steps) while degrading quality by -0.23 GLUE / +0.26 PPL. Because end-to-end ablations also alter downstream optimization trajectories, we additionally ran a same-state 64-step counterfactual fork check: in each seed, we took the first 4 chronologically encountered candidates that passed all other admission checks but were rejected solely by the cumulative budget (12 pairs total; 3 × 4 × 2 × 64 × 262,144 = 402,653,184 token positions), branching from identical model, optimizer, and data-cursor states into an "accept" vs. "retain incumbent" branch for 64 updates. Accepting these candidates worsened validation NLL in 9/12 events (with 3 slightly improved), yielding per-seed mean ΔNLLs of +0.013, +0.006, +0.008 nats/token (overall mean +0.009 ± 0.0036), confirming that the cumulative budget filters updates that pass single-step checks yet hurt short-horizon progress.
  2. Bridge cost and practical role of DyClust-Lite: The bridge itself accounts for only 0.38% (1.1 min) of full-run time (mostly offset when removed due to slightly higher acceptance, leaving 0.2 min net savings), while reducing transient post-event NLL spikes (+0.014 vs. +0.083). Meanwhile, DyClust-Lite achieves nearly identical final quality to Full (-0.04 GLUE / +0.05 PPL in 289.8 min) without output-drift prechecks, shadow gates, or the bridge, demonstrating that the primary gains remain intact when omitting these auxiliary guardrails.
AuthorsOpenReview ↗

Response to Reviewer hhJ8 (3)

3. Schedule-Matched, Partially Coupled, and 5-Seed Confirmatory Penalized Comparisons (W2, Q3)

To eliminate optimization-schedule confounding in the original cyclic Decoupled-3State, we added three cached controls strictly matched to the 659.022M unique-scalar budget (with physical replicas, optimizer states, and sync traffic accounted separately):

  • Matched-Schedule: all three independent states evaluate updates synchronously at every opportunity, matching total candidate-evaluation and refit work budgets;
  • Partial (Route+Place): routing and placement share one partition while parameter sharing optimizes a separate partition;
  • Penalized-Independent: retains three separate partitions while adding explicit shared-base and owner-access cost penalties to the update objectives. Each of the four controller families selected hyperparameters on disjoint pilot runs under an identical search budget (6 candidates × 2 seeds × 0.25B = 3B per family; 12B pilot tokens total, isolated from test sets).

(1) 3-Seed Multi-Method Descriptive Overview:

State-Coordination Design (E=128, 3B)GLUE ↑PPL ↓Bases/tokenOwners/tokenPrefill / SwitchFull-Run TimeF+B wire / Switch
Original cyclic Decoupled-3State85.3224.361.541.471.10×327.0 min0.91
Matched-Schedule (sync 3-state)85.3424.341.511.441.11×326.6 min0.89
Partial: Shared Route+Place, Separate Share85.4024.301.491.211.17×312.3 min0.75
Penalized-Independent (access-penalized)85.3924.311.281.221.23×301.7 min0.75
DyClust (Single Grouping State)85.4224.281.231.191.26×292.0 min0.73

(Paired Switch full-run time is 356.0 min, giving a DyClust full-run training speedup of 356.0 / 292.0 = 1.219×.)

Matched-Schedule performs similarly to the cyclic baseline, ruling out update-cadence confounding; Partial lowers Owners/token to 1.21 and captures most wire savings (0.75), but leaves Bases/token at 1.49; and Penalized-Independent substantially narrows the execution gap, serving as a strong competitive alternative.

(2) 5-Seed Confirmatory Paired Comparison Against Penalized-Independent:

We evaluated DyClust against Penalized-Independent across the 5 prespecified paired seeds (per-seed paired differences: GLUE −0.02, +0.06, +0.05, +0.03, +0.03; PPL +0.01, −0.06, −0.04, −0.02, −0.04):

DyClust vs. Penalized-Independent (5 Paired Seeds)Paired EstimateDescriptive Two-Sided 95% CIOne-Sided Bonferroni Bound
GLUE difference (prespecified margin >−0.20> -0.20)+0.030 pts[−0.008, +0.068]LCB −0.018
WikiText PPL difference (prespecified margin <+0.25< +0.25)−0.030[−0.063, +0.003]UCB +0.011
Prefill throughput gain (prespecified >0> 0)+2.44%[+1.72%, +3.16%]LCB +1.54%
Full-run training throughput gain (prespecified >0> 0)+3.32%[+2.85%, +3.79%]LCB +2.73%

Under paired tt-inference with Bonferroni correction across the four directional tests, access-penalized separate states are competitive alternatives; under the tested setup, a single grouping state preserves quality within prespecified margins while delivering modest but consistent cross-seed prefill (+2.44%) and full-run training (+3.32%) speedups without extra cross-partition penalty hyperparameters.

Reviewer_3A5UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DyClust-MoE framework that coordinates expert representation, routing, and placement through a shared layer-local grouping state. Experts within each group use shared base matrices and low-rank residuals. A mass-distilled hierarchical router maintains exactly k selected experts per token under capacity constraints. Regrouping candidates undergo reconstruction and output checks. A versioned transaction publishes the updated representation, router, and placement together. The evaluation combines controlled small-model experiments, mechanism-level interventions and distribution shifts. The paper reports up to 84.68% reduction in stored inference-representation scalars, 1.26× prefill throughput, and 1.22× full-run training throughput relative to controlled same-EE Switch with small quality differences.

Critical strengths
  1. The cached Decoupled-3State comparison is informative. The decoupled controller obtains slightly better individual objectives but touches more bases and owners per token. And the metadata-only replay reduces the shared or cached execution ratio when tensors, assignments, and owners are held identical. These results support an explanation for the observed benefit that the grouping-induced execution pattern rather than metadata handling alone.

  2. The paper reports a substantial quality–storage trade-off with systems accounting. The large reduction in stored representation is accompanied by small measured quality changes, full-run training gains that include regrouping overhead. The paper also avoids equating scalar compression with proportional HBM or FLOP savings. This makes the practical contribution more credible than a compression ratio reported in isolation.

Critical weaknesses
  1. Decoupled-3State's poorer combined execution behavior demonstrates a limitation of this particular independently optimized controller, but does not establish that separate states themselves are the limiting factor. The controller could retain separate partitions while incorporating explicit penalties for bases touched, owners touched, or communication cost into its updates. The existing results support the proposed design, but a broader claim that state unification is necessary remains underdetermined.

  2. The OLMoE and Qwen transfers preserve native unnormalized routing weights and evaluate all expert logits. But the primary method uses normalized hierarchical routing and its associated TV decomposition. They limit the extent to which the transfer results validate the full method. The reported outcomes therefore establish a post-continuation trade-off, leaving the quality and adaptation cost immediately after conversion less clear.

Questions for authors

How much of the systems gain remains when the selector/support restriction is disabled while preserving the compact representation and placement procedure? This would clarify the contribution of grouped routing when all native logits are already evaluated.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3A5U (1)

We thank the reviewer for the positive assessment of our systems accounting and experimental controls, as well as the incisive questions on access-penalized separate states and isolating support restriction when all native logits are evaluated. We address these points below through a 5-seed confirmatory comparison, a 2×22 \times 2 orthogonal execution replay, and public-model conversion/recovery curves.

1. Access-Penalized Independent States and 5-Seed Confirmatory Comparison (W1)

We agree with your logical distinction: unpenalized independent controllers do not establish the exclusive necessity of a shared partition. On two nodes (8×A100, E=128, 3B continuation, all matched to 659.022M unique scalars with caching enabled), we implemented a cadence- and work-matched 3-state baseline (Matched-Schedule), a partially coupled baseline (Shared Route+Place, Separate Representation), and an access-penalized independent controller (Penalized-Independent) incorporating explicit shared-base and owner-access cost penalties into its update objectives (hyperparameters tuned per family under identical disjoint pilot budgets).

(1) 3-Seed Multi-Method Descriptive Overview:

State-Coordination Design (E=128, 3B)GLUE ↑PPL ↓Bases/tokenOwners/tokenPrefill / SwitchFull-Run Train / Switch
Matched-Schedule (3 independent states)85.3424.341.511.441.11×1.090×
Shared Route+Place, Separate Representation85.4024.301.491.211.17×1.140×
Access-Penalized Independent (Penalized)85.3924.311.281.221.23×1.180×
Single Grouping State (DyClust)85.4224.281.231.191.26×1.219×

Confirming your intuition, adding explicit base- and owner-access penalties reduces Bases/token and Owners/token to 1.28 / 1.22 and recovers most of the prefill and training speedups (1.23× / 1.180×).

(2) 5-Seed Confirmatory Paired Comparison (DyClust vs. Penalized-Independent):

We evaluated statistical differences across 5 prespecified paired seeds (seed-level paired differences for quality, log-ratios for throughput; per-seed GLUE diffs: −0.02, +0.06, +0.05, +0.03, +0.03; PPL diffs: +0.01, −0.06, −0.04, −0.02, −0.04):

DyClust vs. Penalized-Independent (5 Paired Seeds)Paired EstimateDescriptive Two-Sided 95% CIOne-Sided Bonferroni Bound
GLUE difference (prespecified margin >−0.20> -0.20)+0.030 pts[−0.008, +0.068]LCB −0.018
WikiText PPL difference (prespecified margin <+0.25< +0.25)−0.030[−0.063, +0.003]UCB +0.011
Prefill throughput gain (prespecified >0> 0)+2.44%[+1.72%, +3.16%]LCB +1.54%
Full-run training throughput gain (prespecified >0> 0)+3.32%[+2.85%, +3.79%]LCB +2.73%

Based on Bonferroni-corrected bounds across the four prespecified directional tests, we scope our claim accordingly: properly access-penalized separate states are highly competitive; under our tested budget and implementation, a single grouping state avoids extra cross-partition penalty tuning while maintaining quality within prespecified tolerances and delivering modest, consistent prefill (+2.44%) and full-run training (+3.32%) speedups.

AuthorsOpenReview ↗

Response to Reviewer 3A5U (2)

2. Disabling Support Restriction and 2×22 \times 2 Orthogonal Execution Replay (Question & W2)

Native Routing-Weight Semantics and Scalar Inventories. Both OLMoE-1B-7B-0924 and Qwen1.5-MoE-A2.7B set norm_topk_prob=false officially: a full softmax is computed over all routed expert logits, and selected expert probabilities are applied directly without renormalization (Qwen also has an uncompressed shared branch excluded from top-4 competition and updated identically across both branches). We separate whole-model vs. routed-module (including selector) scalar reductions:

Public ModelRouted Experts / top-kkNative Whole-Model ScalarsCompact Whole-Model ScalarsWhole-Model ReductionRouted-Module Reduction
OLMoE-1B-7B64 / 86.919162B1.584269B77.10%82.81%
Qwen1.5-MoE-A2.7B60 / 4 (+shared branch)14.315784B4.413131B69.17%79.49%

Under this contract, evaluating all expert logits fixes the global softmax denominator, whereas the selector's group support restriction determines which experts are eligible to be selected and executed. For a fixed input and full-logit softmax vector pp, the unnormalized sparse routing weights of accepted sets TT and AA satisfy the exact L1L_1 identity:

∥p⊙1T−p⊙1A∥1=p(T∖A)+p(A∖T).\|p \odot \mathbf{1}_T - p \odot \mathbf{1}_A\|_1 = p(T \setminus A) + p(A \setminus T).

(1) Controlled Replay Disabling Support Restriction (answering your Question): Across all 10 paired 4B checkpoints on two nodes (8×A100), we held Compact weights, groups, learned owner maps, full-logit evaluation, and capacity repair fixed, toggling only group support restriction (Support-on) vs. opening all EE experts to selection (Support-off):

Public Model & Routing Mode (8×A100 Replay)ΔPPL (vs. Paired Native)Bases/tokenOwners/tokenPrefillDecodeF+B wire
OLMoE, Support-on (default)+0.121.351.291.19×1.15×0.76
OLMoE, Support-off (restriction disabled)+0.102.842.161.06×1.05×0.91
Qwen, Support-on (default)+0.141.621.411.17×1.14×0.77
Qwen, Support-off (restriction disabled)+0.122.582.041.05×1.04×0.90

Even when all native logits are computed, disabling support restriction substantially increases execution dispersion (OLMoE Bases/token rises from 1.35 to 2.84 and Owners/token from 1.29 to 2.16), dropping prefill speedups from 1.19× / 1.17× to 1.06× / 1.05× and raising F+B wire from 0.76 / 0.77 to 0.91 / 0.90, while improving mean ΔPPL by 0.02 — demonstrating that support restriction trades a slight 0.02 PPL routing cost for the bulk of the communication and throughput gains.

(2) 2×22 \times 2 Orthogonal Replay (Representation Execution × Support Restriction): To further separate shared-base execution from support restriction and test their interaction, on the same 10 OLMoE Compact checkpoints we crossed "materialized full expert matrices reconstructed from the compact weights" vs. "direct shared-base + low-rank factor execution" (holding inputs, logits, accepted edges, owners, and wire bytes identical within each support mode, verified for numerical output parity; reporting geometric means and log-space 95% CIs across the 10 pairs):

Representation Execution (OLMoE Compact)Group Support RestrictionPrefill / Paired NativeDescriptive 95% CIF+B wire / Paired Native
Materialized expert matricesDisabled (Support-off)1.025×[1.014, 1.036]0.91
Shared bases + low-rank factorsDisabled (Support-off)1.060×[1.049, 1.071]0.91
Materialized expert matricesEnabled (Support-on)1.142×[1.132, 1.152]0.76
Shared bases + low-rank factorsEnabled (Support-on)1.190×[1.178, 1.202]0.76

Paired contrasts highlight both conditional contributions and positive synergy:

  • Under shared-base execution, enabling support restriction increases prefill throughput by +12.26% (95% CI [11.57%, 12.96%]);
  • With support restriction disabled, shared-base execution outperforms materialized matrices by +3.41% ([3.12%, 3.71%]);
  • With support restriction enabled, the gain of shared-base execution over materialized matrices expands to +4.20% ([3.89%, 4.52%]), because concentrating active experts into fewer groups increases on-chip shared-base reuse.
AuthorsOpenReview ↗

Response to Reviewer 3A5U (3)

3. Immediate Conversion Loss and Recovery Horizon on Public Models (W2)

Comparing paired Compact and Native branches at matched continuation token counts (0B is deterministic conversion of the fixed public checkpoint):

Public ModelQuality Contrast (Compact - Native)0B (Immediate)1B tokens4B tokens
OLMoE-1B-7BWikiText ΔPPL ↓+1.84+0.17±0.04+0.12±0.06
OLMoE-1B-7BMMLU (percentage points) ↑−3.44−0.24±0.18−0.18±0.27
Qwen1.5-MoE-A2.7BWikiText ΔPPL ↓+1.46+0.16±0.04+0.14±0.06
Qwen1.5-MoE-A2.7BCMMLU (percentage points) ↑−3.12−0.23±0.19−0.19±0.25

One-shot conversion incurs visible 0B losses of +1.84 / +1.46 PPL (+0.28 mean ΔPPL on OLMoE at 0.5B), whereas by the prespecified 1B checkpoint all four endpoints satisfy our Bonferroni-corrected margins (ΔPPL<0.25\Delta\text{PPL} < 0.25 and task drop <0.60< 0.60 points).

Reviewer_GXY9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes DyClust-MoE, which uses one balanced expert partition per layer for three purposes: a shared group base plus rank-r residual per expert, group-first routing repaired to exact-k, and group placement that coalesces tokens sent to the same owner. The authors conducted extensive experiments  to demonstrate the effectiveness of the method.

Critical strengths
  1. The paper is well-motivated, using a shared grouping state to coordinate expert sharing, routing, and placement. Ablation studies demonstrate that this design facilitates performance gains that exceed what could be achieved by optimizing each local objective in isolation.
  2. The empirical results are promising in efficiency and effectiveness. There is a significant reduction in parameter storage requirements alongside an improvement in end-to-end throughput. I appreciate that the experiments decouple the contributions and overheads associated with storage, FLOPs, and communication, making it easy to discern the sources of the observed gains.
  3. The evaluation is quite comprehensive, covering controlled comparisons, frozen-interface replay, distribution shifts, and transfer to a common model.
Critical weaknesses
  1. I feel quite difficult to read the paper, the presentation has a large room to improve. There are too many dense notations, numerous thresholds, and extensive details in the main text. Furthermore, key definitions and experimental setup details are scattered across the main text and the appendix, forcing the reader to constantly flip back and forth to piece everything together.
  2. Comparing the proposed method with methods like DeepSeek-style constrained routing would better show the contributions of this work. Also, a full-expert baseline would help isolate the specific gains from shared baseline representations.
  3. Experts initialized as dense clones with rank-8 perturbations make the base plus rank-8 representation nearly exact by construction, so the near-parity result may not generalize.
Questions for authors

Please see weakness above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer GXY9 (1)

We thank the reviewer for recognizing the paper's motivation, storage and throughput gains, and decoupled systems evaluation. Alongside reorganizing the main text to present a cleaner hierarchy, placing a self-contained baseline control matrix at the start of the evaluation section, and moving low-level engineering details to the appendix to improve readability (W1), we conducted targeted experiments addressing constrained routing / full-expert baselines (W2) and initialization generalization (W3).

1. Full-Expert Baseline and DeepSeek-Style Device-Constrained Routing (W2)

Using an independent 3-seed paired cohort on two nodes (8×A100-80GB, E=128, top-2, 3B tokens per branch; reported separately from the original 6-seed cohort), we added two controlled families:

  1. Full-Expert + Grouped Route/Place: retains all 4,301.677M uncompressed independent expert scalars while enabling our grouped hierarchical router and topology-aware placement, directly isolating the contribution of the shared-base representation.
  2. Controlled Device-Preferred Routing (inspired by DeepSeek-V2 device-level routing constraints, Dai et al., 2024): because capping top-2 routing at 2 devices imposes no binding constraint, we implemented a non-degenerate single-preferred-device baseline (Device-Preferred) that selects one preferred device by maximum in-device expert affinity and performs top-2 selection within it, while retaining our common capacity factor and zero-drop reserve/repair mechanism (allowing overflow repair across devices when saturated). We evaluated this under both full experts and our compact representation + placement:
E=128 Baseline (3B tokens)Unique Stored ScalarsGLUE ↑WikiText PPL ↓Prefill / SwitchF+B wire / Switch
Switch (Full-Expert + standard flat routing)4,301.677M85.5024.121.00×1.00
Full-Expert + Grouped Route/Place4,301.677M85.4724.151.17×0.74
Full-Expert + Device-Preferred4,301.677M85.2524.381.13×0.69
Shared + Device-Preferred659.022M85.0624.551.19×0.68
DyClust (Shared Base + Grouped Route + Place)659.022M85.4224.281.26×0.73
  • Isolating gains from shared-base representations: Comparing Switch, Full-Expert + Grouped Route/Place, and DyClust shows that grouped routing and placement alone capture the primary wire reduction (0.74) and a 1.17× prefill speedup without weight compression; adding the co-partitioned shared-base + low-rank representation cuts unique stored scalars by 84.68%, lowers inference peak HBM from 8.71 to 7.81 GB/GPU, and adds a further 7.69% prefill speedup (1.26 / 1.17 - 1). In a frozen-interface kernel replay holding inputs, accepted edges, and owners fixed, shared-base execution achieves 1.06±0.01× expert-stage throughput and 0.91±0.02× DRAM read bytes relative to materialized independent expert matrices (with replicas, activations, and executed FLOPs accounted separately).
  • Quality–communication trade-off vs. device-constrained routing: Full-Expert and Shared Device-Preferred touch only 1.06 and 1.05 owners/token on average (with 5%–6% of tokens overflowing via capacity repair, below DyClust's 1.19 owners/token), yielding lower wire volume (0.68–0.69). However, restricting top-2 selection to a single preferred device narrows expert-combination flexibility, degrading quality and throughput under this controlled setup (Shared + Device-Preferred trails DyClust by -0.36 GLUE and +0.27 PPL, with 1.19× vs. 1.26× prefill). DyClust achieves a better balance between communication reduction and routing expressivity.
AuthorsOpenReview ↗

Response to Reviewer GXY9 (2)

2. Three Complementary Tests of Generalization Beyond Rank-8 Clone Perturbations (W3)

To verify whether near-parity relies on dense cloning with tiny 10^-3 rank-8 perturbations, we evaluated three complementary regimes:

  1. Full-Rank High-Noise Upcycling (breaking initial low-rank structure): At E=128, we replaced the 10^-3 rank-8 perturbation with independent full-rank Gaussian perturbations of relative norm 0.05 (giving an initial lossy conversion error of 0.049 relative to the raw uncompressed experts, accounted separately from the 0.03 per-event local threshold used during training), training all three methods for 3B tokens from the same perturbed parent:
Method (E=128, Full-Rank Perturbation 0.05)GLUE ↑WikiText PPL ↓
Switch85.49±0.3024.14±0.13
Static OC-Share84.62±0.3525.06±0.24
DyClust85.39±0.3124.32±0.15

Without tiny low-rank perturbations, DyClust remains within -0.10±0.04 GLUE and +0.18±0.03 PPL of Switch, while outperforming static OC-Share by +0.77 GLUE.

  1. From-Scratch Random Initialization (no dense parent): At E=32, we removed the dense parent entirely and trained directly from random initialization for 4B tokens to test standalone learnability (DyClust and OC-Share start from identical initial compact parameterizations). Switch reaches 79.14±0.42 GLUE / 31.45±0.28 PPL, while DyClust reaches 79.02±0.39 GLUE / 31.65±0.27 PPL, substantially ahead of static OC-Share (78.26±0.44 / 32.64±0.31).

  2. Compression and Recovery on Mature Pretrained Public MoEs: OLMoE-1B-7B is a natively from-scratch pretrained MoE, whereas Qwen1.5-MoE-A2.7B (14.3B total / 2.7B active parameters; top-4 over 60 routed experts plus an uncompressed shared branch) was upcycled from Qwen-1.8B — so only OLMoE serves as strictly dense-origin-free mature evidence. High-ratio one-shot conversion of these specialized models incurs visible 0B (zero-training) ΔPPL losses of +1.84 (OLMoE) and +1.46 (Qwen), which recover to +0.17±0.04 / +0.16±0.04 at 1B and +0.12±0.06 / +0.14±0.06 at 4B paired continuation tokens, transparently characterizing both immediate conversion loss and recovery cost.

AuthorsOpenReview ↗

Kind Request for Reviewer Follow-up

Dear Reviewer GXY9, as the discussion period is ending soon, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify any remaining questions. Thank you!

I appreciate the authors' detailed responses. I am inclined to keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.