- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes DyClust-MoE, a framework that uses a dynamically updated expert grouping state to coordinate parameter sharing, routing, and device placement. Experts within each group share dense base matrices while retaining expert-specific low-rank residuals. Candidate groupings are generated using weight sketches, activation statistics, and co-routing information, followed by exact refitting and several admission checks. The method further includes mass-distilled group routing, a cyclic reserve with bounded assignment repair, topology-aware placement, and a bridge-and-publication mechanism for updating the grouping during training. The authors report substantial reductions in stored representation scalars together with relatively small quality changes compared with controlled Switch baselines. They also report improvements in prefill throughput, full-run training throughput, and communication volume, and evaluate transfers to OLMoE and Qwen1.5-MoE.
Critical strengths
S1. The paper studies an important MoE systems problem: parameter sharing, routing, and physical placement are coupled in practice, but are often optimized separately. Using a common grouping structure provides a plausible way to reduce both representation redundancy and communication, and the reported reductions in accessed bases, owners, and wire traffic are consistent with this motivation.
S2. The evaluation considers several system-level quantities beyond parameter count, including communication traffic, HBM usage, executed FLOPs, serving performance, and full-run training throughput. The frozen-interface experiments and the Decoupled-3State comparison are useful attempts to identify where the measured efficiency gains come from.
Critical weaknesses
W1. The method is substantially more complicated than what is justified by the current evidence. It combines deterministic sketch-based grouping, balanced assignment, repeated FP64 SVD refits, swap search, multiple reconstruction and routing admission gates, shadow router updates, a 64-slice bridge, cyclic reserve assignment, bounded repair, topology-aware placement, and transactional state publication. The paper specifies the corresponding hyperparameters and thresholds, but the rationale for many of their values is unclear. In particular, the reconstruction, cumulative-distortion, output-drift, selector, and bridge thresholds appear to be empirical design choices rather than consequences of a common optimization principle.
The existing ablations do not sufficiently establish that these mechanisms are necessary. The paper studies grouping statistics, hierarchical routing, rank, group size, bridge length, and several other components, but I could not find end-to-end experiments that remove or substantially vary the key admission gates while reporting final quality, acceptance rate, and complete training cost. The theoretical results also provide limited justification here: most establish local feasibility or implementation properties through standard results or properties of the construction, rather than guarantees on grouping quality, optimization stability, or the benefit of the complete dynamic procedure.
W2. The experiments do not cleanly isolate the paper's central claim that representation, routing, and placement should use exactly one grouping state. Decoupled-3State is a useful control, but it also changes how a fixed proposal budget is allocated across representation, routing, and placement updates. Therefore, the comparison changes both state coupling and the optimization schedule. A partially coupled baseline, such as one that shares routing and placement while keeping parameter-sharing groups independent, would provide a more direct test of the claimed advantage of a single partition. A matched comparison with the closest dynamic expert-clustering architecture would also make the incremental contribution easier to assess.
The generalization evidence is also incomplete. The primary MoE experiments start from a dense parent trained for 100B tokens, clone its FFNs into experts with small perturbations, and then continue training for 3B tokens. This initialization is favorable to a shared-base representation because the experts initially have closely related parameters. The public-model transfers partially address this concern, but OLMoE and Qwen are evaluated after an additional 4B-token continuation, retain full expert-logit computation, and are not evaluated with downstream fine-tuning after compression. The paper therefore does not show the immediate compression loss, how much continuation is required for recovery, or whether downstream adaptability and full dynamic-training efficiency are preserved on these pretrained MoE models.
Questions for authors
Q1. What is the empirical or theoretical basis for the specific reconstruction, cumulative-distortion, output-drift, selector-KL, retained-mass, recall, and bridge thresholds? How sensitive are the results to these values?
Q2. Can the authors provide end-to-end ablations that remove the cumulative distortion ledger or the interpolation bridge, while reporting final quality, regrouping acceptance rate, and full-run training cost?
Q3. Can the authors compare against a partially coupled design in which routing and placement share a grouping state but parameter-sharing groups are optimized separately? This would help determine whether one common partition is actually necessary.
Q4. How does DyClust-MoE compare under matched training, parameter, and hardware budgets with the closest dynamic expert-clustering and structured-compression methods?
Q5. How sensitive is the method to expert specialization? Can the authors report inter-expert similarity and reconstruction statistics after continuation training, or evaluate a setting in which the experts were not initialized by cloning the same dense FFN?
Q6. For OLMoE and Qwen, what is the model quality immediately after compression, before the additional 4B-token continuation? A quality-versus-continuation curve would help separate compression error from subsequent recovery.
Q7. Can the authors provide downstream fine-tuning results on at least one compressed public MoE model, together with full-run training throughput and peak regrouping-event memory? These results would clarify whether both adaptation capability and the dynamic training procedure scale beyond the primary small-model experiments.
Q8. The anonymous code repository was inaccessible to me at the time of review. Can the authors verify its availability and provide the configurations and scripts required to reproduce the grouping, routing, repair, placement, bridge, and main system results?
7 comments
Response to Reviewer hhJ8 (2)
2. End-to-End Mechanism Ablations, Short-Horizon Checks, and DyClust-Lite (W1, Q2)
In this 3-seed cohort, Full DyClust achieves 85.42±0.24 GLUE and 24.28±0.11 WikiText PPL versus 85.50±0.28 and 24.12±0.14 for paired Switch (paired difference -0.08±0.04 GLUE, +0.16±0.03 PPL), with 99 regrouping opportunities per seed. We evaluated end-to-end variants removing the interpolation bridge (retaining endpoint checks, optimizer resets, and LR ramp rules), removing the cumulative distortion ledger, removing all quality admission gates and the bridge, as well as a streamlined DyClust-Lite variant (retaining dynamic grouping, refits, per-event/cumulative reconstruction budgets, selector mass distillation, and runtime capacity repair, while removing output-drift prechecks, shadow admission gates, and the interpolation bridge):
| End-to-End Variant | ΔGLUE (vs. Full) | ΔPPL (vs. Full) | Accepted Events (3 seeds) | Mean Accept Rate | Mean Full-Run Time | Max Post-Event ΔNLL |
|---|---|---|---|---|---|---|
Full DyClust | 0 | 0 | 25 / 28 / 31 | 28.28% | 292.0 min | +0.014 |
| w/o Interpolation Bridge | −0.09±0.03 | +0.11±0.03 | 27 / 30 / 33 | 30.30% | 291.8 min | +0.083 |
| w/o Cumulative Ledger | −0.23±0.04 | +0.26±0.04 | 42 / 46 / 50 | 46.46% | 309.4 min | +0.038 |
| w/o Quality Gates & Bridge | −0.51±0.06 | +0.59±0.05 | 99 / 99 / 99 | 100.00% | 326.4 min | +0.310 |
DyClust-Lite (Streamlined) | −0.04±0.03 | +0.05±0.02 | 27 / 30 / 33 | 30.30% | 289.8 min | +0.060 |
| Tightened Gates () | −0.06±0.03 | +0.07±0.02 | 11 / 14 / 17 | 14.14% | 282.8 min | +0.008 |
| Relaxed Gates () | −0.07±0.03 | +0.08±0.02 | 38 / 42 / 46 | 42.42% | 304.2 min | +0.026 |
(Note: "Max Post-Event ΔNLL" is measured on a fixed held-out validation batch as the maximum NLL over the first 32 updates after publication minus pre-publication NLL, reporting the maximum across the three runs as an operational transition-smoothness diagnostic.)
A mutually exclusive critical-path wall-clock breakdown separates direct mechanism overhead from net runtime shifts induced by altered acceptance rates:
| Training Phase Breakdown | Full DyClust | w/o Bridge | w/o Cumulative Ledger |
|---|---|---|---|
| Ordinary training steps | 251.0 min | 250.9 min | 252.0 min |
| Proposal, pre-screen, and SVD refit | 32.6 min | 33.3 min | 47.6 min |
| Extra bridge overhead | 1.1 min | 0.0 min | 1.8 min |
| Other regrouping and publication work | 2.3 min | 2.6 min | 3.0 min |
| Initialization, in-run validation & saving | 5.0 min | 5.0 min | 5.0 min |
| Total | 292.0 min | 291.8 min | 309.4 min |
- Role of the cumulative ledger & same-state short-horizon counterfactual check: Removing the cumulative ledger raises the acceptance rate to
46.46%, adding17.4 minto full-run time (15.0 minfrom proposal/SVD refit and2.4 minfrom ordinary, bridge, and publication steps) while degrading quality by-0.23GLUE /+0.26PPL. Because end-to-end ablations also alter downstream optimization trajectories, we additionally ran a same-state 64-step counterfactual fork check: in each seed, we took the first 4 chronologically encountered candidates that passed all other admission checks but were rejected solely by the cumulative budget (12 pairs total;3 × 4 × 2 × 64 × 262,144 = 402,653,184token positions), branching from identical model, optimizer, and data-cursor states into an "accept" vs. "retain incumbent" branch for 64 updates. Accepting these candidates worsened validation NLL in 9/12 events (with 3 slightly improved), yielding per-seed mean ΔNLLs of+0.013, +0.006, +0.008 nats/token(overall mean+0.009 ± 0.0036), confirming that the cumulative budget filters updates that pass single-step checks yet hurt short-horizon progress. - Bridge cost and practical role of
DyClust-Lite: The bridge itself accounts for only0.38%(1.1 min) of full-run time (mostly offset when removed due to slightly higher acceptance, leaving0.2 minnet savings), while reducing transient post-event NLL spikes (+0.014vs.+0.083). Meanwhile,DyClust-Liteachieves nearly identical final quality to Full (-0.04GLUE /+0.05PPL in289.8 min) without output-drift prechecks, shadow gates, or the bridge, demonstrating that the primary gains remain intact when omitting these auxiliary guardrails.
Response to Reviewer hhJ8 (3)
3. Schedule-Matched, Partially Coupled, and 5-Seed Confirmatory Penalized Comparisons (W2, Q3)
To eliminate optimization-schedule confounding in the original cyclic Decoupled-3State, we added three cached controls strictly matched to the 659.022M unique-scalar budget (with physical replicas, optimizer states, and sync traffic accounted separately):
Matched-Schedule: all three independent states evaluate updates synchronously at every opportunity, matching total candidate-evaluation and refit work budgets;Partial (Route+Place): routing and placement share one partition while parameter sharing optimizes a separate partition;Penalized-Independent: retains three separate partitions while adding explicit shared-base and owner-access cost penalties to the update objectives. Each of the four controller families selected hyperparameters on disjoint pilot runs under an identical search budget (6 candidates × 2 seeds × 0.25B = 3Bper family;12Bpilot tokens total, isolated from test sets).
(1) 3-Seed Multi-Method Descriptive Overview:
State-Coordination Design (E=128, 3B) | GLUE ↑ | PPL ↓ | Bases/token | Owners/token | Prefill / Switch | Full-Run Time | F+B wire / Switch |
|---|---|---|---|---|---|---|---|
Original cyclic Decoupled-3State | 85.32 | 24.36 | 1.54 | 1.47 | 1.10× | 327.0 min | 0.91 |
Matched-Schedule (sync 3-state) | 85.34 | 24.34 | 1.51 | 1.44 | 1.11× | 326.6 min | 0.89 |
Partial: Shared Route+Place, Separate Share | 85.40 | 24.30 | 1.49 | 1.21 | 1.17× | 312.3 min | 0.75 |
Penalized-Independent (access-penalized) | 85.39 | 24.31 | 1.28 | 1.22 | 1.23× | 301.7 min | 0.75 |
DyClust (Single Grouping State) | 85.42 | 24.28 | 1.23 | 1.19 | 1.26× | 292.0 min | 0.73 |
(Paired Switch full-run time is 356.0 min, giving a DyClust full-run training speedup of 356.0 / 292.0 = 1.219×.)
Matched-Schedule performs similarly to the cyclic baseline, ruling out update-cadence confounding; Partial lowers Owners/token to 1.21 and captures most wire savings (0.75), but leaves Bases/token at 1.49; and Penalized-Independent substantially narrows the execution gap, serving as a strong competitive alternative.
(2) 5-Seed Confirmatory Paired Comparison Against Penalized-Independent:
We evaluated DyClust against Penalized-Independent across the 5 prespecified paired seeds (per-seed paired differences: GLUE −0.02, +0.06, +0.05, +0.03, +0.03; PPL +0.01, −0.06, −0.04, −0.02, −0.04):
DyClust vs. Penalized-Independent (5 Paired Seeds) | Paired Estimate | Descriptive Two-Sided 95% CI | One-Sided Bonferroni Bound |
|---|---|---|---|
| GLUE difference (prespecified margin ) | +0.030 pts | [−0.008, +0.068] | LCB −0.018 |
| WikiText PPL difference (prespecified margin ) | −0.030 | [−0.063, +0.003] | UCB +0.011 |
| Prefill throughput gain (prespecified ) | +2.44% | [+1.72%, +3.16%] | LCB +1.54% |
| Full-run training throughput gain (prespecified ) | +3.32% | [+2.85%, +3.79%] | LCB +2.73% |
Under paired -inference with Bonferroni correction across the four directional tests, access-penalized separate states are competitive alternatives; under the tested setup, a single grouping state preserves quality within prespecified margins while delivering modest but consistent cross-seed prefill (+2.44%) and full-run training (+3.32%) speedups without extra cross-partition penalty hyperparameters.
Response to Reviewer hhJ8 (1)
We thank the reviewer for the thorough feedback regarding mechanism complexity, single-state attribution, and pretrained-model transfer evaluation. Following your suggestions, we added end-to-end gate/bridge ablations with same-state short-horizon counterfactual checks, schedule-matched and partially coupled baselines, budget-matched compression comparisons, and immediate-compression recovery and downstream fine-tuning evaluations on public MoE models.
Our refined core conclusion is strictly scoped as follows: under the evaluated parameter budgets, training horizons, and hardware topologies, coordinating parameter sharing, routing support, and physical placement through a single grouping state yields consistent system-efficiency gains within prespecified quality tolerances (while independent states regularized with explicit cross-objective access penalties are also competitive alternatives, meaning state unification is not the sole viable path); meanwhile, the roles of the cumulative distortion budget and the interpolation bridge are isolated separately through end-to-end ablations and same-state short-horizon checks.
Common Protocol for New Experiments. Unless noted otherwise, new small-model runs use an independent 3-seed paired cohort on two nodes totaling 8×A100-80GB GPUs (default
E=128,top-2,3B tokensper branch, sharing the dense parent, data order, optimizer, and evaluation setup; reported separately from the original 6-seed cohort). In Section 3, the confirmatory comparison betweenDyClustand the strongest decoupled baseline (Penalized-Independent) was prespecified at 5 paired seeds (its first 3 seeds enter the multi-method descriptive overview, and confirmatory inference was executed after all 5 seeds completed; sample sizes and statistics are reported separately per cohort). Quality entries report between-seed sample SDs, and contrasts report SDs of paired seed differences. Full-run training throughput divides total effective training tokens by total wall-clock time (including initial conversion, inspection of all accepted and rejected candidates, SVD refits, bridges, synchronization, and scheduled in-training validation/checkpointing; GLUE fine-tuning, standalone downstream evaluations, and system replays are accounted separately). Paired throughput intervals are computed on seed-level log-ratios.1. Empirical-Budget Framing of Thresholds and Global Sensitivity (W1, Q1)
We agree with your critique and frame all admission thresholds explicitly as empirical quality-and-overhead budgets calibrated on disjoint pilot data, rather than constants derived from a unified optimization principle: reconstruction error bounds parameter approximation, output drift checks local function change, selector gates limit routing shift, the cumulative ledger caps repeated per-layer refactorization budgets, and bridge gates bound transient path loss fluctuations. Correspondingly, our theoretical statements are strictly limited to optimal continuous refit updates under a fixed partition (with other variables held fixed), reserve capacity feasibility under stated assumptions, and atomic epoch publication consistency.
To evaluate sensitivity to these values, we uniformly scaled all "allowable error margins" by α∈{0.5,1,2} (for lower-bound thresholds τ such as retained mass and recall, we scaled the error slack via 1−α(1−τ); bridge NLL and PPL thresholds were scaled identically, while hard capacity and consistency invariants remained unchanged):
End-to-end sensitivity results are included in the table below: across this 4× span (α∈[0.5,2.0]), mean quality shifts relative to the default remain within
0.07GLUE points and0.08PPL under the tested setup.