- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a CH-MoE routing framework to separate hard execution budgets from the gradient-based learning credit. It has two key components:
- CH-Global executes the four highest-ranked experts for each token. The fifth-ranked expert’s routing score participates in a backward-only normalization attachment, but that expert is not executed. The attachment uses an analytic cardinality-event probability to supply an additional training signal.They report CH-Global improves perplexity, downstream accuracy, and load balance compared to that of ReMoE and Default MoE. They then use a four-arm ablation to separate the attachment and the penalty gradient.
- CH-Primary allocates three/four experts across a batch to enforce ~ 90% of the top-4's expert call budget. CH-Primary reports throughput improvements over Top-4 on both a single A100 and a physical two-node setup. On a single A100, it also outperforms Scalar-90, a structurally matched baseline with the same expert-call budget.
Critical strengths
- The paper provides clear execution semantics for experts along with informative components controls. The formulation needs some presentation work, as it requires tracking multiple policies and models across the methods section, but overall it clearly distinguishes executed work from how backward credit is applied. And the paper is very clear how this work fits as a contribution through the combination of established parts, which is still very useful.
- The paper has strong results throughout its well-designed experiments. Controlling the structure of the CH-Primary vs Scalar 90 provides a fair and strong baseline for evaluating the two. The results themselves are good. The physical node measurements are also very convincing for just the CH-Primary.
Critical weaknesses
-
The learning mechanism provides a very small gain over the two other methods and it is quite unclear where the larger quality gains come from. PPL improvements are modest in table 2 over the same router w/o their intervention. Table S5 shows that the gains are similar when both of the gradients are disabled. Section 5.2 makes larger claims therefore than what their results show, specifically around the hard execution budget which Default MoE already has. The claims need to either be weakened or new results must be produced.
-
There are no quality results to accompany the speed results. Without this, one cannot assess the practical importance of CH-Primary. A comparison of the proposed method and Scalar-90/Top-4, measuring both quality and speed under the same conditions, would materially strengthen the paper.
Questions for authors
- What is the leading explanation for the quality improvements? What other differences separate All-Q-Off from Default MoE?
- Is there a quality degradation with the speed improvements?
- How much do the quality results change with changes in batching/batch ordering?
7 comments
Response to Reviewer D9Hf (Part 2/3: Weakness 2 & Question 2 — Joint Quality-Speed Evaluation Under Identical Conditions)
Thank you again, Reviewer D9Hf, for highlighting the need to pair our latency measurements with quality metrics under identical deployment conditions.
Weakness 2 & Question 2: Joint Quality–Speed Evaluation Under Identical Deployment Conditions
Weakness 2 & Question 2 Summary
Speed results lacked paired quality evaluation under identical conditions. Does the speedup come with any quality degradation?
To provide a direct, apples-to-apples answer, we evaluated CH-Primary and Top-4 side-by-side within a unified evaluation suite where all evaluations share the exact same causal attention, expert execution backend, and precision stack:
- Matched Models & Training: Both share the identical 36-layer architecture (
hidden = 3072,intermediate = 12288,32heads, 16 experts at MoE layers{8, 17, 26, 35},n_required = 4) trained in paired runs across 3 seeds (42,123,777) on the same 240M-token corpus (14,649steps,lr = 1.5e-4,weight_decay = 0.01,bf16). - Shared Evaluation Stack: Both arms run at
batch_size = 8with left-padding (1for valid tokens,0for padding) and intra-sequence causal masking, sharing the exact same precision and backend (BF16dense,FP32MoE,TF32off,float32_precision = highest,grouped_bmm, no CUDA Graph), identicalbenchmark_sha256data across all 7 tasks (7/7), and identical continuation log-likelihood scoring (accuracy_normalizedheadline):
Table R1: Unified Deployment Evaluation (36-Layer Paired Models, 3 Seeds, batch_size = 8)
| Category | Metric | Top-4 | CH-Primary | Relative Change |
|---|---|---|---|---|
| Quality | Corpus PPL ↓ | 60.49 | 59.31 | -1.94% (3-seed CIs < 0) |
| 7-Task NLL ↓ | Baseline | Lower on 6/7 tasks | -0.5% to -1.6% | |
| 7-Task Macro Acc (%) ↑ | 36.13% | 36.01% | -0.12 pp | |
| Inference | TTFT (Prefill) ↓ | 356 ms | 224 ms | -37% |
| TBT (Decode) ↓ | 29.4 ms | 27.1 ms | -8% | |
| End-to-End Generation Latency ↓ | 4116 ms | 3697 ms | -10% | |
| Generation Throughput ↑ | 502 tok/s | 555 tok/s | +11% | |
| PPL Scoring Throughput ↑ | 5,743 tok/s | 8,797 tok/s | +53% | |
| Peak Memory ↓ | Baseline | Reduced | -5% | |
| Routing Load | Active Slot Rate | 1.000 | 0.900 | -10% active slots |
| Load CV ↓ | 1.13 | 0.55 | -52% | |
| Effective Experts ↑ | 8.7 / 16 | 13.8 / 16 | +59% | |
| Dead Expert Fraction ↓ | 2–5% | 0% | Eliminated |
Within this unified causal-attention and execution stack (batch_size = 8, grouped_bmm), executing 10% fewer active expert slots (active_slot_rate 0.900 vs. 1.000) delivers -37% TTFT, +11% generation throughput, -52% Load CV, and -1.94% Corpus PPL (60.49 → 59.31, 3-seed CIs < 0), alongside a minor -0.12 pp change in discrete 7-task Macro Accuracy (36.13% → 36.01%).
In addition, under our aligned 4.38B/120M full-sequence protocol comparing Top-4, Scalar-90, and CH-Primary on the identical Triton backend (B=8, L=512), CH-Primary similarly achieves the highest throughput (15,627 ± 256 tok/s vs. 14,051 ± 215 for Scalar-90 and 13,339 ± 269 for Top-4), the lowest Validation PPL (132.54 vs. 134.55 and 137.00), and matched 7-task accuracy (29.28%).
We appreciate this suggestion, as this unified evaluation makes the practical deployment trade-off of CH-Primary clear.
Response to Reviewer D9Hf (Part 3/3: Question 3 — Sensitivity to Batching Method and Batch Order)
Thank you again, Reviewer D9Hf, for raising the important practical question of how batch composition and ordering affect CH-Primary.
Question 3: Sensitivity of Quality Results to Batching Method and Batch Order
Question 3 Summary
How much do the quality results change when the batching method or the order of batches is altered?
Because CH-Primary enforces micro-batch quotas, altering batch composition changes token-to-expert assignments. To quantify this sensitivity, we evaluated CH-Primary across 3 seeds (42, 123, 777) under 8 different batching conditions on our 7-task reasoning suite, holding model weights, prompts, scorers, and backend strictly constant:
Table R2: Impact of Batching Perturbations on 7-Task Macro Accuracy (CH-Primary, 3-Seed Mean)
| Batching Condition | Macro-7 Acc (%) | Change vs. Original B16 |
|---|---|---|
| Original B16 (Baseline) | 29.1183% | — |
| Reverse Batch Execution Order | 29.1183% | 0.0000 pp |
| Reverse Row Order within Batches | 29.2300% | +0.1117 pp |
| Re-batch, Fixed Permutation 101 | 29.1344% | +0.0162 pp |
| Re-batch, Fixed Permutation 202 | 29.1657% | +0.0474 pp |
| Re-batch, Fixed Permutation 303 | 29.1838% | +0.0656 pp |
| Change to B8 | 29.1428% | +0.0245 pp |
| Change to B32 | 29.1262% | +0.0079 pp |
Key Findings:
- Batch Execution Order Invariance: Reversing the execution order of identical batches produces exact determinism (
0.0000 ppchange across all 9 checkpoints when evaluation feedback buffers are frozen). - Bounded Mean Shifts: Across all 8 perturbations (reversing intra-batch rows, reshuffling batches, or changing batch sizes across
8,16, and32), the 3-seed mean Macro-7 accuracy stayed tightly bounded between29.1183%and29.2300%(maximum shift+0.1117 pp). - Bounded Single-Seed Shifts: Individual seed variations exhibited minor bidirectional shifts strictly within -0.1322 pp to +0.1680 pp (all single-seed absolute changes
< 0.17 pp).
These results indicate small changes under the tested conditions rather than universal invariance across arbitrary deployment distributions. We will include Table R2 in the revised Appendix. We sincerely thank you for your thoughtful review, helpful feedback, and constructive assessment of our work.
Response to Reviewer D9Hf (Part 1/3: Weakness 1 & Question 1 — Primary Drivers of Quality Gains, Table S5, & Default MoE Differences)
We sincerely thank Reviewer D9Hf for the positive assessment ("Weak Accept") and for recognizing our execution semantics, the
Scalar-90control, and our physical multi-node measurements. We organize our response into three parts matching your Weaknesses and Questions.Weakness 1 & Question 1: Primary Drivers of Quality Gains, Claim Moderation, & Differences Between
All-Q-OffandDefault MoEWe appreciate your careful reading and fully agree: because
Default MoEalready enforces a hard expert-call budget, "enforcing a hard budget" alone is not a unique source of quality improvement. We will moderate our claims in Section 5.2 accordingly.1. Primary Source of Quality Gains vs. Incremental Role of the Analytic Attachment: We agree that CH-MoE's primary quality advantage over
Default MoEstems from the holistic architectural co-design of our shared routing infrastructure, while the analytic attachment provides a modest, penalty-free additional learning gain on top of that foundation:19701External Comparison, 260M/5B): Under identical frozen scorers (L=512,B=16 for PPL; L=256,B=8 for Macro Acc), moving fromDefault MoE(27.453Test PPL,45.258%Macro Acc) toAll-Q-Off (00)(26.894PPL,45.793%Acc) already captures the majority of the gain (-0.559 PPL, +0.535 pp Acc), outperforming bothDefault MoEandReMoE(27.084PPL,45.320%Acc) without Q enabled. Enabling the analytic attachment (AttachOnly (10)) adds a clean incremental refinement (26.885PPL,45.937%Acc).19701,28463,39173, 260M/5B): Relative to the sharedAll-Q-Offbaseline (26.933PPL,45.257%Acc,0.0352Load CV), enabling the analytic attachment in main-paper Table 2 (Attach Only:26.905PPL,45.317%Acc,0.0326Load CV) and in our controlled surrogate ablation (Q-Only:26.922PPL,45.350%Acc,0.0332Load CV) yields small, consistent gains across all three metrics without auxiliary penalties—whereas replacing it with the tested comparable-strengthSimple Surrogate(sample, averaging the 12 unexecuted experts' routing scores) drops Macro Acc to44.497%(26.931PPL,0.0336Load CV). The two runs were executed on different A100 server node groups and yielded slightly different results (0.017PPL and0.033 ppMacro Accuracy differences); we report both separately for transparency.OFF) degrades validation PPL from132.54 ± 1.68to138.69 ± 1.97and lowers 7-task macro accuracy from29.28%to29.12%.Default MoE, and the analytic attachment as a safe, penalty-free learning refinement.2. Key Architectural Differences Between
All-Q-OffandDefault MoE: As reflected in Table S5 (All-Q-Off (00)vs.Default MoE), although both execute 4 experts per token, they differ fundamentally in three components (not yet isolated one-by-one, and explicitly scoped as a combined architectural difference in revised Section 5.2 and Table S5):All-Q-Offrenormalizes merge weights strictly over actually executed experts, whereasDefault MoEapplies global softmax weights and injects EMA default output estimates of unexecuted experts into the forward pass.All-Q-Offapplies a learned, token-adaptive output gain gt=2σ(wg⊤ht+bg), whichDefault MoElacks.Default MoEuses EMA to estimate unexecuted expert outputs, whereasAll-Q-Offuses dual-timescale (fast/slow) EMA to track realized loads and update proportional selection biases bj, shifting routing selection probabilities rather than perturbing forward features.Thank you again for encouraging a clearer, well-scoped attribution of our architectural gains.