- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines the trade-off between model capacity and active computation in sparse MoE models. The authors compare each MoE configuration with two dense baselines, one matched by total parameter count and the other by active FFN compute. On WikiText-103, the MoE models generally perform worse than parameter-matched dense models in terms of perplexity, but some achieve better results when active FFN compute is matched, at the expense of additional parameters. The paper also investigates how expert count, routing, load balancing, and batch size affect these comparisons, alongside measurements of inference latency and memory usage.
Critical strengths
1.The two-sided evaluation provides a clear account of the trade-off between stored capacity and active computation. The matched comparisons and explicit accounting make the reported results easier to interpret. 2.The experimental controls are useful. The batch-size analysis demonstrates how protocol differences can reverse a comparison, and the latency measurements prevent nominal FLOP savings from being confused with actual runtime improvements.
Critical weaknesses
1.The conceptual novelty is limited. Separating parameter count from active computation is well established in the MoE literature, including Clark et al. (2022) and Artetxe et al. (2022). The proposed framework mainly organizes existing comparison principles. The paper does not sufficiently establish what new scientific understanding follows from this organization.
2.The experimental evidence is too narrow for the broader claims. Controlled quality comparisons cover only WikiText-103 at approximately 96M and 203M parameters, with a fixed training-token budget. Most comparisons rely on one seed, and the replicate study excludes the dense references. Furthermore, active-FFN matching does not establish equal end-to-end training cost. The unusual latency differences across model scales also require clarification.
Questions for authors
I have three questions listed below.
1.Beyond combining two established matching criteria, what conclusions does MoE-TAX enable that cannot be obtained from existing routed-model scaling and compute-efficiency studies?
2.Do the parameter-matched and compute-matched conclusions persist under longer training budgets and additional datasets? Can the authors provide uncertainty estimates for the dense references?
3.What explains the substantial decrease in measured MoE latency from S to M scale under the reported profiling setup? Were precision, kernel implementation, and measurement conditions identical?
3 comments
Reply to the author response
Thank you for the detailed response and for the new controlled latency benchmark. I appreciate the transparent correction. However, the response does not resolve my main concerns, and I keep my rating.
Thank you for the follow-up. We have now completed nine longer-budget runs covering both dense references and three seeds, an official-test evaluation, and a first matched training experiment on C4. These add experimental evidence beyond our previous response.
Q1. What scientific understanding does MOE-TAX add?
Our contribution is a controlled empirical characterization of when the two resource comparisons support different conclusions. The new results identify three concrete boundaries:
- The AF gain changes substantially during training. Within the same six-epoch trajectories, its mean decreases from 1.690% at the halfway checkpoint to 0.645% at completion, while the TP deficit and AF advantage both persist across all three seeds on the training-tail validation split. Thus, the ordering can persist while the size of the benefit contracts.
- The two conclusions have different robustness. On the official test split, MoE remains worse than TP-dense in all three seeds, whereas its smaller AF advantage reverses in one seed. The completed C4 triplet recovers both sides of the trade-off on a separately trained corpus. Reporting these outcomes together distinguishes a consistent TP deficit from an AF benefit that depends on the evaluation setting.
- Matching forward arithmetic does not precisely match training arithmetic. An executed operation audit finds only 0.030% extra full-model forward matrix FLOPs for E4 versus AF4, but 3.376% extra forward-plus-backward matrix FLOPs with our gradient-checkpoint configuration. The trace identifies different recomputation work. These counts include attention and the output head, but exclude non-matrix operations and optimizer work.
Prior work establishes the distinction between stored parameters and active computation. Our empirical contribution quantifies how the quality comparison responds to training progress, evaluation and execution choices, within the tested scope.
Q2. Longer training, additional datasets and dense-reference uncertainty.
The new six-epoch study uses 43,110 updates and 706.3M sampled sequence tokens. TP-dense, E4-MoE and AF4-dense share the training recipe, batch size and minibatch indices within each seed, using an independent data RNG. All nine runs reached the fixed final endpoint.
| Completed experiment | Seeds | TP PPL | E4-MoE PPL | AF4 PPL |
|---|---|---|---|---|
| Six-epoch WikiText, training-tail validation | 3 | 21.892 ± 0.085 | 22.443 ± 0.020 | 22.589 ± 0.030 |
| Same final checkpoints, official WikiText test | 3 | 22.388 ± 0.047 | 22.900 ± 0.083 | 22.944 ± 0.096 |
| New C4 training, official-validation prefix | 1 | 52.437 | 54.578 | 55.735 |
Uncertainty is sample SD across seeds 0/1/2, not a significance claim. At the six-epoch validation endpoint, the paired TP penalty is 2.519% and AF gain is 0.645%. On the official test, these become 2.287% and 0.190%; individual AF gains are +0.444%, +0.416% and -0.290%. We report the reversal explicitly. Test evaluation uses fixed final checkpoints, training-compatible tokenization and no test-based checkpoint selection. These are GPT-2 subword block perplexities.
The C4 experiment trains all three models from scratch for 353.2M sampled sequence tokens, using a fixed 117.7M-token training prefix and a separate 204,800-token official-validation prefix. Its MoE has 4.08% higher PPL than TP-dense and 2.08% lower PPL than AF-dense. This extends the observed ordering beyond WikiText in one completed seed; the remaining C4 repeats are running and are not included in this claim.
We also completed historical-style three-epoch repeats including the dense references: TP/MoE/AF PPLs are 24.297 ± 0.025 / 25.224 ± 0.038 / 25.813 ± 0.035, with a paired AF improvement of 0.588 ± 0.022 PPL. This row combines archived and new runs under the original global-RNG convention and best-recorded endpoint; its variability can include execution differences. We keep it separate from the new fixed-final study. The six-epoch halfway checkpoint is not an independent three-epoch experiment.
We will clarify that historical validation held out the last 400 cached training-stream blocks, sampling used replacement, and AdamW beta2 was 0.999. Our conclusions concern finite-budget training.
Q3. The S-to-M latency anomaly.
The controlled rerun does not reproduce the anomaly: E4 latency increases from 95.26 to 176.47 ms and E8 from 101.62 to 180.57 ms. All configurations use one idle RTX 5090, FP32 with TF32 disabled, the same expert-loop implementation, batch 32 and length 512. We will replace the affected S/M values. The old separate-session profiles lack sufficient metadata to establish the cause. The new MoE/AF latency ratios remain 1.15-1.39, so active-FFN matching still does not imply equal runtime.
We would appreciate your reassessment in light of these new completed experiments.
Response to Reviewer QXv9
Thank you for recognizing the two-sided accounting and controls. Your questions prompted further analysis of the training records and a new controlled latency benchmark.
Q1. What scientific understanding does the framework add?
Separating parameters from computation is established in prior work. We empirically characterize how quality, resource premiums and protocol choices interact for the same sparse model. Matching equations alone do not determine these observations:
These findings complement routed-model scaling studies. We will distinguish the reusable evaluation procedure from empirical magnitudes that depend on dataset, budget and implementation.
Q2. Longer budgets, additional datasets and uncertainty.
We additionally examined all 44 common logged checkpoints of the S-scale, batch-32, seed-0 runs. Both top-2 MoEs have higher PPL than TP dense throughout; from step 1000 through 21555, both outperform their respective AF references at all 43 common checkpoints. The two-sided ordering therefore persists across the recorded trajectory after its emergence, rather than appearing only at the selected best checkpoint. The batch-16 scale check also retains the four-expert TP gap at 96M and 203M: 4.9% and 5.1%, respectively.
Completed paired longer-budget or additional-dataset experiments are not available. The six-epoch dense result (23.27 to 21.10 PPL) demonstrates remaining optimization headroom, but cannot determine the extended-budget ranking without corresponding MoE and AF runs. We will explicitly scope the numerical quality conclusions to WikiText-103 and the stated training budget.
Table 7 repeats five MoE configurations with seeds 0 and 1; their ordering is unchanged, with a maximum reported absolute PPL change of 0.23. Dense-reference repeats are unavailable, so we cannot provide their training-seed uncertainty estimates. We will clarify that neither these MoE-only repeats nor checkpoints along one trajectory establish statistical significance of the smaller MoE-dense gaps.
Q3. Why did latency decrease sharply from S to M? Were conditions identical?
We addressed this concern with a new controlled S/M benchmark. The archived profiles came from separate runs and lack sufficient device/backend metadata to establish the cause of the discrepancy. The source-code audit found the same explicit expert loop and no explicit mixed-precision switch; it cannot verify identical historical runtime conditions.
The new measurements use one idle RTX 5090, FP32 with TF32 disabled, PyTorch 2.13.0/CUDA 13.0, one CPU thread, batch 32 and sequence length 512. We keep the same model implementation and use fixed synthetic token IDs with newly initialized models, as an architecture-level forward benchmark. Each model receives 3 warmups followed by two blocks of 10 forwards, with CUDA synchronization around each timed block.
Values are mean milliseconds per forward over both timed blocks. The large absolute S-to-M latency decrease is not reproduced. We will replace the affected S/M measurements and avoid treating the archived cross-run difference as an established scale effect; the unremeasured L profile will be identified separately.
The controlled results retain the relevant systems finding: MoE/AF latency ratios range from 1.15 to 1.39 across all four pairs, despite matched active FFN arithmetic. Thus, active-FFN matching does not establish equal runtime. This correction concerns the systems profiles; the separately trained TP/AF quality comparisons are unchanged.