- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ProFiMoE is a training-free inference scheme for frozen MoE models. Each routed expert runs either at Full fidelity or as a cheaper Light subnet (a prefix of reordered channels). A promotion score decides which experts run Full under a budget, and the output is rescaled to compensate. On three MoE models it beats prior expert pruning and skipping methods at aggressive ratios, with lower time-to-first-token than the unpruned model.
Critical strengths
- Training-free and easy to apply: the Full expert is unchanged, and an error bound motivates the promotion score. Beyond compression, the same per-token choice between widths could also be used to select dimensions in elastic-width models.
- Broad accuracy evaluation: two model families, many ratios and task types, and gains over the strongest baseline in most settings, largest at aggressive ratios.
Critical weaknesses
- The baseline comparison is not fully controlled. Baseline numbers are taken from the ACE evaluation record while ProFiMoE is newly run, so the two may not share the same setup, and no variance over seeds is reported, which makes the small gains at moderate ratios hard to judge. There is also no comparison with a smaller unpruned model of similar compute.
- The inference settings are narrow. Latency is measured on one model, one GPU and one ratio, the serving framework is not named, and generation settings are not given, e.g. whether thinking mode is used and what sampling strategy is applied. For models that support thinking, it is unclear whether the method still works with thinking on, where long generation makes decode speed matter most. Per-token decode time also gets worse for shorter inputs, which the paper does not discuss; a likely cause is that the extra cost of scoring experts and running separate Full and Light kernels outweighs the FFN savings in these settings.
Questions for authors
- Can you re-run the baselines in your own setup and report variance over several seeds?
- How does the pruned model compare with a smaller unpruned model of similar compute?
- Which framework and generation settings were used? Does the method still help with thinking on, and how much time do expert scoring and the split into Full and Light kernels take?
5 comments
Response to Reviewer pbpG Part II
Q3 (also addressing W2: inference framework, generation configuration, latency breakdown, and thinking mode)
We thank the reviewer for the detailed attention to the inference deployment environment and to long-form generation. We describe the complete configuration, the operator-level timing breakdown, and the behavior under thinking mode as follows.
The paper already clearly reports that the latency experiments use NVIDIA A100, Qwen3-30B-A3B, and 60% target effective FFN pruning. The remaining implementation details, generation parameters, and component timings below are taken from the actual measured environment.
| Item | Actual setting |
|---|---|
| GPU | NVIDIA A100 |
| CUDA / PyTorch | CUDA 12.4 / PyTorch 2.1 |
| Serving framework | vLLM 0.11.0 |
| Full/Light execution | Two independent dense GroupGEMMs; Full and Light share the same inference path |
| Expert dispatch / merge | Expert dispatch and result merging performed on GPU |
| Scoring / sorting / Top-P selection | Scoring, sorting, and cumulative Top-P selection performed on GPU |
| CUDA Graph | Enabled |
| Warm-up and measurement | 10 warm-up runs, then 50 measured runs, reporting the mean |
| Generation setting | Greedy decoding, temperature=0, top-p=1.0, thinking off |
The component timings in the table below have been re-organized according to the updated total TPOT.
| Input length | Batch | BF16 TPOT | ProFiMoE TPOT | Expert scoring | Sorting / Top-P | Full GEMM | Light GEMM | Dispatch / merge | OMR | Other overhead |
|---|---|---|---|---|---|---|---|---|---|---|
| 1024 | 1 | 87.1 ms | 70.5 ms | 1.2 ms | 0.5 ms | 23.0 ms | 27.0 ms | 4.5 ms | 0.3 ms | 14.0 ms |
Thinking mode. We measured thinking off and thinking on under the same checkpoint, the same tasks, and an explicit mode configuration. With thinking on, the seven-task average accuracy of Qwen3-30B-A3B is 83.39 for BF16 and 74.50 for ProFiMoE at 60% pruning.
| Model | Mode | Configuration | Accuracy mean |
|---|---|---|---|
| Qwen3-30B-A3B | Thinking off | BF16 | 80.67 |
| Qwen3-30B-A3B | Thinking off | ProFiMoE, 60% pruning | 66.16 |
| Qwen3-30B-A3B | Thinking on | BF16 | 83.39 |
| Qwen3-30B-A3B | Thinking on | ProFiMoE, 60% pruning | 74.50 |
We wish you continued success in research and all the best in life.
The authors of ProFiMoE
Follow-up on Our Rebuttal Response
Dear Reviewer,
Thank you again for your valuable comments and suggestions.
As the rebuttal period is approaching its end, we would like to kindly follow up on our responses. We sincerely hope that our clarifications have adequately addressed your concerns.
If you have any remaining questions or concerns, we would greatly appreciate it if you could let us know. We would be happy to provide further clarification before the rebuttal period concludes.
Thank you very much for your time and consideration.
Best regards,
The Authors
Response to Reviewer pbpG Part I
We thank the reviewer for the high evaluation of the practicality and the comprehensiveness of the evaluation of this work. The reviewer appreciates the "training-free and easy to apply" property of the method, as well as the broad accuracy evaluation covering multiple model families, multiple tasks, and clear advantages at aggressive pruning ratios. Regarding the reviewer's concerns about baseline reproduction, multi-seed variance, and inference settings, we reply one by one with additional data.
Q1 (also addressing W1: baseline evaluation control and multi-seed variance)
We thank the reviewer for the attention to the controlled nature of the baseline comparison and to run-to-run stability. We describe the reproduction of the experimental environment and the variance reporting as follows.
ProFiMoE strictly reuses ACE's model version, data splits, prompt template, decoding scheme, answer-extraction script, and evaluation pipeline; we also completed an ACE reproduction based on its public code repository. In particular, ACE's GSP and RCR contribution estimators are computed entirely from frozen model parameters, use no calibration samples, and involve no stochastic training procedure. Given a fixed checkpoint, inputs, budget threshold, router configuration, and greedy decoding, ACE's expert ranking and skip decisions are deterministic and do not vary with the random seed. Therefore, ACE itself exhibits no seed-induced ranking variance; the ACE paper adopts a single deterministic evaluation with seed 42 and consequently does not report multi-seed standard deviations. The revision will clearly distinguish the publicly recorded values of the paper from our reproduction verification, and note the data sources.
To avoid confusing the single reported values of the paper with the statistics of our reproduction, the BF16 and ACE rows in the table keep single-run results: BF16 is 80.67%; ACE is 74.30% at the 50% target pruning ratio and 63.35% at 60%. ProFiMoE includes 5 random seeds: at 50% the mean is 77.19% with a sample standard deviation of 0.13 percentage points; at 60% the mean is 66.20% with a sample standard deviation of 0.14 percentage points. The corresponding single-run ProFiMoE values reported in the main text are 77.13% and 66.16%; the 77.19%/66.20% shown in the table are the reproduction means, with standard deviations 0.13/0.14 reported alongside them.
Q2 (also addressing W1: comparison with a smaller unpruned dense model at matched compute)
We thank the reviewer for this highly valuable reference dimension. We add a side-by-side comparison with an unpruned dense model at matched compute.
This experiment answers whether compressing compute while retaining the capacity of a larger MoE model is more valuable than directly adopting a smaller dense model from the same family. We use Qwen3-1.7B dense as the same-family reference and compare it with the unpruned Qwen3-30B-A3B and with 50%-pruned ProFiMoE (about 1.7B actually activated parameters).