- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies the effect of different pruning strategies on accuracy/verbosity of 3 reasoning LLMs. It compares Expert, FFN, and Layer pruning over a very extensive number of samples and various benchmarks like AIME, Minerva, HumanEval+, and LiveCodeBench. The manuscript finds very different responses to the different pruning strategies relative to the unpruned model, and quantifies/analyzes their outcomes extensibly. The paper finds that the coverage of problems the pruned models solve can survive or exceed the original (unpruned model) despite overall falling accuracy and that the different pruning methods yield distinctly different responses in the models.
Critical strengths
- There is an extensive empirical basis behind the work with very high pass@k rates evaluated. At this scale pass@k is a useful lens, and separating success frequency from the reachable set is a real conceptual contribution.
- The study is carefully designed and its results thoroughly discussed going well beyond aggregate curves.
- I think it's an interesting comparison and while I don't know the literature well, I have not yet come across such a comparison that studies the different impact all the pruning methods have. Perhaps there exists previous literature on this that I'm not aware of, so I trust the other reviewers experience on this aspect.
Critical weaknesses
- The key motivation for pruning is to improve the efficiency of serving the LLMs and/or their serving speed. Because of that I think a study comparing different pruning methods should also consider this aspect in the study design (i.e. how much efficiency/speed is traded for the accuracy/verbosity?). All these methods trade a variable number of active parameters in the LLMs and this trade is not discussed or reflected in the paper. This is a major silent omission and the main reason I don't rate this paper at an accept level.
- The study mostly concerns two MoE LLMs which makes it difficult to tell what aspects of the observed phenomena are model specific vs general tendencies unfortunately limiting the usefulness of the study.
Questions for authors
- Did you consider the resource efficiency (speed/active parameters) of LLMs in your study, do these considerations influence the findings/results?
- Can you give total parameters, active parameters per token and approximate FLOPs per token for every checkpoint, and replot Fig. 1 & 3 against those?
- Can you explain figure 3 which has both the token cap and the accuracy on the y axis? I don't understand how to read both axes in the plot.
8 comments
Response to Reviewer bJ9L (2/7)
Q2. Resource accounting and resource comparisons in Figures 1 and 3
A2. Following this suggestion, we have redrawn Figures 1 and 3 against total parameters, active parameters, estimated FLOPs, and measured decoding time. These views make the resource differences between Expert, FFN, and Layer pruning visible alongside their quality results. They cover the original models and 10%, 30%, and 50% pruning on both main backbones; the tables below provide the values used on each resource axis.
Parameter counts
Following Table 1 and Appendix A.2, we count parameters from the actual dimensions of the retained weights and modules. Total parameters include embeddings and the output head; shared weights are counted once. Vision and auxiliary prediction modules are excluded. Active parameters count the weights used for each token, including attention, routing, always-active modules, selected experts, and the output head. The supplementary GLM layer-pruning models at 10% and 30% use the same counting rules.
Estimated computation
We calculate FLOPs from the actual dimensions of all 20 configurations, counting one multiply–accumulate as two operations. For an input of S tokens and an output of 128 tokens, the model processes the prompt once (prefill), then performs 127 further forward passes to generate the remaining tokens (decode). The output head is evaluated at 128 positions. We report input processing, decoding, and their total separately for S = 512 and 4,096.
These estimates use explicit architectural conventions: Qwen’s gated delta network is counted using its equivalent recurrent formulation; GLM’s multi-head latent attention (MLA) is counted in expanded form during input processing and in compressed latent form during decoding. The estimates describe the operations under these conventions. Direct latency measurements additionally capture the effects of GPU kernels and execution overhead.
Measured latency
Each complete model runs on one dedicated H20 GPU, using vLLM 0.25.1 and BF16. Prefix caching is disabled, and configurations within each model family use identical prompts. We test 512- and 4,096-token inputs, exactly 128 output tokens per request, and either one request at a time or eight simultaneous requests. Each condition uses two warmup batches followed by five measured batches; a batch contains one or eight requests, respectively.
Time to first token (TTFT) runs from request submission to the first output token. Time per subsequent output token (TPOT) is the time from the first to the last output token divided by 127. For each metric, we average the request times within each batch, then report the median and interquartile range of the five batch means. The interquartile range spans the 25th to 75th percentiles. Throughput is total output tokens divided by the complete batch’s elapsed time, including input processing. Timing includes engine scheduling and delivery of tokens to the CPU; it excludes model loading, tokenization, warmup, and HTTP transport. Fixing output length makes speed comparable across configurations.
Response to Reviewer bJ9L (3/7)
Parameters, estimated computation, and measured latency
The rows below use 512 input tokens, exactly 128 output tokens, and one request at a time. The FLOP columns show input processing (Prefill), the subsequent 127 generation steps (Decode), and their sum (Request total). Active parameters are counted per token. TPOT is the median time per subsequent output token, measured as described above. B denotes billion parameters; TFLOPs denotes 10¹² floating-point operations per request, a count rather than a rate.
Qwen3.6-35B-A3B
| Configuration | Total (B) | Active (B) | Prefill (TFLOPs) | Decode (TFLOPs) | Request total (TFLOPs) | TPOT (ms/token) |
|---|---|---|---|---|---|---|
| Original | 34.66 | 2.95 | 2.575 | 0.774 | 3.349 | 4.65 |
| Expert, 10% pruning | 31.39 | 2.94 | 2.573 | 0.774 | 3.347 | 4.99 |
| FFN, 10% pruning | 31.63 | 2.84 | 2.466 | 0.747 | 3.214 | 4.53 |
| Layer, 10% pruning | 31.30 | 2.70 | 2.318 | 0.710 | 3.027 | 4.49 |
| Expert, 30% pruning | 24.97 | 2.94 | 2.569 | 0.773 | 3.341 | 5.02 |
| FFN, 30% pruning | 25.06 | 2.61 | 2.231 | 0.689 | 2.920 | 4.99 |
| Layer, 30% pruning | 24.57 | 2.22 | 1.803 | 0.581 | 2.384 | 3.46 |
| Expert, 50% pruning | 18.54 | 2.94 | 2.564 | 0.772 | 3.336 | 4.83 |
| FFN, 50% pruning | 18.49 | 2.38 | 1.995 | 0.630 | 2.626 | 4.58 |
| Layer, 50% pruning | 17.84 | 1.73 | 1.288 | 0.452 | 1.740 | 2.74 |
GLM-4.7-Flash
| Configuration | Total (B) | Active (B) | Prefill (TFLOPs) | Decode (TFLOPs) | Request total (TFLOPs) | TPOT (ms/token) |
|---|---|---|---|---|---|---|
| Original | 29.94 | 3.58 | 3.467 | 1.059 | 4.526 | 5.84 |
| Expert, 10% pruning | 27.34 | 3.58 | 3.467 | 1.059 | 4.526 | 5.80 |
| FFN, 10% pruning | 27.14 | 3.36 | 3.241 | 1.003 | 4.244 | 6.75 |
| Layer, 10% pruning† | 26.77 | 3.23 | 3.100 | 0.955 | 4.055 | 5.55 |
| Expert, 30% pruning | 21.69 | 3.58 | 3.466 | 1.058 | 4.524 | 6.15 |
| FFN, 30% pruning | 21.40 | 2.91 | 2.777 | 0.887 | 3.664 | 5.96 |
| Layer, 30% pruning† | 21.05 | 2.61 | 2.440 | 0.769 | 3.208 | 4.23 |
| Expert, 50% pruning | 16.05 | 3.58 | 3.464 | 1.058 | 4.522 | 6.13 |
| FFN, 50% pruning | 15.80 | 2.46 | 2.324 | 0.775 | 3.099 | 5.72 |
| Layer, 50% pruning | 14.70 | 1.92 | 1.705 | 0.561 | 2.267 | 3.14 |
† Nominal 10% and 30% layer pruning remove 5/47 and 14/47 GLM layers (10.64% and 29.79%). Displayed FLOPs are rounded independently. The full benchmark also includes 4,096-token inputs and eight simultaneous requests.
At 50% pruning, total parameters are similar across structures, while expert pruning keeps active parameters and estimated FLOPs close to the original. Layer pruning roughly halves request FLOPs and lowers TPOT from 4.65 to 2.74 ms on Qwen and 5.84 to 3.14 ms on GLM. FFN pruning reduces estimated computation with a smaller change in measured TPOT.
We report timing variability alongside the medians. For example, GLM at 50% FFN pruning with 512 input tokens and eight simultaneous requests has a median TTFT of 603.64 ms and an interquartile range of 206.40–616.88 ms. This setting shows substantial prefill variability, which the interquartile range makes visible alongside the median.
Response to Reviewer bJ9L (1/7)
We appreciate the reviewer's recognition of the extensive sampling and the value of separating success frequency from the observed solved set. We agree that this distinction should be connected directly to model size, active computation, and the cost of obtaining a correct answer.
Q1. Resource efficiency and its effect on the conclusions
A1. The resource measurements clarify the practical conclusion: expert pruning saves weight memory while best preserving coverage in these comparisons; layer pruning provides the clearest acceleration, with a much larger coverage loss; FFN pruning reduces computation without a proportional gain in measured speed. We compare the original models and 10%, 30%, and 50% pruning on both main backbones, with broadly comparable total-parameter savings at each ratio.
Weight savings and coverage retention
For example, at 50% pruning, the remaining parameters are similar in total but differ in how many are used per token (B denotes billions):
Mean coverage retention is approximately 100% under expert pruning on both models. FFN pruning retains 48.4% on Qwen and 18.8% on GLM; layer pruning retains 9.4% and 5.4%, respectively. For each of four tasks, we divide pruned pass@k by original pass@k, then average the ratios. Here k is the number of attempts per problem: 2,048 for mathematics and 512 for code.
The table below gives absolute latency for the original models and the 50% configurations. Each configuration runs on one H20 GPU in BF16 with vLLM, using 512 input tokens, exactly 128 output tokens, and one request at a time. We report medians of five runs after two warmups. Time to first token (TTFT) measures the wait from request submission to the first output token, including processing the input. Time per output token (TPOT) is the average time to generate each subsequent token: the elapsed time from the first to the last token divided by 127. Output throughput is generated tokens per second, including input-processing time; 1.00× is the original model’s throughput.
Layer pruning reduces both TTFT and TPOT substantially, whereas expert pruning remains close to the originals and FFN pruning yields little throughput improvement in this setting. Across both input lengths (512 and 4,096 tokens), with either one request at a time or eight simultaneous requests, 50% layer pruning reaches 1.70–1.99 times the original throughput. Combined with the coverage results, these measurements favor expert pruning for memory-constrained deployment and identify the stronger speed–coverage tradeoff of layer pruning. They also show why FLOP savings and measured acceleration should be reported separately.
Sampling cost then determines which pruning ratios remain useful. On Qwen AIME, reaching 90% of the original model's coverage at k = 2,048 requires 20 candidates for the original model and 22 after 50% expert pruning, at an estimated generated-token cost of 1.12 times the original. At 80% pruning, the same target requires 1,319 candidates and 59.63 times the tokens. These results support choosing a ratio that meets the memory constraint while retaining affordable sampling and adequate selected-answer accuracy.