acceptodds
Withdrawn from ICLR 2027

Rethinking LLM Pruning via Test-Time Scaling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model pruning aims to compress large language models while preserving their capabilities, yet its effects on reasoning remain incompletely understood. We revisit expert, FFN and layer pruning through test-time scaling on mathematical reasoning and coding benchmarks. Pass@ curves distinguish success frequency from finite-budget reasoning coverage. Pruned models can preserve or exceed the original model's coverage even when sampling efficiency declines. Expert pruning more consistently preserves accuracy and coverage than FFN or layer pruning in the core matched-rate comparisons. Expert selection further affects both coverage and the sampling and token costs of reaching it, with advantages that vary across backbones. As removal increases, retained coverage can require substantially greater generated-token work, while severe pruning can also degrade generation stability. A mean-accuracy-matched uniform-loss reference does not fully explain the observed coverage losses. Models pruned from larger backbones can also retain broader reasoning coverage than native smaller models, although these comparisons do not match total parameters or training compute. Together, these findings provide an empirical guide to pruning that considers reasoning coverage, sampling efficiency and generation stability jointly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, MoE, Model Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bJ9LOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the effect of different pruning strategies on accuracy/verbosity of 3 reasoning LLMs. It compares Expert, FFN, and Layer pruning over a very extensive number of samples and various benchmarks like AIME, Minerva, HumanEval+, and LiveCodeBench. The manuscript finds very different responses to the different pruning strategies relative to the unpruned model, and quantifies/analyzes their outcomes extensibly. The paper finds that the coverage of problems the pruned models solve can survive or exceed the original (unpruned model) despite overall falling accuracy and that the different pruning methods yield distinctly different responses in the models.

Critical strengths
  • There is an extensive empirical basis behind the work with very high pass@k rates evaluated. At this scale pass@k is a useful lens, and separating success frequency from the reachable set is a real conceptual contribution.
  • The study is carefully designed and its results thoroughly discussed going well beyond aggregate curves.
  • I think it's an interesting comparison and while I don't know the literature well, I have not yet come across such a comparison that studies the different impact all the pruning methods have. Perhaps there exists previous literature on this that I'm not aware of, so I trust the other reviewers experience on this aspect.
Critical weaknesses
  • The key motivation for pruning is to improve the efficiency of serving the LLMs and/or their serving speed. Because of that I think a study comparing different pruning methods should also consider this aspect in the study design (i.e. how much efficiency/speed is traded for the accuracy/verbosity?). All these methods trade a variable number of active parameters in the LLMs and this trade is not discussed or reflected in the paper. This is a major silent omission and the main reason I don't rate this paper at an accept level.
  • The study mostly concerns two MoE LLMs which makes it difficult to tell what aspects of the observed phenomena are model specific vs general tendencies unfortunately limiting the usefulness of the study.
Questions for authors
  • Did you consider the resource efficiency (speed/active parameters) of LLMs in your study, do these considerations influence the findings/results?
  • Can you give total parameters, active parameters per token and approximate FLOPs per token for every checkpoint, and replot Fig. 1 & 3 against those?
  • Can you explain figure 3 which has both the token cap and the accuracy on the y axis? I don't understand how to read both axes in the plot.
8 comments
AuthorsOpenReview ↗

Response to Reviewer bJ9L (1/7)

We appreciate the reviewer's recognition of the extensive sampling and the value of separating success frequency from the observed solved set. We agree that this distinction should be connected directly to model size, active computation, and the cost of obtaining a correct answer.

Q1. Resource efficiency and its effect on the conclusions

A1. The resource measurements clarify the practical conclusion: expert pruning saves weight memory while best preserving coverage in these comparisons; layer pruning provides the clearest acceleration, with a much larger coverage loss; FFN pruning reduces computation without a proportional gain in measured speed. We compare the original models and 10%, 30%, and 50% pruning on both main backbones, with broadly comparable total-parameter savings at each ratio.

Weight savings and coverage retention

For example, at 50% pruning, the remaining parameters are similar in total but differ in how many are used per token (B denotes billions):

  • Qwen3.6-35B-A3B: expert, FFN, and layer pruning leave 18.54B, 18.49B, and 17.84B total parameters, respectively. Active parameters per token are 2.94B, 2.38B, and 1.73B in the same order.
  • GLM-4.7-Flash: the corresponding totals are 16.05B, 15.80B, and 14.70B; active parameters per token are 3.58B, 2.46B, and 1.92B.

Mean coverage retention is approximately 100% under expert pruning on both models. FFN pruning retains 48.4% on Qwen and 18.8% on GLM; layer pruning retains 9.4% and 5.4%, respectively. For each of four tasks, we divide pruned pass@k by original pass@k, then average the ratios. Here k is the number of attempts per problem: 2,048 for mathematics and 512 for code.

The table below gives absolute latency for the original models and the 50% configurations. Each configuration runs on one H20 GPU in BF16 with vLLM, using 512 input tokens, exactly 128 output tokens, and one request at a time. We report medians of five runs after two warmups. Time to first token (TTFT) measures the wait from request submission to the first output token, including processing the input. Time per output token (TPOT) is the average time to generate each subsequent token: the elapsed time from the first to the last token divided by 127. Output throughput is generated tokens per second, including input-processing time; 1.00× is the original model’s throughput.

BackboneConfigurationFirst token: TTFT (ms)Subsequent tokens: TPOT (ms/token)Throughput (× original)
Qwen3.6-35B-A3BOriginal56.094.651.00×
Qwen3.6-35B-A3BExpert, 50% pruning56.064.830.97×
Qwen3.6-35B-A3BFFN, 50% pruning58.774.581.01×
Qwen3.6-35B-A3BLayer, 50% pruning33.112.741.70×
GLM-4.7-FlashOriginal57.535.841.00×
GLM-4.7-FlashExpert, 50% pruning60.126.130.95×
GLM-4.7-FlashFFN, 50% pruning57.725.721.02×
GLM-4.7-FlashLayer, 50% pruning32.673.141.85×

Layer pruning reduces both TTFT and TPOT substantially, whereas expert pruning remains close to the originals and FFN pruning yields little throughput improvement in this setting. Across both input lengths (512 and 4,096 tokens), with either one request at a time or eight simultaneous requests, 50% layer pruning reaches 1.70–1.99 times the original throughput. Combined with the coverage results, these measurements favor expert pruning for memory-constrained deployment and identify the stronger speed–coverage tradeoff of layer pruning. They also show why FLOP savings and measured acceleration should be reported separately.

Sampling cost then determines which pruning ratios remain useful. On Qwen AIME, reaching 90% of the original model's coverage at k = 2,048 requires 20 candidates for the original model and 22 after 50% expert pruning, at an estimated generated-token cost of 1.12 times the original. At 80% pruning, the same target requires 1,319 candidates and 59.63 times the tokens. These results support choosing a ratio that meets the memory constraint while retaining affordable sampling and adequate selected-answer accuracy.

AuthorsOpenReview ↗

Response to Reviewer bJ9L (2/7)

Q2. Resource accounting and resource comparisons in Figures 1 and 3

A2. Following this suggestion, we have redrawn Figures 1 and 3 against total parameters, active parameters, estimated FLOPs, and measured decoding time. These views make the resource differences between Expert, FFN, and Layer pruning visible alongside their quality results. They cover the original models and 10%, 30%, and 50% pruning on both main backbones; the tables below provide the values used on each resource axis.

Parameter counts

Following Table 1 and Appendix A.2, we count parameters from the actual dimensions of the retained weights and modules. Total parameters include embeddings and the output head; shared weights are counted once. Vision and auxiliary prediction modules are excluded. Active parameters count the weights used for each token, including attention, routing, always-active modules, selected experts, and the output head. The supplementary GLM layer-pruning models at 10% and 30% use the same counting rules.

Estimated computation

We calculate FLOPs from the actual dimensions of all 20 configurations, counting one multiply–accumulate as two operations. For an input of S tokens and an output of 128 tokens, the model processes the prompt once (prefill), then performs 127 further forward passes to generate the remaining tokens (decode). The output head is evaluated at 128 positions. We report input processing, decoding, and their total separately for S = 512 and 4,096.

These estimates use explicit architectural conventions: Qwen’s gated delta network is counted using its equivalent recurrent formulation; GLM’s multi-head latent attention (MLA) is counted in expanded form during input processing and in compressed latent form during decoding. The estimates describe the operations under these conventions. Direct latency measurements additionally capture the effects of GPU kernels and execution overhead.

Measured latency

Each complete model runs on one dedicated H20 GPU, using vLLM 0.25.1 and BF16. Prefix caching is disabled, and configurations within each model family use identical prompts. We test 512- and 4,096-token inputs, exactly 128 output tokens per request, and either one request at a time or eight simultaneous requests. Each condition uses two warmup batches followed by five measured batches; a batch contains one or eight requests, respectively.

Time to first token (TTFT) runs from request submission to the first output token. Time per subsequent output token (TPOT) is the time from the first to the last output token divided by 127. For each metric, we average the request times within each batch, then report the median and interquartile range of the five batch means. The interquartile range spans the 25th to 75th percentiles. Throughput is total output tokens divided by the complete batch’s elapsed time, including input processing. Timing includes engine scheduling and delivery of tokens to the CPU; it excludes model loading, tokenization, warmup, and HTTP transport. Fixing output length makes speed comparable across configurations.

AuthorsOpenReview ↗

Response to Reviewer bJ9L (3/7)

Parameters, estimated computation, and measured latency

The rows below use 512 input tokens, exactly 128 output tokens, and one request at a time. The FLOP columns show input processing (Prefill), the subsequent 127 generation steps (Decode), and their sum (Request total). Active parameters are counted per token. TPOT is the median time per subsequent output token, measured as described above. B denotes billion parameters; TFLOPs denotes 10¹² floating-point operations per request, a count rather than a rate.

Qwen3.6-35B-A3B

ConfigurationTotal (B)Active (B)Prefill (TFLOPs)Decode (TFLOPs)Request total (TFLOPs)TPOT (ms/token)
Original34.662.952.5750.7743.3494.65
Expert, 10% pruning31.392.942.5730.7743.3474.99
FFN, 10% pruning31.632.842.4660.7473.2144.53
Layer, 10% pruning31.302.702.3180.7103.0274.49
Expert, 30% pruning24.972.942.5690.7733.3415.02
FFN, 30% pruning25.062.612.2310.6892.9204.99
Layer, 30% pruning24.572.221.8030.5812.3843.46
Expert, 50% pruning18.542.942.5640.7723.3364.83
FFN, 50% pruning18.492.381.9950.6302.6264.58
Layer, 50% pruning17.841.731.2880.4521.7402.74

GLM-4.7-Flash

ConfigurationTotal (B)Active (B)Prefill (TFLOPs)Decode (TFLOPs)Request total (TFLOPs)TPOT (ms/token)
Original29.943.583.4671.0594.5265.84
Expert, 10% pruning27.343.583.4671.0594.5265.80
FFN, 10% pruning27.143.363.2411.0034.2446.75
Layer, 10% pruning†26.773.233.1000.9554.0555.55
Expert, 30% pruning21.693.583.4661.0584.5246.15
FFN, 30% pruning21.402.912.7770.8873.6645.96
Layer, 30% pruning†21.052.612.4400.7693.2084.23
Expert, 50% pruning16.053.583.4641.0584.5226.13
FFN, 50% pruning15.802.462.3240.7753.0995.72
Layer, 50% pruning14.701.921.7050.5612.2673.14

† Nominal 10% and 30% layer pruning remove 5/47 and 14/47 GLM layers (10.64% and 29.79%). Displayed FLOPs are rounded independently. The full benchmark also includes 4,096-token inputs and eight simultaneous requests.

At 50% pruning, total parameters are similar across structures, while expert pruning keeps active parameters and estimated FLOPs close to the original. Layer pruning roughly halves request FLOPs and lowers TPOT from 4.65 to 2.74 ms on Qwen and 5.84 to 3.14 ms on GLM. FFN pruning reduces estimated computation with a smaller change in measured TPOT.

We report timing variability alongside the medians. For example, GLM at 50% FFN pruning with 512 input tokens and eight simultaneous requests has a median TTFT of 603.64 ms and an interquartile range of 206.40–616.88 ms. This setting shows substantial prefill variability, which the interquartile range makes visible alongside the median.

Reviewer_buaqOpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the effects of structured LLM pruning on reasoning capabilities through the lens of test-time scaling. The authors investigate three pruning strategies, namely expert, FFN, and layer pruning, across multiple language model backbones and mathematical reasoning and coding benchmarks. Rather than relying solely on single-sample accuracy, the paper analyzes pass@k curves to distinguish reasoning coverage from sampling efficiency.

The main findings suggest that pruning can substantially reduce pass@1 while preserving or even improving high-budget pass@k, that expert pruning generally preserves reasoning coverage more effectively than FFN or layer pruning under nominally matched removal ratios, and that retained coverage may require significantly more sampling attempts and generated tokens. The authors further investigate the effects of expert selection strategies, generation stability, and the differences between pruned larger models and native smaller models.

Overall, the paper provides an extensive empirical characterization of how structured pruning affects reasoning coverage, sampling efficiency, and generation behavior.

Critical strengths

1. An informative perspective on reasoning capability after pruning.

The paper highlights an important distinction between single-sample accuracy and high-budget reasoning coverage. The observation that heavily pruned models can retain or even exceed their original coverage despite reduced pass@1 is interesting and challenges the conventional interpretation of pruning-induced capability degradation. This perspective provides useful insights into evaluating compressed reasoning models beyond standard accuracy metrics.

2. Extensive empirical evaluation across pruning strategies and model families.

The paper presents a broad experimental investigation covering expert, FFN, and layer pruning across multiple model backbones and mathematical reasoning and coding benchmarks. The additional analyses of expert selection, token efficiency, and generation stability provide a more comprehensive characterization of pruning behavior than evaluations based solely on accuracy or perplexity.

Critical weaknesses

W1. The comparison across pruning strategies lacks resource-matched evaluation.

One of the central conclusions is that expert pruning preserves reasoning capabilities more effectively than FFN or layer pruning. However, the three strategies are compared primarily under the same nominal structural removal ratios, which do not correspond to equivalent reductions in model parameters, active parameters, FLOPs, or inference latency. In particular, removing routed experts from an MoE model can substantially reduce the total parameter count without proportionally reducing the computation per generated token. Consequently, the observed advantage of expert pruning may partly reflect differences in the effective compression budgets rather than an intrinsic advantage of the pruning strategy. Without comparisons under matched resource constraints, this central conclusion remains insufficiently supported.

W2. Limited mechanistic understanding and practical validation of the observed reasoning coverage.

The paper presents several interesting empirical observations, but provides limited explanation of why pruning can preserve or increase high-budget reasoning coverage. The uniform-loss reference shows that pruning-induced changes cannot always be explained by a uniform reduction in success probabilities, but it does not identify the underlying mechanisms or establish whether newly solved problems reflect meaningful changes in reasoning behavior. Furthermore, the reported pass@k represents oracle coverage rather than practically achievable accuracy, since identifying a correct solution among many candidates may require an external verifier. Some heavily pruned models also require substantially more generated tokens to recover their original coverage. Without stronger analysis of reasoning mechanisms or evaluation under realistic inference and verification budgets, the practical implications of these observations remain unclear.

Questions for authors

Q1. Fair comparison across pruning strategies. Could the authors compare expert, FFN, and layer pruning under matched total parameter counts, active parameters, FLOPs, or measured inference latency? Would the conclusion that expert pruning better preserves reasoning coverage still hold under equivalent resource budgets?

Q2. Understanding the coverage improvements. For problems that are solved by a pruned model but not by its original backbone within the evaluated sampling budget, could the authors provide a more detailed analysis of the generated solutions? In particular, is there evidence that these improvements correspond to meaningful changes in reasoning behavior, rather than sampling variability or changes in the distribution of successful outputs?

Q3. Practical utility under constrained inference budgets. How do the comparisons change when performance is measured under a fixed total generation-token budget rather than a fixed number of sampled candidates? Could the authors additionally evaluate whether a practical solution-selection mechanism, such as majority voting or a verifier, can recover the apparent coverage advantages without relying on oracle selection?

Q4. Robustness of the key observations. To what extent are the reported coverage improvements stable across independent sampling seeds and decoding configurations? This is particularly relevant for improvements observed on the relatively small AIME benchmark.

9 comments
AuthorsOpenReview ↗

Response to Reviewer buaq (1/8)

We thank the reviewer for recognizing the distinction between single-sample accuracy and reasoning coverage, and the breadth of our empirical evaluation. The questions help clarify how this distinction relates to compression resources, reasoning capacity, and practical inference. Below, we combine the parameter comparisons and new FLOP and latency results with the output-budget experiments, answer-selection results, and decoding controls.

Q1. Comparing pruning structures under matched resource budgets

A1. We agree that the comparison should identify the resources saved. At 10%, 30%, and 50% pruning, the three structures achieve broadly comparable total-parameter savings, but different reductions in active computation. Expert pruning preserves more coverage at these model sizes; layer pruning delivers greater measured acceleration. We have completed analytical FLOP accounting and inference-latency measurements on both main backbones to quantify this tradeoff alongside the quality results.

Total and active parameters

The original Qwen3.6-35B-A3B has 34.66B total parameters and uses 2.95B per token; GLM-4.7-Flash has 29.94B total and uses 3.58B per token. In each cell below, the first number is the remaining total parameters; the second is the active parameters per token. Both are in billions (B).

BackbonePruning ratioExpert: total / activeFFN: total / activeLayer: total / active
Qwen3.6-35B-A3B10%31.39 / 2.9431.63 / 2.8431.30 / 2.70
Qwen3.6-35B-A3B30%24.97 / 2.9425.06 / 2.6124.57 / 2.22
Qwen3.6-35B-A3B50%18.54 / 2.9418.49 / 2.3817.84 / 1.73
GLM-4.7-Flash10%27.34 / 3.5827.14 / 3.3626.77 / 3.23†
GLM-4.7-Flash30%21.69 / 3.5821.40 / 2.9121.05 / 2.61†
GLM-4.7-Flash50%16.05 / 3.5815.80 / 2.4614.70 / 1.92

Counts follow Table 1 and Appendix A.2. †The supplementary GLM layer configurations remove 5/47 and 14/47 layers, respectively; their parameters are counted from the model tensors using the same convention.

Estimated computation and measured throughput

The following table quantifies the compute and speed differences at all three ratios. Each cell contains two values: estimated FLOPs reduction per request (%) / measured throughput relative to the original (×). For example, 48.06% / 1.70× means 48.06% fewer operations and 1.70 times as many output tokens per second. Throughput below 1.00× means slower generation. All rows use one H20 GPU, BF16, vLLM, 512 input tokens, exactly 128 output tokens, and one request at a time. Throughput is output tokens divided by total elapsed time, including input processing (prefill), and is reported as the median of five runs after two warmups. Estimated FLOPs include both input processing and output generation; one multiply–accumulate counts as two operations.

BackbonePruning ratioExpert: FLOPs reduction / throughputFFN: FLOPs reduction / throughputLayer: FLOPs reduction / throughput
Qwen3.6-35B-A3B10%0.08% / 0.94×4.05% / 1.02×9.61% / 1.04×
Qwen3.6-35B-A3B30%0.24% / 0.93×12.83% / 0.92×28.83% / 1.33×
Qwen3.6-35B-A3B50%0.40% / 0.97×21.61% / 1.01×48.06% / 1.70×
GLM-4.7-Flash10%0.02% / 1.01×6.24% / 0.86×10.40% / 1.06×
GLM-4.7-Flash30%0.05% / 0.95×19.05% / 0.98×29.12% / 1.37×
GLM-4.7-Flash50%0.09% / 0.95×31.53% / 1.02×49.92% / 1.85×
AuthorsOpenReview ↗

Response to Reviewer buaq (2/8)

Coverage at comparable model sizes

Coverage retained at 10%, 30%, and 50% pruning. Each entry is the equal-weight mean of the four task-wise ratios of pruned to original pass@k, expressed as a percentage; 100% means retaining the original coverage. Mathematics uses k = 2,048 and code uses k = 512, with an 8,192-token response limit. “—” means the corresponding quality result is unavailable.

ModelPruning ratioExpert retention (%)FFN retention (%)Layer retention (%)
Qwen3.6-35B-A3B10%100.2499.4195.61
Qwen3.6-35B-A3B30%100.2288.8356.71
Qwen3.6-35B-A3B50%100.5548.379.35
GLM-4.7-Flash10%102.3899.1396.56
GLM-4.7-Flash30%101.3274.00—
GLM-4.7-Flash50%100.4018.785.38

At 50% pruning, expert pruning retains approximately 100% of the original coverage on both backbones, compared with 48.4% and 18.8% for FFN pruning, and 9.4% and 5.4% for layer pruning. Thus, similar total-parameter savings can entail very different coverage losses. Together with the resource measurements, this supports choosing expert pruning when weight memory is the main constraint and coverage retention is a priority.

We will include these coverage-retention, parameter, FLOP, and throughput tables at all three ratios in the revised resource analysis, with the accounting definitions and measurement protocol documented in the appendix.

AuthorsOpenReview ↗

Response to Reviewer buaq (3/8)

Q2. Reasoning capacity and the interpretation of newly solved problems

A2. We address this question by comparing which problems are solved, how often they are solved, and whether the accepted answers have valid derivations. Large-k pass@k probes reasoning capacity—the range of problems a model can solve—following Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? [5]. It separates a lower frequency of correct answers from a smaller observed solved set.

Coverage also sets the ceiling for selecting a correct answer from a given candidate pool: a selector can succeed only if the pool contains one. To examine what newly covered problems mean for reasoning, we identified all cases solved after pruning but not by the original model within the same sampling budget, then checked their saved derivations.

Checking newly covered problems

We compared AIME solved sets at 2,048 candidates per problem, with an 8,192-token limit per response. The comparison covers 56 pruned configurations of the main backbones, including Expert, FFN, Layer, and expert-selection variants, plus the 122B comparison:

BackbonePruned configurations examinedNewly covered problem–configuration pairs
Qwen3.6-35B-A3B260
GLM-4.7-Flash301
Qwen3.5-122B-A10B11

In each of the two newly covered cases, the original model has 0/2,048 correct candidates and the pruned model has 1/2,048. Both final answers match the benchmark labels, but the saved derivations contain explicit errors:

Pruned modelProblem and accepted answerEvidence from the generated derivation
GLM-4.7-Flash; FFN, 10% pruningCell-loop partition; 83Its final aggregation states 1 + 5 + 5 + 5 + 5 + 30 + 20 + 1 + 1 + 30 = 83, although the sum is 103.
Qwen3.5-122B-A10B; Expert, 75% pruningCyclic congruence count; 393It replaces the specified cyclic products with arbitrary triples of nonzero positions. It accepts all 210 arrangements of three zeros, two ones, and two twos, whereas exact enumeration gives 42 valid arrangements. Errors across cases cancel in the final total.

The 122B models each cover 28/30 problems, comprising 27 shared problems, one original-only problem, one pruned-only problem, and one covered by neither. Thus, equal coverage need not mean identical solved sets. The two newly covered answers occur only once each and contain derivation errors, so they establish changes in observed answer coverage without demonstrating improved reasoning on those problems. The solution audit provides the case counts and an executable enumeration of the cyclic problem.

Reviewer_cu8wOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the application of pruning—specifically targeting experts, feed-forward networks, and layers—within the context of test-time scaling for mathematical and programming tasks. The research demonstrates that while pruned models exhibit a decline in single-sample accuracy, they retain a degree of coverage capability under constrained computational budgets.

Critical strengths
  1. The paper is well-written, easy to follow, and features clear illustrations.、

  2. The evaluation effectively distinguishes single-sample accuracy from finite-budget coverage, providing useful insights into how structured pruning affects reasoning performance.

  3. Results on mathematics and coding benchmarks, combined with analyses of sampling costs and generation stability, highlight the value of this study for both dense models and MoE models.

  4. The experiments encompass a diverse range of pruning structures, backbone model architectures, and task types, ensuring that the research findings have broader applicability rather than being confined to a single scenario.

  5. The paper rigorously defines several key concepts, such as "oracle coverage" versus answer selection, "accepted" answers versus valid reasoning processes, and token-based computational cost versus hardware execution efficiency.

Critical weaknesses
  1. The comparison with highly relevant studies needs further clarification. Wang et al. [1] and ShortOPD [2] also examine pruning under test-time scaling and repeated sampling, so the authors should explain which findings overlap with these studies and what additional insights this paper provides.

  2. The experiments include several recent model families, but omit newer architectures such as Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash. Since pruning behavior may depend on the underlying architecture, additional results on these models would help establish generalizability.

  3. The main experiments focus on mathematical reasoning and code generation, leaving generalization to other reasoning tasks unclear. I suggest adding representative experiments on BBEH [3] and MuSR [4] to provide a more comprehensive evaluation.

[1] Wang K, Lyu T, Su G, et al. When fewer layers break more chains: Layer pruning harms test-time scaling in LLMs [C]. COLM 2026.

[2] Zhang Q, Yuan Q, Lin H, et al. ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation [J]. arXiv preprint arXiv:2607.13124, 2026.

[3] Kazemi M, Fatemi B, Bansal H, et al. Big-bench extra hard [C]. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 26473-26501.

[4] Sprague Z, Ye X, Bostrom K, et al. Musr: Testing the limits of chain-of-thought with multistep soft reasoning [C]. International Conference on Learning Representations. 2024, 2024: 14670-14728.

Questions for authors
  1. What are the primary distinctions between this study and previous research on pruning within the context of "test-time scaling"?

  2. When balancing model accuracy against sampling costs, how do the authors suggest selecting a pruning configuration? Are there any practical guidelines or principles to follow?

5 comments
AuthorsOpenReview ↗

Response to Reviewer cu8w (1/4)

We appreciate the reviewer's recognition of the distinction between single-sample accuracy and finite-budget coverage, and of the breadth of the empirical analysis. We agree that the relationship to closely related work and the implications for choosing a pruning configuration deserve a more direct explanation.

Q1. Relationship to previous studies of pruning and test-time scaling

A1. The main distinction is our analysis of test-time scaling in mixture-of-experts (MoE) models, particularly after expert pruning. Many recent leading language models use MoE architectures, making it important to understand whether removing experts preserves reasoning coverage and how much sampling is needed to access it. Expert pruning changes the pool of experts available to the router, introducing a compression choice specific to MoE models alongside reductions in FFN width and model depth. Our study directly examines how these different structural changes affect test-time scaling.

Wang et al. [1] study parallel and sequential test-time scaling under layer pruning in dense models. Our layer-pruning results are consistent with their findings, while our comparison of expert, FFN, and layer pruning on the same MoE backbones shows that the observed loss of coverage depends strongly on the structure removed. Expert pruning can retain high-budget coverage at similar total model sizes where FFN or layer pruning loses substantially more coverage. Expert pruning achieves these storage savings while preserving active computation, which helps interpret its stronger coverage retention. The MoE analysis extends the scope of the layer-pruning findings to a distinct and widely used model architecture and compression strategy.

ShortOPD [2] observes that repeated sampling can recover successful outputs after pruning and uses this observation to motivate on-policy distillation. We acknowledge this overlap. Our study focuses on reasoning coverage before recovery training, especially under expert pruning, and compares original and pruned models at the same number of sampled answers per problem. This differs from the illustrative comparison in ShortOPD's Table 1, which compares increasing sampling budgets for the pruned model against the original model's greedy score. Our comparison tests how much of the original model's coverage remains after pruning when both models receive the same number of attempts, and how the sampling cost of reaching that coverage changes.

We will cite both Wang et al. [1] and ShortOPD [2] and incorporate this comparison into the revised Related Work, so readers can more clearly understand the shared motivation and the differences in research scope and evaluation.

AuthorsOpenReview ↗

Response to Reviewer cu8w (2/4)

Q2. Coverage of model architectures

A2. We agree that evaluating Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash would strengthen the evidence across model architectures. Both are recent releases, and test-time scaling evaluations of their original and pruned variants require substantial computation because each configuration must generate many candidates. We do not currently have sufficient resources to complete these evaluations within the rebuttal period.

Our existing Qwen3.5-122B-A10B results show that the separation between single-sample accuracy and high-budget coverage also occurs at a substantially larger model scale. After 75% expert pruning, AIME pass@1 decreases from 65.05% to 52.98%, while coverage remains at 28 out of 30 problems at k = 2,048. Minerva coverage increases slightly despite lower pass@1, whereas coverage decreases on both code benchmarks. The table below compares all four tasks using the same number of attempts per problem for both models.

Qwen3.5-122B-A10B: original versus 75% expert pruning. Here k is the number of attempts per problem, matched between models in each row. LiveCodeBench uses k = 256, the largest budget available for both models. The Expert columns report 75% pruning; all scores are percentages. Source: Section 3.3 and Appendix B.5.

TaskkOriginal pass@1 (%)Original pass@k (%)Expert pass@1 (%)Expert pass@k (%)
AIME 20262,04865.0593.3352.9893.33
Minerva2,04860.4077.9442.5779.78
HumanEval+51290.2799.3984.1896.34
LiveCodeBench25655.0383.4328.8563.63

Together with Qwen3.6-35B-A3B, GLM-4.7-Flash, and Gemma-4-26B-A4B-it, this brings the MoE evidence to four backbones. FFN and layer-pruning experiments on the dense Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B, and Llama-3.1-8B-Instruct models further examine behavior beyond expert routing. These experiments also identify differences in pruning sensitivity: Gemma loses coverage earlier under expert pruning than Qwen3.6 and GLM. The cross-model analysis therefore shows both the recurring separation between accuracy and coverage and how the usable pruning range varies by architecture.

In the revised manuscript, we will highlight the existing 122B results and cross-architecture comparison in the discussion of model scope, making both the evidence at larger scale and the differences in pruning sensitivity easier to locate.

AuthorsOpenReview ↗

Response to Reviewer cu8w (3/4)

Q3. Generalization beyond mathematics and code

A3. The additional evaluations show that expert pruning retains higher pass@1 than FFN and layer pruning at both 10% and 30% on all four tested tasks and both principal backbones. We compare 14 configurations on MuSR [4], BBEH [3], MMLU-Pro, and GPQA-Diamond, including the two original models. The table combines all 756 MuSR and 4,520 BBEH questions, and 24-question subsets of MMLU-Pro and GPQA-Diamond.

All configurations use the same questions, prompts, and decoding settings, with eight responses per question, temperature 0.7, top-p 0.95, top-k 20, and an 8,192-token output limit. We report pass@1 as the mean correct fraction among the eight responses, with equal weight per question. This estimates single-response accuracy; no candidate selection is applied. BBEH uses its official scorer, while MuSR extracts the answer using fixed rules and counts extraction failures as incorrect. MMLU-Pro and GPQA-Diamond are scored using the final boxed option; the MMLU-Pro subset contains six questions each from biology, chemistry, computer science, and physics. Invalid and truncated responses remain in the denominator for all four tasks.

Additional-task results (pass@1, %). Column headers specify full benchmarks versus exploratory subsets and their question counts.

BackboneConfigurationMuSR (full, 756)BBEH (full, 4,520)MMLU-Pro (subset, 24)GPQA-Diamond (subset, 24)
Qwen3.6-35B-A3BOriginal81.8936.3691.6766.15
Qwen3.6-35B-A3BExpert, 10% pruning82.4736.6591.1564.06
Qwen3.6-35B-A3BExpert, 30% pruning83.8334.1691.1560.94
Qwen3.6-35B-A3BFFN, 10% pruning82.2632.4985.4262.50
Qwen3.6-35B-A3BFFN, 30% pruning80.3419.3675.5252.08
Qwen3.6-35B-A3BLayer, 10% pruning79.7526.6983.3358.33
Qwen3.6-35B-A3BLayer, 30% pruning42.467.4841.1525.52
GLM-4.7-FlashOriginal62.0013.5175.5253.12
GLM-4.7-FlashExpert, 10% pruning63.5913.3275.5245.83
GLM-4.7-FlashExpert, 30% pruning60.2012.2071.3543.75
GLM-4.7-FlashFFN, 10% pruning56.5111.8869.2737.50
GLM-4.7-FlashFFN, 30% pruning43.117.0035.9415.62
GLM-4.7-FlashLayer, 10% pruning45.828.2155.7341.67
GLM-4.7-FlashLayer, 30% pruning7.690.844.6910.42

The additional tasks show that accuracy loss depends strongly on which structure is removed. For example, at 30% pruning on BBEH, Qwen retains 34.16% pass@1 with expert pruning, compared with 36.36% for the original model, 19.36% for FFN pruning, and 7.48% for layer pruning. The corresponding GLM scores are 12.20%, 13.51%, 7.00%, and 0.84%. On this task, removing experts preserves most of the original accuracy, while reducing FFN width or model depth incurs much larger losses.

Generation stability helps interpret the larger losses under layer pruning. Under the shared 8,192-token output cap, GLM with 30% layer pruning truncates 89.88% of MuSR responses, 91.58% of BBEH responses, 77.08% of MMLU-Pro responses, and 78.65% of GPQA-Diamond responses. These low scores accompany a pronounced failure to finish within the available response length, motivating the joint use of accuracy, length, and truncation to select a configuration.

We will add this four-task comparison to the revised experimental section, together with its implications for accuracy and generation stability, and include the evaluation protocol and truncation statistics in the appendix.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to all reviewers (1/3)

We thank all three reviewers for their careful reading and constructive feedback. We appreciate the positive assessments of our distinction between single-sample accuracy and finite-budget reasoning coverage, the breadth of the evaluation across pruning structures and model families, and the extensive sampling experiments. The reviews also recognize the value of the analyses of expert selection, token efficiency, and generation stability.

Across expert, feed-forward network (FFN), and layer pruning, we ask: when accuracy falls, how much reasoning capacity remains accessible through sampling, and what does it cost to obtain successful answers? Comparing these structures reveals how the answer depends on whether we remove experts, reduce FFN width, or remove layers. Here, pass@1 measures single-response accuracy, while pass@k measures the probability of obtaining at least one correct answer in k attempts per problem, averaged across problems. When k equals the full number of sampled responses per problem, pass@k is simply the fraction of problems with at least one correct response in that pool: the observed coverage.

A recurring issue across the reviews is how these findings should inform a resource-constrained choice of pruning structure and ratio. The questions concern both whether the structures are compared under equivalent resource savings and how accuracy should be balanced against sampling cost. We address these related concerns by separating total model parameters, active parameters per token, and the sampling needed to obtain a usable answer. Total parameters determine weight storage; active parameters are those used to process each token.

AuthorsOpenReview ↗

Response to all reviewers (2/3)

Weight memory and active parameters

Parameter comparison at 10%, 30%, and 50% pruning. The table below compares the original models and all three pruning structures at these pruning ratios on the two backbones used in the main comparison. All counts are in billions (B) and follow Appendix A.2. Remaining total and active parameter counts come from Table 1, except for the two marked GLM layer-pruning configurations, which are computed from the models used in our supplementary evaluations under the same definition. Removed totals are computed by subtracting the displayed remaining totals from the corresponding original totals. The ratio refers to experts, FFN width, or layers, rather than the fraction of all model parameters removed.

BackboneConfigurationRemaining total (B)Removed total (B)Active per token (B)
Qwen3.6-35B-A3BOriginal34.660.002.95
Qwen3.6-35B-A3BExpert, 10% pruning31.393.272.94
Qwen3.6-35B-A3BFFN, 10% pruning31.633.032.84
Qwen3.6-35B-A3BLayer, 10% pruning31.303.362.70
Qwen3.6-35B-A3BExpert, 30% pruning24.979.692.94
Qwen3.6-35B-A3BFFN, 30% pruning25.069.602.61
Qwen3.6-35B-A3BLayer, 30% pruning24.5710.092.22
Qwen3.6-35B-A3BExpert, 50% pruning18.5416.122.94
Qwen3.6-35B-A3BFFN, 50% pruning18.4916.172.38
Qwen3.6-35B-A3BLayer, 50% pruning17.8416.821.73
GLM-4.7-FlashOriginal29.940.003.58
GLM-4.7-FlashExpert, 10% pruning27.342.603.58
GLM-4.7-FlashFFN, 10% pruning27.142.803.36
GLM-4.7-FlashLayer, 10% pruning†26.773.173.23
GLM-4.7-FlashExpert, 30% pruning21.698.253.58
GLM-4.7-FlashFFN, 30% pruning21.408.542.91
GLM-4.7-FlashLayer, 30% pruning†21.058.892.61
GLM-4.7-FlashExpert, 50% pruning16.0513.893.58
GLM-4.7-FlashFFN, 50% pruning15.8014.142.46
GLM-4.7-FlashLayer, 50% pruning14.7015.241.92

† GLM layer pruning at nominal 10% and 30% removes 5/47 and 14/47 layers, respectively (10.64% and 29.79%), because layers are indivisible. The 10% configuration has the same parameter counts as the rounded 11% row in Table 1. The 30% configuration is an additional entry from the supplementary evaluations.

At each of the three pruning ratios, expert, FFN, and layer pruning achieve broadly comparable reductions in total parameters. The main difference is in active parameters: expert pruning leaves them nearly unchanged, whereas FFN and layer pruning progressively reduce them as the pruning ratio increases.

This difference helps explain why FFN and layer pruning can degrade performance more severely despite comparable total-parameter savings. At 50% pruning, mean coverage retention is approximately 100% for expert pruning, compared with 48.4% and 18.8% for FFN pruning and 9.4% and 5.4% for layer pruning on Qwen and GLM, respectively. For each of the four tasks, we divide the pruned model’s pass@k by the original model’s pass@k, then average these four ratios. We use k = 2,048 attempts per problem for mathematics and k = 512 for code.

This combination of weight savings and coverage retention makes expert pruning particularly relevant when deployment is constrained by weight memory.

AuthorsOpenReview ↗

Response to all reviewers (3/3)

Computation and measured speed

We have now completed analytical FLOP accounting and inference-latency measurements for 20 configurations: the original models and all three pruning structures at 10%, 30%, and 50% on Qwen3.6-35B-A3B and GLM-4.7-Flash. Each model runs on a single H20 GPU in BF16 with vLLM. We test 512- and 4,096-token inputs, exactly 128 output tokens per request, and two workloads: one request at a time or eight simultaneous requests. FLOPs count floating-point operations and are estimated from the model architecture; latency and output throughput (generated tokens per second) are measured directly.

With 512-token inputs, 128 output tokens, and one request at a time, 50% expert pruning leaves computation and throughput close to the originals. At the same ratio, layer pruning reduces estimated FLOPs per request by 48.06% and 49.92% and increases output throughput to 1.70 and 1.85 times the originals on Qwen and GLM, respectively. FFN pruning reduces estimated FLOPs by 21.61% and 31.53%, but throughput remains at 1.01 and 1.02 times the originals.

These results distinguish the memory savings of expert pruning from the more pronounced acceleration, and greater coverage loss, of layer pruning.

Sampling cost and answer selection

Sampling cost and answer selection complete this resource comparison. For Qwen on AIME, 50% expert pruning reaches the shared target of 90% of the original model's coverage at k = 2,048 with an estimated 1.12 times the original model's generated tokens; at 80% pruning, this rises to 59.63 times.

We additionally compare coverage and majority-vote accuracy under the same cumulative output-token budget per question, replaying recorded generation batches. At 1,048,576 output tokens per AIME question, Qwen with 50% expert pruning achieves 91.78% coverage and 84.85% majority-vote accuracy, compared with 7.97% / 3.74% for FFN pruning and 0.77% / 0.00% for layer pruning. GLM likewise retains substantially higher coverage and voting accuracy under expert pruning than under FFN or layer pruning. These comparisons connect the choice of pruning structure to model size, token cost, and selected-answer accuracy.

The supporting tables appear directly in the responses below. Revised Figure 1 and Figure 3 are also available on anonymous GitHub. Both now include resource-based horizontal axes, and Figure 3 separates length, accuracy/coverage, and truncation into aligned rows. We will incorporate these figures and the completed resource, token-budget, and answer-selection comparisons into the revised manuscript, together with the task-specific guidance they support.

Dear Reviewers,

Thank you for your time and effort in reviewing this paper. The authors have posted responses to your reviews. If you have not yet done so, please read their responses and leave a follow-up comment visible to the authors, indicating which concerns have been addressed and which remain unresolved.

Please also update your review and rating if your assessment has changed. Your feedback will help the authors address any remaining questions during the discussion period.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.