acceptodds
Under review as a conference paper at ICLR 2027

WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

Abstract

Pruning is a promising approach for improving the efficiency of large language models (LLMs). Static structured pruning is hardware-friendly, but its input-agnostic computation allocation often degrades model quality at high sparsity. Dynamic pruning adapts computation to individual tokens, but coarse-grained layer-level routing limits allocation flexibility, and finer-grained decisions introduce irregular execution that can offset computational savings. To address these challenges, we present WIDE, the first end-to-end differentiable framework that combines token-wise structured dynamic width pruning with GPU kernel co-design for both prefill and decoding. WIDE uses differentiable routers to select GQA-aligned query-head groups and GPU-tile-aligned FFN-channel groups for each token. To execute these decisions efficiently, it reorders routing masks to cluster active tokens within each group, enabling hardware-agnostic block-level skipping and architecture-aware skipping of inactive loads and tensor-core operations. Experiments on the Llama3 and Qwen3 families demonstrate improved quality retention over representative static and dynamic pruning methods. At 50% target routing sparsity, WIDE improves Llama3.1-8B average zero-shot accuracy by up to 20.26 points over the strongest dynamic-depth baseline under calibration-only settings, while retaining 97.6%/88.2% of dense Qwen3-14B likelihood/generation performance after LoRA tuning. Separate latency benchmarks demonstrate end-to-end speedups of up to for prefill and for decoding over the dense baseline. These results demonstrate that pruning–kernel co-design pushes the frontier of token-wise dynamic structured pruning, combining improved quality retention with practical acceleration across both prefill and decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models (LLMs), LLM inference, Efficient LLM, LLM Pruning, Dynamic Structured Pruning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YRmvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WIDE, a token-level dynamic width pruning method for efficient LLM inference. It dynamically selects groups of attention heads and FFN channels for different tokens and develops customized GPU kernels to reduce the overhead of sparse computation. Experiments on several LLMs evaluate both model quality and inference performance, including prefill and decoding. The combination of adaptive pruning and kernel optimization is the main contribution.

Critical strengths
  1. The method combines adaptive pruning with systems optimization. Rather than focusing only on reducing model computation, the paper also develops customized GPU kernels to improve execution efficiency. This makes the work practically relevant to LLM inference.

  2. The evaluation covers multiple models and inference stages. The experiments include several LLM families and evaluate both prefill and decoding, providing a broader assessment than results from a single model or inference stage.

  3. The paper reports actual inference measurements. Measuring runtime is important because nominal sparsity and theoretical FLOPs do not necessarily translate into real speedups. The systems evaluation is a meaningful strength of this work.

Critical weaknesses
  1. The comparisons do not fully establish advantages under matched compute budgets. Methods with the same nominal sparsity can have different effective FLOPs and execution costs. Comparisons at matched measured latency or effective compute would make the claimed advantages more convincing.

  2. The latency evaluation covers a limited range of workloads. The reported speedups may depend on batch size, prompt length, and serving conditions. More diverse workload measurements are needed to establish how broadly the gains generalize.

  3. Differences in training and recovery settings complicate some quality comparisons. Following each baseline’s original recipe is understandable, but different training budgets and configurations make it difficult to attribute all performance differences to the proposed method itself.

Questions for authors
  1. Can you compare WIDE with the strongest baselines at matched measured latency or effective FLOPs, and clarify how differences in training and recovery budgets affect the quality comparisons?

  2. How do the end-to-end speedups change with batch size and prompt length? Can you provide results under additional serving settings to show that the gains are not specific to the current workload?

4 comments
AuthorsOpenReview ↗

Response to Reviewer YRmv

We sincerely appreciate the reviewer for the detailed comments. Please find our point-by-point responses from the revised PDF, general responses, and below.

W1:The comparisons do not fully establish advantages under matched compute budgets. Methods with the same nominal sparsity can have different effective FLOPs and execution costs. Comparisons at matched measured latency or effective compute would make the claimed advantages more convincing., W3:Differences in training and recovery settings complicate some quality comparisons. Following each baseline’s original recipe is understandable, but different training budgets and configurations make it difficult to attribute all performance differences to the proposed method itself., and Q1:Can you compare WIDE with the strongest baselines at matched measured latency or effective FLOPs, and clarify how differences in training and recovery budgets affect the quality comparisons?


Thank you for the question. The FLOPS–quality Pareto curves and additional experiments are provided in our general response part 1 and P7 Fig.3, P39 Fig.33, 34. Throughout the evaluation, WIDE remains on the Pareto frontier when methods are compared under equal FLOPS in most cases.

For training budgets, WIDE directly follows SkipGPT's settings, so we have added experiments with other static depth and width pruning methods (P29 Table 11, 12). From the results, increasing the calibration set provides little benefit to Shortened-llm-style methods, whereas end-to-end trained DDP and WIDE benefit from larger budgets. At matched S1 budgets of at least 3k steps, WIDE consistently outperforms DDP. As for S2, WIDE shows a similar trend that 3k steps LoRA tuning can yield clear gains.

For matched budget baselines, we also added DDP-10k to the main tables to match WIDE’s overall training budget. In this case, WIDE still outperforms the stronger baseline. (P6 Table 1, P7 Table 2)

W2:The latency evaluation covers a limited range of workloads. The reported speedups may depend on batch size, prompt length, and serving conditions. More diverse workload measurements are needed to establish how broadly the gains generalize. and Q2:How do the end-to-end speedups change with batch size and prompt length? Can you provide results under additional serving settings to show that the gains are not specific to the current workload?


Thank you for the question. We sampled real RULER SQuAD inputs across different batch size ×\times sequence length settings to measure WIDE’s layer-wise sparsity and speedup, as well as the end-to-end speedup (P10 Fig.9, P 35 Fig.26-32). The results below report attention sparsity / FFN sparsity / end-to-end speedup for each sample group:

Qwen3-14B WIDE 50% sparsity:

InputsPrefillDecode
1$\times$4,09661.6% / 33.4% / 1.38×63.1% / 34.0% / 1.38×
1$\times$8,19261.4% / 33.5% / 1.44×62.5% / 33.0% / 1.35×
1$\times$16,38461.9% / 32.8% / 1.50×58.8% / 32.3% / 1.32×
2$\times$4,09661.7% / 33.3% / 1.41×62.0% / 35.0% / 1.21×
2$\times$8,19261.7% / 33.2% / 1.45×61.1% / 33.8% / 1.23×
4$\times$4,09661.7% / 33.2% / 1.42×58.4% / 34.8% / 1.10×

Qwen3-14B WIDE 50% uniform sparsity:

InputsPrefillDecode
1$\times$4,09648.2% / 47.9% / 1.55×48.8% / 48.5% / 1.55×
1$\times$8,19248.2% / 47.9% / 1.61×49.7% / 49.4% / 1.54×
1$\times$16,38448.1% / 48.0% / 1.64×50.6% / 47.9% / 1.49×
2$\times$4,09648.2% / 47.7% / 1.60×48.9% / 49.4% / 1.28×
2$\times$8,19248.1% / 48.0% / 1.63×48.6% / 48.9% / 1.29×
4$\times$4,09648.1% / 47.7% / 1.61×49.1% / 49.1% / 1.19×

With the current implementation, a small batch in decode GEMM may leave only one M-axis tile containing all tokens. Since WIDE does not constrain token or tile-level activation patterns, the co-activation at low sparsity reduces the amount of work that can be skipped. Nevertheless, layers with moderate or high sparsity can still benefit from acceleration.

Reviewer_YRmvOpenReview ↗

Response to the author rebuttal

Thank you for the additional experiments. But some of my concerns remains. Specifically, can you compare WIDE’s quality with strong baselines at matched measured latency for prefill and decoding? And how the speedups hold in serving workloads with mixed lengths of prompt or larger batches. Finally, can you show the gpu hours of DDP-10k for both training stages, so the actual training cost can be compared with WIDE?

AuthorsOpenReview ↗

Response to Reviewer YRmv

Thank you for the follow-up comments! We have added the speedup–quality comparison at the end of the PDF (P42 Fig.35) and conducted additional decoding tests at larger batch sizes. For all methods, we use the base transformers implementation with graph capture and torch.compile. In this setup, mixed-length inputs are represented using left padding and padding positions to make compatible with static KV Cache, which are directly consumed by the attention kernel, where the linear projection directly process the padded sequences (P25 Line 1332). We therefore focus on larger-batch decoding and reduce the KV cache length to 256 for all methods so that the larger batches fit in GPU memory. The results are shown below:

BatchWIDEWIDE-uniform
12852.4% / 35.7% / 0.99×44.1% / 45.9% / 0.99×
25652.3% / 35.9% / 1.00×43.8% / 45.0% / 1.00×
51252.2% / 36.0% / 1.11×43.6% / 44.5% / 1.14×
1,02452.2% / 36.1% / 1.13×43.8% / 44.6% / 1.18×
2,04852.3% / 36.2% / 1.19×44.0% / 45.1% / 1.25×

The corresponding layer-wise analysis is provided in P43 Fig.36 37. As discussed in our response to 4z3X, co-activation has a noticeable impact roughly over M∈[8,256]M\in[8, 256] under the current implementation. We observe speedups when the input token count is below or above this range.

Finally, we have also added the DDP-10k training costs to the general comment and the corresponding tables in the PDF (P26 Table 6, 7).

Thank you for providing the detailed results. Now most of my concerns have been addressed. Therefore, I would like to keep my positive score for this submissioin.

Reviewer_4z3XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a token-wise structured width pruning method, where lightweight differentiable routers decide which attention heads and tile-aligned FFN channel groups are used for compute, on a per-token basis. It combines this technique with a systems contribution, where optimized kernels sort routing masks and skip inactive work for improved efficiency, allowing for speedups of up to 1.68x for prefill and 1.55x for decoding.

Critical strengths
  1. The problem presented in the paper is well-motivated, and WIDE is a practical approach to compromise between flexibility and making efficient use of hardware.
  2. The paper uses kernel optimization techniques to make their method highly practical, staying compatible with CUDA graphs and achieving strong speedups in real-world scenarios.
  3. WIDE clearly outperforms existing previous dynamic pruning methods and establishes a new state-of-the-art.
Critical weaknesses
  1. The sparsity definition is misleading and causes unfair comparisons with other methods. WIDE targets the average skip rates across attention and FFN blocks, averaged across layers and tokens. According to Figures 13 and 14, the learned routers end up around 65% sparse in attention and 30% sparse in FFN. However, the FFN makes up the vast majority of FLOPs and parameter count in most LLMs [1, 2]. In addition, only the q layer in the attention block is pruned, while the k and v layers are left untouched. In other words, even at "50% sparsity" this method is not actually pruning 50% of active parameters or FLOPs (I would guess closer to 35% but that is a simple estimate). Meanwhile, other methods you compare against, such as static depth pruning, DDP, etc, truly remove 50% of parameters/FLOPs, so this comparison overstates your improvements against these methods.
  2. Conceptually, this sounds very similar to the existing subfield of converting dense models into MoEs, and this paper should discuss the similarities and differences between Dense-to-MoE and WIDE more thoroughly.
  3. Your efficiency gains are weaker than existing methods. Static baselines achieve higher prefill and decode speeds, and they also have memory reductions, while WIDE must still keep every parameter in memory.
  4. The training cost of WIDE is unclear. Even though stage 1 only trains the routers, it still needs to backpropagate through the entire model, which can be expensive. More comparisons between the pruning cost of WIDE and the pruning costs of existing baselines are needed.

[1] COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens, https://arxiv.org/abs/2509.06836.

[2] Accelerating Large Language Models through Partially Linear Feed-Forward Network, https://arxiv.org/abs/2501.10054.

Questions for authors
  1. Could the authors report per-token compute (FLOPs or active parameters) on each evaluation task for WIDE and every baseline, and compare methods at matched compute rather than matched nominal sparsity? The uniform-allocation variant already approximates this and might belong in the main tables.
  2. How is "50% sparsity" defined for each baseline (parameter count, number of layers, slicing ratio)? What does 50% mean for the attention-only and FFN-only variants in Table 7?
  3. What are the wall-clock time and GPU-hours of S1 and S2 for WIDE and for each baseline? Does WIDE’s calibration-only advantage persist when all methods get the same training budget?
  4. How much does realized sparsity vary across tasks, inputs, and between prefill and decoding?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 4z3X (part 1)

We sincerely appreciate the reviewer for the detailed comments. Please find our point-by-point responses from the revised PDF, general responses, and below.

W1:The sparsity definition is misleading and causes unfair comparisons with other methods. WIDE targets the average skip rates across attention and FFN blocks, averaged across layers and tokens. According to Figures 13 and 14, the learned routers end up around 65% sparse in attention and 30% sparse in FFN. However, the FFN makes up the vast majority of FLOPs and parameter count in most LLMs [1, 2]. In addition, only the q layer in the attention block is pruned, while the k and v layers are left untouched. In other words, even at "50% sparsity" this method is not actually pruning 50% of active parameters or FLOPs (I would guess closer to 35% but that is a simple estimate). Meanwhile, other methods you compare against, such as static depth pruning, DDP, etc, truly remove 50% of parameters/FLOPs, so this comparison overstates your improvements against these methods. and Q1:Could the authors report per-token compute (FLOPs or active parameters) on each evaluation task for WIDE and every baseline, and compare methods at matched compute rather than matched nominal sparsity? The uniform-allocation variant already approximates this and might belong in the main tables.


Thank you for the question, the uniform sparsity indeed provides a closer match to other baselines. Using the setup in P27 Table 8, we added WIDE uniform results for all backbones and constructed FLOPS–quality Pareto plots from task-wise sparsity (P7 Fig.3, P39 Fig.33, 34, see also our general response part 1). WIDE remains on the frontier at comparable FLOPS in most cases.

For the pruned modules, please refer to the actual GQA export settings for each methods' official codebase in our general response part 1. Static depth pruning removes the KV cache of pruned layers, where most other methods retain full key-value projections and cache in GQA models. Besides, although FFN/MoE accounts for most LLM parameters, the attention and FFN compute shares vary with sequence length because attention has quadratic complexity about sequence length. Their prefill and decode balance points can be calculated as follows:

For a dense GQA Transformer layer, let B be batch size, T sequence length, D hidden size, H_q the number of query heads, H_k the number of key-value heads, and s the up-scale factor of intermediate dimension. The main per-layer FLOPS are:

W_qW_kW_vW_oAttentionW_upW_gateW_down
2BTD22BTD^22BTD2HkHq2BTD^2\frac{H_k}{H_q}2BTD2HkHq2BTD^2\frac{H_k}{H_q}2BTD22BTD^24BT2D4BT^2D2sBTD22sBTD^22sBTD22sBTD^22sBTD22sBTD^2

Where attention accounts for 4BTD2(1+HkHq)+4BT2D4BTD^2(1 + \frac{H_k}{H_q}) + 4BT^2D FLOPS and FFN for 6sBTD26sBTD^2 (excluding SwiGLU), yielding the ratio 2D(1+HkHq)+2T:3sD2D(1+\frac{H_k}{H_q}) + 2T : 3sD, whose shares vary with sequence length. COMPACT and the other works mainly consider particular short lengths (e.g., <300 tokens) to calculate the FLOPS of each part. For Llama3.1-8B and Qwen3-14B, the ideal equal-compute lengths are 16,384 and 19,968 tokens for full attention, and 32,768 / 39,936 for causal attention, respectively. The FLOPS from attention can exceed FFN beyond these sequence lengths.

For decode, consider an ideal memory-bound setting where I/O fully hides computation. Ignoring activation reads and writes, latency is proportional to weight/cache reads. The main memory traffic is as follows:

W_qW_kW_vW_oAttentionW_upW_gateW_down
D2D^2D2HkHqD^2\frac{H_k}{H_q}D2HkHqD^2\frac{H_k}{H_q}D2D^22LDHkHq2LD\frac{H_k}{H_q}sD2sD^2sD2sD^2sD2sD^2

Where the attention : FFN ratio is 2D(1+HkHq)+2LHkHq:3sD2D(1 + \frac{H_k}{H_q}) + 2L\frac{H_k}{H_q}: 3sD. Their costs are comparable at 65,536 tokens for Llama3.1-8B and 99,840 for Qwen3-14B. Given the current hardware limitations, our evaluations on RULER benchmark and end-to-end throughput are mainly limited to 16k context.

AuthorsOpenReview ↗

Response to Reviewer 4z3X (part 2)

W2:Conceptually, this sounds very similar to the existing subfield of converting dense models into MoEs, and this paper should discuss the similarities and differences between Dense-to-MoE and WIDE more thoroughly.


This is an interesting question. Indeed, WIDE’s FFN pruning is partly similar to MoE, from their token-wise routing and dispatch of weight groups. The appendix also tests an equivalent grouped GEMM implementation for WIDE (P21 Fig.10). However, there are still several crucial differences between WIDE and Dense-to-MoE:

  • Each WIDE FFN router can independently select zero-to-all weight groups per token, which is typically at finer granularity than Dense-to-MoE (32–128 vs. 512–2048). Dense-to-MoE generally uses shared experts, top-k routing, and weighted accumulation, with the fixed top-k budget across models or layers.
  • WIDE uses preprocessing (e.g., mask reordering) + dense GEMM with an early-exit prologue, where Dense-to-MoE generally reuses grouped-GEMM-like implementations. For WIDE, the former is more efficient.
  • WIDE requires no offline reordering or splitting of FFN channels into experts, where Dense-to-MoE need to group experts offline using clustering or scoring metrics before training a top-k router.

Besides, WIDE also jointly learns dynamic sparsity allocation across attention and FFN end to end. Thus, despite similarities in FFN computation, the approaches differ. Our initial version already included CMoE as a baseline (P28 Table 10), where WIDE outperforms CMoE when both prune only 50% of the FFN.

W3:Your efficiency gains are weaker than existing methods. Static baselines achieve higher prefill and decode speeds, and they also have memory reductions, while WIDE must still keep every parameter in memory.


Thank you for raising this point. In my view, this is rather a trade-off shared by dynamic pruning methods:

  • Without fine-grained asynchronous weight offloading, both dynamic structured methods such as MoD and SkipGPT and dynamic unstructured methods such as TEAL and WINA need to keep their weights on-device. In return, online decisions allow methods to preserve more quality than static methods under a similar pruning granularity.
  • More broadly, static vs. dynamic and structured vs. unstructured pruning offer different trade-offs. Static pruning can reduce weight storage and simplify acceleration, but typically incurs a larger quality loss. Dynamic pruning retains the weights but adapts computation to each token. Likewise, unstructured pruning offers finer neuron-wise control and potentially better quality retention, while structured pruning makes acceleration applicable to a broader range of workloads.

The practical speed gap also depends on the exported model and inference implementation. Our general response part 1 analyzes the exported structures, while the updated P9 Fig.7 and the layer-wise measurements using real checkpoints and sampled inputs (P10 Fig.9, P35 Fig.26) provide further comparisons:

  • For GQA models, DDP retains the full key-value projections and KV cache, as WIDE does. Moreover, its query-head pruning does not constrain how retained heads map to KV groups, and its exported weight dimensions can be unaligned. Our measurements of both the original and dimension-aligned implementations show lower speedups than the theoretical simulations. In particular, when GQA attention optimizations cannot be used during decoding, DDP does not outperform WIDE in some settings.
  • WINA obtain acceleration in more restricted decoding settings, their speedup decreases further when the additional runtime sorting is included.

Thus, in settings favorable to WIDE, its speedup can be close to that of static width pruning. The remaining gap can be further reduced with larger group size, and more comprehensive kernel implementations. (WIDE is insensitive to group size after LoRA (P8 Fig.4))

AuthorsOpenReview ↗

Response to Reviewer 4z3X (part 3)

W4:The training cost of WIDE is unclear. Even though stage 1 only trains the routers, it still needs to backpropagate through the entire model, which can be expensive. More comparisons between the pruning cost of WIDE and the pruning costs of existing baselines are needed. and Q3:What are the wall-clock time and GPU-hours of S1 and S2 for WIDE and for each baseline? Does WIDE’s calibration-only advantage persist when all methods get the same training budget?


Thank you for the question. We have added the training budgets and costs for each method to our general response part 2 and P26 Table 6, 7. WIDE uses the same default training budget as SkipGPT. Its end-to-end training does take longer than that of the other methods. However, several methods without back-propagation still require about half of WIDE's S1 GPU hours despite using less calibration data. The overall cost of a "training-free" method therefore depends on its calibration procedure. WIDE also takes longer than the end-to-end DDP baseline in their default settings, where their costs are broadly comparable when using the same training budget.

We also compare the effect of training budgets in the general response part 2 and P29 Table 11, 12. In our setting, Shortened-taylor gains little from increasing the S1 calibration budget, whereas DDP and WIDE do benefit from longer S1 and S2 training, with most gains obtained by increasing the number of steps from 1k to 3k. At the same S1 training budget, WIDE retains a clear advantage over DDP once training reaches at least 3k steps.

Q2:How is "50% sparsity" defined for each baseline (parameter count, number of layers, slicing ratio)? What does 50% mean for the attention-only and FFN-only variants in Table 7?


Thank you for the question. The target sparsity is defined according to the pruning units of each method:

  • For static depth pruning methods such as Shortened-llm, it is the fraction of layers removed.
  • For Tyr-the-Pruner and DDP, it is the fraction pruned along the feature dimensions of W_q / W_o / W_down. Pruning these channels also removes the corresponding channels in other operations.
  • For SkipGPT and WIDE, it is a global target. In WIDE, the attention and FFN routers each produce an execution mask of shape [B,T,NG][B, T, N_G], where NGN_G is the number of groups. We first compute the average group skip rate across layers for each module, then average the attention and FFN rates with equal weight (P4 Line 206). For the uniform variant, each module instead approaches the target independently.

For the attention-only and FFN-only WIDE variants in Table 7 (now P28 Table 10), we impose the 50% group-skipping constraint only on the selected module and keep the other module dense. This allows comparison with methods that prune only attention or only FFN. Such constraint is similar to the per-module constraint in the uniform variant.

Q4:How much does realized sparsity vary across tasks, inputs, and between prefill and decoding?


Thank you for the question. The initial paper already included case studies of WIDE's routing behavior (P31 Routing Behavior and Visualizations). These show the layer-wise attention/FFN allocation of Llama3.1-8B on WikiText2, the layer-wise prefill/decode allocation on GSM8K, and token-level examples in Fig.17-23. We have now added task-average sparsity results for two backbones (P34 Fig.24, 25), together with layer-wise prefill/decode sparsity and speedups for samples from RULER SQuAD and WikiText2 (P35 Full Layer-wise Speedup Results). These results show that:

  • Across backbones, layer-wise attention/FFN allocation is significantly different.
  • For the same backbone, the average allocation is stable across tasks.
  • Within a task, routing can differ substantially across inputs and tokens. In the case studies, the router tends to retain semantically rich content, such as nouns and verbs (P31 Line 1661).
  • Allocation also varies by layer between prefill and decode. For example, decode FFN sparsity of Qwen3-14B tends to be more concentrated in the first few layers than prefill.

Together, these observations indicate that model differences and the semantics and context of individual tokens influence WIDE's routing, while the average sparsity of each task remains close to the target.

Reviewer_mYX8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose WIDE, a token-wise dynamic width pruning method for LLM inference. They develop GPU kernels for both prefill and decode for their methods. Unlike previous methods (SkiptGPT and D-LLM) that skip whole layers or sublayers, WIDE routes each token over two kinds of groups: Attention and FFN with lightweight routers. The routers are trained with Gumbel Softmax. On the systems side, WIDE sorts the routing mask per group so that active tokens are clustered into tile-aligned prefixes.

Critical strengths
  1. The authors report both measured speedups an FLOP counts, which shows their method can be useful in practice as well.

  2. The kernel co-design in this approach seems sound and also well tested.

  3. The accuracy gains are pretty large in comparison to other depth/sublayer routing methods at the same sparsity.

  4. The paper includes some useful analysis. The group-size ablation shows that after LoRA, quality becomes insensitive to G. That is what justifies the larger, kernel-friendly G.

Critical weaknesses
  1. “50% sparsity” does not mean matched compute across methods. The target is mask sparsity averaged equally over attention and FFN. WIDE puts about 66% on attention and only about 28% on FFN (Table 6, Fig. 13), and Wk/Wv are never pruned. An accuracy vs. measured-speedup (or FLOPs) Pareto plot would settle the question.

  2. Speed and quality are measured on different configurations.

  3. A smaller dense model appears to dominate. Dense Llama3.2-3B scores 64.91 on the same seven tasks (Table 10). Llama3.1-8B with WIDE(32) at 50% after LoRA scores 64.82 (Table 1). Serving smaller dense models are much easier (less memory on GPU is needed), with smaller KV cache and hidden dimensions for activations. This limits how much the compression method is useful in practice.

  4. WINA looks stronger for decoding. It is training-free, yet reaches 68.52 vs. 61.84 for WIDE, with similar decode speedup (1.46× vs. 1.54×). So WIDE’s advantage rests on prefill.

  5. No reported training costs.

Questions for authors
  1. How would your method perform in comparison to others under a accuracy v.s measured speedup (or FLOPs) Pareto plot?

  2. Can you please explain what are the exact configs used for the quality and speed measurements in the work? Quality uses at most 4,096 tokens on short tasks (Appendix C), while latency is reported at 16k context. With about 66% attention sparsity, long-context retrieval could suffer, and there are no RULER, LongBench, or needle tests.

  3. What are the exact training costs (time and memory) for your method? And how does it compare to other methods?

Note for authors: If your method is pareto optimal in Q1 and my other questions are addressed, I am open to raising your score.

6 comments
AuthorsOpenReview ↗

Response to Reviewer mYX8 (part 1)

We sincerely appreciate the reviewer for the detailed comments. Please find our point-by-point responses from the revised PDF, general responses, and below.

W1:“50% sparsity” does not mean matched compute across methods. The target is mask sparsity averaged equally over attention and FFN. WIDE puts about 66% on attention and only about 28% on FFN (Table 6, Fig. 13), and Wk/Wv are never pruned. An accuracy vs. measured-speedup (or FLOPs) Pareto plot would settle the question. and Q1:How would your method perform in comparison to others under a accuracy v.s measured speedup (or FLOPs) Pareto plot?


Thank you for the question. We constructed FLOPS–quality Pareto plots using the existing uniform variants (P7 Fig.3, P39 Fig.33, 34, see also our general response part 1). WIDE (128) remains on the Pareto frontier at matched FLOPS in most cases.

Among the main baselines, only static depth pruning removes the corresponding layers’ KV cache on the three GQA models. Tyr-the-Pruner and DDP retain full KV projections and cache for GQA, as does WIDE. Please find detailed pruning implementation in each baselines' official codebase from our general response part 1.

W2:Speed and quality are measured on different configurations. and Q2:Can you please explain what are the exact configs used for the quality and speed measurements in the work? Quality uses at most 4,096 tokens on short tasks (Appendix C), while latency is reported at 16k context. With about 66% attention sparsity, long-context retrieval could suffer, and there are no RULER, LongBench, or needle tests.


Thank you for the question. The original end-to-end speed benchmark used batch size 1, sequence length 16,384, and 50% sparsity (P9 Line 455), whereas training was limited to 4,096 tokens (P26 Table 5). To address this mismatch, we evaluated the 50% pruned Qwen3-14B on RULER 16k with lm-evaluation-harness (P10 Table 3). We also measured WIDE (128)’s realized layer-wise sparsity and speedup using real RULER SQuAD and WikiText2 samples at six batch ×\times sequence-length settings: [1, 4,096], [1, 8,192], [1, 16,384], [2, 4,096], [2, 8,192], and [4, 4,096] (P10 Fig.9, P35 Full Layer-wise Speedup Results). The RULER 16k results using the default metrics:

MethodsS-NIAH-1S-NIAH-2S-NIAH-3MK-NIAH-1MK-NIAH-2MK-NIAH-3MQMVVTCWEFWEQA1QA2Avg.
Dense100.00100.00100.00100.00100.00100.00100.0099.80100.0094.0491.1371.6058.8093.49
Shortened-taylor97.8096.4085.6094.8069.6012.6084.7075.0044.5611.3646.7336.1328.0060.25
DDP100.0099.8099.6069.2054.405.2089.8086.4077.8820.1678.0042.7032.6065.83
DDP-10k100.0097.0098.2064.2034.406.8077.8585.7054.3621.8272.7339.2828.8060.09
SkipGPT100.0099.8091.4097.2079.8064.8097.1592.7598.2843.2281.5364.8045.2081.23
WIDE (128)100.00100.00100.0097.4098.4099.0097.9594.4096.0456.2685.4769.0252.0088.15
WIDE (128)U^\text{U}100.0099.8099.8098.2098.4074.4096.3592.3097.8057.4688.2062.4251.6085.90

WIDE achieves the best long-context performance with over 60% attention sparsity, which is better than its uniform variant even in long-context evaluation.

AuthorsOpenReview ↗

Response to Reviewer mYX8 (part 2)

W3:A smaller dense model appears to dominate. Dense Llama3.2-3B scores 64.91 on the same seven tasks (Table 10). Llama3.1-8B with WIDE(32) at 50% after LoRA scores 64.82 (Table 1). Serving smaller dense models are much easier (less memory on GPU is needed), with smaller KV cache and hidden dimensions for activations. This limits how much the compression method is useful in practice.


We thank the reviewer for this question. In fact, prior work has already discussed pruning + recovery vs. training a small model from scratch under matched training budgets. For example, paper [1] and [2] report that pruning requires substantially less compute than training a new small model and can outperform it with carefully designed calibration data, pruning designs, and training strategies. Although our end-to-end approach uses a larger budget than other pruning methods, its data budget, mid-training and post-training FLOPS, and data / training diversity remain lower than those of training from scratch. Under our training setup, pruned Llama3.1-8B and dense Llama3.2-3B therefore do not constitute an apple-to-apple comparison.

Besides, as post-training compression methods, WIDE and other pruning approaches also allow sparsity to be adjusted manually. As shown in P8 Fig.4(c), each method has favorable regions on its sparsity–accuracy curve, offering efficiency gains with relatively small quality losses. This provides a more flexible throughput–accuracy trade-off than training a smaller model.

Regarding memory and efficiency, different pruning methods offer different trade-offs among speed, accuracy, and on-device memory. In practical deployments, these methods are often combined to take advantage of their respective strengths. For example, paper [3] combines static and dynamic pruning across prefill and decoding, but such combinations are standing as a separate topic that required to be further investigation.

[1] Compact Language Models via Pruning and Knowledge Distillation. https://arxiv.org/abs/2407.14679

[2] Small LLMs: Pruning vs. Training from Scratch. https://arxiv.org/abs/2606.14150

[3] Unified Static-Dynamic Pruning for Efficient LLM Inference. https://arxiv.org/abs/2607.21985

W4:WINA looks stronger for decoding. It is training-free, yet reaches 68.52 vs. 61.84 for WIDE, with similar decode speedup (1.46× vs. 1.54×). So WIDE’s advantage rests on prefill.


Thank you for the question. WINA is a representative activation-based dynamic unstructured pruning method, it calibrates activation thresholds offline and skips neurons online:

  • Such fine-grained unstructured pruning provides the finest control and thus the highest accuracy.
  • As a trade-off, these methods can benefit from TEAL's sparse GEMV only for batch=1 decoding, WINA is likewise limited to this setting.
  • Furthermore, WINA also adds runtime top-k sorting, where our previous simulation used direct TEAL-like threshold filtering. Adding WINA’s extra operation can reduce end-to-end decode speedup from 1.46x to 0.97x in the same setting, where WINA itself only reports their speedup of sparse GEMV vs. dense GEMV.
  • Despite using much less calibration data, the training-free WINA still requires 48.1% of WIDE’s total GPU hours in S1 training.

In fact, WIDE’s coarser structured pruning is more likely to be a trade-off: it offers better accuracy than static methods while retaining broader acceleration capability similar to other structured pruning. As a result, WIDE and WINA perform as another example under the common static–dynamic and structured–unstructured pruning trade-offs.

W5:No reported training costs. and Q3:What are the exact training costs (time and memory) for your method? And how does it compare to other methods?


Thank you for the question. Our general response (part 2) and P26 Table 6, 7 report training costs for Llama3.1-8B / Qwen3-14B at 50% sparsity. Some training-free methods have limited GPU-hour advantage over our end-to-end training, while WIDE can also retain most of its performance with a lower training budget (please see general response part 2 and P29 Table 11, 12), which can further reduce such cost gap. In addition, part of the extra time cost in WIDE is caused by the limited device memory per GPU, this constraint further translates into our FSDP2 strategy.

I thank the authors for their thorough reply. Some of my concerns are addressed, but two main things remain:

  1. The new Pareto plots use FLOPs, while my main question was about measured speedup. Could you add accuracy vs. measured end-to-end speedup for the same methods?
  2. On the smaller dense model: Llama-3.2-3B already exists, so the cost of training a small model from scratch does not apply to someone choosing between it and a pruned Llama-3.1-8B, and WIDE still keeps all 8B weights in GPU memory. On the point that sparsity can be adjusted manually, does changing the target sparsity require retraining the routers?

I will raise my score if these issues are addressed.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (part 1)

We sincerely appreciate all reviewers for the constructive reviews and insightful comments. We are glad to see the broad agreement that:

  • The co-design of pruning and GPU kernels has practical value (Reviewer mYX8, 4z3X, and YRmv)
  • Measuring inference speed is a strength of the paper (Reviewer mYX8, 4z3X, and YRmv)
  • WIDE improves quality over existing dynamic depth and sublayer pruning methods (Reviewer mYX8 and 4z3X)

Below we address two common questions shared across reviewers.

GQ1: Quality comparison at matched sparsity/FLOPS

We thank the reviewers for raising this important issue. By default, SkipGPT and WIDE decouple attention and FFN, allowing their routers to learn uneven sparsity allocations across the two modules. We therefore included the uniform variants of WIDE and DDP with matched 10k S1 budget, and calculated each method's pruned FLOPS from its task-wise sparsity allocation, pairing these with downstream performance in Pareto plots (detailed in P7 Fig.3, P39 Fig.33, 34). The results at 50% target sparsity are shown below, with Qwen3-14B averaged FLOPS / scores over all 8 + 3 tasks:

Llama3.1-8B50%50% uniform
Dense100.00% / 71.55 (100.00%)
Shortened-ppl53.06% / 47.26 (66.05%)
Shortened-taylor53.06% / 49.97 (69.83%)
CoopPruner53.06% / 49.63 (69.36%)
Týr-the-Pruner54.63% / 53.46 (74.72%)
DDP58.11% / 54.64 (76.36%)
DDP-10k54.69% / 51.64 (72.18%)
SkipGPT63.54% / 61.60 (86.10%)57.53% / 58.39 (81.61%)
WIDE (128)64.70% / 64.34 (89.92%)57.11% / 61.62 (86.13%)
Qwen3-14B50%50% uniform
Dense100.00% / 76.63 (100.00%)
Shortened-ppl52.48% / 34.22 (44.65%)
Shortened-taylor52.48% / 39.37 (51.38%)
CoopPruner52.48% / 37.37 (48.76%)
DDP55.56% / 55.51 (72.44%)
DDP-10k53.89% / 58.09 (75.81%)
SkipGPT57.31% / 60.18 (78.53%)53.40% / 59.41 (77.53%)
WIDE (128)61.59% / 72.65 (94.81%)54.60% / 71.85 (93.76%)

The following is the results for new settings in P6 Table 1, P7 Table 2:

MethodsS1-25%S2-25%S1-50%S2-50%
DDP64.35 (89.93%)64.24 (89.79%)53.05 (74.14%)54.64 (76.36%)
DDP-10k62.02 (86.68%)62.98 (88.02%)50.66 (70.81%)51.64 (72.18%)
SkipGPT51.68 (72.23%)69.48 (97.11%)42.52 (59.42%)61.60 (86.10%)
WIDE (128)63.65 (88.96%)69.53 (97.18%)61.49 (85.94%)64.34 (89.92%)
WIDE (128)U^\text{U}69.61 (97.30%)69.27 (96.82%)50.99 (71.27%)61.62 (86.13%)
MethodsS1-50%S2-50%
DDP48.03 (74.00%)48.88 (75.30%)
DDP-10k45.29 (69.78%)46.61 (71.81%)
SkipGPT36.74 (56.60%)54.33 (83.70%)
WIDE (128)55.42 (85.38%)58.11 (89.52%)
WIDE (128)U^\text{U}51.99 (80.10%)56.14 (86.49%)
MethodsS1-25%-Like.S1-25%-Gen.S2-25%-Like.S2-25%-Gen.S1-50%-Like.S1-50%-Gen.S2-50%-Like.S2-50%-Gen.
DDP65.26 (88.71%)73.23 (86.37%)68.77 (93.48%)74.62 (88.00%)53.00 (72.04%)52.20 (61.56%)55.74 (75.77%)54.88 (64.72%)
DDP-10k66.08 (89.82%)76.96 (90.76%)69.13 (93.96%)77.82 (91.78%)55.42 (75.34%)60.38 (71.21%)57.79 (78.55%)58.91 (69.48%)
SkipGPT61.63 (83.77%)38.34 (45.21%)71.96 (97.80%)76.67 (90.42%)35.15 (47.77%)8.09 (9.54%)57.20 (77.74%)68.12 (80.34%)
WIDE (128)73.11 (99.37%)78.57 (92.67%)74.33 (101.03%)81.46 (96.08%)69.24 (94.11%)70.81 (83.52%)71.82 (97.62%)74.85 (88.28%)
WIDE (128)U^\text{U}72.29 (98.26%)78.30 (92.35%)73.58 (100.02%)81.46 (96.08%)68.73 (93.42%)69.57 (82.05%)71.37 (97.01%)73.14 (86.26%)

It is clear that WIDE remains on the Pareto frontier when actual FLOPS and matched training budget are considered, with a larger advantage at high sparsity, such as 50%, and on Qwen3-14B.

We also clarify the evaluated methods' actual pruning settings during evaluation and inference on the Llama3.1-8B, Llama3.2-3B, and Qwen3-14B, based on their official repository implementations for GQA model. Here, T denotes pruning and F denotes keeping the dense component:

MethodsW_qW_kW_vKV CacheW_oAttentionW_upW_gateW_down
Shortened-llmTTTTTTTTT
DDPTFFFTTTTT
WINATTTFTFTTT
SkipGPTTFFFTTTTT
WIDETFFFTTTTT

Tyr-the-Pruner and DDP prune W_k, W_v, and the corresponding KV cache for MHA, but their GQA evaluation code retains the full W_k, W_v, and KV cache. Their search operates on attention query head or W_o channels at head granularity and on FFN intermediate channels. Pruned W_k and W_v are exported only for MHA (H_q == H_k). Activation-based unstructured methods such as WINA modify only the hidden-size dimension of the linear projections.

AuthorsOpenReview ↗

General Response (part 2)

GQ2: Training costs and budgets

We thank the reviewers for this question. Using the Llama3.1-8B / Qwen3-14B training logs at 50% sparsity, we summarized each method's training costs in P26 Table 6, 7. SkipGPT and WIDE finish training in FSDP2, and apply FFN gradient checkpointing in Llama3.1-8B S2 and Qwen3-14B S1, as well as attention + FFN checkpointing in Qwen3-14B S2. DDP adopts full gradient checkpointing in all stages with distributed data parallel, which is same as other baselines in stage 2 LoRA.

Llama3.1-8BS1-BudgetsS1-GPUsS1-GPU HoursS1-GPU Mem.S2-BudgetsS2-GPUsS2-GPU HoursS2-GPU Mem.
Shortened-ppl1010.007h19.37GiB16 x 10,000412.130h13.71GiB
Shortened-taylor1011.025h20.12GiB16 x 10,000412.130h13.71GiB
CoopPruner1041.024h24.24GiB16 x 10,000412.210h13.73GiB
Týr-the-Pruner1,024112.467h38.92GiB16 x 10,000414.010h14.18GiB
DDP16 x 1,00043.747h23.98GiB16 x 10,000424.930h18.44GiB
DDP-10k16 x 10,000427.930h23.98GiB16 x 10,000424.930h18.44GiB
WINA20111.350h39.80GiB16 x 10,0004116.000h27.30GiB
WIDE (128)16 x 10,000423.600h38.01GiB16 x 10,000436.460h26.73GiB
Qwen3-14BS1-BudgetsS1-GPUsS1-GPU HoursS1-GPU Mem.S2-BudgetsS2-GPUsS2-GPU HoursS2-GPU Mem.
Shortened-ppl1010.007h28.37GiB16 x 10,000414.667h18.72GiB
Shortened-taylor1010.092h29.12GiB16 x 10,000414.667h18.72GiB
CoopPruner1041.024h28.37GiB16 x 10,000414.800h18.72GiB
DDP16 x 1,00043.881h31.55GiB16 x 10,000431.200h32.33GiB
DDP-10k16 x 10,000430.730h31.55GiB16 x 10,000431.040h32.33GiB
WIDE (128)16 x 10,000433.060h37.92GiB16 x 10,000447.860h29.58GiB

From the table, WIDE has the highest total GPU-hour cost in S1, which is mainly because we adopt the 10,000 training steps used by SkipGPT, where the "training-free" Tyr-the-Pruner and WINA still require about half as many GPU hours. Given the methods' natural compatibility with existing parallel training pipelines, WIDE still keeps its scale-up potential.

Beyond the above comparison, further 50% pruned Qwen3-14B experiments on S1 and S2 budget sensitivity show that DDP and WIDE do benefit from larger training budgets, but obtain most of the gains by 3k steps. At matched training budgets from 3k steps onward, WIDE substantially outperforms DDP. Details are available in P29 Table 11, 12:

Qwen3-14B S1BudgetWikiText2ARC-EARC-CBoolQWinograndePIQAOpenbookQAHellaswagMMLUIFEvalMBPP+GSM8KLike. Acc.Gen. Acc.
Shortened-taylor10 (default)2.27e+727.2326.5347.4649.2551.0830.6025.5426.3811.090.000.3035.50 (48.26%)3.79 (4.47%)
Shortened-taylor128 / 256 / 5124.78e+727.7726.9641.9847.9851.4131.8025.3925.409.980.000.6834.84 (47.35%)3.55 (4.19%)
DDP1k (default)48.3955.5536.5181.5957.0666.7034.4049.6142.6046.7650.7959.0553.00 (72.04%)52.20 (61.56%)
DDP3k295.9060.3140.5282.5057.9367.2430.8051.7041.5853.0460.8463.6054.07 (73.50%)59.16 (69.78%)
DDP5k31.9261.7841.2183.8557.5368.1733.8052.2149.4550.4660.3164.7456.00 (76.12%)58.50 (69.00%)
DDP10k28.3862.6239.7682.2957.6968.2833.0052.7746.9954.5260.0566.5655.42 (75.34%)60.38 (71.21%)
WIDE (128)1k25.6976.0146.8475.6258.4074.3137.6060.6346.6534.9358.9954.0559.51 (80.89%)49.32 (58.17%)
WIDE (128)3k22.0582.8756.1485.0764.4075.8945.4065.5560.6257.1172.2271.9466.99 (91.06%)67.09 (79.13%)
WIDE (128)5k21.5384.0958.9586.6665.1177.2044.8066.0664.0854.7176.1976.1168.37 (92.93%)69.00 (81.38%)
WIDE (128)10k (default)19.8584.5560.4987.5565.1178.1245.0068.1565.0058.2275.9278.3169.24 (94.11%)70.81 (83.52%)
Qwen3-14B S2BudgetWikiText2ARC-EARC-CBoolQWinograndePIQAOpenbookQAHellaswagMMLUIFEvalMBPP+GSM8KLike. Acc.Gen. Acc.
WIDE (128)1k16.9284.7262.8888.2567.5678.5045.2070.3467.6156.5674.3380.2870.63 (96.01%)70.39 (83.02%)
WIDE (128)3k15.9285.6061.3488.8366.4579.3746.2071.1067.8960.0778.3082.1870.85 (96.30%)73.52 (86.71%)
WIDE (128)5k15.4385.8563.1388.6565.5079.5447.2071.7668.4059.3376.4584.1571.26 (96.86%)73.31 (86.46%)
WIDE (128)10k (default)15.1186.3664.0788.2567.5678.3448.8072.5668.6660.2579.1085.2171.82 (97.62%)74.85 (88.28%)
AuthorsOpenReview ↗

Modification Lists of PDF

In response to the reviewers' questions, we updated the initial PDF using results from existing checkpoints and additional training and evaluation. The main changes are summarized as follows:

Major Revision


Task-wise Quality-FLOPS Pareto

  • We measured the task-wise sparsity of the baselines and WIDE at different target sparsities and calculated the corresponding FLOPS based on each method's pruning implementation during evaluation. We use these results to present a complete Pareto frontier (P7 Performance across Matched Computation, Fig.3, P39 Fig.33, 34)

Training Budget & Costs

  • We used training logs to report the computing resources, number of training samples, and total GPU hours for each method trained on Llama3.1-8B / Qwen3-14B at 50% sparsity (P26 Table 6, 7)
  • We conducted additional experiments on the S1 and S2 training budgets of different methods, comparing how the number of training steps and calibration data size affect final performance (P29 Table 11, 12)

New Settings & Baselines

  • We introduced a uniform-constrained WIDE (128) for Llama3.1-8B, Llama3.2-3B, and Qwen3-14B. The constraint requires attention and FFN to reach the target sparsity independently, rather than share a budget, to better align with the baselines' actual sparsity. This variant was already evaluated on Llama3.1-8B in the initial version (P27 Line 1406). We also added DDP trained for 10k steps in stage 1 as a stronger baseline setting (P6 Table 1, P7 Table 2, P27 Table 9, P40 Table 14, 15, P41 Table 16, 17, 18)

Long Context Evaluation

  • We added RULER 16k to evaluate the long-context capabilities of different pruning methods on Qwen3-14B, using the default lm-evaluation-harness settings (P9 Long-context Evaluation, P10 Table 3)

Throughput Evaluation on Real Trace

  • We extracted real 4k, 8k, and 16k samples from Wikitext2 for Llama3.1-8B and RULER SQuAD for Qwen3-14B, grouped them into different batch sizes, and evaluated WIDE (128)'s layer-wise sparsity and corresponding speedup on these samples (P10 Fig.9, P 35 Full Layer-wise Speedup Results, Fig.26-32)

Minor Revision


  • We introduced Dense-to-MoE methods earlier and outlined a key difference from WIDE: fixed top-k routing budgets vs. fully flexible online per-token, per-layer routing (zero-to-all) (P3 Line 134)
  • We introduced DDP with 10k S1 training steps (DDP-10k) and uniform-constrained WIDE (WIDE$^U$) earlier in the main experimental section (P7 Line 354)
  • We rechecked the configurations used to simulate end-to-end speedup at 50% sparsity. DDP used a default setting that pruned GQA KV groups proportionally, while WINA used a default TEAL-like threshold-based setting. These settings led to overestimated speedups under the Llama3.1-8B configuration. We corrected DDP's configuration and clarified WINA's test implementation (P9 Line 467, Fig.7)
  • We clarified that the exports of Tyr-the-Pruner and DDP on GQA models retain the full KV projections and cache (P25 Line 1321)
  • We moved the previous layer error accumulation analysis entirely to the appendix (P30 Line 1577, Table 13)
  • We added figures showing WIDE's average sparsity allocation on downstream tasks for Llama3.1-8B and Qwen3-14B, calculated over all samples in each task (P31 Line 1670, P34 Fig.24, 25)
AuthorsOpenReview ↗

Summary of Rebuttal

Dear ACs,

Thank you and the reviewers for your valuable time. To assist with evaluation, we provide a concise summary of the review and discussion process thus far, along with the updates we have made during the rebuttal process.

Paper Summary


This work introduces WIDE, a token-wise structured dynamic width pruning framework for LLM, along with two-stage end-to-end differentiable training. We provide:

  • Finer-grained end-to-end training: Building on SkipGPT's two-stage framework, WIDE extends layer-wise pruning to attention KV groups and contiguous FFN intermediate-channel groups.
  • Unified CuTe/Python DSL kernel libraries for routing-based structured dynamic pruning: The mask reordering preprocessing enables grouped tokens to activate the same weight group, enabling GEMM-tile-level skipping, along with additional support to finer-grained hardware-aligned activation-load skipping and MMA skipping, selected through unified autotuning.
  • Comprehensive quality evaluation: We evaluate Llama3.1-8B, Llama3.2-3B, and Qwen3-14B on WikiText2 and up to eight likelihood and three generation tasks, and study the effect of group size. WIDE remains competitive on the matched-FLOPS and speedup–quality Pareto plots added during rebuttal.
  • Comprehensive inference benchmarking: We report kernel-level speedup, end-to-end speedup with random masks, and layer-wise speedup on real RULER and WikiText2 sample traces. We also added an analysis of co-activation in dynamic structured pruning during rebuttal.

Reviewers agree that WIDE's co-design of pruning and GPU kernels has practical value (Reviewer mYX8, 4z3X, and YRmv), measurable inference speedup (Reviewer mYX8, 4z3X, and YRmv), as well as improved quality over existing dynamic depth and sublayer pruning methods (Reviewer mYX8 and 4z3X).

Rebuttal Status Summary


During rebuttal, we addressed three shared concerns by new experiments and analysis:

  • Quality comparisons and Pareto plots at matched theoretical FLOPS (reviewers mYX8, 4z3X, and YRmv).
  • Long-context evaluation on RULER 16k. Prefill/decode layer-wise sparsity and speedup on real RULER and WikiText2 traces across batch sizes and sequence lengths, and sparsity allocation across tasks (reviewers mYX8, 4z3X, and YRmv).
  • Training budgets and costs for WIDE and the baselines, with quality comparison under matched training budgets, and sensitivity to stage 1 and stage 2 budgets (reviewers mYX8, 4z3X, and YRmv).

We also clarified individual questions about:

  • Pruning versus a smaller dense model, such as Llama3.2-3B (reviewer mYX8).
  • Comparison with activation-based dynamic unstructured pruning, such as WINA (reviewer mYX8).
  • Differences between WIDE and Dense-to-MoE methods (reviewer 4z3X).
  • On-device memory and speedup trade-offs between static and dynamic pruning (reviewer 4z3X).
  • The definition of target sparsity across baselines and our attention-only and FFN-only evaluations (reviewer 4z3X).

Follow-up discussions focused on:

  • Quality–speedup Pareto comparisons using measured speedup (reviewers mYX8, 4z3X, and YRmv).
  • Decoding speedups at larger batch sizes and the effects of co-activation (reviewers 4z3X and YRmv).
  • The feasibility of supporting multiple sparsity levels with one training run (reviewer mYX8).

The remaining concerns were the dynamic methods' on-device weight requirements (reviewer mYX8), and the effect of co-activation on decoding speedup (reviewer 4z3X). To investigate these, we conducted exploratory Llama3.1-8B experiments on asynchronous FFN weight offloading and partial static width pruning. The latter permanently removes selected weight groups from some layers while largely preserving quality, and provides speedup in the range affected by co-activation.

Overall, WIDE is a structured dynamic width pruning method. Like MoD, D-LLM, and SkipGPT, it adapts computation dynamically, while extending pruning to finer-grained attention head and FFN channel groups with full acceleration support. Compared with static pruning, WIDE retains the full weights on-device, and its current implementation experiences co-activation under certain workloads. In return, it retains more quality and can approach static width pruning's speedup in favorable workloads, where the quality advantage is particularly pronounced at moderate to high sparsity and on Qwen3-14B.

Best regards,

Authors of Submission 715

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.