- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WIDE, a token-level dynamic width pruning method for efficient LLM inference. It dynamically selects groups of attention heads and FFN channels for different tokens and develops customized GPU kernels to reduce the overhead of sparse computation. Experiments on several LLMs evaluate both model quality and inference performance, including prefill and decoding. The combination of adaptive pruning and kernel optimization is the main contribution.
Critical strengths
-
The method combines adaptive pruning with systems optimization. Rather than focusing only on reducing model computation, the paper also develops customized GPU kernels to improve execution efficiency. This makes the work practically relevant to LLM inference.
-
The evaluation covers multiple models and inference stages. The experiments include several LLM families and evaluate both prefill and decoding, providing a broader assessment than results from a single model or inference stage.
-
The paper reports actual inference measurements. Measuring runtime is important because nominal sparsity and theoretical FLOPs do not necessarily translate into real speedups. The systems evaluation is a meaningful strength of this work.
Critical weaknesses
-
The comparisons do not fully establish advantages under matched compute budgets. Methods with the same nominal sparsity can have different effective FLOPs and execution costs. Comparisons at matched measured latency or effective compute would make the claimed advantages more convincing.
-
The latency evaluation covers a limited range of workloads. The reported speedups may depend on batch size, prompt length, and serving conditions. More diverse workload measurements are needed to establish how broadly the gains generalize.
-
Differences in training and recovery settings complicate some quality comparisons. Following each baseline’s original recipe is understandable, but different training budgets and configurations make it difficult to attribute all performance differences to the proposed method itself.
Questions for authors
-
Can you compare WIDE with the strongest baselines at matched measured latency or effective FLOPs, and clarify how differences in training and recovery budgets affect the quality comparisons?
-
How do the end-to-end speedups change with batch size and prompt length? Can you provide results under additional serving settings to show that the gains are not specific to the current workload?
4 comments
Response to the author rebuttal
Thank you for the additional experiments. But some of my concerns remains. Specifically, can you compare WIDE’s quality with strong baselines at matched measured latency for prefill and decoding? And how the speedups hold in serving workloads with mixed lengths of prompt or larger batches. Finally, can you show the gpu hours of DDP-10k for both training stages, so the actual training cost can be compared with WIDE?
Response to Reviewer YRmv
Thank you for the follow-up comments! We have added the speedup–quality comparison at the end of the PDF (P42 Fig.35) and conducted additional decoding tests at larger batch sizes. For all methods, we use the base transformers implementation with graph capture and torch.compile. In this setup, mixed-length inputs are represented using left padding and padding positions to make compatible with static KV Cache, which are directly consumed by the attention kernel, where the linear projection directly process the padded sequences (P25 Line 1332). We therefore focus on larger-batch decoding and reduce the KV cache length to 256 for all methods so that the larger batches fit in GPU memory. The results are shown below:
| Batch | WIDE | WIDE-uniform |
|---|---|---|
| 128 | 52.4% / 35.7% / 0.99× | 44.1% / 45.9% / 0.99× |
| 256 | 52.3% / 35.9% / 1.00× | 43.8% / 45.0% / 1.00× |
| 512 | 52.2% / 36.0% / 1.11× | 43.6% / 44.5% / 1.14× |
| 1,024 | 52.2% / 36.1% / 1.13× | 43.8% / 44.6% / 1.18× |
| 2,048 | 52.3% / 36.2% / 1.19× | 44.0% / 45.1% / 1.25× |
The corresponding layer-wise analysis is provided in P43 Fig.36 37. As discussed in our response to 4z3X, co-activation has a noticeable impact roughly over under the current implementation. We observe speedups when the input token count is below or above this range.
Finally, we have also added the DDP-10k training costs to the general comment and the corresponding tables in the PDF (P26 Table 6, 7).
Thank you for providing the detailed results. Now most of my concerns have been addressed. Therefore, I would like to keep my positive score for this submissioin.
Response to Reviewer YRmv
We sincerely appreciate the reviewer for the detailed comments. Please find our point-by-point responses from the revised PDF, general responses, and below.
Thank you for the question. The FLOPS–quality Pareto curves and additional experiments are provided in our general response part 1 and
P7 Fig.3, P39 Fig.33, 34. Throughout the evaluation, WIDE remains on the Pareto frontier when methods are compared under equal FLOPS in most cases.For training budgets, WIDE directly follows SkipGPT's settings, so we have added experiments with other static depth and width pruning methods (
P29 Table 11, 12). From the results, increasing the calibration set provides little benefit to Shortened-llm-style methods, whereas end-to-end trained DDP and WIDE benefit from larger budgets. At matched S1 budgets of at least 3k steps, WIDE consistently outperforms DDP. As for S2, WIDE shows a similar trend that 3k steps LoRA tuning can yield clear gains.For matched budget baselines, we also added DDP-10k to the main tables to match WIDE’s overall training budget. In this case, WIDE still outperforms the stronger baseline. (
P6 Table 1, P7 Table 2)Thank you for the question. We sampled real RULER SQuAD inputs across different batch size × sequence length settings to measure WIDE’s layer-wise sparsity and speedup, as well as the end-to-end speedup (
P10 Fig.9, P 35 Fig.26-32). The results below report attention sparsity / FFN sparsity / end-to-end speedup for each sample group:Qwen3-14B WIDE 50% sparsity:
Qwen3-14B WIDE 50% uniform sparsity:
With the current implementation, a small batch in decode GEMM may leave only one M-axis tile containing all tokens. Since WIDE does not constrain token or tile-level activation patterns, the co-activation at low sparsity reduces the amount of work that can be skipped. Nevertheless, layers with moderate or high sparsity can still benefit from acceleration.