- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue of significant accuracy loss caused by standard 2:4 pruning by proposing an overlapping pruning OP, which expands the selection space for valid masks by having adjacent selection windows share candidate positions. This method retains the same number of selectors, the four-consecutive structure, and the 2-bit index for each retained weight, while primarily altering how the selectors are connected. The authors model mask selection as a matching problem and utilize matroid structures to achieve an accurate and efficient solution. Experiments show that OP can significantly improve the pruning quality of the standard 2:4 algorithm and can be combined with existing importance scoring and quantification methods. The overhead associated with selecting a network is close to the standard 2:4 workload.
Critical strengths
The method features an innovative design and a unique approach. Authors start with selector connectivity and expands the feasible mask space through overlapping candidate windows, while preserving the original selector count, fan-in, and 2-bit index. It links mask flexibility with hardware implementation, providing a distinctive approach to improving the quality of 2:4 pruning.
The authors model mask selection as maximum-weight matching and utilize the properties of transversal matroids to prove that greedy selection is optimal for a given saliency objective. For OP-F2D, they further accelerate the feasibility check using union-find, limiting the time required for mask generation to approximately 1.3–1.6 times that of the 2:4 top-k method, thereby balancing solution quality and computational overhead.
Across multiple models and different saliency scorers, OP-F2D significantly improves the standard 2:4 perplexity, reaching levels close to 8:16, with a corresponding increase in zero-shot accuracy. It maintains its advantage even under FP8 and NVFP4 quantization, indicating that this sparsity pattern can be combined with other compression methods.
Critical weaknesses
The current hardware evaluation covers only the selection network and assumes that the activations for the entire tile are already accessible to all selectors. The overhead associated with inter-lane data supply, the operand collector, and weight/index packing, which are required for actual execution, has not been evaluated, nor has the throughput of a complete sparse GEMM or LLM inference been demonstrated. Therefore, while the PPA of a local selector is close to 2:4, this is not sufficient to prove that the complete system achieves the same efficiency.
The 4:8 and 8:16 selectors are implemented as 8:1 and 16:1 selectors, respectively; however, based on the valid range of ordered indices, it can be deduced that each selector actually needs to cover only 5 and 9 candidate positions, respectively. It remains unclear whether a more streamlined implementation would narrow the PPA gap. Additionally, the approximately 0.05% per-MAC area overhead is derived from combining 130nm measurements with the area ratio of another design at 15nm; this is an order-of-magnitude estimate and should not be equated with actual measured results for a complete MAC.
Questions for authors
Can you provide an execution verification that includes real-world data supply constraints? For example, a complete datapath that includes lane partitioning, an operand collector, index decoding, and packing, or a cycle-accurate simulation, and compare latency, throughput, and data movement overhead with the 2:4 configuration. Aside from the network, which component do the authors believe is most likely to become a bottleneck?
Do N:M hardware baselines take advantage of the valid candidate range of ordered indexes? For 4:8 and 8:16, the number of candidates for each ordered selector can be reduced to 5 and 9, respectively; after adopting this type of implementation, how much of the OP’s area, delay, and power advantages remain? How should the scope of application be limited for the approximately 0.05% per-MAC overhead?
Can OP be combined with ALPS and/or SparseGPT? For example, in the case of SparseGPT, how do overlapping constraints interact with adaptive mask selection, error compensation, and channel-reorder (it was introduced in GPTQ, but the principle is universal)? I hope the authors will discuss the key modifications, as well as whether weight reconstruction might reduce or increase OP’s relative gains compared to the 2:4 ratio. Additionally, how do the search spaces of OP’s selector and MaskLLM compare in terms of scale?
Can OP be extended to activation sparsity? Appendix G mentions this approach, but activation masks typically need to be generated online. Aside from mask generation, is it necessary to adjust the index encoding, data selection, and the dataflow for subsequent computations? Could these additional overheads potentially offset the benefits gained from sparsity? We hope the authors will discuss the applicable conditions and the main obstacles.
Questions 3 and 4 are intended primarily for discussion; however, if resources permit, experimental evidence is welcome.
4 comments
[Response to Reviewer ZxUp 2/2]
Q3: Combining OP with SparseGPT and ALPS, and the search space compared with MaskLLM.
Results: We ran SparseGPT and ADMM-Grad (General Response, Table R1). At tile 64, OP-F2D recovers 80.3% of the 2:4-to-unstructured gap under both methods.
ALPS/ADMM: ALPS uses ADMM to jointly optimize mask selection and error compensation. In their weight update, a projection operator is used to project an input matrix onto the set of feasible masks for the given sparsity pattern. The paper suggests that extending compatibility to N:M sparsity (and other patterns) requires only replacing this projection to the respective feasible mask set. To apply OP, we can similarly replace it with the feasible mask set of OP. This requires a mask calculation on every update step.
SparseGPT changes: OP is complementary to SparseGPT because it modifies mask constraints without changing the underlying reconstruction procedure.
- Adaptive mask selection: Replace SparseGPT’s independent 2:4 mask selection with OP’s joint, exact mask selection per tile, using SparseGPT’s current Hessian-aware scores. Subsequent tile masks are selected after error compensation, preserving adaptivity.
- Error compensation: Once an OP-valid mask is selected, SparseGPT’s existing Hessian-based weight updates can proceed without changing the reconstruction equations, while preserving the fixed mask.
- Channel reordering: Weights and Hessian information must be reordered consistently, while OP constraints must remain valid in the final inference layout. As shown in Appendix N, OP-aware permutations can reduce selector contention, whereas structure-unaware permutations may degrade quality.
Does weight reconstruction reduce OP's gains? In absolute terms, reconstruction shrinks the 2:4 perplexity gap. In relative terms, OP-F2D still recovers 80.3% of the gap under both SparseGPT and ADMM-Grad. Reconstruction and a larger set of allowed masks therefore add up.
Search space vs MaskLLM. OP and MaskLLM differ along two separate axes.
(i) Feasible space. MaskLLM chooses masks within the 2:4 family, where each disjoint group of four weights selects one of 6 masks. This gives 6^(N/4) full-model masks for N weights. OP enlarges the family itself. OP-F2D admits ~8,500× more masks per 64-weight tile (Table R6). Because tiles are independent, this advantage compounds to roughly (8.5×10^3)^(N/64) over the full model with OP-F2D.
(ii) Objective. MaskLLM optimizes the end-to-end language-modeling loss, so its choice in each group accounts for interactions with all other groups in the model. Its feasible set factors over groups, but its objective does not. OP instead maximizes a fixed, additive saliency score. Overlapping windows couple decisions within a tile only through feasibility, and the greedy matroid pass returns the exact optimum of this objective.
MaskLLM therefore performs approximate search with a global objective over a restricted space. OP performs exact search with a local objective over a much larger space. OP-F2D matches or improves on MaskLLM (6.70 vs. 6.72 on Llama 2 7B; 5.74 vs. 5.85 on Llama 2 13B) without mask training. This suggests the 2:4 feasible set limits quality even when masks are trained end to end. The two approaches are complementary: MaskLLM-style end-to-end mask learning could be applied over the OP family. We leave this to future work.
Table R6: Comparison of feasible mask space sizes at 50% sparsity (L=64).
| Method | Feasible masks | Relative to MaskLLM |
|---|---|---|
| MaskLLM (2:4) | 2.82×10^12 | 1x |
| OP-1D (1x64) | 1.38×10^15 | ~490x |
| OP-F2D (1x64) | 2.41×10^16 | ~8,500x |
Q4: Extension to activation sparsity.
Yes. OP changes only which inputs each selector is wired to, so it works for whichever operand is sparse. For activation sparsity, each token's activation vector is tiled along the reduction dimension. It is compressed to L/2 kept values with 2-bit indices, and the weights become the dense input to the selection network (Figure 1e with the operands swapped). The index format, selector wiring and dataflow stay the same, so the costs in Table 2 apply directly.
What changes is that scoring, mask selection and encoding run online for every token. The run-time optimized solver for 1x16 tiles was designed for this setting (Appendix G.4). At L=16, the 8 KiB lookup table generates OP-F2D masks for a 4096×4096 tensor in 2.49 ms (Table 9) and allocates only the output mask (Table 10). Each tile costs a fixed amount, so mask generation grows only with the number of activations.
Gains should appear in compute-bound settings not in memory-bound decoding, because the dense weights are still fetched. We will add this discussion to the paper and leave the full evaluation to future work.
Post-rebuttal decision
After reading the author's rebuttal, I decide to keep my score unchanged.
Thank you for reading our rebuttal and for your continued support of our paper. We appreciate your constructive feedback, which helped us improve the manuscript.
[Response to Reviewer ZxUp 1/2]
We thank the reviewer for the careful review and for recognizing OP's selector-level design, its matroid-based optimality, and its gains across scorers and quantization formats.
W1 / Q1: System-level execution and bottlenecks. We extended the hardware analysis beyond the selector network (General Response, Hardware (i)–(iii)):
We have not yet shown end-to-end sparse GEMM or LLM inference throughput. Because OP's data movement matches 2:4 and its selector fits within the cycle, we expect throughput to match 2:4, apart from the cross-lane inputs above.
W2 / Q2: Optimized N:M baselines and the 0.05% estimate. The reviewer is correct that ordered selectors for 4:8 and 8:16 need only 5 and 9 candidates (this is not the current 2:4 design). Even with this optimization, 4:8 and 8:16 still need 3 and 4 index bits per kept weight (⌈log₂5⌉ and ⌈log₂9⌉). That means 96 and 128 metadata bits per 1×64 tile, compared with 64 for 2:4 and OP. Fan-in is also still 5 and 9 rather than 4. OP's advantage over N:M therefore does not depend on the unoptimized baseline.
Below is the hardware efficiency data for the 5 and 9 variants with the ASAP7 7nm process. While these variants reduce the hardware cost of 4:8 and 8:16 sparsity, they require 3 and 4 index bits which would impact the rest of the sparse matrix multiplication data path. Additionally, the 5 variant for 4:8 results in significant increase in worst-case delay (1.42 - 1.60x) and the 9 variant for 8:16 results in significant increase in all hardware metrics (1.18 - 2.13x).
Table R7: 1×64 normalized hardware data with 5 and 9 variants
We agree that the 0.05% per-MAC figure was an order-of-magnitude estimate. We have replaced it with direct 7nm measurements: OP is within 0.97–1.01× of 2:4's cell area and 0.96–1.02× of its power. We present these as selector-level results, not complete-MAC measurements.