acceptodds
Under review as a conference paper at ICLR 2027

No Man Is an Island: Overlapping Pruning for Near-Unstructured Quality at 2:4 Efficiency

Abstract

Semi-structured 2:4 sparsity is the dominant fine-grained pruning pattern on modern ML accelerators, yet for large language models it degrades quality far more than unstructured pruning at the same 50% sparsity. We trace this gap to mask rigidity: 2:4 partitions weights into disjoint four-element windows, so both selectors draw from the same four inputs, severely constraining the feasible mask space. We propose Overlapping Pruning (OP), an algorithm–hardware co-designed sparsity structure that overlaps neighboring selector windows, enlarging the mask space while preserving 2:4's selectors, 2-bit index, and datapath. At 4-, 8-, and 16-bit, OP-1D and OP-F2D cost at most 2:4's core area, - its delay, and - its power. OP-2D additionally yields transposable masks. Feasible masks are the independent sets of a transversal matroid, so a saliency-ordered greedy pass with a per-candidate feasibility check is exact for every OP pattern; our kernels generate OP-F2D masks within the 2:4 mask-generation cost. One-shot OP-F2D with Wanda++ increases perplexity by at most over unstructured 50%, versus up to for 2:4. OP-1D attains lower perplexity than 4:8 and OP-F2D stays within of 8:16; patterns cost - 2:4's core area and - its delay. OP-F2D matches or improves on reported results from learned-mask and factorized 2:4 methods at a fraction of their pruning cost, with no mask training or added parameters. By relaxing mask geometry alone, OP-F2D closes - of the unstructured-to-2:4 quality gap at 2:4's hardware cost.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Post-Training Pruning, Sparsity, N:M Semi-Structured Pruning, Large Language Models, Efficient ML, Inference Acceleration, Combinatorial Optimization, 2:4

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_ZxUpOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the issue of significant accuracy loss caused by standard 2:4 pruning by proposing an overlapping pruning OP, which expands the selection space for valid masks by having adjacent selection windows share candidate positions. This method retains the same number of selectors, the four-consecutive structure, and the 2-bit index for each retained weight, while primarily altering how the selectors are connected. The authors model mask selection as a matching problem and utilize matroid structures to achieve an accurate and efficient solution. Experiments show that OP can significantly improve the pruning quality of the standard 2:4 algorithm and can be combined with existing importance scoring and quantification methods. The overhead associated with selecting a network is close to the standard 2:4 workload.

Critical strengths

The method features an innovative design and a unique approach. Authors start with selector connectivity and expands the feasible mask space through overlapping candidate windows, while preserving the original selector count, fan-in, and 2-bit index. It links mask flexibility with hardware implementation, providing a distinctive approach to improving the quality of 2:4 pruning.

The authors model mask selection as maximum-weight matching and utilize the properties of transversal matroids to prove that greedy selection is optimal for a given saliency objective. For OP-F2D, they further accelerate the feasibility check using union-find, limiting the time required for mask generation to approximately 1.3–1.6 times that of the 2:4 top-k method, thereby balancing solution quality and computational overhead.

Across multiple models and different saliency scorers, OP-F2D significantly improves the standard 2:4 perplexity, reaching levels close to 8:16, with a corresponding increase in zero-shot accuracy. It maintains its advantage even under FP8 and NVFP4 quantization, indicating that this sparsity pattern can be combined with other compression methods.

Critical weaknesses

The current hardware evaluation covers only the selection network and assumes that the activations for the entire tile are already accessible to all selectors. The overhead associated with inter-lane data supply, the operand collector, and weight/index packing, which are required for actual execution, has not been evaluated, nor has the throughput of a complete sparse GEMM or LLM inference been demonstrated. Therefore, while the PPA of a local selector is close to 2:4, this is not sufficient to prove that the complete system achieves the same efficiency.

The 4:8 and 8:16 selectors are implemented as 8:1 and 16:1 selectors, respectively; however, based on the valid range of ordered indices, it can be deduced that each selector actually needs to cover only 5 and 9 candidate positions, respectively. It remains unclear whether a more streamlined implementation would narrow the PPA gap. Additionally, the approximately 0.05% per-MAC area overhead is derived from combining 130nm measurements with the area ratio of another design at 15nm; this is an order-of-magnitude estimate and should not be equated with actual measured results for a complete MAC.

Questions for authors

Can you provide an execution verification that includes real-world data supply constraints? For example, a complete datapath that includes lane partitioning, an operand collector, index decoding, and packing, or a cycle-accurate simulation, and compare latency, throughput, and data movement overhead with the 2:4 configuration. Aside from the network, which component do the authors believe is most likely to become a bottleneck?

Do N:M hardware baselines take advantage of the valid candidate range of ordered indexes? For 4:8 and 8:16, the number of candidates for each ordered selector can be reduced to 5 and 9, respectively; after adopting this type of implementation, how much of the OP’s area, delay, and power advantages remain? How should the scope of application be limited for the approximately 0.05% per-MAC overhead?

Can OP be combined with ALPS and/or SparseGPT? For example, in the case of SparseGPT, how do overlapping constraints interact with adaptive mask selection, error compensation, and channel-reorder (it was introduced in GPTQ, but the principle is universal)? I hope the authors will discuss the key modifications, as well as whether weight reconstruction might reduce or increase OP’s relative gains compared to the 2:4 ratio. Additionally, how do the search spaces of OP’s selector and MaskLLM compare in terms of scale?

Can OP be extended to activation sparsity? Appendix G mentions this approach, but activation masks typically need to be generated online. Aside from mask generation, is it necessary to adjust the index encoding, data selection, and the dataflow for subsequent computations? Could these additional overheads potentially offset the benefits gained from sparsity? We hope the authors will discuss the applicable conditions and the main obstacles.

Questions 3 and 4 are intended primarily for discussion; however, if resources permit, experimental evidence is welcome.

4 comments
AuthorsOpenReview ↗

[Response to Reviewer ZxUp 1/2]

We thank the reviewer for the careful review and for recognizing OP's selector-level design, its matroid-based optimality, and its gains across scorers and quantization formats.

W1 / Q1: System-level execution and bottlenecks. We extended the hardware analysis beyond the selector network (General Response, Hardware (i)–(iii)):

  • Data supply and packing: OP uses exactly 2:4's compressed format: L/2 values plus 2-bit indices, or 64 metadata bits per 1×64 tile. An OP sparse MMA fetches the same weight fragment, metadata, and activation slice as a 2:4 MMA. Register-file and operand-collector traffic, MAC count, and bytes per output are therefore unchanged. Index decoding differs only in which hardwired port each 2-bit index selects.
  • Lane partitioning: When a tile's activations are split across lanes, OP-1D needs 6 of 128 selector inputs to cross lanes (four 16-wide lanes). OP-F2D needs 16 of 128 (one 4×4 quadrant per lane). All of these are nearest-neighbor.
  • Timing: At 7nm (ASAP7), the 16-bit OP-F2D selector path is 165.5 ps, compared with 158.7 ps for 2:4. That is about 23% of an A100 cycle, so it fits within an existing pipeline stage without lowering frequency.
  • Most likely bottleneck: We expect it to be inter-lane operand delivery for OP-F2D. Where this is a problem, OP-F2D with a 1×16 tile needs only 16 activations at a time. It still closes 77.3% (Wanda) and 73.6% (Wanda++) of the 2:4-to-unstructured gap and beats 4:8 on all 30 model–scorer pairs.

We have not yet shown end-to-end sparse GEMM or LLM inference throughput. Because OP's data movement matches 2:4 and its selector fits within the cycle, we expect throughput to match 2:4, apart from the cross-lane inputs above.

W2 / Q2: Optimized N:M baselines and the 0.05% estimate. The reviewer is correct that ordered selectors for 4:8 and 8:16 need only 5 and 9 candidates (this is not the current 2:4 design). Even with this optimization, 4:8 and 8:16 still need 3 and 4 index bits per kept weight (⌈log₂5⌉ and ⌈log₂9⌉). That means 96 and 128 metadata bits per 1×64 tile, compared with 64 for 2:4 and OP. Fan-in is also still 5 and 9 rather than 4. OP's advantage over N:M therefore does not depend on the unoptimized baseline.

Below is the hardware efficiency data for the 5 and 9 variants with the ASAP7 7nm process. While these variants reduce the hardware cost of 4:8 and 8:16 sparsity, they require 3 and 4 index bits which would impact the rest of the sparse matrix multiplication data path. Additionally, the 5 variant for 4:8 results in significant increase in worst-case delay (1.42 - 1.60x) and the 9 variant for 8:16 results in significant increase in all hardware metrics (1.18 - 2.13x).

Table R7: 1×64 normalized hardware data with 5 and 9 variants

Prec.DesignA_cellA_coreDelayPower
16-bit8:16 (9 variant)1.741.691.491.18
4:8 (5 variant)1.031.091.420.95
8:163.272.591.642.26
4:81.981.581.711.76
OP-F2D0.981.001.040.97
OP-1D0.971.000.960.97
2:4 (control)1.001.001.001.00
8-bit8:16 (9 variant)2.131.591.861.33
4:8 (5 variant)1.161.001.601.01
8:163.502.162.022.47
4:82.091.282.031.84
OP-F2D0.991.000.970.96
OP-1D1.011.000.981.01
2:4 (control)1.001.001.001.00
4-bit8:16 (9 variant)2.011.401.901.38
4:8 (5 variant)1.211.001.451.00
8:163.672.712.082.92
4:81.221.401.872.17
OP-F2D0.991.000.991.02
OP-1D0.991.000.940.99
2:4 (control)1.001.001.001.00

We agree that the 0.05% per-MAC figure was an order-of-magnitude estimate. We have replaced it with direct 7nm measurements: OP is within 0.97–1.01× of 2:4's cell area and 0.96–1.02× of its power. We present these as selector-level results, not complete-MAC measurements.

AuthorsOpenReview ↗

[Response to Reviewer ZxUp 2/2]

Q3: Combining OP with SparseGPT and ALPS, and the search space compared with MaskLLM.

Results: We ran SparseGPT and ADMM-Grad (General Response, Table R1). At tile 64, OP-F2D recovers 80.3% of the 2:4-to-unstructured gap under both methods.

ALPS/ADMM: ALPS uses ADMM to jointly optimize mask selection and error compensation. In their weight update, a projection operator is used to project an input matrix onto the set of feasible masks for the given sparsity pattern. The paper suggests that extending compatibility to N:M sparsity (and other patterns) requires only replacing this projection to the respective feasible mask set. To apply OP, we can similarly replace it with the feasible mask set of OP. This requires a mask calculation on every update step.

SparseGPT changes: OP is complementary to SparseGPT because it modifies mask constraints without changing the underlying reconstruction procedure.

  • Adaptive mask selection: Replace SparseGPT’s independent 2:4 mask selection with OP’s joint, exact mask selection per tile, using SparseGPT’s current Hessian-aware scores. Subsequent tile masks are selected after error compensation, preserving adaptivity.
  • Error compensation: Once an OP-valid mask is selected, SparseGPT’s existing Hessian-based weight updates can proceed without changing the reconstruction equations, while preserving the fixed mask.
  • Channel reordering: Weights and Hessian information must be reordered consistently, while OP constraints must remain valid in the final inference layout. As shown in Appendix N, OP-aware permutations can reduce selector contention, whereas structure-unaware permutations may degrade quality.

Does weight reconstruction reduce OP's gains? In absolute terms, reconstruction shrinks the 2:4 perplexity gap. In relative terms, OP-F2D still recovers 80.3% of the gap under both SparseGPT and ADMM-Grad. Reconstruction and a larger set of allowed masks therefore add up.

Search space vs MaskLLM. OP and MaskLLM differ along two separate axes.

(i) Feasible space. MaskLLM chooses masks within the 2:4 family, where each disjoint group of four weights selects one of 6 masks. This gives 6^(N/4) full-model masks for N weights. OP enlarges the family itself. OP-F2D admits ~8,500× more masks per 64-weight tile (Table R6). Because tiles are independent, this advantage compounds to roughly (8.5×10^3)^(N/64) over the full model with OP-F2D.

(ii) Objective. MaskLLM optimizes the end-to-end language-modeling loss, so its choice in each group accounts for interactions with all other groups in the model. Its feasible set factors over groups, but its objective does not. OP instead maximizes a fixed, additive saliency score. Overlapping windows couple decisions within a tile only through feasibility, and the greedy matroid pass returns the exact optimum of this objective.

MaskLLM therefore performs approximate search with a global objective over a restricted space. OP performs exact search with a local objective over a much larger space. OP-F2D matches or improves on MaskLLM (6.70 vs. 6.72 on Llama 2 7B; 5.74 vs. 5.85 on Llama 2 13B) without mask training. This suggests the 2:4 feasible set limits quality even when masks are trained end to end. The two approaches are complementary: MaskLLM-style end-to-end mask learning could be applied over the OP family. We leave this to future work.

Table R6: Comparison of feasible mask space sizes at 50% sparsity (L=64).

MethodFeasible masksRelative to MaskLLM
MaskLLM (2:4)2.82×10^121x
OP-1D (1x64)1.38×10^15~490x
OP-F2D (1x64)2.41×10^16~8,500x

Q4: Extension to activation sparsity.

Yes. OP changes only which inputs each selector is wired to, so it works for whichever operand is sparse. For activation sparsity, each token's activation vector is tiled along the reduction dimension. It is compressed to L/2 kept values with 2-bit indices, and the weights become the dense input to the selection network (Figure 1e with the operands swapped). The index format, selector wiring and dataflow stay the same, so the costs in Table 2 apply directly.

What changes is that scoring, mask selection and encoding run online for every token. The run-time optimized solver for 1x16 tiles was designed for this setting (Appendix G.4). At L=16, the 8 KiB lookup table generates OP-F2D masks for a 4096×4096 tensor in 2.49 ms (Table 9) and allocates only the output mask (Table 10). Each tile costs a fixed amount, so mask generation grows only with the number of activations.

Gains should appear in compute-bound settings not in memory-bound decoding, because the dense weights are still fetched. We will add this discussion to the paper and leave the full evaluation to future work.

Reviewer_ZxUpOpenReview ↗

Post-rebuttal decision

After reading the author's rebuttal, I decide to keep my score unchanged.

Thank you for reading our rebuttal and for your continued support of our paper. We appreciate your constructive feedback, which helped us improve the manuscript.

Reviewer_QNE7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

Semi-structured 2:4 sparsity is the standard fine-grained pattern supported by current GPUs, but because every block of four weights must keep exactly two, it causes much larger quality loss in LLMs than unstructured 50% pruning. The paper proposes Overlapping Pruning (OP), which keeps the selector count, 4:1 mux, and 2-bit index and only rewires which positions each selector can see, so that neighboring selector windows overlap. This enlarges the set of feasible masks. The authors present three forms step by step: OP-1D (overlap along a row), OP-2D (row and column selectors, transposable masks), and OP-F2D (OP-2D connectivity flattened into one row). They also show that feasible masks form a matroid, so a simple greedy pass gives the optimal mask. Experiments on ten LLMs (6.7B–32B) with Wanda, Wanda++, and RIA show that OP-F2D recovers 75.8–88.8% of the gap between 2:4 and unstructured 50%, reaching quality similar to 8:16.

Critical strengths
  1. The core idea is simple and appealing. The observation that one input of each 2:4 selector is unused, and can instead be wired to a neighboring block, is easy to understand and changes only the wiring while keeping the 2:4 primitives. This matters to my assessment because it suggests a low-cost change to future sparse hardware, rather than a new format with a large overhead.

  2. The quality gain is shown consistently across many settings. The improvement 2:4 → OP-1D → OP-F2D holds for all ten models and all three scoring methods, and OP-F2D matches or improves on learned-mask and factorized 2:4 methods (MaskLLM, ARMOR) without mask training. This consistency makes it reasonable to think that the gain comes from the mask structure itself, not from a specific model or task.

Critical weaknesses

1. Overall, the paper is hard to follow. Several figures and passages are imprecise or under-explained.

  • (A) Figure 2 and Table 4 show the same data (the text says Table 4 “reports the absolute perplexities behind Figure 2”). Given the 9-page limit, this space would have been better used to explain the hardware difference between OP and 8:16 in the main text, which would make the main claim easier to follow.
  • (B) Figure 1 does not help much in understanding the method. In Figure 1(d), the highlighted cells in the OP-2D tile are not explained, so it is hard to see how OP-F2D works. Figure 1(e) is captioned “OP-1D and OP-F2D require only selector rewiring compared with 2:4,” but the figure itself does not show this rewiring; showing the 2:4 and OP-F2D wiring side by side would make the point much clearer. The figure also mixes notation (“M Activations (L = 8)”).
  • (C) Table 2 lacks key information. The caption does not define “Prec.”, “A_cell”, or “A_core”, does not state the process node, and the 2:4 baseline row is omitted from the table (it is only mentioned in the text as 1.00). The main text never mentions that the results are at 130 nm (only in Appendix H), while the only node it mentions is 15 nm, taken from another work, so Table 2 is easy to misread.

2. The hardware comparison with 8:16 is not clear enough to support the main claim. The main claim is “quality similar to 8:16 at the hardware cost of 2:4.” However, the hardware cost is evaluated only for the selector block. It would help to include some comparison at the system level, such as a full sparse MAC unit, a simulation of the whole accelerator, or kernel speed, against 2:4 or 8:16. The paper instead estimates the cost per MAC, but this estimate combines two different processes: the selector’s share of a MAC comes from a 15 nm design in another paper, while the OP overhead is measured at 130 nm. In modern processes, wiring may take a larger share of area and power, and since OP mainly changes the wiring, its overhead over 2:4 might be different from what is reported. Therefore, it is hard to tell how much of the selector-level advantage would remain in a real accelerator.

Questions for authors
  1. How is OP different from methods that run flexible sparse patterns on current GPUs, such as VENOM, NM-SpMM, and SlideSparse (ICML 2026)?
  2. Can OP masks run on current GPUs using a software kernel like VENOM, NM-SpMM methods? If so, how fast is it compared to dense 2:4, and 8:16?
  3. Could you clarify what A_cell and A_core mean in Table 2, and which process node was used for these results?
3 comments
AuthorsOpenReview ↗

[Response to Reviewer QNE7 1/2]

We thank the reviewer for the careful reading. We appreciate that they see OP as a low-cost wiring change to future sparse hardware rather than a new high-overhead format, and that the gains hold across models and scorers. We agree that the presentation could be revised to improve clarity as described below.

W1: Presentation.

(i) Figure 2 and Table 4 repeat the same data. While we agree the data is repeated, we believe the two serve different purposes. Figure 2 visually shows that OP-1D performs better than 4:8 and OP-F2d performs similarly to 8:16. For clarity, we were only able to fit 5 models in Figure 2. Table 4 shows all evaluated models' perplexity data across multiple saliency scores. We intend to keep both for clarity.

(ii) Figure 1. We will revise it as described below:

  • Fig 1d: Recreating a more intuitive figure to show the OP-2D to OP-F2D mapping.
  • Fig 1e: Fixed the typo of "M Activations" to "L Activations"

(iii) Q3: Table 2. We will make Table 2 self-contained:

  • Prec. = operand bit width (precision)
  • A_cell = total standard-cell area after synthesis (logic only).
  • A_core = core area of the placed-and-routed layout (logic + wires).

All values are normalized to 2:4. We plan to add an explicit 1.00 row for 2:4.

The original results used the 130nm SkyWater sky130 PDK. Only Appendix H stated this, which we agree made the table easy to misread. Table 2 will now report new 7nm results (ASAP7, OpenROAD) with the node in the caption. The sky130 results move to the appendix.

W2: Hardware comparison with 8:16, and wiring cost at modern nodes. We have addressed the reviewer's specific concern, that wiring may cost more at modern nodes, by re-running synthesis, placement and routing at 7nm (General Response, Hardware (i)). OP's overhead does not grow. OP-1D and OP-F2D stay within 0.94–1.04× of 2:4 in cell area, delay and power at 4, 8 and 16 bits. This is tighter than our sky130 results. We have removed the per-MAC estimate that combined 15nm and 130nm numbers.

Against 8:16 at the same node:

  • Standard 8:16 (16:1 selectors): 3.27× cell area, 1.64× delay and 2.26× power at 16 bits.
  • Optimized 8:16 (9:1 selectors, suggested by Reviewer ZxUp): data to be provided.
  • Metadata: Both standard and optimized 8:16 need 4 index bits per kept weight, or 128 metadata bits per 1×64 tile. OP needs 64, the same as 2:4.

Quality stays close to 8:16. With ADMM-Grad at tile 256, OP-F2D beats 8:16 on all five models. With SparseGPT, it is within 0.18 perplexity of 8:16 on every model (Table R1).

Beyond the selector itself (General Response, Hardware (ii)–(iii)):

  • OP uses exactly 2:4's compressed format, so weight, metadata and activation fetches are the same as 2:4.
  • The 16-bit OP-F2D selector path is about 23% of an A100 clock cycle.
  • When a tile's activations are split across lanes, only 6 (OP-1D) or 16 (OP-F2D) of 128 selector inputs cross lanes, all to an adjacent lane.

We have not yet run a full-accelerator simulation. However, because data movement and MAC count match 2:4, the only system-level difference we expect is the small number of cross-lane inputs above.

AuthorsOpenReview ↗

[Response to Reviewer QNE7 2/2]

Q1: How OP differs from VENOM, NM-SpMM and SlideSparse? These methods and OP work on different sides of the same problem. VENOM, NM-SpMM and SlideSparse are software methods that run sparsity ratios other than 2:4's fixed 50% on existing GPUs.

  • VENOM keeps 4 of every M columns in each V×M block and applies 2:4 within them. This runs 2:M patterns (sparsity of 1-2/M), which are sparser than 50%, on 2:4 Sparse Tensor Cores (SPTCs).
  • SlideSparse goes the other way. It splits each (2N−2):2N block, which is less sparse than 50% (e.g., 6:8 at sparsity of 25%), into N−1 overlapping 2:4 windows and duplicates activations to match. This stretches the reduction dimension by a factor of 2 − 2/N, and in return it runs on SPTCs at up to N/(N−1)× dense speed (1.33× at 6:8).
  • NM-SpMM runs vector-wise N:M kernels on FP32 CUDA cores. It gives up SPTC throughput in exchange for any vector-wise N:M ratio.

All three treat today's hardware as fixed and change the sparsity ratio. OP instead fixes the ratio at 50% and asks what small hardware change can bring mask quality close to unstructured pruning. Its answer is overlapping selector windows that keep 2:4's selector count, four-input selectors, 2-bit index and datapath.

SlideSparse also uses overlapping windows, but its overlap is in software. It duplicates data so that each window is still a separate 2:4 group in hardware. OP's overlap is in the selector wiring and adds no data expansion.

Q2: Can OP masks run on current GPUs using a software kernel like VENOM, NM-SpMM methods?

Based on our understanding, NM-SpMM could be used to run OP patterns by setting the vector length to 1. However, it will need to treat OP patterns with tile size 1x64 as 32:64 patterns requiring 6 index bits for each weight. This requirement would lead to the loss of performance. Additionally, NM-SpMM only works with FP32 data.

Note: The approaches are therefore orthogonal: the software methods add ratio flexibility, and OP adds mask flexibility at a fixed ratio. They don't substitute for each other on current GPUs, because OP masks generally violate 2:4 and can't get SPTC speedup through VENOM or SlideSparse. On hardware that supports OP, they could be combined:

  • VENOM's column selection could keep L of every M columns before OP picks masks within them, giving (L/2):M patterns. Detailed analysis left for future work.

  • SlideSparse also uses overlapping windows, but it targets lower sparsity (e.g., 6:8, or 25%). It breaks weights into overlapping 2:4-compliant blocks that run on existing SPTCs. OP targets mask rigidity at a fixed 50% sparsity by changing selector connectivity. SlideSparse thus improves hardware utilization at lower sparsity, while OP improves mask flexibility through a hardware change. Combining them: If OP replaces 2:4 SPTC hardware, SlideSparse's decomposition can use OP-F2D blocks. For example, three overlapping OP-F2D (1×16) blocks keep 24 weights across 32 positions (25% sparsity). This expands computation by 1.5× while keeping OP's 2× native sparse throughput. The theoretical speedup over dense is 1.33×, the same ratio as SlideSparse 6:8 on 2:4 hardware. We evaluate this on Llama 3.1 8B (Table R5).

Table R5: WikiText-2 perplexity (↓) on Llama 3.1 8B at 25% weight sparsity, comparing OP-F2D with structured 6:8 pruning.

Sparsity TypeSparsity RatioPerplexity
Dense0%6.24
Unstructured 25%25%6.55
OP-F2D 24:3225%6.77
SlideSparse 6:825%6.86

At 25% sparsity, OP-F2D (24:32) has 0.09 (1.3%) lower perplexity than SlideSparse 6:8, recovering 29.2% of the gap to unstructured pruning. OP's mask flexibility thus complements SlideSparse's decomposition: it improves quality at lower sparsity while keeping the potential speedup of sparse execution. Actual speedup on OP hardware remains to be evaluated.

Reviewer_QNE7OpenReview ↗

Response to rebuttal

Thank you for your patience while waiting for my response, and for the well-organized rebuttal and revised paper. The new 7nm results, the clearer Table 2 and Figure 1(d), and the new related work section address some of my concerns, so I have raised my score to 3. I do not raise it further because the hardware evaluation is still limited, and the paper is still hard to read in places.

Reviewer_cV2bOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper intoduces a new sparsity scheme that's a superset of 2:4 (or other N:M) sparsity patterns and can reach closer accuracy to unstructured sparsity. It proposes Overlapping Pruning (OP), which keeps 2:4’s 4:1 selectors, selector count, and 2-bit index, but rewires selectors so their candidate windows overlap with their neighbors. It has a 1D, 2D (transposable) and 2D flattened version. Mask selection becomes a coupled problem within each tile. The authors show that feasible masks are the independent sets of a transversal matroid, so a saliency-sorted greedy pass is exact

Critical strengths
  1. The idea is very simple and well motivated. The observation that each 2:4 selector has a redundant input, and that the input can be rewired to a neighbor at no index cost, is clean and easy to follow.

  2. The transversal-matroid argument makes greedy provably optimal for every OP pattern. The union-find oracle makes it cheap.

  3. The quality gains are significant and broad.

Critical weaknesses
  1. The synthesized block assumes all 64 activations of a tile are available to every selector. The paper calls whether real hardware exposes them this way out of scope, and lists the “lane-partitioned operand collector” as future work.

  2. The work cannot be supported on existing hardware.

  3. For one-shot baselines, ADMM [1] and SparseGPT [2] are missing, which are usually stronger variants than Wanda.

  4. The models tested in this paper are old and the behavior of more recent models have changed significantly with sparsity. The newer models are usually harder to prune. I do not expect the authors to evaluate newer models though as the pruning method is not their core contribution.


[1] Boza, Fast and Effective Weight Update for Pruned Large Language Models, TMLR 2024

[2] Frantar and Alistarh, SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot, ICML 2023

Questions for authors
  1. How does your method perform in comparison to ADMM and SparseGPT?

  2. There seems to be a tie between the propsoed method and learnable methods like MaskLLM (which is purely 2:4). Is it possible to extend your approach to a learnable scheme like MaskLLM to see better results? I understand that learnable methods are very expensive and I'm not expecting you to run your method on a learnable framework. But a detailed extension and algorithm to support it would strengthen the work.

4 comments
AuthorsOpenReview ↗

[Response to Reviewer cV2b 1/2]

We thank the reviewer for the positive assessment of OP's simplicity, the matroid-based greedy optimality, and the breadth of the quality gains. We address each concern below.

W1: All 64 activations are assumed to be available to every selector. We now measure this cost directly instead of leaving it as future work (General Response, Hardware (iii)). When a tile's activations are split across lanes, OP-1D needs only 6 of 128 selector inputs to cross lanes (four 16-wide lanes). OP-F2D needs 16 of 128. All of these are nearest-neighbor. For designs that cannot stage a full 1×64 tile, OP-F2D with a 1×16 tile needs only 16 activations at a time. It still closes 77.3% (Wanda) and 73.6% (Wanda++) of the 2:4-to-unstructured gap, and it beats 4:8 on all 30 model–scorer pairs.

W2: Not supported on existing hardware. We agree that OP needs a change to how the selectors are wired, just as 2:4 needed Ampere's sparse tensor cores. However, everything else stays the same (General Response (i)–(ii)):

  • the compressed format (L/2 values plus 2-bit indices)
  • metadata size
  • register-file and operand-collector traffic
  • MAC count

At 7nm (ASAP7), OP's area, delay, and power are within 0.94–1.04× of 2:4. Its selector path takes about 23% of an A100 cycle. OP is therefore a low-cost change to the selector stage, not a new datapath. In contrast, 4:8 and 8:16 need wider indices and larger selectors.

W3 / Q1: Comparison with ADMM and SparseGPT. We added both methods (General Response, Table R1). The ordering 2:4 > OP-1D > OP-F2D (perplexity, lower is better) holds on all five models and all three tile sizes. At tile 64, OP-F2D reduces perplexity relative to 2:4 by 26.3% (SparseGPT) and 23.8% (ADMM-Grad) on average. In both cases, it recovers 80.3% of the gap to unstructured sparsity. With ADMM-Grad at tile 256, OP-F2D beats 8:16 on all five models while keeping 2:4's selector hardware.

W4: Newer models are harder to prune. We agree that pruning sensitivity can vary across model generations. To address this concern, we extended our evaluation to three recent models released in 2026: Qwen 3.5 9B, Qwen 3.8 27B, and Gemma 4 12B. We evaluated each model using SparseGPT, Wanda, and Wanda++ at 50% sparsity.

Table R3: Recently released models included in the additional evaluation.

ModelRelease date
Qwen 3.5 9BMarch 2, 2026.
Qwen 3.8 27BAugust 14, 2026.
Gemma 4 12BJune 3, 2026.

Table R4: WikiText-2 perplexity (↓) at 50% sparsity on recent models. Dense perplexity is shown in parentheses. OP-1D and OP-F2D retain the hardware and index cost of 2:4. Gap closed (↑) measures the fraction of the 2:4-to-unstructured perplexity gap recovered by OP-F2D.

Model (Dense)MethodUnstr. 50%8:164:82:4OP-1D (64)OP-F2D (1x64)∆Rel (↓)Gap closed (↑)
Qwen 3.5 9B (8.65)SparseGPT9.3910.1411.0513.8811.0510.289.5%80.2%
Wanda9.5310.4111.6516.0011.4310.429.4%86.2%
Wanda++9.4210.0310.8013.2410.6410.137.6%81.4%
Qwen 3.8 27B (7.05)SparseGPT7.818.108.549.578.518.174.6%79.5%
Wanda7.848.238.719.968.608.255.2%80.7%
Wanda++7.828.218.6910.008.578.255.5%80.3%
Gemma 4 12B (6.92)SparseGPT8.749.8611.4515.6411.409.9613.9%82.3%
Wanda9.7911.6314.6824.6513.5412.0823.4%84.6%
Wanda++8.829.5810.7514.0310.579.629.0%84.7%

The additional results confirm that the same trend holds across all nine model–method combinations: 2:4 → OP-1D → OP-F2D consistently reduces perplexity. OP-1D matches or outperforms 4:8, while OP-F2D achieves perplexity close to 8:16, without requiring its higher hardware and index cost. Across these recent models, OP-F2D reduces perplexity by 14.6–51.0% relative to 2:4, recovering 79.5–86.2% of the gap to unstructured pruning. The improvements remain consistent across SparseGPT, Wanda, and Wanda++, despite differences in pruning sensitivity. These results further support our central claim: the benefits of OP come from a more flexible sparsity pattern and generalize across newer model architectures and different pruning methods, while preserving the hardware efficiency of 2:4.

AuthorsOpenReview ↗

[Response to Reviewer cV2b 2/2]

Q2: Extension to learnable schemes such as MaskLLM.

We agree that OP and MaskLLM can complement each other. The main challenge is that OP offers greater flexibility and many more valid masks over the same number of weights. For a 1x64 tile, OP-F2D supports 2.41x10^16 distinct valid masks (Table 1). MaskLLM learns a distribution over six valid masks per independent 2:4 group. Applying this approach directly to an entire OP tile would require an impractically large distribution. One possible extension could avoid enumerating all valid OP masks as follows.

Selector-level learning. Each 4:1 OP selector could learn a categorical distribution over its four candidate weights, analogous to MaskLLM's six-way distribution. A 1x64 tile contains 32 selectors, requiring only 128 trainable logits. The original dense weights would remain frozen, and the logits could be initialized randomly or from an existing OP mask.

Differentiable selection and training. Unlike independent 2:4 groups, 4:1 OP selectors share candidate weights, so independent decisions may select the same weight multiple times. To prevent this, Gumbel-Sinkhorn matching [1] and entropy-regularized optimal transport [2] could be adapted to enforce OP's assignment constraints. Each selector would assign one unit of probability mass across its four candidates, while each weight would receive at most one unit across all selectors. An auxiliary row would absorb the remaining unassigned mass. Masked Sinkhorn iterations could produce differentiable soft assignments that approximately satisfy these constraints. The selector logits could be trained using MaskLLM's language-model loss and retained-weight regularizer.

Final mask and guarantees. After training, Gumbel noise would be removed, and OP's exact assignment solver would use the learned selector-specific scores to generate the final mask. The solver would maximize the total learned score while enforcing one selection per selector and at most one selection per weight. If multiple selectors prefer the same weight, the solver would resolve the conflict by finding the highest-scoring valid assignment. This guarantees a valid binary OP mask with exactly 50% sparsity and preserves the existing OP inference format. Every valid OP mask remains representable, although training is not guaranteed to find the lowest-loss mask. The accuracy gains and training cost of this extension remain to be evaluated experimentally.

[1] Mena et al. Learning Latent Permutations with Gumbel-Sinkhorn Networks. ICLR, 2018.

[2] Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. NeurIPS, 2013.

I thank the authors for their lengthy and thorough rebuttal. All my concerns are addressed and I've raised my score.

Also, I encourage the authors to include the additions in the final version of their manuscript.

We sincerely thank the reviewer for their thoughtful consideration and constructive feedback. We will incorporate the suggested additions into the final manuscript.

Comments on the paper

3 comments
AuthorsOpenReview ↗

[General response 1/3] High-Level Summary

We thank the reviewers for their thoughtful and constructive feedback. We are encouraged that reviewers found OP's core idea simple and well motivated (Reviewers cV2b, QNE7, ZxUp), appreciated the theoretical grounding of the mask selection (Reviewers cV2b, ZxUp), and noted the consistency of the quality improvements (Reviewers cV2b, QNE7, ZxUp). To briefly restate the contribution: OP reuses 2:4's redundant selector input to overlap neighboring windows, with no change to selector count, fan-in, or the 2-bit index of 2:4 sparsity. The matroid structure makes greedy mask selection provably exact at near-2:4 cost, and across ten models, three scorers, and quantization, OP-F2D closes 75.8–88.8% of the 2:4-to-unstructured gap. Below, we address the two concerns shared across reviews:

  1. Expanding the scope of the hardware evaluation.
  2. Additional error compensating baselines (SparseGPT and ADMM-Grad).

In the individual reviewer threads, we also address:

  1. Comparison of Overlapping Pruning with prior work (VENOM, NM-SPMM, SlideSparse).
  2. Compatibility of Overlapping Pruning with learned-mask methods (MaskLLM).
  3. Updated figures for clarity.

An updated pdf of the paper has been uploaded to address reviewer questions.

AuthorsOpenReview ↗

[General response 2/3] Hardware Evaluation

Hardware efficiency. We extend the evaluation in three ways.

(i) Modern node. We re-ran synthesis, placement, and routing using the ASAP7 7nm PDK with OpenROAD. At 4, 8, and 16 bits and the same core area as 2:4, OP-1D and OP-F2D are within 0.97–1.01× of 2:4's cell area, 0.94–1.04× of its delay, and 0.96–1.02× of its power. OP's wiring overhead therefore does not grow at a modern node; it is actually smaller than in our sky130 results. For these new results, we used Nvidia's A100 1.41 GHz clock (710 ps cycle). At that clock, the 16-bit OP-F2D selector path is 165.53 ps (2:4: 158.67 ps), about 23% of the cycle. It fits well within an existing pipeline stage without lowering frequency. OP keeps 2:4's MAC count and bytes per output, so its throughput matches 2:4. Because OP's hardware cost now matches 2:4, we will remove the 15nm selector-overhead estimate (Gondimalla et al., 2023).

Table R0: ASAP7 results, normalized to 2:4 (= 1.00). A_cell is the logic only area, and A_core is area of both logic and wiring.

PrecisionDesignA_cellA_coreDelayPower
16-bit8:163.272.591.642.26
4:81.981.581.711.76
OP-F2D0.981.001.040.97
OP-1D0.971.000.960.97
8-bit8:163.502.341.872.35
4:82.091.272.241.89
OP-F2D0.991.000.970.96
OP-1D1.011.000.981.01
4-bit8:163.672.712.082.92
4:82.321.401.872.17
OP-F2D0.991.000.991.02
OP-1D0.991.000.940.99

(ii) Data format. OP stores the same compressed format as 2:4: L/2 values plus a 2-bit index per kept weight, or 64 metadata bits per 1×64 tile. Only the meaning of each index changes. In 2:4, the index selects a position within a 4-wide window shared by two selectors. In OP, value slot k always feeds selector k, and its index selects one of that selector's four hardwired ports. Values and metadata are encoded offline at pruning time. An OP sparse MMA therefore fetches the same weight fragment, metadata, and dense activation slice as 2:4 (all k64 activations for FP8 m16n8k64), so register-file and operand-collector traffic are unchanged.

In contrast, 4:8 and 8:16 need 3 and 4 index bits per kept weight, even with the 5:1/9:1 selectors suggested by ZxUp. Evaluating their performance may require implementation changes and cycle-accurate simulation; OP needs neither.

(iii) Activation routing. OP changes only which staged activation each selector reads. If a tile's 64 activations are split across lanes, some selector inputs read activations held by another lane. We call these cross-lane inputs; 2:4 has none.

OP-1D, with lanes of width w (a multiple of 4): only the two selectors at each lane boundary have a cross-lane input, each reading one activation from the adjacent lane. That is 2(64/w − 1) of 128 selector inputs per tile (6 for four 16-wide lanes), all nearest-neighbor.

OP-F2D, with one 4×4 quadrant of the 8×8 tile per lane: only the 8 boundary-band selectors (Appendix D) have cross-lane inputs, each reading 2 activations from an adjacent quadrant. That is 16 of 128 inputs, all nearest-neighbor. Splits that ignore the fold cost more.

We consider this inter-lane traffic the most likely bottleneck for OP-F2D. Designs that cannot deliver a full 1×64 tile's activations at once can use a 1×16 tile (the flattened 4×4 OP-2D tile). Windows never cross tile boundaries, so this needs only 16 activations at a time. At 1×16, OP-F2D still closes 77.3% (Wanda) and 73.6% (Wanda++) of the 2:4-to-unstructured gap on average (minimum 70.9%). It also has lower perplexity than 4:8 on all 30 model–scorer pairs. Tile size thus trades inter-lane traffic for mask quality, using the same 2:4 selector primitives.

AuthorsOpenReview ↗

[General response 3/3] Additional Baselines

Additional error compensating baselines (SparseGPT and ADMM-Grad)

Comparison with SparseGPT. The results confirm the same consistent perplexity ordering observed with Wanda, RIA, and Wanda++: 2:4 > OP-1D > OP-F2D (lower is better). This ordering holds across all five evaluated models and all three tile sizes (16, 64, and 256). At tile size 64, OP-F2D reduces perplexity relative to conventional 2:4 SparseGPT by 26.3% on average, recovering 80.3% of the 2:4-to-unstructured perplexity gap. For example, perplexity decreases from 11.09 > 8.47 > 7.90 on Llama 1 7B and 15.85 > 12.21 > 10.83 on Llama 3.1 8B when moving from 2:4 to OP-1D to OP-F2D.

Table R1: WikiText-2 perplexity (↓) under SparseGPT at 50% weight sparsity for five representative models.

Sparsity TypeLlama1-7BLlama2-7BLlama3.1-8BQwen2.5-7BGemma2-9B
Dense5.685.476.246.856.84
Unstructured (50%)7.207.009.328.428.50
Structured 2:411.0910.9715.8511.1811.55
Structured 4:88.728.5011.989.539.80
Structured 8:167.857.6410.458.889.11
Overlapping 1D (16)8.728.4212.769.839.97
Overlapping 1D (64)8.478.1712.219.609.77
Overlapping 1D (256)8.378.1112.059.549.73
Overlapping F2D (1x16)8.047.7711.329.279.45
Overlapping F2D (1x64)7.907.5110.839.019.20
Overlapping F2D (1x256)7.767.4310.638.999.17

Comparison with ADMM-Grad. We observe the same consistent ordering, 2:4 > OP-1D > OP-F2D, across all five evaluated models and three tile sizes. At tile size 64, OP-F2D reduces perplexity relative to conventional 2:4 ADMM-Grad by 23.8% on average, recovering 80.3% of the 2:4-to-unstructured perplexity gap. Furthermore, at tile size 256, OP-F2D achieves lower perplexity than conventional 8:16 across all five models while retaining 2:4's selector count, fan-in, and index width.

Table R2: WikiText-2 perplexity (↓) under ADMM at 50% weight sparsity for five representative models.

Sparsity TypeLlama1-7BLlama2-7BLlama3.1-8BQwen2.5-7BGemma2-9B
Dense5.685.476.246.856.84
Unstructured (50%)7.116.838.948.248.27
Structured 2:49.9010.1314.6710.7210.98
Structured 4:88.188.1111.439.329.41
Structured 8:167.627.3910.078.768.82
Overlapping 1D (16)8.328.1411.479.339.53
Overlapping 1D (64)8.147.9211.049.159.33
Overlapping 1D (256)8.077.8711.009.149.28
Overlapping F2D (1x16)7.797.5910.498.939.02
Overlapping F2D (1x64)7.757.3410.058.758.81
Overlapping F2D (1x256)7.577.309.998.698.77

These results further demonstrate that OP's quality improvements are consistent across different pruning methods, including SparseGPT and ADMM-Grad. We will update Table 4 and the corresponding tables in Appendix K of the revised manuscript to include the complete SparseGPT results across the evaluated models and tile sizes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.