What Sets a Block-Sparse Training Speedup? SM Count, the Dense Baseline and the Kernel, on Six GPUs

Dynamic sparse training (DST) rarely delivers wall-clock speedups, because standard implementations multiply dense weights by masks and still execute dense products. We ask what sets the speedup once the sparsity is actually executed. Our instrument is BlockDST, which keeps a constant number of live blocks in every block-row so that our own load-balanced blocked-ELL kernels run all three per-layer products sparse, including a block sampled dense–dense matrix multiplication (block-SDDMM) for the weight gradient. On six NVIDIA graphics processing units (GPUs), the same kernel and shape run $1.08$-$5.13\times$ faster than a dense General Matrix Multiply (GEMM) at 95% sparsity, and most of this spread comes from how fast the dense baseline runs on each card. Our kernel's throughput scales with streaming multiprocessor (SM) count $\times$ clock, so a one-parameter model predicts it on a held-out card to within $0.91$-$1.19\times$ at the reference shape ($0.78$-$1.66\times$ across shapes). That scaling is not a property of block sparsity in general: cuSPARSE's Blocked-ELL kernel, timed on five of the same cards, does not follow it. It runs $1.5\times$ faster on an A100 than on an A40, while ours runs $0.95\times$ as fast, yet its speedup over dense still varies about $2\times$ from card to card. End to end the picture is narrower: on an L40, BlockDST reaches a fixed accuracy $1.44\times$ faster than dense (not significant with three seeds) with $2.1\times$ less memory, but it is slower than dense on an A40, slower than a dense model with the same number of live parameters, and, on a laptop GPU, $1.2$-$1.4\times$ slower than the same block layout without rewiring. A sparse speedup should therefore be reported together with the kernel, the card's SM count and clock, and the dense throughput it was measured against.