acceptodds
Under review as a conference paper at ICLR 2027

What Sets a Block-Sparse Training Speedup? SM Count, the Dense Baseline and the Kernel, on Six GPUs

Abstract

Dynamic sparse training (DST) rarely delivers wall-clock speedups, because standard implementations multiply dense weights by masks and still execute dense products. We ask what sets the speedup once the sparsity is actually executed. Our instrument is BlockDST, which keeps a constant number of live blocks in every block-row so that our own load-balanced blocked-ELL kernels run all three per-layer products sparse, including a block sampled dense–dense matrix multiplication (block-SDDMM) for the weight gradient. On six NVIDIA graphics processing units (GPUs), the same kernel and shape run - faster than a dense General Matrix Multiply (GEMM) at 95% sparsity, and most of this spread comes from how fast the dense baseline runs on each card. Our kernel's throughput scales with streaming multiprocessor (SM) count clock, so a one-parameter model predicts it on a held-out card to within - at the reference shape (- across shapes). That scaling is not a property of block sparsity in general: cuSPARSE's Blocked-ELL kernel, timed on five of the same cards, does not follow it. It runs faster on an A100 than on an A40, while ours runs as fast, yet its speedup over dense still varies about from card to card. End to end the picture is narrower: on an L40, BlockDST reaches a fixed accuracy faster than dense (not significant with three seeds) with less memory, but it is slower than dense on an A40, slower than a dense model with the same number of live parameters, and, on a laptop GPU, - slower than the same block layout without rewiring. A sparse speedup should therefore be reported together with the kernel, the card's SM count and clock, and the dense throughput it was measured against.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
DST, block sparsity, GPU architecture, hardware-native kernels, issue rate scaling, memory efficiency, load balancing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5JbyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BlockDST, which maintains a fixed number of active blocks per block row and uses custom kernels for sparse training. It evaluates performance across six GPUs, fits an empirical throughput model based on SM count and clock frequency, and examines how kernel speedups translate into training time and memory savings.

Critical strengths
  1. The evaluation covers multiple GPU architectures and distinguishes kernel performance from training performance.

  2. The paper clearly reports limitations and negative results, including cases where the proposed approach is slower than alternative methods.

Critical weaknesses
  1. The paper mainly benchmarks a custom kernel without adequately explaining its design or establishing its novelty in the main text.

  2. Scaling with SM count and clock frequency is expected and, by itself, provides little new insight.

  3. The contribution beyond prior work on sparse DST execution, including DynaDiag [1], HASTE [2], and so on, remains unclear.

[1] Abhishek Tyagi et al. Dynamic Sparse Training of Diagonally Sparse Networks. ICML, 2025.

[2] Nasib Ullah et al. HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces. ICML, 2026.

Questions for authors

Please address the concerns in Critical Weaknesses, particularly kernel novelty and the distinction from prior sparse DST systems.

14 comments

Thank you for the review. While W1 and W3 were in the initial submission, key details were too brief or in the appendix; we have moved both into the main text (Secs. 2 and 3, in blue; L = line in the initial submission; "new" marks material added in the revised PDF; table numbers follow the revised PDF, with initial numbers in parentheses).

W1. On kernel novelty, the first part of your question: what is new is covering all three training products on Blocked-ELL, where every block-row holds exactly k blocks. cuSPARSE's Blocked-ELL supports only the non-transposed forward pass, and its SDDMM does not support Blocked-ELL. Our kernels add the transposed product and a block-SDDMM, so all three training products run on one load-balanced layout with no dense W or ∇W, which cuts peak memory 2.1× below dense (L428–429).

The initial submission described the layout and three products in Sec. 3 (L157–199, Fig. 2) and, in the introduction, what existing block-sparse libraries do not provide for this layout (L45–53). The revised Sec. 3 adds a new kernel-design paragraph:

  • Each block-row stores exactly k blocks in one dense (M × k·b) value array plus block-column IDs, with no row pointers or padding.
  • In the forward kernel, each thread block computes one block-row × 128 batch rows using bf16 tiles in shared memory and fp32 WMMA accumulation, giving every thread block identical work.
  • The input gradient walks a block-column index rebuilt only when the mask changes.
  • The weight gradient is a block-SDDMM that writes ∇W only at live blocks, while rewiring updates IDs in place. (App. B.3 was also corrected from an earlier 64-row-tile description; both use 32 warps per SM.)

W2. Sec. 4 of the initial submission already shows why this scaling, which we do not claim alone as our contribution, is informative here:

  1. A standard roofline predicts the opposite: at 16–30 FLOP/byte (L205) against ridge points of 150–290 (Tables 2 and 5, initially 1 and 4), a roofline classifies the kernel as bandwidth-bound. Yet the bandwidth model misses by up to 3.3× (L264–266), and an A40 with half the A100's bandwidth runs our kernel 11% faster (L270).
  2. Block-sparse kernels on the same layout do not generally scale this way: cuSPARSE's throughput per SM-GHz has a median cross-card CV of 33% vs. 8.7% for ours in the initial submission (L292–295), and 41% vs. 2.8% at matched shapes across six GPUs (new App. B.10).
  3. Decomposing the speedup shows that per SM-GHz, the cuBLAS baseline varies 4.53× across our cards while our kernel varies 1.24× (L84–86), so most of the card dependence of a sparse speedup comes from the dense baseline. For example, the L40 and L40S share a die and run our kernel within 4% of each other, yet its speedup is 5.13× vs. 2.76×, motivating our recommendation to report SM count, clock, and dense throughput alongside any sparse speedup (Sec. 4.3).

W3. On the second part of your question, the distinction from prior sparse DST systems is twofold. Of the methods in the new Table 1, only BlockDST runs all three training products on tensor-core blocks without keeping a dense W / ∇W. And whereas the other GPU methods in that table report speedups on one GPU type, we measure how such a speedup varies across six GPUs (time to accuracy on five, three predicted in advance), trace that variation to the dense baseline, and show when it becomes an end-to-end loss.

The initial submission discussed this prior work in Sec. 2 (L120–128), App. O.1–O.2 (L1724–1754), and App. O.3 / Table 30 (initially 23; L1756–1774, L1783–1792). The new Table 1 brings the closest methods into Sec. 2, comparing sparsity pattern, which training products run sparse, whether dense W / ∇W tensors are kept, and hardware evaluated:

  • HASTE (ICML'26) runs its three products on a single extreme multi-label output layer rather than hidden layers of MLPs, ViTs, and LM FFNs (our benchmarks have no such layer, so we do not run it; App. O.3), gathers feature subsets rather than reading 64×64 tensor-core tiles, and reports speedups on a single A100 without studying cross-GPU variation.
  • DynaDiag (ICML'25) uses diagonal sparsity in forward and backward passes (whether dense W / ∇W are kept is not stated) on one GPU. Because its CUDA kernels are unreleased, we run its pattern as a masked baseline (Table 3, initially 2; Fig. 4) and estimate in App. Q (L1850–1853) that its reported 1.59× speedup would close about 1.1 pp of the 8.6 pp gap at 90%.
  • BLaST (2025) uses a variable number of blocks per column on a GH200, running forward and input gradients sparse while keeping a dense W / ∇W.
  • Most other sparse-execution DST works run on CPUs (truly sparse SET, SparseProp; L1747–1749) or time inference only (SRigL; L1725).
  • STrengthen the kernel analysis by developing a performance model grounded in kernel design and hardware characteristics, supported by profiling results and controlled ablation studies. The goal is to explain which design choices drive performance and why. Kernel studies published at computer architecture venues provide useful examples of this approach.

  • Demonstrate the practical significance of the work. Establishing a performance trend is only a first step; the paper should also clarify why the findings matter and what practical capabilities or opportunities they enable.

Thank you for your suggestions. We updated the revised PDF with new App. S (plus new Table 35 and new Fig. 13) to address both points, with its code and results in the supplement (L = line in the initial submission; "new" marks material added in the revised PDF).

1. Grounding the performance model in kernel design, hardware, profiling, and ablations (new App. S): Controlled ablations on all six GPUs show which design choice drives our kernel's performance and why its throughput tracks SM count × clock: it reads tensor-core fragments from shared memory with an 8-way bank conflict, which standard row padding removes.

Each thread reads its fragments with 32 four-byte loads per live block from tiles with 128-byte rows. A warp-wide load touches 8 rows falling on the same 4 of the 32 banks and is served in 8 passes instead of 1, so fragment loads of a live block occupy the SM's shared memory for 8,192 cycles instead of 1,024, which is 4–16× the tensor-core work at datasheet rates (512 cycles on the A100 to 2,048 on the L40). The input- and weight-gradient kernels issue the same loads (not ablated).

We profile in-kernel, reading the SM cycle counter (clock64) in every thread block at three points per live block, and time forward-kernel variants that change one design choice (or two combined) with bit-identical output at the reference shape (95.3% and 89.8% sparse). Across all six GPUs and both densities (new Table 35 and new Fig. 13 show 89.8%):

  • The fragment-load and MMA phase takes 8,200–9,800 of the 11,600–13,500 cycles per live block (67–76%), never below the 8,192-cycle floor.
  • Padding each tile row by 16 bytes makes every load conflict-free and removes 5,700–6,700 cycles from this phase; a thread block then needs 1.7–2.1× fewer cycles, and 1.9–3.4× fewer with 16-byte staging loads (1.21–2.10× and 1.69–3.45× faster than our released kernel). The A100, whose tensor cores do the most work per SM and clock, gains most; the L40 gains least, as its clock drops from 2.45–2.46 to 1.7–2.0 GHz at its 300 W limit. Other single changes matter less (against a template copy of our kernel: 16-byte staging alone 1.16–1.37×, issuing staging loads together 1.08–1.18×).
  • Before these runs we wrote six predictions from the machine code and a small-shape test (not externally timestamped): five held, while the predicted time gain from padding missed on three of the six cards (higher on the A100 and RTX PRO 6000, lower on the L40).

This identifies the per-SM resource left open in the initial submission (L86, L469–471): shared-memory bandwidth (32 banks of 32 bits per clock on all six GPUs). Two checks tie the cycles to the banks rather than the card: the A40 and laptop RTX A500, running identical machine code on 84 and 16 SMs, both take 9,025–9,034 cycles in this phase, and the A100's phase is within 0.4% of the floor. With the fp32 storage of Sec. 4.1, a live block runs at 72–88 FLOP per SM-cycle, matching the fitted k0 = 0.079 TFLOP/s per SM-GHz, or 79 (0.078 at L279–280). Thus throughput follows SM count × clock rather than DRAM bandwidth or the tensor-core rate cuBLAS depends on. Once the conflict is removed, tensor cores matter more as predicted: on the L40S (the L40's die at twice its tensor-core rate), the padded phase takes 3,020 cycles vs. 4,022 on the L40, close to the 1,024-cycle difference in their tensor-core work. The introduction, Sec. 4.1, and Limitations now state this, and Sec. 4.1 gives the bf16-storage gain as under 1.3× (initially 1.12–1.25×, L272–273).

2. Practical significance and capabilities enabled: First, new App. S shows how to find and remove what limits a kernel one can modify, across GPUs and without profiling counters: with padded rows and 16-byte staging, our 89.8%-sparse forward product is 2.02–5.27× faster than dense GEMM on all six GPUs, up from 0.59–2.70× before, when it was slower than dense on the A100, A40, and RTX PRO 6000 (new Table 35; dense GEMM as in new App. B.10). Because a shared-memory-bound kernel takes about the same cycles per live block on every card while cuBLAS tracks the tensor-core rate, a single-card speedup cannot separate a better method from a better-suited GPU (L305–307).

Second, our model lets a practitioner decide before training whether BlockDST will pay off on a GPU: SM count, clock, and one dense GEMM timing predict our kernel's speedup on a held-out card within 0.91–1.19× at the reference shape (L280–281), and the sparsifiable step share and extra steps to target turn it into a training speedup (L419–422). We tested that second step prospectively twice, with predictions timestamped before the runs (new App. H.2–H.3): 9 of 12 and then all 11 scored predictions held, and both tests foresaw where BlockDST is slower than dense.

If our answers, PDF changes, and new App. S address your concerns, we would be grateful if you would consider them in your final assessment.

Reviewer_2CxKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes using adaptive sparse blocks to improve performance and storage overhead of GPU kernels during and after training. The method works by maintaining a fixed number, k, of sparse blocks throughout the entire process, so dense training time and storage overheads are never inherited into the overall distillation process. The authors introduce a parameterized model to study the performance profile of their implementation and understand the dependency of the performance on card-specific features, such as total SM count. The evaluation supports the authors' claims that the proposed method represents an adequate trade-off between dense processing accuracy and block-sparse increased performance.

Critical strengths
  • By far, the best result was the demonstration that accuracy remains on par with dense processing at high multiple sparsity values, as represented in Figure 4 (a). In comparison with the competing implementation, this accuracy metric is noticeably higher and outperforms the other methods in performance, as illustrated in Figure 4 (c).
  • The authors spend ample time supporting their derivation of the performance model and how it is connected to the specific features of the hardware, and how variations in the problem specification may yield substantial trade-offs with respect to the overall improvements of the method.
Critical weaknesses
  • While interesting, the improvements are not surprising and the block sparse contributions by themselves would probably fall below the threshold for acceptance based on the substantial prior work on modeling sparse kernels on GPUs. Sparse blocks naturally straddle the performance barrier between bandwidth-limited large sparse problems and compute-limited dense problems and the work points out naunces of the barrier and how more knowledge of the underlying architecture coupled with the block size and volume are needed to successfully traverse that barrier.
  • I think the connection with learned block sparsity and how that impacts the accuracy of a given model and parity with dense counterparts mitigates my concerns and that aspect of the paper is truly interesting.
  • I would appreciate a cleaner separation between the data presented in section 4 and section 5. Section four is bogged down with justifiable details of the architectures in question but it feels like an extension of the experimental section because of the volume of data to make the points clear. This is my personal aesthetics opinion though so the authors can take this opinion with a grain of salt.
Questions for authors

I was most concerned with this block of text in section 5.5.

On LLaMA-style (Touvron et al., 2023) decoders of about 411M and 604M dense-equivalent parameters with 95%-sparse feed-forward layers and dense attention (WikiText-103 (Merity et al., 2017), one hour, three seeds), BlockDST trains at 1.10× and 1.16× the dense step rate. Neither is a quality gain over the strongest baselines: SET and RigL are 1.1–1.6 pp more accurate on the ViT, and dense and 2:4 reach 5.4–13.6 lower perplexity on the language models (Appendix R).

Can you clarify what I'm supposed to take away from this commentary? It almost feels like an invalidation of some advancements that were presented in section 5. Is the takeaway that the method is more brittle on a wider class of architectures?

I would also consider reworking Figure 2 (a). The number of empty blocks per row in the left figure shows all 14 rows with ragged block assignment, but in the right, all the blocks are filled. Seems like a pictorial discrepancy.

4 comments

Thank you for the positive review and helpful suggestions. W1–W3 and Q1–Q2 below follow the order of the weaknesses and questions in your review, and all changes are in blue in the revised PDF (L = line in the initial submission; "new" marks material added in the revised PDF; table numbers follow the revised PDF, with initial numbers in parentheses).

Q1. The takeaway is scope rather than brittleness. The initial submission gave the explanation and numbers below in App. H (L1403–1413) and App. R (L1888–1908), and we now summarize them in two new sentences at the end of Sec. 5.5.

Specifically, step speedup is bounded by how much of the step is sparsifiable GEMM. Sparse GEMMs are 93% of a dense MLP step, whereas in our LMs only the FFNs are sparse, and they take only f = 0.34–0.37 of a dense step (timed directly in the new App. H.3; App. R.2 had estimated 0.2–0.3 from the step gains). By Amdahl's law, no kernel can then speed up such a step by more than 1/(1−f) ≈ 1.5–1.6×. From these timings alone, our step model predicted the LMs' step gains in advance to within 4%, including a larger model never trained before (new App. H.3). Training is stable in every run, and the accuracy cost comes from 95% sparsity at this scale rather than the block pattern: on the LMs, dense leads by 5.4–12.3 perplexity, and unstructured SET at the same density pays about the same cost (BlockDST matches SET at d_model 2048 and is within one s.d. at 2560). On the ViT at 90% sparsity with matched epochs, BlockDST trails SET and RigL by 1.1–1.6 pp but matches dense (80.41% vs. 79.80%).

Q2. You are right that the two sides of Fig. 2(a) used different scales. Both now plot each block-row's work relative to the busiest row, which sets kernel runtime: 1/5 to 5/5 on the left (with the idle part hatched) and 3/3 in every row on the right. We updated the figure and caption accordingly.

W1. Our contributions beyond prior modeling of sparse GPU kernels are: (1) measuring a 1.08–5.13× speedup spread for one kernel and shape across six GPUs (Fig. 1, L68–71); (2) showing via decomposition that most of this spread comes from the dense baseline (L84–86), e.g., the L40 and L40S share a die and run our kernel within 4% of each other, yet give speedups of 5.13× vs. 2.76× (Table 2, initially 1); (3) showing that the roofline model misses by up to 3.3× (L264–266), while a second kernel follows neither model (L292–295); (4) linking kernel speedup to time-to-accuracy (Sec. 5.2), now tested prospectively (new App. H.2–H.3; general comment, items 2 and 4); and (5) accuracy parity with dense counterparts (Fig. 4(a)), which your next paragraph finds mitigates these concerns (W2). The new Table 1 in Sec. 2 compares the sparse training methods closest to ours: they report speedups on one GPU type, whereas we measure how the speedup varies across six GPUs and trace that variation to the dense baseline. Sec. 2 also places our one-parameter model within prior work on sparse GPU modeling (L142–149), with SM×clock scaling as our kernel's mechanism rather than a novelty claim. If we missed prior cross-GPU work on block-sparse training kernels, we would appreciate a pointer.

W2. As your Strengths paragraph notes, Fig. 4(a) shows accuracy parity with dense counterparts: with equal wall-clock budgets on the L40, BlockDST is the only sparse method that matches dense accuracy at 90% and 95% sparsity (1.22 and 0.53 pp above dense, neither significant after Holm correction) and is 7.9–13.5 pp above every masked baseline (L345–348). This comparison includes each method's speed, shown in Fig. 4(b): BlockDST trains at 1.03–4.04× the dense step rate on the L40 and A40, whereas the masked baselines multiply a dense weight by a mask and gain no throughput (L321–323, L373–374). Per epoch, RigL is 0.7–2.8 pp more accurate (L350; Table 15, initially 12), as a new sentence in the Fig. 4 caption now states, while on the ViT with matched epochs BlockDST is on par with dense (Q1). Fig. 4(c) shows that the speed advantage depends on the GPU (1.08–5.13× at 95% sparsity; W1): on the three cards of the prospective test, which have smaller kernel speedups, our model predicted in advance that BlockDST would be slower than dense end to end, and the runs confirmed it (general comment, item 2).

W3. We agree. Sec. 4 now opens with a new scope sentence (it covers the kernel alone, while Sec. 5 follows its speedup into training), and the A100/A40 hardware details moved to App. B.9.

AuthorsOpenReview ↗

Follow-up for Reviewer 2CxK

Following up briefly on your W1 (which noted that sparse blocks straddle the barrier between bandwidth-limited sparse and compute-limited dense problems), new App. S in the revised PDF (added in reply to Reviewer 5Jby) shows where our kernel sits on all six GPUs at the reference shape.

Using in-kernel cycle counts and forward-kernel variants that change one design choice (or two combined) with bit-identical output, we find that our kernel's throughput follows neither DRAM bandwidth nor the tensor-core rate, but is bound by shared-memory bank conflicts in its tensor-core fragment loads, a cost fixed in cycles by the tile layout and each SM's shared-memory banks. Five of six predictions we wrote before the cluster runs (not externally timestamped) held, and the measured cycles match the constant fitted in Sec. 4.1 (72–88 vs. 79 FLOP per SM-cycle, both with fp32 storage), providing a mechanism for the hardware connection noted in your review.

We would be glad to answer any further questions before the discussion closes.

Thank you for your thorough and thoughtful responses to my inquiries and to the other reviewers. ABased on your feedback, I will maintain my original score.

Thank you for your kind reply and for your constructive feedback, which strengthened our work.

Reviewer_Q9gVOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates what factors determine actual block-sparse training speedups across different GPUs. The authors implement BlockDST, a row-balanced blocked-ELL training system that supports sparse forward, input-gradient, and weight-gradient computations, and they evaluate its performance across six NVIDIA GPUs. Their main findings are that speedups depend heavily on the choice of the dense baseline, that their kernel throughput roughly scales with the SM count multiplied by the clock speed, and that kernel-level improvements result in much smaller gains in end-to-end training time.

I appreciate that the paper is clearly positioned as a measurement study and openly reports unfavorable results. However, my main concerns are about the generality and explanatory depth of their performance model, and whether the end-to-end experiments show enough relevance beyond this specific implementation and workload.

Critical strengths
  1. Careful distinction of performance metrics. The evaluation separates kernel speed, step throughput, and time to a target quality. This is important because just looking at sparse FLOP reductions can greatly overestimate actual training benefits. The cross-card comparisons, static-layout controls, and parameter-matched dense baselines give useful insights into when sparsity actually helps. Also, the paper rightly admits that its main three-seed time-to-target improvement is not statistically conclusive.

  2. Strong reproducibility support from code and artifacts. The supplementary material includes CUDA kernels for all three training parts, dense reference tests, experiment runners, and per-run accuracy-versus-time logs. Re-running the provided scripts successfully reproduces the main scaling exponent and leave-one-card-out predictions. This builds confidence in the analysis, although I haven't personally re-compiled the kernels or run full GPU training.

Critical weaknesses
  1. The main scaling model does not clearly transfer beyond the authors' own implementation. The SM-count-times-clock model fits the paper's kernel measurements well, but it does not predict cuSPARSE's Blocked-ELL performance. In the reported comparison, cuSPARSE gets much higher throughput, although the shapes and precisions are different. Because of this, it is unclear if the scaling model reflects a general property of sparse training or just a bottleneck specific to this implementation.

  2. The end-to-end training results provide limited evidence for practical relevance. The main training experiment uses a very wide CIFAR-10 MLP with batch size 8192 and relatively low target accuracies. The reported 1.44 times speedup is not statistically conclusive across three seeds, reverses on the A40, and does not beat parameter-matched dense or static sparse baselines. The transformer experiments also do not improve accuracy over strong baselines. The paper would be stronger if it showed that the proposed cost model reliably predicts outcomes across more typical model shapes, training phases, and target accuracies.

Questions for authors
  1. Kernel dependence: Under matched matrix shapes and precision, how do the custom and cuSPARSE kernels differ in cross-card scaling? Are there existing measurements that clarify whether the scaling behavior comes from an implementation-specific bottleneck or a fundamental GPU architecture effect in sparse computing?

  2. Prediction scope: Which training outcomes were predicted using parameters fixed before observing those specific outcomes? How accurately does the model predict full forward-plus-backward step throughput across the released configurations?

4 comments

Thank you for the careful review and for running our scripts. We address your questions below with new measurements (L = line in the initial submission; "new" marks material added in the revised PDF).

Q1 / W1. New App. B.10 removes the mismatch in shapes and precision that the initial submission noted for its cuSPARSE test (L471–473): it times both kernels in one job per card on all six GPUs, with the same six layer shapes (batch 8192 and 2048), densities, 64×64 blocks, 16-bit inputs and cuBLAS baseline. It confirms what the initial submission stated (L86–89, L299–301, L469–473): the SM×clock scaling is a property of our kernel.

On your first question, the two kernels scale differently at matched shapes across the six GPUs: the median CV of throughput per SM-GHz is 2.8% for our kernel vs. 41% for cuSPARSE; the A100 runs ours 1.08× and cuSPARSE 1.80× as fast as the A40, although it has only 4% more SM-GHz; and leave-one-card-out prediction (predicted/measured) is 0.93–1.09× for ours vs. 0.45–2.06× for cuSPARSE.

On your second question, these measurements point to a bottleneck of our implementation rather than a general effect of GPU architecture on sparse computing. cuSPARSE's A100 advantage is not a cache effect, as it holds at batch 8192 where activations exceed the A100's L2. Our kernel runs at 3–16% of tensor-core peak (L86–87) with fixed occupancy (App. B.3), so it scales with SM count × clock, whereas cuSPARSE is 1.3–7.1× faster and can use the A100's wider tensor cores (identifying which per-SM resource limits either kernel requires profiling counters we lack administrator rights for, L469–471).

What carries over across kernels is the decomposition (L84–89): speedup is sparse throughput divided by cuBLAS throughput D, the same denominator for any sparse kernel on a card. Because D varies 4.53× per SM-GHz across our six GPUs, any sparse kernel inherits card dependence. At matched shapes and 95% sparsity, cuSPARSE's speedup over dense spans 4.92× to 9.98× across the cards (2.68–5.02× in the initial submission, L300–301).

Q2a. New App. H.2 lists every prediction and whether its inputs were fixed. In the initial submission, only the A40 transfer (App. H.1, L1420–1457) had parameters fixed beforehand; it predicted the direction (a slowdown) and steps-to-target factor (within 3%), but underestimated the step-rate ratio by 21% and 27% (at 95% and 90% sparsity) and the end-to-end ratio by 23% and 25%. We also ran a prospective test on three new GPUs, with predictions timestamped before any run (general comment, item 2): 9 of 12 predictions hold, and BlockDST is slower than dense on all three cards as predicted (the three misses, all on the A100, underestimated BlockDST). In a second test, 11 of 11 scored predictions hold (W2).

Q2b. At the layer level, App. B.5 of the initial submission checked the forward-only sweep against forward+backward timings on five cards, agreeing within 1.00–1.19× (L1009), and the forward model predicts measured forward+backward speedups within 0.87–1.00× (new App. H.2). At batch 8192, all three training products together are within 0.84–1.13× of the bf16 forward speedup on all six cards (0.65–1.01× at batch 2048; new App. B.10).

At the step level, Amdahl's law with the GEMM share of the step (f = 0.93, fitted once on the L40 in App. H, L1403) predicts the step-rate ratio of ten MLP configurations (new Table 23) within −14% to +9%, and time to 0.55 accuracy within −19% to +9%. It replaces the fixed kernel-to-step ratio that caused the A40 miss; because we chose it after that result, these errors are post hoc, which motivated the prospective tests.

W2. The revised PDF addresses your last point, whether the cost model reliably predicts outcomes across more typical model shapes, training phases, and target accuracies, with a second prospective test, timestamped before its runs (new App. H.3): all 11 scored predictions held. From component timings alone, it predicted BlockDST's step-rate gain for decoders with d_model 2048, 2560 and 3072 (the last never trained) as 1.11, 1.18 and 1.23; we measured 1.08, 1.18 and 1.22. Our pre-set targets (0.45–0.55) are reached within 1,000 steps, after one rewiring (L358–360); at 0.60 and 0.65, reached after up to five rewirings, all 8 time ratios on the L40S and RTX PRO 6000 fell within a factor of 1.25, with BlockDST slower than dense, as predicted (an unscored repeat at 0.55 held in 3 of 4).

Ten L40 seeds (new) give 1.49× [1.34, 1.66] over dense (three seeds: 1.44× [0.99, 2.07]). The losses you list are in Limitations (L458–468) and the revised abstract: BlockDST is slower than dense on cards with smaller kernel speedups (A40, RTX A500, and the three new cards; four predicted in advance), and the parameter-matched dense model and never-rewired layout train faster. On transformers, BlockDST does not beat the strongest baselines, but matches dense on the ViT at a tenth of the weights (L448–454).

AuthorsOpenReview ↗

Follow-up for Reviewer Q9gV

Following up briefly on Q1 / W1, new App. S (added to the revised PDF in reply to Reviewer 5Jby; L = line in the initial submission) directly tests the part of your Q1 that our first reply answered only by inference: whether the scaling comes from an implementation-specific bottleneck or a fundamental GPU-architecture effect in sparse computing.

In our first reply, we noted that the measurements pointed to a bottleneck of our implementation and that identifying which per-SM resource limits either kernel (among the candidates in the initial submission, including shared-memory bandwidth, L900–907) would need profiling counters. We have now identified it for our kernel without profiling counters, using forward-kernel variants that change one design choice (or two combined), produce bit-identical output, and are timed on all six GPUs, with in-kernel cycle counts (clock64) for each phase.

Specifically, tensor-core fragments are read from shared-memory tiles with 128-byte rows, causing an 8-way bank conflict: the phase issuing these loads and the tensor-core instructions takes 8,200–9,800 of the 11,600–13,500 cycles per live block on every card, never below the 8,192-cycle conflict floor. Because that cost is fixed in cycles by the tile layout and the 32-bank structure shared by all six GPUs, throughput follows SM count × clock. Padding each row by 16 bytes removes the conflict and speeds up the kernel by 1.21–2.10×, and the padded kernel no longer tracks SM count × clock as closely (at the reference shape with bf16 storage, the CV of throughput per SM-GHz rises to 16–17%, vs. 2.2–2.4% for our unpadded kernel). This confirms that the scaling comes from an implementation-specific bottleneck; identifying cuSPARSE's limiting resource would still require profiling counters. The code, result files, and scripts for new App. S are in the updated supplement.

We would be glad to answer any remaining questions before the discussion closes, and would be grateful if you would take our answers, PDF changes, and these results into account in your final rating.

Thank you authors for the detailed feedback. The kernel scale experiment result solve my previous concerns about cross GPU capability, and the cost model successfully predicted gain on different decoding width. Although I still have some concerns that BlockDST still may not be adopted by model trainers, the paper can help them decide when to use sparse training vs dense models. I would like to raise my score to make more people to try it in the future.

Thank you for your reply and for raising your score. We are glad that the new experiments resolved your concerns about cross-GPU capability. Your constructive feedback led to the matched kernel comparison and the two prospective tests, which made the paper stronger. Helping model trainers decide when to use sparse training rather than dense, as you note, is the use we intend. The kernels, the training code, and the scripts for the cost model and its predictions are in the supplementary material, so others can try BlockDST on their own NVIDIA GPUs.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment

We thank the reviewers for their careful reviews. The revised PDF is uploaded, with changes in blue, and we answer each point under its review.

A note on versions: "Initial submission" is the version the reviews refer to, and "revised PDF" is the revision uploaded with this comment. L gives line numbers in the initial submission. Table numbers follow the revised PDF, which adds eight tables, with the initial number in parentheses where it differs. Section, appendix and figure numbers are unchanged, and "new" marks material added in the revised PDF.

The paper contains measurement study. For one block-sparse training kernel and one shape, the speedup over dense varies 1.08–5.13× across six GPUs, mostly because of the dense baseline. Our claims concern what sets this speedup and how well it can be predicted.

1. Matched kernel comparison (new App. B.10; Q9gV, 5Jby) We timed our kernel and cuSPARSE at identical shapes, densities and 16-bit precision on all six GPUs. Throughput per SM-GHz varies across cards with a median CV of 2.8% for ours and 41% for cuSPARSE, and a one-constant model predicts a held-out card within 0.93–1.09× for ours but only within 0.45–2.06× for cuSPARSE. The SM×clock scaling is therefore a property of our kernel, as the initial submission already states (L86–89).

2. Prospective test (new App. H.2; Q9gV) We predicted training on three GPUs we had never run time-to-target training on, using only released numbers: each card's kernel speedup, Amdahl's law with f = 0.93 fitted on the L40, and the L40's steps-to-target factor. The pass bands were fixed with the predictions (step-rate ratio within a factor of 1.2, time ratio to 0.55 accuracy within 1.25), and the prediction file was timestamped with OpenTimestamps before any run. The file, its proof and a checking script are in the supplementary material, and new App. H.2 states what we had seen before the timestamp. With the protocol of Table 3 (initially 2) and three seeds, predicted vs. measured ratios at 95% and 90% sparsity were:

  • A100: step rate 1.07 vs. 1.18 and 0.50 vs. 0.61; time 0.38 vs. 0.51 and 0.24 vs. 0.32 (1 of 4 within the bands).
  • L40S: step rate 2.46 vs. 2.37 and 1.35 vs. 1.33; time 0.88 vs. 0.87 and 0.65 vs. 0.57 (4 of 4).
  • RTX PRO 6000: step rate 1.75 vs. 1.75 and 0.90 vs. 0.89; time 0.62 vs. 0.65 and 0.43 vs. 0.43 (4 of 4).

So 9 of 12 predictions hold, including all eight on the L40S and RTX PRO 6000. The three misses, all on the A100, underestimated BlockDST, consistent with App. B.5 of the initial submission: our forward-only input understates the A100's forward+backward speedup the most.

3. Ten seeds on the L40 (new Table 18 in App. E, and App. J; Q9gV) BlockDST reaches 0.55 accuracy 1.49× sooner than dense (95% CI [1.34, 1.66]), compared with 1.44× [0.99, 2.07] from the initial three seeds; the interval now excludes 1. The same job also reruns the never-rewired layout and the parameter-matched dense model, which train faster, as in the initial submission.

4. Second prospective test (new App. H.3; Q9gV) Its predictions were also timestamped before the runs, and all 11 scored ones held. From component timings alone, BlockDST's step-rate gain on decoders with d_model 2048, 2560 and 3072 (the last never trained) was predicted as 1.11, 1.18 and 1.23 and measured as 1.08, 1.18 and 1.22 (3 of 3 within a factor of 1.05). At targets 0.60 and 0.65 on the L40S and RTX PRO 6000, 8 of 8 time ratios fell within a factor of 1.25, with BlockDST slower than dense, as predicted (unscored repeat at 0.55: 3 of 4).

Changes in the revised PDF (in blue)

  • Abstract: the matched cuSPARSE contrast, the ten-seed result, and that BlockDST is slower than dense on cards with smaller kernel speedups.
  • Sec. 2: new Table 1 comparing DynaDiag, HASTE and BLaST, brought up from App. O (5Jby, 2CxK).
  • Sec. 3: a new kernel-design paragraph (5Jby).
  • Sec. 4: a new scope, with the hardware detail moved to App. B.9 (2CxK).
  • Sec. 5.2 and new App. H.2–H.3: the step model and the two prospective tests (Q9gV).
  • Sec. 5.5: new part with the takeaway, brought up from App. H and R (2CxK).
  • Fig. 2(a), redrawn, and a new sentence in the Fig. 4 caption (2CxK).
  • Corrections in App. B.3 (kernel resources), App. G.2 (timings without per-call casts) and App. H.1 (cause of the A40 miss).
  • Limitations and Conclusion: the new losses.
  • All new code and results are in the supplementary.
AuthorsOpenReview ↗

General Comment 2 - Further Revision: New App. S

We uploaded a further revision of the PDF in reply to Reviewer 5Jby's follow-up (changes in blue; "new" marks material added in the revised PDF):

  • New App. S (with new Table 35 and new Fig. 13) addresses the follow-up via controlled kernel ablations and in-kernel cycle counts, summarized in the introduction, Sec. 4.1, and Limitations. Sec. 4.1 now gives the bf16-storage gain as under 1.3×, against about 1.8× for a bandwidth-bound kernel, and App. B.1 now says that the exponent remains a fitted parameter, since new App. S finds the kernel limited by shared memory. The supplement includes the code and data of new App. S.
  • Other relevant changes: Sec. 4.1, the residual stated for Fig. 3(a), and the per-card values in Table 6 and new Table 23 now all use the 2.43 GHz RTX PRO 6000 clock of Table 2 (CV 7.3%, k0 = 0.079, residual 17%); the held-out ranges are unchanged. At 90% sparsity, BlockDST and dense tie in time to accuracy 0.50 (Sec. 5.1 and Table 3). The Limitations now state that at batch 8192 the three training products together are within 0.84–1.13× of our forward speedup with bf16 storage, and 0.65–1.01× at batch 2048, as in new App. B.10. Table 29 now states that only the dense and masked arms use bf16 autocast, and new App. B.10 and App. G state that both kernels were timed in one job per card (cuSPARSE runs as a separate program). Smaller corrections are also in blue, and the supplement now contains every script the paper cites.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.