- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces BlockDST, which maintains a fixed number of active blocks per block row and uses custom kernels for sparse training. It evaluates performance across six GPUs, fits an empirical throughput model based on SM count and clock frequency, and examines how kernel speedups translate into training time and memory savings.
Critical strengths
-
The evaluation covers multiple GPU architectures and distinguishes kernel performance from training performance.
-
The paper clearly reports limitations and negative results, including cases where the proposed approach is slower than alternative methods.
Critical weaknesses
-
The paper mainly benchmarks a custom kernel without adequately explaining its design or establishing its novelty in the main text.
-
Scaling with SM count and clock frequency is expected and, by itself, provides little new insight.
-
The contribution beyond prior work on sparse DST execution, including DynaDiag [1], HASTE [2], and so on, remains unclear.
[1] Abhishek Tyagi et al. Dynamic Sparse Training of Diagonally Sparse Networks. ICML, 2025.
[2] Nasib Ullah et al. HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces. ICML, 2026.
Questions for authors
Please address the concerns in Critical Weaknesses, particularly kernel novelty and the distinction from prior sparse DST systems.
14 comments
-
STrengthen the kernel analysis by developing a performance model grounded in kernel design and hardware characteristics, supported by profiling results and controlled ablation studies. The goal is to explain which design choices drive performance and why. Kernel studies published at computer architecture venues provide useful examples of this approach.
-
Demonstrate the practical significance of the work. Establishing a performance trend is only a first step; the paper should also clarify why the findings matter and what practical capabilities or opportunities they enable.
Thank you for your suggestions. We updated the revised PDF with new App. S (plus new Table 35 and new Fig. 13) to address both points, with its code and results in the supplement (L = line in the initial submission; "new" marks material added in the revised PDF).
1. Grounding the performance model in kernel design, hardware, profiling, and ablations (new App. S): Controlled ablations on all six GPUs show which design choice drives our kernel's performance and why its throughput tracks SM count × clock: it reads tensor-core fragments from shared memory with an 8-way bank conflict, which standard row padding removes.
Each thread reads its fragments with 32 four-byte loads per live block from tiles with 128-byte rows. A warp-wide load touches 8 rows falling on the same 4 of the 32 banks and is served in 8 passes instead of 1, so fragment loads of a live block occupy the SM's shared memory for 8,192 cycles instead of 1,024, which is 4–16× the tensor-core work at datasheet rates (512 cycles on the A100 to 2,048 on the L40). The input- and weight-gradient kernels issue the same loads (not ablated).
We profile in-kernel, reading the SM cycle counter (clock64) in every thread block at three points per live block, and time forward-kernel variants that change one design choice (or two combined) with bit-identical output at the reference shape (95.3% and 89.8% sparse). Across all six GPUs and both densities (new Table 35 and new Fig. 13 show 89.8%):
- The fragment-load and MMA phase takes 8,200–9,800 of the 11,600–13,500 cycles per live block (67–76%), never below the 8,192-cycle floor.
- Padding each tile row by 16 bytes makes every load conflict-free and removes 5,700–6,700 cycles from this phase; a thread block then needs 1.7–2.1× fewer cycles, and 1.9–3.4× fewer with 16-byte staging loads (1.21–2.10× and 1.69–3.45× faster than our released kernel). The A100, whose tensor cores do the most work per SM and clock, gains most; the L40 gains least, as its clock drops from 2.45–2.46 to 1.7–2.0 GHz at its 300 W limit. Other single changes matter less (against a template copy of our kernel: 16-byte staging alone 1.16–1.37×, issuing staging loads together 1.08–1.18×).
- Before these runs we wrote six predictions from the machine code and a small-shape test (not externally timestamped): five held, while the predicted time gain from padding missed on three of the six cards (higher on the A100 and RTX PRO 6000, lower on the L40).
This identifies the per-SM resource left open in the initial submission (L86, L469–471): shared-memory bandwidth (32 banks of 32 bits per clock on all six GPUs). Two checks tie the cycles to the banks rather than the card: the A40 and laptop RTX A500, running identical machine code on 84 and 16 SMs, both take 9,025–9,034 cycles in this phase, and the A100's phase is within 0.4% of the floor. With the fp32 storage of Sec. 4.1, a live block runs at 72–88 FLOP per SM-cycle, matching the fitted k0 = 0.079 TFLOP/s per SM-GHz, or 79 (0.078 at L279–280). Thus throughput follows SM count × clock rather than DRAM bandwidth or the tensor-core rate cuBLAS depends on. Once the conflict is removed, tensor cores matter more as predicted: on the L40S (the L40's die at twice its tensor-core rate), the padded phase takes 3,020 cycles vs. 4,022 on the L40, close to the 1,024-cycle difference in their tensor-core work. The introduction, Sec. 4.1, and Limitations now state this, and Sec. 4.1 gives the bf16-storage gain as under 1.3× (initially 1.12–1.25×, L272–273).
2. Practical significance and capabilities enabled: First, new App. S shows how to find and remove what limits a kernel one can modify, across GPUs and without profiling counters: with padded rows and 16-byte staging, our 89.8%-sparse forward product is 2.02–5.27× faster than dense GEMM on all six GPUs, up from 0.59–2.70× before, when it was slower than dense on the A100, A40, and RTX PRO 6000 (new Table 35; dense GEMM as in new App. B.10). Because a shared-memory-bound kernel takes about the same cycles per live block on every card while cuBLAS tracks the tensor-core rate, a single-card speedup cannot separate a better method from a better-suited GPU (L305–307).
Second, our model lets a practitioner decide before training whether BlockDST will pay off on a GPU: SM count, clock, and one dense GEMM timing predict our kernel's speedup on a held-out card within 0.91–1.19× at the reference shape (L280–281), and the sparsifiable step share and extra steps to target turn it into a training speedup (L419–422). We tested that second step prospectively twice, with predictions timestamped before the runs (new App. H.2–H.3): 9 of 12 and then all 11 scored predictions held, and both tests foresaw where BlockDST is slower than dense.
If our answers, PDF changes, and new App. S address your concerns, we would be grateful if you would consider them in your final assessment.
Thank you for the review. While W1 and W3 were in the initial submission, key details were too brief or in the appendix; we have moved both into the main text (Secs. 2 and 3, in blue; L = line in the initial submission; "new" marks material added in the revised PDF; table numbers follow the revised PDF, with initial numbers in parentheses).
W1. On kernel novelty, the first part of your question: what is new is covering all three training products on Blocked-ELL, where every block-row holds exactly k blocks. cuSPARSE's Blocked-ELL supports only the non-transposed forward pass, and its SDDMM does not support Blocked-ELL. Our kernels add the transposed product and a block-SDDMM, so all three training products run on one load-balanced layout with no dense W or ∇W, which cuts peak memory 2.1× below dense (L428–429).
The initial submission described the layout and three products in Sec. 3 (L157–199, Fig. 2) and, in the introduction, what existing block-sparse libraries do not provide for this layout (L45–53). The revised Sec. 3 adds a new kernel-design paragraph:
W2. Sec. 4 of the initial submission already shows why this scaling, which we do not claim alone as our contribution, is informative here:
W3. On the second part of your question, the distinction from prior sparse DST systems is twofold. Of the methods in the new Table 1, only BlockDST runs all three training products on tensor-core blocks without keeping a dense W / ∇W. And whereas the other GPU methods in that table report speedups on one GPU type, we measure how such a speedup varies across six GPUs (time to accuracy on five, three predicted in advance), trace that variation to the dense baseline, and show when it becomes an end-to-end loss.
The initial submission discussed this prior work in Sec. 2 (L120–128), App. O.1–O.2 (L1724–1754), and App. O.3 / Table 30 (initially 23; L1756–1774, L1783–1792). The new Table 1 brings the closest methods into Sec. 2, comparing sparsity pattern, which training products run sparse, whether dense W / ∇W tensors are kept, and hardware evaluated: