- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces Permutation-Packable N:M sparsity (PP-N:M), a restricted family of bidirectionally balanced sparse supports that become block diagonal after row and column permutations. This structure is closed under the matrix products used by Newton–Schulz iterations, enabling Muon updates without generating values at inactive coordinates. Building on this representation, S2S Muon combines component-wise Newton–Schulz updates, row–column equilibration, and Adam-reference magnitude scaling. Custom PP-linear kernels preserve compact weights and gradients throughout forward, backward, and optimizer computation. Experiments cover CIFAR-100, 300-epoch ImageNet-1K training, and 350M/1B language models. On ImageNet, S2S reaches 78.21% accuracy versus 77.31% for projected Muon, while remaining below dense Muon’s 79.28%. Packed execution reduces measured peak memory, with throughput gains depending on the workload.
Critical strengths
PP-N:M connects local sparsity constraints, exact packing, and optimizer compatibility.
Computing packed weight gradients directly avoids a common gap between sparse parameter storage and actual training-memory savings.
The paper reports negligible or inconsistent PRE gains in vision, the remaining accuracy gap to dense Muon, and throughput regressions.
Critical weaknesses
The comparisons primarily establish savings against projected Muon and a dense-masked implementation of S2S. They do not show the quality–memory–time trade-off against an established compact sparse-training implementation. Such alternatives could be preferable when the goal is training under a fixed resource budget. Existing approaches such as SLoPe https://arxiv.org/pdf/2405.16325 make compact sparse execution a relevant practical comparison.
Questions for authors
Can the authors provide one representative comparison of complete S2S against a competitive compact sparse implementation and a smaller dense model?
3 comments
ViT update: faster training, lower memory, better sparse-model accuracy, and faster 2:4 inference
Thank you again for recognizing our support construction and compact gradient computation. Your request for a clearer quality–memory–time comparison, together with Reviewer Hau7's request for a timing breakdown, motivated a concrete improvement that we would like to share.
The original PP-NM update issued 144 independent Newton–Schulz calls per step (72 matrices × 2 packed components). We implemented batched NS, grouping equal-shaped components while preserving the update rule and independent per-component computation. The PP-linear forward/backward kernels remain unchanged, and no CUDA Graphs or compilation are used for this training optimization.
| Metric | Original component-wise NS | Batched NS |
|---|---|---|
| Matrix optimizer (ms/update) | 108.482 ± 1.307 | 2.214 ± 0.006 |
| Whole update (ms/update) | 225.422 ± 0.126 | 115.686 ± 0.372 |
| Training throughput (images/s) | 283.912 ± 0.158 | 553.235 ± 1.781 |
| Peak allocated (GiB) | 0.8244 | 0.8203 |
ViT on three RTX 5090 GPUs; mean ± SE over paired fresh-process runs, with 50 warmup and 200 measured updates, micro/global batch 16/64.
PP-NM's whole-step throughput reaches 1.95× its original level, with essentially unchanged peak allocated memory. All 250 per-update losses and final model, optimizer and RNG states are bitwise identical in our validation, and checkpoint save/resume agrees exactly.
In a separate matched backend comparison, PP-linear improves throughput by 23.0% for PP-NM and 21.9% for S2S, while reducing peak allocated memory by 16.4% and 15.2%, respectively, relative to their corresponding dense+mask implementations.
These efficiency gains complement the completed ImageNet comparison: PP-NM and S2S with PP-linear achieve 56.185% and 60.284% top-1 accuracy, versus 48.803% for Projected Muon, under the method-specific reference-throughput-derived update budgets described in our earlier response. (19 Nov 2026 update: We subsequently reran both PP-NM Muon seeds under the recalibrated budget. Revised Table 28 reports 57.542% at 735,899 updates per seed, up from the 56.185% quoted here.) Together with the compact-training and smaller-dense controls already reported, these results strengthen the evidence for the practical quality–memory–time tradeoff. Full measurement and validation details are provided in our supplementary response to Reviewer Hau7.
Taken together, our ViT results demonstrate gains in all four practical dimensions: higher training throughput, lower training memory, better accuracy than the evaluated sparse baseline, and faster native 2:4 inference. The training gains are relative to each optimizer's dense+mask backend, the accuracy comparison is against Projected Muon under the budgets above, and the previously measured inference gain is 34.1% over same-weight dense execution with CUDA Graphs enabled for both routes.
Your recognition of the underlying contribution has been encouraging, and your feedback has helped us turn that contribution into stronger practical results. We will incorporate these improvements into the revision and continue optimizing throughput. We hope this progress addresses your concerns and strengthens your confidence in the work, and would be grateful if you would consider it in a more favorable final assessment.
Reviewer 5dPk, our revised manuscript is ready — we welcome your feedback
Dear Reviewer 5dPk,
Thank you again. We have uploaded a revised PDF and invite you to download and read it. We have improved the language and organization and substantially expanded the experiments, as summarized below.
Addressing your quality–memory–time comparison request
Appendix D.14, Table 28 compares ten ViT configurations × two seeds, including complete S2S, adapted compact SLoPe, and smaller dense models. All 20 endpoints use full 50,000-image validation. Selected rows are:
| Method | Updates/seed | Top-1 (%) | Allocated/reserved GiB |
|---|---|---|---|
| S2S / PP-linear (Ours) | 623,494 | 60.284 ± 0.616 | 0.845/0.930 |
| Adapted SLoPe | 736,404 | 44.525 ± 0.101 | 1.307/1.400 |
| Smaller dense Muon | 993,555 | 59.303 ± 0.373 | 0.533/0.545 |
| Full-size dense Muon | 537,476 | 62.025 ± 0.311 | 0.949/0.980 |
Accuracy is mean ± SE. Counts reflect a calibrated 24-hour RTX 5090 reference budget, not actual accelerated training time; global batch and assigned counts remain fixed. Memory is measured separately in single-GPU eager execution, micro/global batch 16/64. SLoPe uses double-pruned backward computation and rank-24 adapters for the final 7,365 updates (1%); memory includes the maximum across ordinary, adapter-activation, and adapter-enabled phases. Its budget uses paired timing against Top2 AdamW with 99%/1% phase weights, and its LR is inherited from Top2 AdamW. Smaller models halve depth from 12 to 6, retain width, and inherit dense LRs. These choices are documented in D.14.
S2S exceeds adapted SLoPe in accuracy with lower measured memory, and exceeds smaller dense Muon in accuracy while using more memory. Full-size dense Muon remains most accurate. The comparison is specific to this ViT protocol and adaptation.
Additions prompted by other reviewers
We added phase/kernel timing, indexing controls, initialization-cost scaling (Appendices E.3, B.1), and the distinction from MPDCompress (Section 5). Broader evaluation includes five methods × three seeds on LoveDA segmentation and six methods × three seeds on FineWeb 124M (Appendices D.11–D.12, Tables 25–26). FineWeb screens four LRs with two selection seeds per method before 18 formal runs of 2.517B tokens each and separate final testing.
Further replication and implementation improvements
The full 300-epoch ImageNet comparison now covers three common seeds for all seven methods. Tables 2 and 13 report mean ± SE and all 21 endpoints. S2S reaches 78.24 ± 0.08%, versus 77.28 ± 0.05% projected Muon and 77.95 ± 0.08% SMET; dense Muon remains higher at 79.12 ± 0.17%. This strengthens the main result separately from the two-seed reference-budget study.
As discussed in our earlier supplementary response, the reviewers' efficiency questions, particularly Hau7's timing request, led us to identify overhead from separate NS calls for packed components. Batching equal-shaped components while retaining the update recipe and PP-linear kernels gives 1.95× whole-update throughput over component-wise execution (Appendix E.3). We then compared both backends with the same batched optimizer. Validation includes component/shared-gradient checks, checkpoint recovery, and 250-update trajectories (E.1).
After optimization, both PP-NM seeds were rerun from initialization at a newly calibrated 735,899 updates, reaching 57.542 ± 0.920% top-1 (Table 28); the other nine configurations retain their budgets/results. Additional evidence includes full-length PRE/amplitude controls on ImageNet and 350M LM with two paired seeds, a separately tuned five-configuration direction/amplitude study, connectivity/routing controls, and full-length NS-precision comparisons (Appendices D.2–D.6, D.15).
PP-NM Muon is, to our knowledge, the first N:M-sparse Muon optimizer preserving support closure throughout NS iterations, with S2S further improving its update. On ViT, PP-linear raises PP-NM/S2S training throughput by 23.0%/21.9% and lowers allocated memory by 16.4%/15.2%, versus their respective dense-mask implementations (Table 5). Both improve accuracy over projected Muon (Table 2). Native 2:4 inference gives 1.34× ViT throughput and 12.5% lower allocated memory than dense execution of identical sparse weights, with CUDA Graphs on both routes (Table 36).
With less than a day remaining, we hope these revisions address your concerns. Please raise any remaining questions before the deadline so we can respond promptly. We appreciate your positive assessment. If the added comparisons and improvements sufficiently strengthen the paper, we would be grateful if you would consider a higher score. Thank you again.
Dear Reviewer 5dPk,
Thank you very much for your constructive review and for recognizing our support construction and compact gradient computation. Following your suggestion, we have added a representative ImageNet experiment comparing complete S2S against an adapted compact SLoPe implementation and smaller dense baselines to clarify the quality–memory–time trade-offs.
Below are our detailed responses to your specific points:
Response: Thank you for this valuable suggestion. We agree that comparisons with Projected Muon and dense-mask S2S alone do not fully capture the trade-offs against compact sparse training and smaller dense models. We have therefore conducted a representative ImageNet experiment evaluating complete S2S, an adapted compact SLoPe implementation, and smaller dense models trained with both AdamW and Muon.
1. Implementation Details of Adapted SLoPe
Our SLoPe adaptation trains from initialization with native compact sparse execution via
cuSPARSELtand double-pruned backward computation. It retains rank-24 low-rank adapters during the final 1% of updates (7,365 of 736,404). While SLoPe's final model combines a 2:4 sparse backbone with dense adapters (unlike S2S, whose eligible weights remain strictly representable as 2:4), it serves as a competitive compact-training baseline. We will include it in the related work section.2. Joint ImageNet Quality–Time–Memory Comparison (RTX 5090)
(19 Nov 2026 update: Revised Appendix D.14, Table 28 updates the PP-NM Muon row after NS batching and recalibration: 57.542% mean top-1 at 735,899 updates per seed, compared with 56.185% at 546,232 updates in the row below. Table 28 in revised article also gives the memory-measurement conditions.)
Note on Budgets & Protocols: "24 h" denotes a calibrated update budget. Memory is measured in single-GPU eager execution (taking the maximum across ordinary and adapter windows for SLoPe). Results report mean ± SE across seeds 20260925 and 20260926 on all 50,000 ImageNet validation images.
3. Key Findings & Trade-offs
Thank you once again for your positive assessment and helpful suggestions. We are very glad that our work and supplementary experiments addressed your concerns. Please feel free to let us know if you have any further questions—we would be more than happy to discuss them. Thank you for your valuable contribution to improving our paper!