acceptodds
Under review as a conference paper at ICLR 2027

Permutation-Packable N:M Sparsity for Sparse-to-sparse Muon

Abstract

Applying Muon to sparse vision Transformers creates a mismatch: its Newton–Schulz (NS) iterations can generate updates at pruned weights, which projection then discards, while dense training tensors can limit the memory benefits of sparsity. We introduce Permutation-Packable N:M (PP-N:M), a locally balanced support that is N:M in both directions and packs exactly into dense active matrices. Its closure under NS cubic products enables matrix updates entirely on active coordinates. On this support, S2S Muon constructs update directions through pre-NS row–column equilibration and component-wise NS, with a second-moment-based Adam reference controlling their magnitude. To retain compact storage beyond the optimizer, we develop PP-linear kernels that fuse channel indexing with matrix products and compute packed weight gradients directly. In a seven-method ImageNet-1K comparison at 300 epochs over two seeds, ViT with S2S Muon reaches 78.21% mean top-1 accuracy versus 77.31% for projected Muon, a gain of 0.90 percentage points, while retaining half of the eligible weights. Complementary CIFAR-100 experiments reach % (mean standard error) across three seeds, 3.76 percentage points above projected Muon. As a cross-domain test, 350M and 1B language models improve perplexity over projected Muon by 2.59% and 2.31%, respectively, at a fixed 1.008B-token budget. On RTX 5090, PP-linear reduces peak allocated training memory for single-momentum Muon+PP-N:M by 18.29% on ViT and 40.22% on a 1B language model relative to projected Muon. With the complete S2S Muon update rule held fixed, it saves 15.17% and 29.09%, respectively, relative to a dense-masked implementation. These memory benefits come with workload- and implementation-dependent throughput trade-offs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Muon, Semi-Structure Sparsity, N:M Sparsity, ViT, Optimizer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5dPkOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Permutation-Packable N:M sparsity (PP-N:M), a restricted family of bidirectionally balanced sparse supports that become block diagonal after row and column permutations. This structure is closed under the matrix products used by Newton–Schulz iterations, enabling Muon updates without generating values at inactive coordinates. Building on this representation, S2S Muon combines component-wise Newton–Schulz updates, row–column equilibration, and Adam-reference magnitude scaling. Custom PP-linear kernels preserve compact weights and gradients throughout forward, backward, and optimizer computation. Experiments cover CIFAR-100, 300-epoch ImageNet-1K training, and 350M/1B language models. On ImageNet, S2S reaches 78.21% accuracy versus 77.31% for projected Muon, while remaining below dense Muon’s 79.28%. Packed execution reduces measured peak memory, with throughput gains depending on the workload.

Critical strengths

PP-N:M connects local sparsity constraints, exact packing, and optimizer compatibility.

Computing packed weight gradients directly avoids a common gap between sparse parameter storage and actual training-memory savings.

The paper reports negligible or inconsistent PRE gains in vision, the remaining accuracy gap to dense Muon, and throughput regressions.

Critical weaknesses

The comparisons primarily establish savings against projected Muon and a dense-masked implementation of S2S. They do not show the quality–memory–time trade-off against an established compact sparse-training implementation. Such alternatives could be preferable when the goal is training under a fixed resource budget. Existing approaches such as SLoPe https://arxiv.org/pdf/2405.16325 make compact sparse execution a relevant practical comparison.

Questions for authors

Can the authors provide one representative comparison of complete S2S against a competitive compact sparse implementation and a smaller dense model?

3 comments

Dear Reviewer 5dPk,

Thank you very much for your constructive review and for recognizing our support construction and compact gradient computation. Following your suggestion, we have added a representative ImageNet experiment comparing complete S2S against an adapted compact SLoPe implementation and smaller dense baselines to clarify the quality–memory–time trade-offs.

Below are our detailed responses to your specific points:

Response: Thank you for this valuable suggestion. We agree that comparisons with Projected Muon and dense-mask S2S alone do not fully capture the trade-offs against compact sparse training and smaller dense models. We have therefore conducted a representative ImageNet experiment evaluating complete S2S, an adapted compact SLoPe implementation, and smaller dense models trained with both AdamW and Muon.

1. Implementation Details of Adapted SLoPe

Our SLoPe adaptation trains from initialization with native compact sparse execution via cuSPARSELt and double-pruned backward computation. It retains rank-24 low-rank adapters during the final 1% of updates (7,365 of 736,404). While SLoPe's final model combines a 2:4 sparse backbone with dense adapters (unlike S2S, whose eligible weights remain strictly representable as 2:4), it serves as a competitive compact-training baseline. We will include it in the related work section.

2. Joint ImageNet Quality–Time–Memory Comparison (RTX 5090)

(19 Nov 2026 update: Revised Appendix D.14, Table 28 updates the PP-NM Muon row after NS batching and recalibration: 57.542% mean top-1 at 735,899 updates per seed, compared with 56.185% at 546,232 updates in the row below. Table 28 in revised article also gives the memory-measurement conditions.)

Method / ExecutionEligible-Weight SparsityReference Time Budget / BasisUpdates per SeedTop-1 Accuracy (%) ↑Eager Peak Allocated (GiB) ↓Eager Peak Reserved (GiB) ↓
Dense AdamWDense (0%)24 h872,28351.221±0.26751.221 \pm 0.2671.0271.068
Dense MuonDense (0%)24 h537,47662.025±0.31162.025 \pm 0.3110.9490.980
Top2 AdamW2:4 (50%)24 h706,63444.936±0.17844.936 \pm 0.1781.0681.119
Projected Muon2:4 (50%)24 h615,51248.803±0.28748.803 \pm 0.2870.9911.014
PP-NM Muon (Ours)2:4 (50%)24 h546,23256.185±0.33356.185 \pm 0.3330.8060.852
S2S Muon (Ours, dense-mask)2:4 (50%)24 h490,20959.082±0.43459.082 \pm 0.4341.0081.084
S2S Muon (Ours, PP-linear)2:4 (50%)24 h623,49460.284±0.61660.284 \pm 0.6160.8450.930
Adapted SLoPe (w/ adapters)2:4 backbone + dense adapters24 h736,40444.525±0.10144.525 \pm 0.1011.3071.400
Smaller Dense AdamWDense (0%)24 h1,510,61347.925±0.15147.925 \pm 0.1510.5730.598
Smaller Dense MuonDense (0%)24 h993,55559.303±0.37359.303 \pm 0.3730.5330.545

Note on Budgets & Protocols: "24 h" denotes a calibrated update budget. Memory is measured in single-GPU eager execution (taking the maximum across ordinary and adapter windows for SLoPe). Results report mean ±\pm SE across seeds 20260925 and 20260926 on all 50,000 ImageNet validation images.

3. Key Findings & Trade-offs

  • Quality & Memory Balance: Complete S2S with PP-linear achieves 60.284% Top-1 accuracy, substantially outperforming adapted SLoPe (44.525%), smaller dense AdamW (47.925%), and surpassing smaller dense Muon (59.303%) by 0.98 percentage points. While smaller dense Muon uses less memory due to its halved depth (6 blocks vs. 12), S2S retains full architectural depth while leveraging 2:4 sparsity. Furthermore, PP-linear reduces S2S's peak allocated memory by 16.1% compared to dense-mask execution, consistent with our paper's 15.17% reduction.
  • Interpreting the SLoPe Comparison: Our paired calibration yields a modest 1.042×1.042\times throughput gain for SLoPe over Top2 AdamW. Larger speedups are not expected on ViT because sparse-kernel efficiency depends heavily on matrix shapes (SLoPe, Sec. 2.4 & App. O): here, sparsity accelerates forward and input-gradient products, whereas weight gradients remain dense and sparse-format maintenance adds overhead (taking 53.934 ms/update vs. 46.900 ms for dense execution before adapters in a matched diagnostic). On the quality side, retaining SLoPe's final-1% adapter schedule translates to only 7,365 adapter updates (~0.37 ImageNet epochs at batch 64). Without separate ViT hyperparameter tuning or a longer adapter-phase ablation, this result reflects our specific limited-budget adaptation rather than SLoPe's upper-bound accuracy.

Thank you once again for your positive assessment and helpful suggestions. We are very glad that our work and supplementary experiments addressed your concerns. Please feel free to let us know if you have any further questions—we would be more than happy to discuss them. Thank you for your valuable contribution to improving our paper!

AuthorsOpenReview ↗

ViT update: faster training, lower memory, better sparse-model accuracy, and faster 2:4 inference

Thank you again for recognizing our support construction and compact gradient computation. Your request for a clearer quality–memory–time comparison, together with Reviewer Hau7's request for a timing breakdown, motivated a concrete improvement that we would like to share.

The original PP-NM update issued 144 independent Newton–Schulz calls per step (72 matrices × 2 packed components). We implemented batched NS, grouping equal-shaped components while preserving the update rule and independent per-component computation. The PP-linear forward/backward kernels remain unchanged, and no CUDA Graphs or compilation are used for this training optimization.

MetricOriginal component-wise NSBatched NS
Matrix optimizer (ms/update)108.482 ± 1.3072.214 ± 0.006
Whole update (ms/update)225.422 ± 0.126115.686 ± 0.372
Training throughput (images/s)283.912 ± 0.158553.235 ± 1.781
Peak allocated (GiB)0.82440.8203

ViT on three RTX 5090 GPUs; mean ± SE over paired fresh-process runs, with 50 warmup and 200 measured updates, micro/global batch 16/64.

PP-NM's whole-step throughput reaches 1.95× its original level, with essentially unchanged peak allocated memory. All 250 per-update losses and final model, optimizer and RNG states are bitwise identical in our validation, and checkpoint save/resume agrees exactly.

In a separate matched backend comparison, PP-linear improves throughput by 23.0% for PP-NM and 21.9% for S2S, while reducing peak allocated memory by 16.4% and 15.2%, respectively, relative to their corresponding dense+mask implementations.

These efficiency gains complement the completed ImageNet comparison: PP-NM and S2S with PP-linear achieve 56.185% and 60.284% top-1 accuracy, versus 48.803% for Projected Muon, under the method-specific reference-throughput-derived update budgets described in our earlier response. (19 Nov 2026 update: We subsequently reran both PP-NM Muon seeds under the recalibrated budget. Revised Table 28 reports 57.542% at 735,899 updates per seed, up from the 56.185% quoted here.) Together with the compact-training and smaller-dense controls already reported, these results strengthen the evidence for the practical quality–memory–time tradeoff. Full measurement and validation details are provided in our supplementary response to Reviewer Hau7.

Taken together, our ViT results demonstrate gains in all four practical dimensions: higher training throughput, lower training memory, better accuracy than the evaluated sparse baseline, and faster native 2:4 inference. The training gains are relative to each optimizer's dense+mask backend, the accuracy comparison is against Projected Muon under the budgets above, and the previously measured inference gain is 34.1% over same-weight dense execution with CUDA Graphs enabled for both routes.

Your recognition of the underlying contribution has been encouraging, and your feedback has helped us turn that contribution into stronger practical results. We will incorporate these improvements into the revision and continue optimizing throughput. We hope this progress addresses your concerns and strengthens your confidence in the work, and would be grateful if you would consider it in a more favorable final assessment.

AuthorsOpenReview ↗

Reviewer 5dPk, our revised manuscript is ready — we welcome your feedback

Dear Reviewer 5dPk,

Thank you again. We have uploaded a revised PDF and invite you to download and read it. We have improved the language and organization and substantially expanded the experiments, as summarized below.

Addressing your quality–memory–time comparison request

Appendix D.14, Table 28 compares ten ViT configurations × two seeds, including complete S2S, adapted compact SLoPe, and smaller dense models. All 20 endpoints use full 50,000-image validation. Selected rows are:

MethodUpdates/seedTop-1 (%)Allocated/reserved GiB
S2S / PP-linear (Ours)623,49460.284 ± 0.6160.845/0.930
Adapted SLoPe736,40444.525 ± 0.1011.307/1.400
Smaller dense Muon993,55559.303 ± 0.3730.533/0.545
Full-size dense Muon537,47662.025 ± 0.3110.949/0.980

Accuracy is mean ± SE. Counts reflect a calibrated 24-hour RTX 5090 reference budget, not actual accelerated training time; global batch and assigned counts remain fixed. Memory is measured separately in single-GPU eager execution, micro/global batch 16/64. SLoPe uses double-pruned backward computation and rank-24 adapters for the final 7,365 updates (1%); memory includes the maximum across ordinary, adapter-activation, and adapter-enabled phases. Its budget uses paired timing against Top2 AdamW with 99%/1% phase weights, and its LR is inherited from Top2 AdamW. Smaller models halve depth from 12 to 6, retain width, and inherit dense LRs. These choices are documented in D.14.

S2S exceeds adapted SLoPe in accuracy with lower measured memory, and exceeds smaller dense Muon in accuracy while using more memory. Full-size dense Muon remains most accurate. The comparison is specific to this ViT protocol and adaptation.

Additions prompted by other reviewers

We added phase/kernel timing, indexing controls, initialization-cost scaling (Appendices E.3, B.1), and the distinction from MPDCompress (Section 5). Broader evaluation includes five methods × three seeds on LoveDA segmentation and six methods × three seeds on FineWeb 124M (Appendices D.11–D.12, Tables 25–26). FineWeb screens four LRs with two selection seeds per method before 18 formal runs of 2.517B tokens each and separate final testing.

Further replication and implementation improvements

The full 300-epoch ImageNet comparison now covers three common seeds for all seven methods. Tables 2 and 13 report mean ± SE and all 21 endpoints. S2S reaches 78.24 ± 0.08%, versus 77.28 ± 0.05% projected Muon and 77.95 ± 0.08% SMET; dense Muon remains higher at 79.12 ± 0.17%. This strengthens the main result separately from the two-seed reference-budget study.

As discussed in our earlier supplementary response, the reviewers' efficiency questions, particularly Hau7's timing request, led us to identify overhead from separate NS calls for packed components. Batching equal-shaped components while retaining the update recipe and PP-linear kernels gives 1.95× whole-update throughput over component-wise execution (Appendix E.3). We then compared both backends with the same batched optimizer. Validation includes component/shared-gradient checks, checkpoint recovery, and 250-update trajectories (E.1).

After optimization, both PP-NM seeds were rerun from initialization at a newly calibrated 735,899 updates, reaching 57.542 ± 0.920% top-1 (Table 28); the other nine configurations retain their budgets/results. Additional evidence includes full-length PRE/amplitude controls on ImageNet and 350M LM with two paired seeds, a separately tuned five-configuration direction/amplitude study, connectivity/routing controls, and full-length NS-precision comparisons (Appendices D.2–D.6, D.15).

PP-NM Muon is, to our knowledge, the first N:M-sparse Muon optimizer preserving support closure throughout NS iterations, with S2S further improving its update. On ViT, PP-linear raises PP-NM/S2S training throughput by 23.0%/21.9% and lowers allocated memory by 16.4%/15.2%, versus their respective dense-mask implementations (Table 5). Both improve accuracy over projected Muon (Table 2). Native 2:4 inference gives 1.34× ViT throughput and 12.5% lower allocated memory than dense execution of identical sparse weights, with CUDA Graphs on both routes (Table 36).

With less than a day remaining, we hope these revisions address your concerns. Please raise any remaining questions before the deadline so we can respond promptly. We appreciate your positive assessment. If the added comparisons and improvements sufficiently strengthen the paper, we would be grateful if you would consider a higher score. Thank you again.

Reviewer_i15KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce an efficient Muon-like optimizer that obtains training-memory benefits by leveraging N:M sparsity. They propose using permutations to obtain a novel packing format that leverages block-diagonal elements which preserves sparsity after Muon update. They also implement fused kernels for channel indexing and matrix multiplication to obtained the reported memory savings.

Critical strengths

1/ The overall approach is reasonable and the implementation makes sense.

2/ The paper is a full package from (i) noting the lack of well designed sparse optimizers and highlighting the challenges with implementing them to (ii) proposing a sparse optimizer for Muon with memory training benefits.

3/ The experimental results are strong.

Critical weaknesses

1/ It's not clear that sparse second-order optimizers outperform dense first-order optimizers. I would appreciate more pointers from the authors about the usefulness of such approaches compared to say dense AdamW.

2/ The novelty can be broken down to (i) introducing permutations for N:M sparsity to obtain a block-diagonal representation, which has roughly appeared in papers such that [1] and (ii) noting the sparse-to-sparse update under this format, which is truly unique to sparse optimizers. I would appreciate more pointers from the authors on novelty in addition to the hardware implementation.

3/ This optimizer should be generalizable to other architectures. Why did the authors limit the title and experiments to ViTs?

4/ Do authors obtain runtime improvements in addition to memory footprint reduction?

[1] MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression. Lazar Supic, Rawan Naous, Ranko Sredojevic, Aleksandra Faust, Vladimir Stojanovic

Questions for authors

Please see the weaknesses section.

5 comments

Dear Reviewer i15K,

Thank you very much for your detailed and insightful review. We appreciate your constructive feedback regarding our comparison with dense baselines, methodological novelty, task scope, and runtime trade-offs. We have addressed your points below and incorporated new evaluation results (including a semantic segmentation benchmark and explicit inference throughput numbers) to clarify our approach.


Weakness 1 — Usefulness Compared with Dense AdamW

Reviewer's Comment: I would appreciate more pointers from the authors about the usefulness of such approaches compared to say dense AdamW.

Response: Thank you for this comment. We acknowledge that our sparse methods (PP-NM Muon and S2S Muon) do not match the accuracy of dense AdamW on ImageNet, and that a quality gap remains at language-model scales.

(19 Nov 2026 update: Under the calibrated 24-hour RTX 5090 reference budget, PP-NM Muon and S2S Muon reached 57.54% and 60.28% mean ImageNet top-1 accuracy, respectively, both above dense AdamW at 51.22% and below full-size dense Muon at 62.03% (revised Table 28).)

Our primary objective is to train highly accurate models under the hardware-supported 2:4 constraint, enabling their deployment with faster inference. While dense AdamW yields higher accuracy, it cannot leverage NVIDIA's hardware-accelerated 2:4 Sparse Tensor Cores. To demonstrate the value of our method within this hardware-supported format, we compare against Top2 AdamW (our 2:4 sparse AdamW baseline, where support remains fixed at initialization and AdamW optimizes the retained weights).

As shown below, our methods achieve significantly higher accuracy than Top2 AdamW under the exact same 2:4 hardware constraint, while simultaneously unlocking the inference speedups of native 2:4 execution.

ImageNet-1K Quality & RTX 5090 Inference

MethodEligible-Weight SparsityTop-1 Accuracy (%) ↑RTX 5090 Inference Throughput (Representative)*
Dense AdamW0%78.70±0.1578.70 \pm 0.151.00×1.00\times (Reference)
Top2 AdamW50% (2:4)76.41±0.1176.41 \pm 0.11≈1.34×\approx 1.34\times*
PP-NM Muon (Ours)50% (2:4)77.54±0.0577.54 \pm 0.05≈1.34×\approx 1.34\times*
S2S Muon (Ours)50% (2:4)78.24±0.0878.24 \pm 0.08≈1.34×\approx 1.34\times*

Quality: 300 ImageNet epochs, mean ±\pm SE over three training seeds. PP-NM Muon corresponds to the Muon+PP method in the manuscript.

*Inference Ratios: These are format-level representative values rather than per-method measurements. Because models trained with different optimizers share the same architecture, precision, and native 2:4 backend, they exhibit similar inference throughput. We apply a representative ratio: our ViT benchmark demonstrates 1.34×1.34\times native throughput relative to dense execution of the same weights on an RTX 5090 (batch 16, BF16, CUDA Graphs). The 1.00×1.00\times entry serves as this dense baseline rather than a separately timed Dense AdamW checkpoint.

Summary: Our contribution lies in improving model quality within a hardware-supported sparse format rather than claiming universal superiority over dense AdamW. We apologize that the reviewed manuscript omitted these inference results, and we will update the paper to explicitly clarify this quality-versus-inference-speed trade-off.


Weakness 2 — Novelty and MPDCompress

Reviewer's Comment: I would appreciate more pointers from the authors on novelty in addition to the hardware implementation.

Response: Thank you for pointing out MPDCompress; we will gladly include it in our related work discussion.

We agree that block-diagonal representations via permutations have precursors in the literature (e.g., MPDCompress). However, MPDCompress relies on randomly permuted block masks that do not guarantee local N:M counts in the original coordinates. In contrast, PP-N:M enforces strict local N:M balance: for NN dividing MM, it uses K=M/NK = M/N dense components where exactly NN indices from every consecutive group of MM are assigned to each component along both axes.

Our core technical novelty is satisfying this hardware constraint jointly with support preservation throughout Muon's Newton-Schulz (NS) transform. This avoids the off-support updates typically discarded by standard N:M projection, enabling:

  1. Active-coordinate Muon updates, and
  2. Compact storage for weights, gradients, and optimizer states throughout training.

While MPDCompress focuses primarily on compression and efficient execution, our goal is to preserve Muon's optimization effectiveness when training accurate N:M models for faster inference, with training-resource savings as an additional benefit. We view the structural block representation as a foundation, while the active-only matrix update recipe constitutes our primary methodological contribution.

Weakness 3 — Architectural and Task Scope

Reviewer's Comment: This optimizer should be generalizable to other architectures. Why did the authors limit the title and experiments to ViTs?

Response: Thank you for encouraging us to clarify and broaden our scope. Our title, "Permutation-Packable N:M Sparsity for Sparse-to-Sparse Muon," is not restricted to ViTs, and our manuscript already includes 350M and 1B LLaMA-style language models (Section 4.4).

To further demonstrate architectural generality, we have evaluated our method on semantic segmentation using LoveDA with a SegFormer-B0-sized model. Unlike standard ViTs for classification, SegFormer combines a hierarchical multi-scale Transformer encoder with a lightweight segmentation decoder, extending our evaluation to a dense prediction task. All sparse methods use identical architectures and trainable parameter counts (2,297,063), trained from scratch for 30,000 updates under a unified protocol.

Semantic Segmentation (LoveDA)

MethodEligible-Weight SparsityValidation mIoU (%) ↑
Dense AdamWDense (0%)38.440±0.21038.440 \pm 0.210
Dense MuonDense (0%)45.905±0.39945.905 \pm 0.399
Top2 AdamW2:4 (50%)39.618±0.35639.618 \pm 0.356
Projected Muon (row-wise Top2)2:4 (50%)42.638±0.42242.638 \pm 0.422
PP-NM Muon (Ours)2:4 (50%)44.855±0.47244.855 \pm 0.472

Mean ±\pm SE over three seeds. Dense references use the same architecture with 3,452,135 trainable parameters; parameter counts are matched across the three sparse methods. PP-NM Muon outperforms Projected Muon and Top2 AdamW by 2.22 and 5.24 percentage points, respectively. Learning rates were independently selected using our standard validation protocol.

Furthermore, we observe a consistent qualitative pattern across tasks: wherever the dense Muon–AdamW performance gap is larger, our PP-NM Muon gains over sparse baselines also increase:

TaskDense Muon − Dense AdamWPP-NM Muon (Ours) − Projected MuonPP-NM Muon (Ours) − Top2 AdamW
CIFAR-100+4.49 pp+3.89 pp+4.90 pp
ImageNet-1K+0.42 pp+0.26 pp+1.13 pp
LoveDA Segmentation+7.46 pp+2.22 pp+5.24 pp

This trend indicates that PP-NM effectively preserves Muon's optimization advantages across diverse architectures and tasks. We will integrate these segmentation results into the revised manuscript.


Weakness 4 — Runtime Improvements

Reviewer's Comment: Do authors obtain runtime improvements in addition to memory footprint reduction?

Response: Thank you for this great question. We find it helpful to clearly distinguish between training (using compact PP-linear operations) and inference (using the resulting hardware-legal 2:4 weights).

The table below reports both metrics on an RTX 5090:

RTX 5090: Training (PP-linear) vs. Inference (Native 2:4 Execution of PP-NM Weights)

(19 Nov 2026 update: The 0.85× ViT training throughput below compares the original PP-NM Muon with projected Muon. With PP-linear held fixed, batching Newton–Schulz calls made PP-NM Muon 1.95× as fast as its original implementation. Revised Table 5 compares training backends within each optimizer: PP-linear increases throughput by 23.0% for PP-NM Muon and 21.9% for S2S Muon over their respective dense-mask implementations.)

WorkloadTraining Throughput (PP-linear)*Inference Throughput (Native 2:4)*
ViT0.85±0.01×0.85 \pm 0.01\times1.34×1.34\times
1B Language Model0.96±0.00×0.96 \pm 0.00\times1.50×1.50\times

*Training Column: PP-NM Muon / Projected Muon throughput, mean ±\pm SE over three fresh processes (ViT uses micro/global batch 16/64; LM uses 4/32 with context 512). Measurements cover complete eager updates with synthetic inputs (10 warmup, 30 timed steps). A rounded SE of 0.00 denotes low variance rather than zero uncertainty.

*Inference Column: Steady-state native 2:4 / dense throughput ratio for identical PP-NM weights (median of three processes using CUDA Graphs). One-time weight conversion and graph capture are excluded.

Takeaway: While PP-NM Muon incurs a modest training overhead on current hardware configurations, native 2:4 inference achieves substantial speedups (1.34×1.34\times to 1.50×1.50\times) over dense execution of the same weights. Our primary motivation is providing a reliable training path for accurate N:M models to unlock these hardware-accelerated inference benefits, with training-memory savings acting as a valuable secondary advantage. We will add this discussion and table to the final paper.

We hope this response, along with our additional experiments and clarifications, addresses your concerns. We are more than happy to answer any further questions you may have, and we kindly ask that you consider raising your score if you find our revisions satisfactory. Thank you again for your valuable time and constructive feedback.

AuthorsOpenReview ↗

An update on the PP-linear throughput weakness you identified: PP-linear now delivers both training throughput and memory advantages on ViT (part1)

Thank you for asking whether our approach offers runtime improvements in addition to memory savings. Your concern, together with Reviewer Hau7's request for a fine-grained timing breakdown, directly motivated a concrete improvement to our ViT implementation. We would like to share the resulting measurements and explain how they strengthen the practical case for our methods.

The timing breakdown and subsequent paired experiment identified substantial removable overhead in the optimizer's Newton–Schulz (NS) execution. The original PP-NM update issued 144 independent NS calls per step (72 matrices × 2 packed components, with five iterations per call); the matrix-optimizer stage took 108.5 ms within a 225.4 ms update. We implemented a batched-NS variant of PP-NM that groups equal-shaped components along a batch dimension. The PP-linear forward/backward kernels remain unchanged, as do the BF16 NS coefficients, per-component normalization and epsilon, single-momentum Nesterov update, scaling, clipping and weight decay. Each component still undergoes its own matrix products and normalization, with no cross-component mixing.

The original component-wise implementation provided a direct correspondence to the reference update for verification. S2S's dedicated packed optimizer already grouped equal-shaped components, but its FP32 NS and pre-equilibration differ from PP-NM's recipe. We therefore transferred only the batching strategy, preserving PP-NM's numerical recipe. All training measurements below use eager execution, without CUDA Graphs or compilation.

Paired NS optimization with PP-linear held fixed

MetricOriginal component-wise NSBatched NS
Matrix optimizer (ms/update)108.482 ± 1.3072.214 ± 0.006
Whole update (ms/update)225.422 ± 0.126115.686 ± 0.372
Training throughput (images/s)283.912 ± 0.158553.235 ± 1.781
Peak allocated (GiB)0.82440.8203

These are mean ± SE over paired fresh-process runs on three RTX 5090 GPUs, with 50 warmup and 200 measured updates per process. Whole-update throughput is measured without stage instrumentation; matrix-optimizer intervals come from separate instrumented runs. The results show:

  • 1.95× whole-step training throughput, with a 98.0% reduction in the matrix-optimizer interval.
  • Preserved numerical behavior in the tested ViT setting: all 250 per-update losses and final complete model, optimizer and RNG states are bitwise identical between the original and batched versions; optimizer checkpoint save/resume also agrees exactly.
  • Retained allocated-memory savings: batched PP-NM uses 18.7% less peak allocated memory than the matched Projected Muon measurement. Its own peak allocated memory changes only slightly; peak reserved memory increases from 0.8691 to 0.8809 GiB.

Matched training comparison: dense+mask versus PP-linear

We also measured both execution backends under a common protocol. Both PP-NM rows use batched BF16 NS, and both S2S rows retain their existing FP32 grouped optimizer. Within each optimizer, architecture, initialization, support, inputs and update recipe are fixed; required gradient packing and weight writeback are included.

OptimizerTraining executionThroughput (images/s)Peak allocated (GiB)
PP-NM Muon, batched NSdense+mask450.22 ± 3.520.9807
PP-NM Muon, batched NSPP-linear553.71 ± 2.010.8203
S2S Muondense+mask400.69 ± 1.231.0219
S2S MuonPP-linear488.52 ± 1.750.8669

Thus, PP-linear delivers both higher training throughput and lower allocated memory for both optimizers in this ViT setting: throughput improves by 23.0% for PP-NM and 21.9% for S2S, while allocated memory falls by 16.4% and 15.2%, respectively, relative to their corresponding dense+mask implementations. This is a within-optimizer backend comparison, distinct from the PP-NM-versus-Projected-Muon comparison in our earlier response.

These measurements use ViT, GPU-resident 224×224 synthetic images, micro/global batch 16/64, BF16 forward/backward and FP32 master weights, gradients and state. Each configuration runs in a fresh process on each of three RTX 5090 GPUs, with 50 warmup and 200 measured updates; the second GPU reverses the order. The table reports mean ± SE for throughput, with identical memory peaks across repeats. Full measurement and validation details are provided in our supplementary response to Reviewer Hau7.

Reviewer_Hau7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Permutation-Packable N (PP-N) sparsity, a structured sparse representation that satisfies bidirectional N constraints and can be rearranged into independent dense matrix blocks through row and column permutations. The key theoretical property is that these supports are closed under the cubic matrix products used by Muon's Newton–Schulz iterations, allowing matrix-valued optimizer updates to operate exclusively on active coordinates without generating updates at pruned weights. The authors also propose Sparse-to-Sparse Muon (S2S Muon), a second-moment-based update magnitude control.

The method is evaluated on CIFAR-100 and ImageNet-1K using Vision Transformers and on OpenWebText using 350M and 1B-parameter language models. At 50% sparsity, S2S Muon achieves 78.21% ImageNet-1K top-1 accuracy, compared with 77.31% for projected Muon, and improves language-model perplexity relative to the corresponding sparse baselines.

Critical strengths
  1. The paper talks about a class of bidirectionally N sparse supports that can be packed into dense blocks and remain closed under the cubic matrix products used by Muon's Newton–Schulz iterations. This is interesting because conventional sparse supports can produce off-support updates that must subsequently be discarded. The proposed construction enables active-only matrix optimization and compact optimizer states, connecting sparsity structure with optimizer design rather than considering them independently.

  2. The authors evaluate the method on ImageNet-1K, CIFAR-100, and language modeling at 350M and 1B scales. S2S Muon improves ImageNet-1K accuracy by 0.90 percentage points over projected Muon at 50% sparsity and achieves improved perplexity on both language models. The implementation also demonstrates substantial peak GPU memory reductions, reaching 29.09% for the 1B language model when comparing identical S2S update rules. Importantly, the paper reports throughput alongside memory consumption, allowing the practical tradeoffs to be assessed rather than relying exclusively on theoretical FLOP reductions.

Critical weaknesses
  1. Although PP-N reduces the arithmetic complexity of the linear layers and Newton–Schulz updates, the proposed PP-linear kernels do not consistently improve end-to-end training throughput. (On RTX 5090, only 0.85× the throughput of projected Muon on ViT-S/16, while only 0.82× the throughput of its dense-masked counterpart on the 1B language model)

The authors fuse channel gathering and scattering into custom Triton kernels, eliminating explicit activation permutation buffers. Nevertheless, indexed memory access and reduced kernel efficiency may introduce significant overhead. The paper does not provide a sufficiently detailed breakdown isolating indexed-access costs, effective memory bandwidth, matrix-multiplication utilization, and other sources of inefficiency.

This is important because compact execution is a central motivation of the proposed representation. Without demonstrating when the approach produces net throughput benefits, or identifying the regimes in which permutation-related costs outweigh arithmetic savings, its practical advantage over existing sparse and dense implementations remains uncertain.

  1. The primary accuracy comparisons use matched training epochs, steps, or token budgets, while the throughput experiments demonstrate that PP-linear can be slower than the corresponding reference implementations. Improvements in accuracy at a fixed sparsity level do not necessarily translate into better accuracy under a fixed wall-clock training budget.

For instance, the 1B language-model implementation achieves only 0.82× the throughput of its dense-masked counterpart. At equal wall-clock time, the faster implementation could process approximately 22% more tokens, potentially changing the relative optimization outcomes.

An iso-latency comparison against dense Muon, projected Muon, and alternative N sparse implementations would clarify whether the proposed accuracy improvements compensate for indexing and kernel overhead. This limitation is consequential because the paper's practical motivation depends on improving the tradeoff among optimization quality, computational cost, and memory usage, rather than accuracy or memory in isolation.

Questions for authors

Q1 Although PP-linear fuses channel gathering and scattering into its Triton kernels, several configurations experience a throughput reduction relative to their references. Could the authors provide a more detailed latency breakdown isolating the costs of indexed channel access, forward and backward matrix multiplications, weight-gradient computation, and optimizer updates?

Q2 Given that the proposed packed implementation is slower in several configurations, could the authors provide accuracy or perplexity comparisons under matched wall-clock training budgets?

Q3 The paper employs a magnitude-guided balanced assignment heuristic to construct PP-N:M supports before training. Could the authors quantify the initialization-time cost of finding these assignments as model width and the number of layers increase?

6 comments

Dear Reviewer Hau7,

Thank you very much for your thorough, constructive, and detailed review. We truly appreciate your careful reading regarding our runtime trade-offs, wall-clock budget comparisons, and initialization costs. Below, we provide detailed clarifications and new measurement breakdowns to address each of your questions, all of which will be incorporated into the revised manuscript.


Weakness 1 / Question 1 — Training Throughput and Latency Breakdown

Reviewer's Comment: Although PP-linear fuses channel gathering and scattering into its Triton kernels, several configurations experience a throughput reduction relative to their references. Could the authors provide a more detailed latency breakdown isolating the costs of indexed channel access, forward and backward matrix multiplications, weight-gradient computation, and optimizer updates?

Response: Thank you for this valuable suggestion. We completely agree that reduced arithmetic and memory footprint do not automatically translate into faster wall-clock training, as hardware utilization and kernel overhead also play major roles.

To provide a transparent breakdown, we organize our analysis around the RTX 5090 training throughput measurements reported in the paper (retaining the exact models, batch sizes, numerical precision, and implementations).

1. Training Throughput Summary (RTX 5090)

(19 Nov 2026 update: The 0.85× ViT training throughput below compares the original PP-NM Muon with projected Muon. With PP-linear held fixed, batching Newton–Schulz calls made PP-NM Muon 1.95× as fast as its original implementation. Revised Table 5 compares training backends within each optimizer: PP-linear increases throughput by 23.0% for PP-NM Muon and 21.9% for S2S Muon over their respective dense-mask implementations. See the ViT follow-up, Part 1, in this thread.)

WorkloadComparison: PP-linear / ReferenceReference ThroughputPP-Linear ThroughputPaper RatioRemeasured Ratio
ViTPP-NM Muon (Ours) / Projected Muon325.08±1.27325.08 \pm 1.27275.56±5.81275.56 \pm 5.810.85×0.85\times0.87×0.87\times
ViTS2S Muon (Ours): PP-linear / Dense+mask370.22±7.15370.22 \pm 7.15469.45±3.88469.45 \pm 3.881.27×1.27\times1.31×1.31\times
1B LMPP-NM Muon (Ours) / Projected Muon9,551.63±4.909,551.63 \pm 4.909,197.47±19.929,197.47 \pm 19.920.96×0.96\times0.96×0.96\times
1B LMS2S Muon (Ours): PP-linear / Dense+mask10,186.13±0.7510,186.13 \pm 0.758,364.15±5.708,364.15 \pm 5.700.82×0.82\times0.82×0.82\times

Throughput: Images/s for ViT and tokens/s for LM, mean ±\pm SE over three fresh processes. The remeasured column confirms that uninstrumented runs reproduce the original trends. (Note: PP-NM Muon is labeled Muon+PP in the paper).

2. Detailed Stage Timings (ms/update)

ViT — Stage Timing Breakdown

ImplementationForwardBackwardExternal Gather/ScatterMatrix OptimizerOther + ResidualWhole Update
Projected Muon34.56±0.9434.56 \pm 0.9461.43±1.4961.43 \pm 1.49—105.03±2.64105.03 \pm 2.642.91±0.102.91 \pm 0.10203.93±5.15203.93 \pm 5.15
PP-NM Muon (Ours, PP-linear)28.02±0.3828.02 \pm 0.3885.72±0.1185.72 \pm 0.11—115.61±3.61115.61 \pm 3.613.46±0.173.46 \pm 0.17232.83±4.20232.83 \pm 4.20
S2S Muon (Ours, dense+mask)36.92±1.1936.92 \pm 1.1965.03±1.4665.03 \pm 1.4642.81±1.8342.81 \pm 1.8323.80±0.8623.80 \pm 0.865.39±0.195.39 \pm 0.19173.95±5.29173.95 \pm 5.29
S2S Muon (Ours, PP-linear)27.57±0.0427.57 \pm 0.0485.66±0.1385.66 \pm 0.13—19.49±0.8519.49 \pm 0.853.34±0.073.34 \pm 0.07136.07±1.06136.07 \pm 1.06

1B LM — Stage Timing Breakdown

ImplementationForwardBackwardExternal Gather/ScatterMatrix OptimizerOther + ResidualWhole Update
Projected Muon302.90±0.48302.90 \pm 0.48635.33±0.15635.33 \pm 0.15—772.54±0.84772.54 \pm 0.8410.84±0.1110.84 \pm 0.111,721.61±1.121,721.61 \pm 1.12
PP-NM Muon (Ours, PP-linear)419.14±0.12419.14 \pm 0.121,092.04±0.041,092.04 \pm 0.04—274.61±1.94274.61 \pm 1.9410.25±0.0210.25 \pm 0.021,796.04±2.031,796.04 \pm 2.03
S2S Muon (Ours, dense+mask)331.49±0.14331.49 \pm 0.14670.31±0.16670.31 \pm 0.16160.28±0.18160.28 \pm 0.18442.28±0.18442.28 \pm 0.188.74±0.018.74 \pm 0.011,613.09±0.341,613.09 \pm 0.34
S2S Muon (Ours, PP-linear)419.13±0.03419.13 \pm 0.031,091.84±0.061,091.84 \pm 0.06—442.29±0.22442.29 \pm 0.2210.45±0.0510.45 \pm 0.051,963.72±0.281,963.72 \pm 0.28

Takeaways from the Breakdown:

  • PP-NM Muon on ViT: Forward saves 6.5 ms, but backward adds 24.3 ms and the matrix optimizer adds 10.6 ms, yielding a 28.9 ms longer update because NS is invoked separately on each component per matrix.
  • S2S on ViT: Eliminating external shadow gather/scatter saves 42.8 ms and the optimizer saves 4.3 ms, outweighing the +11.3 ms+11.3\text{ ms} forward/backward cost for a 37.9 ms faster overall update.
  • S2S on 1B LM: The backward cost (+509.2 ms+509.2\text{ ms}) exceeds the external gather/scatter savings (160.3 ms160.3\text{ ms}), leading to the 0.82×0.82\times throughput.

3. Packed-Kernel Profiling & Indexed-Access Diagnostic

Isolating the packed GPU kernels from profiler traces (mean ±\pm SE in ms/update):

  • ViT-S/16 (PP-NM / S2S): Forward kernels take 17.729/17.709 ms17.729 / 17.709\text{ ms}; input-gradient kernels take 16.127/16.132 ms16.127 / 16.132\text{ ms}; weight-gradient kernels take 55.588/55.629 ms55.588 / 55.629\text{ ms}.
  • 1B LM (PP-NM / S2S): Forward kernels take 336.502/336.834 ms336.502 / 336.834\text{ ms}; input-gradient kernels take 501.359/502.108 ms501.359 / 502.108\text{ ms}; weight-gradient kernels take 389.412/390.312 ms389.412 / 390.312\text{ ms}.

Comparing original indexed layouts against index-free contiguous layouts across seven projection shapes (under warmed CUDA Graph replay) isolates the indexed-access difference:

  • ViT: Forward: 33.8–108.8 μs33.8\text{--}108.8\ \mu\text{s}; input-gradient: 24.6–89.7 μs24.6\text{--}89.7\ \mu\text{s}; weight-gradient: 114.4–114.7 μs114.4\text{--}114.7\ \mu\text{s}.
  • 1B LM: Forward: 53.5–340.6 μs53.5\text{--}340.6\ \mu\text{s}; input-gradient: 95.5–374.0 μs95.5\text{--}374.0\ \mu\text{s}; weight-gradient: 57.3–422.4 μs57.3\text{--}422.4\ \mu\text{s}.

(We regret that our rented cloud servers do not expose hardware performance counters for direct DRAM bandwidth and Tensor Core utilization measurements, so we rely on the timing and profiler trace evidence above). We will incorporate these complete latency breakdowns and per-shape diagnostics into the revised appendix.


Weakness 2 / Question 2 — Quality Under a Matched Training-Time Budget

Reviewer's Comment: Given that the proposed packed implementation is slower in several configurations, could the authors provide accuracy or perplexity comparisons under matched wall-clock training budgets?

Response: Thank you for this important suggestion. We agree that accuracy at equal epochs does not fully establish performance at equal training time. We have completed an ImageNet comparison across ten configurations (two seeds each), assigning method-specific update counts calibrated from RTX 5090 reference throughput.

Joint ImageNet Quality–Time–Memory Comparison (RTX 5090)

(19 Nov 2026 update: The PP-NM Muon row below reports our original result of 56.185% at 546,232 updates per seed. Following Hau7’s timing question, we batched the Newton–Schulz calls, recalibrated the 24-hour reference budget on real ImageNet inputs, and retrained both seeds from scratch. PP-NM Muon reached 57.542% mean top-1 at 735,899 updates per seed. See revised Appendix D.14, Table 28.)

Method / ExecutionEligible-Weight SparsityReference Time Budget / BasisUpdates per SeedTop-1 Accuracy (%) ↑Eager Peak Allocated (GiB) ↓Eager Peak Reserved (GiB) ↓
Dense AdamWDense (0%)24 h872,28351.221±0.26751.221 \pm 0.2671.0271.068
Dense MuonDense (0%)24 h537,47662.025±0.31162.025 \pm 0.3110.9490.980
Top2 AdamW2:4 (50%)24 h706,63444.936±0.17844.936 \pm 0.1781.0681.119
Projected Muon2:4 (50%)24 h615,51248.803±0.28748.803 \pm 0.2870.9911.014
PP-NM Muon (Ours, PP-linear)2:4 (50%)24 h546,23256.185±0.33356.185 \pm 0.3330.8060.852
S2S Muon (Ours, dense-mask)2:4 (50%)24 h490,20959.082±0.43459.082 \pm 0.4341.0081.084
S2S Muon (Ours, PP-linear)2:4 (50%)24 h623,49460.284±0.61660.284 \pm 0.6160.8450.930
Adapted SLoPe (w/ adapters)2:4 backbone + dense adapters24 h736,40444.525±0.10144.525 \pm 0.1011.3071.400
Smaller Dense AdamWDense (0%)24 h1,510,61347.925±0.15147.925 \pm 0.1510.5730.598
Smaller Dense MuonDense (0%)24 h993,55559.303±0.37359.303 \pm 0.3730.5330.545

Key Findings: Under these calibrated budgets, PP-NM Muon reaches 56.185%56.185\%, outperforming Projected Muon (48.803%48.803\%) by +7.38 pp despite having fewer updates. (19 Nov 2026 update: “Fewer updates” refers to the original budget. In revised Table 28, PP-NM Muon reaches 57.542% at 735,899 updates versus 48.803% at 615,512 updates for projected Muon, a lead of 8.739 percentage points.)Complete S2S with PP-linear reaches 60.284%60.284\%, surpassing dense-mask S2S (59.082%59.082\%), adapted SLoPe (44.525%44.525\%), and smaller dense Muon (59.303%59.303\%), demonstrating a favorable quality–memory–time trade-off. We will add this time-budgeted comparison and protocol description to the revised manuscript.

Question 3 — Cost and Necessity of Magnitude-Guided Initialization

Reviewer's Comment: The paper employs a magnitude-guided balanced assignment heuristic to construct PP-N:M supports before training. Could the authors quantify the initialization-time cost of finding these assignments as model width and the number of layers increase?

Response: Thank you for suggesting this clarification. Our selector favors large retained squared weight magnitude under PP balance constraints without searching over all possible N:M masks. For an initial weight matrix W∈Rm×nW \in \mathbb{R}^{m \times n} under 2:4 sparsity:

  1. It evaluates the three complementary column pairings (01∣23)(01\mid23), (02∣13)(02\mid13), and (03∣12)(03\mid12) across groups of four columns.
  2. For each pairing, it computes each row's retained-energy difference Δr=∑j∈C0Wrj2−∑j∈C1Wrj2\Delta_r = \sum_{j \in C_0} W_{rj}^2 - \sum_{j \in C_1} W_{rj}^2 and assigns the top two rows per group of four to component 0 (and the rest to component 1).
  3. It selects the candidate pairing maximizing total retained energy, which then stays fixed throughout training.

Algorithmic Complexity

Because there are only three candidates and row sorts have length 4, energy accumulation and scoring require O(mn)O(mn) operations per matrix, with no full-matrix sorting or iterative optimization. Across LL square projections of width dd, the total complexity is O(Ld2)O(Ld^2)—linear in the number of projections and quadratic in model width. Compared to a single forward matrix multiplication (O(Tmn)O(Tmn) for TT tokens), the selection lacks the token factor TT and is paid only once at initialization.

Measured Initialization Costs on RTX 5090 (Unified in Seconds)

ModelEligible Projection MatricesMagnitude Selection Only (s)Complete Weight/Support Preparation (s)
ViT-S/16720.089 (0.086–0.089)0.718 (0.696–0.748)
350M LM1960.291 (0.288–0.293)4.195 (4.110–5.315)
1B LM1540.289 (0.287–0.291)10.708 (10.589–13.332)

Values are median (min–max) over five repetitions. Notably, the 1B model's 10.708 s total preparation time is dominated by weight generation, transfers, RMS restoration, and cache construction—the magnitude selection stage itself takes only 0.289 s across all 154 matrices.

Width & Layer Scaling Sweep: Empirical scaling sweeps confirm practical efficiency: for 12 retained square projections, increasing width from 256256 to 40964096 changes median warmed whole-preparation time from 0.028 s0.028\text{ s} to 1.076 s1.076\text{ s}. At a fixed shape of 2816×10242816 \times 1024, increasing projection count from 66 to 4848 changes preparation time from 0.086 s0.086\text{ s} to 0.696 s0.696\text{ s}.

Is Magnitude Guidance Necessary?

Magnitude guidance is optional. Random locally balanced assignments preserve PP legality, exact packing, and NS support closure identically. In a 30-epoch ImageNet control:

  • Magnitude-guided PP: 67.98±0.11%67.98 \pm 0.11\% Top-1
  • Random balanced PP: 67.85±0.00%67.85 \pm 0.00\% Top-1

We will incorporate this complexity analysis, unified timing table, scaling sweep, and random-initialization comparison into the revised paper, clarifying that magnitude guidance is a lightweight, optional heuristic rather than a strict requirement.

We hope this response, along with our new stage-level latency breakdowns, budget-matched comparisons, and initialization complexity analyses, thoroughly addresses your concerns. We are happy to answer any further questions you may have, and we kindly ask that you consider raising your score if you find our revisions satisfactory. Thank you again for your valuable time and constructive feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.