acceptodds
Under review as a conference paper at ICLR 2027

Towards 1:4 semi-structured 75% sparsity via Cannistraci-Hebb N:M dynamic sparse-to-sparse training

Abstract

Improving the computational and energy efficiency of large language models motivates semi-structured N:M sparsity, but preserving model quality becomes difficult when moving from 2:4 (50%) to 1:4 (75%) sparsity. We propose CHTsNM, a sparse-to-sparse pretraining framework that combines Cannistraci–Hebb topology evolution, topology-aware optimization, and contextually modulated low-rank compensation. CHTsNM directly trains the executed sparse backbone while evolving legal row-wise or transposable block-wise supports. Its Topology-Aware Nesterov Optimizer (TANO) stores moments on active coordinates, retains the history of surviving connections, and resets newly activated states. Contextually Modulated LoRA (CoMoLoRA) supplies input-adaptive residual directions through rank-space modulation. These components address connectivity, optimization, and the capacity lost under a sparse constraint. In the completed two-seed LLaMA main study, CoMoLoRA reduces mean perplexity in all twelve auxiliary/no-auxiliary configuration pairs, with larger gains at 1:4 than at 2:4. At 350M and 1:4, the gains are 2.124 and 2.423 perplexity points for row-wise and block-wise supports. Matched 130M controls show complementary benefits from TANO and residual compensation, while packed FP32 moments reduce eligible-matrix state storage by 75%. These results support combining effective optimization, compact optimizer state, and residual compensation as a practical route toward high-quality 1:4 sparse pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Semi-Structured Sparsity, Dynamic Sparse Training, Contextual Modulation branch, Low rank branch, Optimizer

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_M47nOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper deals with semi-structured sparsity, specifically 1:4 in LLMs. Authors propose CHTsNM, a sparse-to-sparse pretraining framework combining Cannistraci-Hebb topology evolution, topology-aware optimization, and contextually modulated low-rank compensation. Furthermore, the paper introduces Topology-Aware Nesterov Optimizer (TANO), and Contextually Modulated LoRA (CoMoLoRA). CoMoLoRA reduces perplexity in twelve settings.

Critical strengths
  • The paper is well motivated
  • The paper achieves improvement over baselines, and tries to cover both improvement in accuracy and performance on hardware
Critical weaknesses
  • Unclear memory and speed benefits: The paper does not clearly explain what the claimed 75% memory reduction includes or what it is compared against. It is also unclear how 1:4 sparsity is implemented and how the reported speedups are measured.

  • Insufficient evidence from the experiments: The paper should compare against SMET and GaLore under the same memory budget. It is also unclear whether 20K updates are enough to draw conclusions about final model quality, and the number of training tokens should be reported. The larger-model results in Table 6 use 2:4 rather than the proposed 1:4 pattern, leaving its performance at larger scales unclear. These gaps make it difficult to judge whether the method improves on existing methods.

Questions for authors

Clarity

“moments reduce eligible-matrix state storage by 75%” — Is this true, and which data format is used to achieve this reduction? How is it measured? What is the 75% reduction relative to: the total memory occupied by the weights or something else? Is index overhead included? Are activations taken into account? The activations are dense, correct?

The Cannistraci–Hebb family should be explained before it is mentioned, rather than assuming that the reader knows what it is. Could you clarify it?

CoMoLoRA — LoRA usually involves a frozen backbone. Is that the case here, or are you referring to methods that replace a dense matrix with a low-rank matrix without retaining a frozen full matrix?

“N active entries in every column of each M×M block” — For clockwise 1:4, does this mean that 1 out of 16 elements is nonzero, or that 4 out of 16 are nonzero?

A better explanation of the baselines would be helpful. Even a one-sentence explanation of each baseline in the main text would improve the reader’s understanding of the paper. For example, what is STE?

Soundness

2:4 sparsity is directly supported by NVIDIA hardware, but how is 1:4 implemented? This is not trivial, as there is no direct hardware support. Does this assume that a hardware vendor will follow the trend and develop hardware that supports it? Alternatively, is a specialized kernel implemented? Does your method outperform the approach in the “VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores” (https://arxiv.org/abs/2310.02065), which includes a low-level kernel implementation? How do your speedups compare to those of VENOM?

Why are newly activated coordinates initialized to zero? SMET (https://arxiv.org/abs/2606.00888) has shown that this causes spikes in the loss function when training with DST. Does the same hold in your setting? Why are the moments initialized to zero for new weights? Is there any evidence that this is superior?

Where does Equation 7 come from? Is it based on previous research, or was it selected arbitrarily? Is it a contribution of this paper?

In general, how does your method compare to SMET or GaLore (https://arxiv.org/abs/2403.03507) under the same memory budget?

Aren’t 20K updates too few to observe convergence? How many tokens are used for training in Table 1? This is what is usually reported, for example, in the GaLore paper.

In Table 1, how is it possible that 2:4 outperforms the dense baseline? This does not seem to make sense. How many tokens were the models trained on? If trained to convergence, I would expect the dense baseline to perform better.

What is the motivation behind clockwise 1:4? As far as I know, this pattern cannot achieve speedups even with the latest hardware. Don’t blockwise kernels for Tensor Cores still need to fill in the zeros within the blocks passed directly to Tensor Cores for MMA operations?

In Table 6, why not show 1:4 instead of 2:4 if the paper focuses on 1:4 rather than 2:4? Does this mean that 1:4 does not scale well to larger models?

In Section 4.6, how does the kernel achieve a speedup with 1:4? What exactly are you measuring when referring to inference? Is this just prefill at a particular batch size, i.e., a simple forward pass? Is any decoding performed?

4 comments
AuthorsOpenReview ↗

Comment (1/4) for Reviewer M47n

Dear Reviewer, thank you for this valuable feedback. In the revised manuscript, we will reorganize the paper to improve readability and balance with a streamlined Methodology section, an expanded Experimental Setup section, and new figures produced during this rebuttal that illustrate different aspects of CHTsNM's performance in comparison to existing methods.

C1: We agree that these claims need distinct accounting. The 75% reduction applies to the first- and second-moment buffers of eligible matrices at 1:4 density, not total training memory. Likewise, sparse connectivity does not by itself imply faster hardware execution. We now distinguish packed-state storage (Q1), the training implementation and observed runtime (Q6), and measured 2:4 versus estimated 1:4 inference acceleration (Q14). This separates the demonstrated benefits from hardware-dependent projections.

C2. In response to this valuable feedback, we have strengthened evidence with equal-budget learning-rate sweeps, a longer SMET-style comparison with measured peak memory, and additional billion-parameter 1:4 experiments. The revisions distinguish performance at a fixed training budget from convergence, and comparable measured memory from an exactly matched memory constraint. (Q9) reports the SMET comparisons and the remaining GaLore limitation; (Q10) gives the training-token budgets; (Q13) presents the new 1B and 7B evidence. These additions support the practical value of CHTsNM.

Q1. Yes. Apologies for the confusion, there is no reduction in total GPU memory; the 75% saving follows directly from storing only active-coordinate moments. For an eligible matrix with D coordinates, two dense FP32 moment buffers occupy 8D bytes. Physically packing moments for the D4 active coordinates reduces this to 2D bytes: a 75% reduction in those moment buffers. We measure this from allocated tensor sizes; masking dense buffers alone would not achieve it. This saving follows from the density and is not claimed as an independent methodological innovation.

The 75% excludes coordinate metadata, weights, gradients and activations. Table 4 separately includes uint8 coordinate metadata and the moments of other parameters, giving a 49.5% reduction in the reported optimizer-state-plus-metadata allocation. Activations remain dense; neither percentage is a claim about total GPU memory.

Q2: We will define the Cannistraci–Hebb family at first mention in our revision. These dynamic sparse training methods learn network connectivity by periodically removing connections and regrowing others using scores derived from the local connectivity structure. In CHTsNM, soft magnitude-based removal and Cannistraci–Hebb-guided regrowth operate within legal N:M sparsity patterns, a topology mechanism inherited from CHTs24.

Q3: CoMoLoRA jointly trains the sparse backbone and an additive low-rank branch. The backbone is neither frozen nor replaced by a low-rank matrix. Unlike standard LoRA fine-tuning, it supplies additional capacity during sparse pretraining, with input-dependent modulation of the low-rank features. Its parameters and computation are additional to the N:M sparse backbone.

Q4: Interpreting “clockwise” as “block-wise”, it means four active entries out of sixteen. Specifically, in each 4 x 4 block, there are 4 active entries—with exactly one in every row and every column—resulting in a density of 25% (75% sparsity). We will make this explicit in the pattern definition.

Q5: We agree and will add a short definition and implementation reference for each baseline in the main text. STE (straight-through estimator) applies an N:M mask during the forward pass and uses a surrogate gradient through the discrete masking operation during backpropagation. We will also distinguish mask-projection methods from explicit topology-regrowth methods, and label our SMET-style adaptations separately from the original published method.

AuthorsOpenReview ↗

Comment (2/4) for Reviewer M47n

Soundness

Q6: Our 1:4 training enforces legal masks on dense weight tensors; it does not depend on a custom NVIDIA 1:4 kernel. The quality results therefore do not assume future hardware support. Packed optimizer-state storage can reduce state memory independently of whether the matrix multiplication uses sparse hardware.

Figure 1 reports observed total runtime, including initialization and evaluation, rather than a native 1:4 kernel speedup. For example, at 350M row-wise 1:4 and LR 0.004, TANO reaches PPL 21.58 in 321.73 minutes, compared with SR-STE’s 21.83 in 358.89 minutes on H200. These are single-seed, implementation-level observations; they do not isolate the cause of the runtime difference or establish a universal speed advantage.

Figure 1: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig1_Quality_vs_Time_60_130_350M.png

VENOM uses a different V:N:M representation and specialized kernels to exploit NVIDIA Sparse Tensor Cores. We have not performed a matched kernel-level comparison. Separate inference evidence and the scope of the vendor compiler estimates are specified in (Q14).

Q7: Zero initialization avoids adding an arbitrary weight contribution at the instant of regrowth. It does not guarantee a smooth refresh, because removing connections can still perturb the network. New-coordinate moments are reset because there is no continuing active-state history to preserve; retained coordinates keep their coordinate-consistent states. This is a state-consistency rationale, not proof that zero moments are universally optimal.

We directly tested zero versus historical-value regrowth, with and without a 10-update SMET-style warmup, on LLaMA-130M over three seeds (Figure 4). At row-wise 2:4, zero initialization without warmup gives the lowest mean final PPL, 26.330, compared with 26.426 for zero plus warmup. At row-wise 1:4, zero plus warmup instead improves the mean from 30.322 to 30.084. Thus, the benefit of warmup depends on the setting; the results do not support blanket superiority of zero initialization without warmup. Figure 4: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig4_CHTsNM_TANO_Regrowth_Initialization_SE.png (Q9) reports a broader comparison with SMET-style Adam adaptations. Whether refresh-induced loss spikes are absent or isolate the effect of moment initialization will be further studied.

Q8: Equation 7 is inspired by the input-dependent contextual modulation in TRANSPONDER (Zhang et al., 2025b). CoMoLoRA adapts this principle to low-rank features for sparse-backbone compensation. The gate 1+G(x) bounds modulation around one, while zero-initializing the controller’s output layer makes the branch begin as ordinary additive LoRA. Our contribution is this adaptation, rather than contextual modulation itself. We will clarify the attribution and design rationale in revision.

Reference: From Transformer to Transponder: Introducing Contextual Modulation Training for Residual Learning in LLMs. URL: https://www.preprints.org/manuscript/202506.0120

Zhang, Y.; Gu, W.; Hu, W.; Li, J.; Cannistraci, C. V. From Transformer to Transponder: Introducing Contextual Modulation Training for Residual Learning in LLMs. Preprints 2025, 2025060120. https://doi.org/10.20944/preprints202506.0120.v2

AuthorsOpenReview ↗

Comment (3/4) for Reviewer M47n

Q9: Compared against CHTsNM Adam SMET over the same settings, CHTsNM TANO achieves a lower mean (three seeds) PPL in 15 of 16 matched-LR comparisons at 60M and 13 of 16 at 130M. At each method’s best tested LR, CHTsNM TANO outperforms CHTsNM Adam SMET (the best SMET-variation) in six of eight configurations, trailing two 130M row-wise settings. With CoMoLoRA, CHTsNM TANO achieves the best PPL across all eight combinations.

(Figure 2) illustrates comparison between CHTsNM TANO and different SMET-variations. On a LLaMA-130M, 20K-step experiment with two paired seeds, we compared TANO, CHT+Adam, Adam+SMET with random regrowth, and CHT+Adam+SMET. At similar VRAM memory budgets of 14.41–14.70 GiB, TANO reaches lower mean PPL of 22.18 versus 22.31 for CHT+Adam+SMET at row-wise 2:4. TANO’s total runtime was 1.1% and 1.6% faster. This supports competitive quality and runtime at comparable measured memory.

Figure 2: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig2_TANO_SMET__Quality_Time.png

For block-wise (1:4 & 2:4) configurations, CHTsNM TANO achieves lower held-out PPL than CHTsNM ADAM SMET at each method’s best tested learning rate across all three model sizes. Results are averaged over three seeds for 60M and 130M, with one seed available for 350M:

Figure 5: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig5_bw_TANO_vs_SMET.png

Among row-wise configurations, CHTsNM ADAM SMET achieves lower mean PPL at 130M with LR 0.004 than CHTsNM TANO at its best tested learning rate. CHTsNM TANO, however, achieves lower best-observed PPL in all learning rates, both configurations at 60M and 350M. Results are averaged over three seeds for 60M and 130M, and one seed for 350M:

Figure 6: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig6_rw_TANO_SMET.png

Q10: We agree that <=20K training steps may not establish convergence. Table 1 compares methods at fixed training budgets: approximately 655M token slots for 60M/130M and 1.31B for 350M, using global batch 256 and sequence length 256. These counts include padding; the separate 100M-token budget is held-out evaluation. We will report these budgets explicitly and frame the results as performance at the tested training horizons, rather than fully converged model quality. A longer 50K-update 1B experiment is presented in (Q13).

Q11: We found strong discoveries after extending a Learning Rate sweep. We are thankful to the reviewer as we have now developed dense TANO (which did not exist before) which provides evidence to CHTsNM TANO's Epitopological advantage over Adam (more in Revision). As shown in (Figure 7), dense TANO sets a stronger (lower-PPL) baseline, which substantially benefits 2:4 sparse TANO configurations, allowing them to outperform dense Adam at certain learning rates:

Figure 7: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig7_DenseAdam_vs_Sparse60_130_350M.png

Q12: Interpreting “clockwise” as block-wise 1:4, the motivation is to preserve a legal sparse pattern under transposition: each 4×4 block contains one active entry per row and column (allowing the same support structure in both forward and backward operations). The review is correct that this does not guarantee hardware acceleration. Ordinary block-sparse kernels using dense MMA indeed still compute zeros inside retained blocks; our pattern is sparsity within blocks, rather than removal of entire blocks. Our measured NVIDIA inference results concern 2:4; the reported 1:4 speedups are MOFFET software simulated estimates, not device measurements. We will clarify this distinction and the structural motivation in the revision.

Q13: Thank you for your suggestion. We now include LLaMA-1B block-wise 1:4 experiments: at 50K steps, TANO achieves a held-out PPL 1.1% better (lower) than SR-STE. These experiments extend our 1:4 evaluation to billion-parameter training, demonstrating that block-wise 1:4 CHTsNM remains competitive with SR-STE at the 1B scale while delivering modest measured throughput and memory gains. (we will conduct more seeds in the following two weeks)

Figure 3: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig3_50k_1B_SRSTE_TANO.png

For LLaMA-7B, we added 1:4 retraining after SparseGPT pruning with knowledge distillation (Figure 8). After 2,600 updates, CHTsNM reaches WikiText-2/OpenWebText PPL 16.669/13.678, compared with 110.316/408.032 immediately after 1:4 pruning. This demonstrates substantial quality recovery through additional training, rather than an equal-compute comparison with pruning alone. The result remains below dense-model quality and is distinct from the 1B pretraining experiment.

Figure 8: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig8_Table8_7B_Retraining_After_Pruning.png

AuthorsOpenReview ↗

Comment (4/4) for Reviewer M47n

Q14: To clarify— the Moffett 1:4 results are compiler-estimated full-forward/prefill latencies, not device measurements. The compiler schedules active weights in eligible operations, reducing weight traffic and multiplication. Tests use BF16, sequence lengths 128/256, and batch one per core across four cores. At length 256, estimated backbone speedups are 2.67×/2.89× for 1B/7B, or 2.40×/2.69× with CoMoLoRA (Figure 9). Separately, (Figure 10) distinguishes measured 2:4 from modelled 1:4 RTX 5080 latencies at batch eight and sequence length 256, citing nmSPARSE as the N:M kernel reference [1]. Neither experiment includes autoregressive decoding or KV-cache-based generation. We will clarify these scopes in Section 4.6.

[1] nmSPARSE — reference method and replication target. Bin Lin, Ningxin Zheng, Lei Wang, Shijie Cao, Lingxiao Ma, Quanlu Zhang, Yi Zhu, Ting Cao, Jilong Xue, Yuqing Yang, and Fan Yang. Efficient GPU Kernels for N:M-Sparse Weights in Deep Learning. Proceedings of Machine Learning and Systems, volume 5, 2023.

Figure 9: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig9_MOFFET_Compiler_Inference.png

Figure 10: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig10_Measured_2to4_Modelled_1to4_Inference.png

Reviewer_8FR2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes CHTsNM, a sparse-to-sparse pretraining recipe aimed at moving N:M semi-structured sparsity from 2:4 to 1:4 (75%). It keeps the CHTs24 Cannistraci-Hebb topology evolution (soft magnitude removal plus CH-score legal regrowth, row-wise or transposable block-wise masks, and an optional "motif pattern revisitation" refinement).

Critical strengths

The 1:4 numbers, if they hold under fair baselines, are notable. At 350M, row-wise 1:4 CHTsNM+CoMoLoRA reaches 22.45 PPL (Table 1), lower than every 2:4 baseline other than CHTsNM itself (the best being CAST-style row 2:4 at 22.96 and SR-STE at 22.98), at half the backbone density plus a ~1% dense branch. The 2×22\times 2 ablation (Table 2) suggests the optimizer and residual gains are largely complementary (5.0% and 8.7% individually, 12.4% combined).

Critical weaknesses

The presentation of this manuscript: difficult to follow despite careful reading. The method takes about two pages and the experimental setup a single paragraph; most training details, baseline provenance and evaluation settings are only in Appendix A. The 17-page submission contains no figures.

Other weaknesses:

  1. The headline gains look like a generic optimizer/recipe effect rather than a sparse-specific one, and baselines are not tuned equally. Without any branch, 2:4 CHTsNM beats the dense Adam model at 60M and 130M (e.g., 26.43 vs. 26.58, Table 1) and on GPT-NeoX (27.06 vs. 27.85, Table 7), which suggests under-optimized baselines rather than a benefit of sparsity. The CHTs24 row in Table 1 is about 2 PPL worse than Adam on the same sparse backbone in Tables 2-3, without explanation. Baselines use published configurations "without additional per-method tuning" (Sec. 4), while the TANO recipe is the authors' own, and at the headline row 1:4 setting the SMET-style adaptation even has a lower mean than TANO (Table 3). Without a dense-TANO baseline and an equal-budget sweep for STE/SR-STE/Adam, one cannot tell how much of the 1:4 improvement comes from sparse training, which is the paper's central claim.

  2. Evidence for 1:4 is thin, and the experiments are under-described and give little insight. All 1:4 pretraining stops at 350M and 20K updates; the 1B and 7B studies are 2:4 only. CHTsNM uses two seeds with SDs up to 1.42, so some of the "twelve" improvements overlap (e.g., 31.00±0.4231.00\pm0.42 vs. 30.48±0.7830.48\pm0.78). Only perplexity is reported, and the 1:4 speedup is a compiler estimate on a single accelerator. The setup is one paragraph that defers every recipe to the appendix, each results subsection mostly restates its table, and the paper has no figures (no training curves, no topology or gate analysis). It is not shown why TANO helps (App. B.5 states its ingredients are never separated) or what CoMoLoRA's modulation learns; at the 1% budget used throughout, plain additive branches match or beat it (Table 5).

Questions for authors
  1. Have you trained a dense model with the TANO recipe (same lr, no bias correction, BF16 without master weights, shape scaling) at 60M/130M/350M, and what PPL does it reach? If TANO improves dense by a similar margin, the sparse-vs-dense gap at 1:4 is larger than Table 1 implies.
2 comments
AuthorsOpenReview ↗

Comment (1/2) for Reviewer 8FR2

C1: Dear Reviewer, thank you for this valuable feedback. In the revised manuscript, we will reorganize the paper to improve readability and balance with a streamlined Methodology section, an expanded Experimental Setup section, and new figures produced during this rebuttal that illustrate different aspects of CHTsNM's performance in comparison to existing methods.

C2: Across row-wise and block-wise 1:4 and 2:4 sparsity, CHTsNM TANO achieves better mean PPL than CHTs24 in all 16 matched-LR comparisons at 60M and 13 of 16 at 130M (averaged over three seeds, error bars showing standard error). Comparing each method at its best tested learning rate, TANO achieves lower mean PPL in all eight model–configuration combinations even without extra contextual modulation from CoMoLoRA.

Figure 2: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%202/Fig2_TANO_CHTs24_BestLR.png

Across 60M and 130M models with row-wise and block-wise 1:4 and 2:4 sparsity, CHTsNM TANO generally achieves a lower mean PPL than STE, SR-STE, and CAST-style baselines across three seeds. Other than two near-tied 130M row-wise configurations, CHTsNM TANO outperforms STE. With CoMoLoRA, CHTsNM TANO achieves the lowest best PPL across all combinations.

Figure 3: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%202/Fig3_TANO_STE_BestLR.png

Compared against CHTsNM Adam SMET over the same settings, CHTsNM TANO achieves a lower mean (three seeds) PPL in 15 of 16 matched-LR comparisons at 60M and 13 of 16 at 130M. At each method’s best tested LR, CHTsNM TANO outperforms CHTsNM Adam SMET (the best SMET-variation) in six of eight configurations, trailing two 130M row-wise settings. With CoMoLoRA, CHTsNM TANO achieves the best PPL across all eight combinations.

Figure 4: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%202/Fig4_TANO_SMET_BestLR.png

At 60M, in an equal-budget, four-LR, three-seed comparison, TANO achieves lower PPL than all tested sparse baselines even without CoMoLoRA. These results demonstrate the benefits of the optimizer itself, rather than attributing gains to sparsity alone. The table below shows our complete results:

Figure 5: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%202/Fig5_T1_60M_Mean_SE.png

On 130M LLaMA (four-LR, three-seed equal-budget sweep), dense TANO improves best mean PPL from 24.35 to 22.87. The Table linked below shows CHTsNM TANO leads even with branch-free baselines in block-wise settings (while SR-STE or Adam+SMET leads row-wise), and TANO+CoMoLoRA achieves the lowest best-LR PPL across all four sparse configurations—confirming benefits beyond sparsity alone.

Figure 6: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%202/Fig6_T1_130M_Mean_SE.png

Dense-TANO does not exist in literature and the most widespread optimizer (because of speed and stability) for dense models is Dense-Adam, which is used in sparse training studies as a baseline for comparison with sparse models. Therefore also in this study we keep dense-Adam as a baseline for comparison with dense models in all tables. However, to address your comment we also add dense-TANO, which now represents an innovation of our study valid also for dense models. The results show two main finding:

(1) Our optimizer TANO also improves dense-Adam training. We thank the Reviewer for this suggestion that turned out to be a new discovery. Dense TANO reaches lower best-observed mean PPL than dense Adam by 0.82 at 60M and 1.48 at 130M, comparing each method at its best tested LR. (Points show mean ± standard error across three seeds).

Figure 1: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%202/Fig1_Dense_TANO_vs_Adam_SE.png

(2) We cannot claim that TANO is topologically aware but we can claim that there is an advantage of TANO with respect to Adam, when applied to dynamic sparse epitopological training driven by CHTsNM , because TANO’s sparse performance remains closer to the respective dense version. Therefore we reduce our claims accordingly, and we rename TANO as Epitopological Nesterov Optimization (ENO). Epitopological indicates there is an emerging advantage in reducing the gap from dense model performance when the method is applied to dynamic sparse training via CHTsNM. We thank the Reviewer, again we recognize the Reviewer’s merit to attain this discovery. The results and discussion of this test are provided below in reply to your Q1.

AuthorsOpenReview ↗

Comment (2/2) for Reviewer 8FR2

C2: To address the limitation, we conducted an additional LLaMA-1B block-wise 1:4 experiment on OpenWebText, comparing CHTsNM with TANO against SR-STE at a shared learning rate. The accompanying figure reports held-out perplexity over 50,000 steps, extending our original 1:4 evaluation to both a larger model scale and a longer training horizon. These results show that block-wise 1:4 CHTsNM remains competitive with SR-STE at the 1B scale while providing modest measured throughput and memory gains.

Figure 7: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%202/Fig7_50k_1B_SRSTE_TANO.png

Q1: We trained Dense-TANO and compared it with CHTsNM-TANO, Dense-Adam and CHTsNM-Adam. The learning rate is the same (LR=0.002) which is the best learning rate we found during LR sweeping. All the other settings are as for reviewer request in the comment above. The results in the two figures below (one for row wise and the other for block wise) show that when subtracting Sparse - Dense Adam, and Sparse - Dense TANO, the gap between Sparse and Dense model is smaller for TANO. This implies that we cannot claim that TANO is topologically aware but we can claim that there is an advantage of TANO, when applied to dynamic sparse epitopological training driven by CHTsNM , because TANO’s sparse performance remains closer to the respective dense TANO. Therefore, we reduce our claims accordingly, and we rename TANO as Epitopological Nesterov Optimization (ENO). Epitopological indicates that we have evidence that ENO offers an advantage in reducing the gap from dense model performance when the method is applied to dynamic sparse training via CHTsNM. We thank the Reviewer, again we recognize the Reviewer’s merit to attain this discovery.

Figure 8: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%202/Fig8_Rowwise_1in4_Dense_Sparse_PPL_Table.png

Figure 9: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%202/Fig9_Blockwise_1in4_Dense_Sparse_PPL_Table.png

Reviewer_zWUrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors focus on semi-structured sparse training of LLMs, particularly the challenge of maintaining model quality under 1:4 sparsity. Specifically, the authors propose CHTsNM, a dynamic sparse-to-sparse training framework that combines Cannistraci–Hebb topology evolution with two components: a Topology-Aware Nesterov Optimizer (TANO) to manage optimizer states under changing sparse connectivity, and Contextually Modulated LoRA (CoMoLoRA) to compensate for the representational capacity lost through sparsification. Experiments across different model sizes and sparsity patterns demonstrate improved perplexity compared with several sparse training baselines, while reducing optimizer-state storage and showing potential inference efficiency gains.

Critical strengths
  1. The paper addresses the challenging problem of training LLMs under 1:4 semi-structured sparsity, where maintaining model quality becomes considerably more difficult than under the more commonly studied 2:4 setting.
  2. The coordinate-consistent state remapping mechanism is clearly described and addresses a practical issue in dynamic sparse training.
Critical weaknesses
  1. The proposed framework largely integrates existing techniques. The Cannistraci–Hebb topology evolution mechanism is inherited from CHTs24, while optimizer-state management and active-only state storage have been investigated in prior dynamic sparse training methods, particularly SMET.
  2. The main 1:4 pretraining experiments are conducted on relatively small models ranging from 60M to 350M parameters, mostly using a context length of 256 tokens. Although the paper includes experiments on 1B and 7B models, these use 2:4 sparsity rather than the proposed 1:4 setting.
Questions for authors
  1. Could the authors clarify the main methodological innovation of CHTsNM beyond combining the existing CHTs24 topology mechanism, active-only optimizer-state management, and context-dependent low-rank compensation?
  2. The 1B and 7B experiments primarily focus on 2:4 sparsity, while the reported 1:4 acceleration relies on compiler estimates. Could the authors clarify which conclusions regarding billion-parameter 1:4 training and actual inference acceleration are directly supported by the current experiments?
1 comment
AuthorsOpenReview ↗

Comment (1/1) for Reviewer zWUr

C1: Our contribution is TANO's combination of preconditioning before Nesterov momentum, matrix-shape scaling, and coordinate-consistent state remapping under N:M topology changes. These are innovations in semi-structured N:M dynamic sparse training that were never explored in SMET and other previous studies. To quantify gains over CHTs24, we ran equal-budget, four-LR sweeps across three seeds: without an auxiliary branch, CHTsNM already achieves lower best-observed mean PPL across all configurations, demonstrating the value of the integrated recipe beyond its inherited topology. Full 60M/130M breakdowns are in Q1 (Figures 3–4).

C2: We now extend the 1:4 evaluation to LLaMA-1B with block-wise sparsity. At 50K steps and LR 0.002 (which we adopt because it offers the best performance during LR sweeping on LLaMA-60M/130M), CHTsNM TANO achieves held-out PPL 17.07, comparable to SR-STE's 17.27, with 13.1% lower observed mean GPU memory usage and 5.2% higher sampled training throughput on one A100 per method. These results are currently single-seed (the other two seeds are very expensive; we will provide them in the coming weeks), however they provide direct evidence of billion-parameter 1:4 training.

Figure 1: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%201/Fig1_1B_LR002_50K%20SRSTE%20TANO.png

We also added 1:4 results at 7B, following Table 8's retraining-after-pruning protocol (SparseGPT 1:4 pruning baseline + CHTsNM retraining). We selected the two best 2:4 methods (SparseGPT and CHTsNM), and did not consider CAST because its performance was significantly lower in 2:4 tests and we prioritized resources given the reply window. The 1:4 results confirm CHTsNM's promise, widening its advantage over SparseGPT.

Figure 2: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%201/Fig2_Table8_7B_Retraining_After_Pruning.png

Q1: CHTsNM's innovation over CHTs24 is the TANO update: preconditioning before Nesterov momentum, plus matrix-shape scaling and coordinate-consistent state remapping under N:M topology changes. Our LR sweep results demonstrate superior performance across a broad LR range, suggesting that CHTsNM offers more consistent pre-training performance than CHTs24 regardless of learning-rate tuning. This is particularly important for very large models, where exhaustive LR search is difficult in practice. For the 60M model, CHTsNM (TANO) and CHTsNM (TANO) + CoMoLoRA outperform CHTs24 at every tested learning rate and layout/sparsity configuration:

Figure 3: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%201/Fig3_60M_TANO_vs_CHTs24.png

Similarly for the 130M model, both CHTsNM (TANO) and CHTsNM (TANO) + CoMoLoRA outperform CHTs24 at every tested learning rate and layout/sparsity configuration:

Figure 4: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%201/Fig4_130M_TANO_vs_CHTs24.png

Q2: We now (1) clarify the Moffett compiler estimation and (2) add new hardware speedup estimates from the nmSPARSE package on RTX 5080 GPUs. nmSPARSE — reference method and replication target. Bin Lin, Ningxin Zheng, Lei Wang, Shijie Cao, Lingxiao Ma, Quanlu Zhang, Yi Zhu, Ting Cao, Jilong Xue, Yuqing Yang, and Fan Yang. Efficient GPU Kernels for N:M-Sparse Weights in Deep Learning. Proceedings of Machine Learning and Systems, volume 5, 2023. NVIDIA accelerates only 2:4 sparsity (native Sparse Tensor Cores), inference-only—no 1:4 support. Our 1:4 speedups are therefore Moffett compiler estimates, scheduling the mask so only the one non-zero weight per 4-element group is fetched and multiplied. On Moffett, the 1B forward pass reaches 2.67× (backbone) / 2.40× (with CoMoLoRA); 7B reaches 2.89× / 2.69×. Training is not accelerated—no hardware supports real sparse 2:4/1:4 training, so training uses dense tensors and speedups are inference-only. These are single forward passes (prefill) with no autoregressive decoding or KV-cache; we will state both caveats in Section 4.6 and Limitations.

Both 1B and 7B were trained without CoMoLoRA to give TANO baseline results. The Moffett tables also report a CoMoLoRA variant, quantifying the dense branch's added latency over the backbone.

Figure 5: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%201/Fig5_b_MOFFET_inference_speed.png

The table below reports RTX 5080 full-forward inference via nmSPARSE at batch size 8, sequence length 256, all speedups relative to dense and no autoregressive decoding. Divergence in speed of 2:4 vs 1:4 is explained by the eligible runtime fraction: at 60M only 53.5% of the forward pass is in sparsifiable projection GEMMs, leaving 1:4 little room; by 1B that fraction hits 93.1%, widening 1:4's advantage over 2:4.

Figure 6: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%201/Fig6_Measured_and_Modelled_black.png

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.