- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper deals with semi-structured sparsity, specifically 1:4 in LLMs. Authors propose CHTsNM, a sparse-to-sparse pretraining framework combining Cannistraci-Hebb topology evolution, topology-aware optimization, and contextually modulated low-rank compensation. Furthermore, the paper introduces Topology-Aware Nesterov Optimizer (TANO), and Contextually Modulated LoRA (CoMoLoRA). CoMoLoRA reduces perplexity in twelve settings.
Critical strengths
- The paper is well motivated
- The paper achieves improvement over baselines, and tries to cover both improvement in accuracy and performance on hardware
Critical weaknesses
-
Unclear memory and speed benefits: The paper does not clearly explain what the claimed 75% memory reduction includes or what it is compared against. It is also unclear how 1:4 sparsity is implemented and how the reported speedups are measured.
-
Insufficient evidence from the experiments: The paper should compare against SMET and GaLore under the same memory budget. It is also unclear whether 20K updates are enough to draw conclusions about final model quality, and the number of training tokens should be reported. The larger-model results in Table 6 use 2:4 rather than the proposed 1:4 pattern, leaving its performance at larger scales unclear. These gaps make it difficult to judge whether the method improves on existing methods.
Questions for authors
Clarity
“moments reduce eligible-matrix state storage by 75%” — Is this true, and which data format is used to achieve this reduction? How is it measured? What is the 75% reduction relative to: the total memory occupied by the weights or something else? Is index overhead included? Are activations taken into account? The activations are dense, correct?
The Cannistraci–Hebb family should be explained before it is mentioned, rather than assuming that the reader knows what it is. Could you clarify it?
CoMoLoRA — LoRA usually involves a frozen backbone. Is that the case here, or are you referring to methods that replace a dense matrix with a low-rank matrix without retaining a frozen full matrix?
“N active entries in every column of each M×M block” — For clockwise 1:4, does this mean that 1 out of 16 elements is nonzero, or that 4 out of 16 are nonzero?
A better explanation of the baselines would be helpful. Even a one-sentence explanation of each baseline in the main text would improve the reader’s understanding of the paper. For example, what is STE?
Soundness
2:4 sparsity is directly supported by NVIDIA hardware, but how is 1:4 implemented? This is not trivial, as there is no direct hardware support. Does this assume that a hardware vendor will follow the trend and develop hardware that supports it? Alternatively, is a specialized kernel implemented? Does your method outperform the approach in the “VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores” (https://arxiv.org/abs/2310.02065), which includes a low-level kernel implementation? How do your speedups compare to those of VENOM?
Why are newly activated coordinates initialized to zero? SMET (https://arxiv.org/abs/2606.00888) has shown that this causes spikes in the loss function when training with DST. Does the same hold in your setting? Why are the moments initialized to zero for new weights? Is there any evidence that this is superior?
Where does Equation 7 come from? Is it based on previous research, or was it selected arbitrarily? Is it a contribution of this paper?
In general, how does your method compare to SMET or GaLore (https://arxiv.org/abs/2403.03507) under the same memory budget?
Aren’t 20K updates too few to observe convergence? How many tokens are used for training in Table 1? This is what is usually reported, for example, in the GaLore paper.
In Table 1, how is it possible that 2:4 outperforms the dense baseline? This does not seem to make sense. How many tokens were the models trained on? If trained to convergence, I would expect the dense baseline to perform better.
What is the motivation behind clockwise 1:4? As far as I know, this pattern cannot achieve speedups even with the latest hardware. Don’t blockwise kernels for Tensor Cores still need to fill in the zeros within the blocks passed directly to Tensor Cores for MMA operations?
In Table 6, why not show 1:4 instead of 2:4 if the paper focuses on 1:4 rather than 2:4? Does this mean that 1:4 does not scale well to larger models?
In Section 4.6, how does the kernel achieve a speedup with 1:4? What exactly are you measuring when referring to inference? Is this just prefill at a particular batch size, i.e., a simple forward pass? Is any decoding performed?
4 comments
Comment (2/4) for Reviewer M47n
Soundness
Q6: Our 1:4 training enforces legal masks on dense weight tensors; it does not depend on a custom NVIDIA 1:4 kernel. The quality results therefore do not assume future hardware support. Packed optimizer-state storage can reduce state memory independently of whether the matrix multiplication uses sparse hardware.
Figure 1 reports observed total runtime, including initialization and evaluation, rather than a native 1:4 kernel speedup. For example, at 350M row-wise 1:4 and LR 0.004, TANO reaches PPL 21.58 in 321.73 minutes, compared with SR-STE’s 21.83 in 358.89 minutes on H200. These are single-seed, implementation-level observations; they do not isolate the cause of the runtime difference or establish a universal speed advantage.
VENOM uses a different V:N:M representation and specialized kernels to exploit NVIDIA Sparse Tensor Cores. We have not performed a matched kernel-level comparison. Separate inference evidence and the scope of the vendor compiler estimates are specified in (Q14).
Q7: Zero initialization avoids adding an arbitrary weight contribution at the instant of regrowth. It does not guarantee a smooth refresh, because removing connections can still perturb the network. New-coordinate moments are reset because there is no continuing active-state history to preserve; retained coordinates keep their coordinate-consistent states. This is a state-consistency rationale, not proof that zero moments are universally optimal.
We directly tested zero versus historical-value regrowth, with and without a 10-update SMET-style warmup, on LLaMA-130M over three seeds (Figure 4). At row-wise 2:4, zero initialization without warmup gives the lowest mean final PPL, 26.330, compared with 26.426 for zero plus warmup. At row-wise 1:4, zero plus warmup instead improves the mean from 30.322 to 30.084. Thus, the benefit of warmup depends on the setting; the results do not support blanket superiority of zero initialization without warmup. Figure 4: https://anonymous.4open.science/r/iclr27_rebuttal_images-4EC2/Rebuttal%20Figures/Reviewer%203/Fig4_CHTsNM_TANO_Regrowth_Initialization_SE.png (Q9) reports a broader comparison with SMET-style Adam adaptations. Whether refresh-induced loss spikes are absent or isolate the effect of moment initialization will be further studied.
Q8: Equation 7 is inspired by the input-dependent contextual modulation in TRANSPONDER (Zhang et al., 2025b). CoMoLoRA adapts this principle to low-rank features for sparse-backbone compensation. The gate 1+G(x) bounds modulation around one, while zero-initializing the controller’s output layer makes the branch begin as ordinary additive LoRA. Our contribution is this adaptation, rather than contextual modulation itself. We will clarify the attribution and design rationale in revision.
Reference: From Transformer to Transponder: Introducing Contextual Modulation Training for Residual Learning in LLMs. URL: https://www.preprints.org/manuscript/202506.0120
Zhang, Y.; Gu, W.; Hu, W.; Li, J.; Cannistraci, C. V. From Transformer to Transponder: Introducing Contextual Modulation Training for Residual Learning in LLMs. Preprints 2025, 2025060120. https://doi.org/10.20944/preprints202506.0120.v2
Comment (3/4) for Reviewer M47n
Q9: Compared against CHTsNM Adam SMET over the same settings, CHTsNM TANO achieves a lower mean (three seeds) PPL in 15 of 16 matched-LR comparisons at 60M and 13 of 16 at 130M. At each method’s best tested LR, CHTsNM TANO outperforms CHTsNM Adam SMET (the best SMET-variation) in six of eight configurations, trailing two 130M row-wise settings. With CoMoLoRA, CHTsNM TANO achieves the best PPL across all eight combinations.
(Figure 2) illustrates comparison between CHTsNM TANO and different SMET-variations. On a LLaMA-130M, 20K-step experiment with two paired seeds, we compared TANO, CHT+Adam, Adam+SMET with random regrowth, and CHT+Adam+SMET. At similar VRAM memory budgets of 14.41–14.70 GiB, TANO reaches lower mean PPL of 22.18 versus 22.31 for CHT+Adam+SMET at row-wise 2:4. TANO’s total runtime was 1.1% and 1.6% faster. This supports competitive quality and runtime at comparable measured memory.
For block-wise (1:4 & 2:4) configurations, CHTsNM TANO achieves lower held-out PPL than CHTsNM ADAM SMET at each method’s best tested learning rate across all three model sizes. Results are averaged over three seeds for 60M and 130M, with one seed available for 350M:
Among row-wise configurations, CHTsNM ADAM SMET achieves lower mean PPL at 130M with LR 0.004 than CHTsNM TANO at its best tested learning rate. CHTsNM TANO, however, achieves lower best-observed PPL in all learning rates, both configurations at 60M and 350M. Results are averaged over three seeds for 60M and 130M, and one seed for 350M:
Q10: We agree that <=20K training steps may not establish convergence. Table 1 compares methods at fixed training budgets: approximately 655M token slots for 60M/130M and 1.31B for 350M, using global batch 256 and sequence length 256. These counts include padding; the separate 100M-token budget is held-out evaluation. We will report these budgets explicitly and frame the results as performance at the tested training horizons, rather than fully converged model quality. A longer 50K-update 1B experiment is presented in (Q13).
Q11: We found strong discoveries after extending a Learning Rate sweep. We are thankful to the reviewer as we have now developed dense TANO (which did not exist before) which provides evidence to CHTsNM TANO's Epitopological advantage over Adam (more in Revision). As shown in (Figure 7), dense TANO sets a stronger (lower-PPL) baseline, which substantially benefits 2:4 sparse TANO configurations, allowing them to outperform dense Adam at certain learning rates:
Q12: Interpreting “clockwise” as block-wise 1:4, the motivation is to preserve a legal sparse pattern under transposition: each 4×4 block contains one active entry per row and column (allowing the same support structure in both forward and backward operations). The review is correct that this does not guarantee hardware acceleration. Ordinary block-sparse kernels using dense MMA indeed still compute zeros inside retained blocks; our pattern is sparsity within blocks, rather than removal of entire blocks. Our measured NVIDIA inference results concern 2:4; the reported 1:4 speedups are MOFFET software simulated estimates, not device measurements. We will clarify this distinction and the structural motivation in the revision.
Q13: Thank you for your suggestion. We now include LLaMA-1B block-wise 1:4 experiments: at 50K steps, TANO achieves a held-out PPL 1.1% better (lower) than SR-STE. These experiments extend our 1:4 evaluation to billion-parameter training, demonstrating that block-wise 1:4 CHTsNM remains competitive with SR-STE at the 1B scale while delivering modest measured throughput and memory gains. (we will conduct more seeds in the following two weeks)
For LLaMA-7B, we added 1:4 retraining after SparseGPT pruning with knowledge distillation (Figure 8). After 2,600 updates, CHTsNM reaches WikiText-2/OpenWebText PPL 16.669/13.678, compared with 110.316/408.032 immediately after 1:4 pruning. This demonstrates substantial quality recovery through additional training, rather than an equal-compute comparison with pruning alone. The result remains below dense-model quality and is distinct from the 1B pretraining experiment.
Comment (4/4) for Reviewer M47n
Q14: To clarify— the Moffett 1:4 results are compiler-estimated full-forward/prefill latencies, not device measurements. The compiler schedules active weights in eligible operations, reducing weight traffic and multiplication. Tests use BF16, sequence lengths 128/256, and batch one per core across four cores. At length 256, estimated backbone speedups are 2.67×/2.89× for 1B/7B, or 2.40×/2.69× with CoMoLoRA (Figure 9). Separately, (Figure 10) distinguishes measured 2:4 from modelled 1:4 RTX 5080 latencies at batch eight and sequence length 256, citing nmSPARSE as the N:M kernel reference [1]. Neither experiment includes autoregressive decoding or KV-cache-based generation. We will clarify these scopes in Section 4.6.
[1] nmSPARSE — reference method and replication target. Bin Lin, Ningxin Zheng, Lei Wang, Shijie Cao, Lingxiao Ma, Quanlu Zhang, Yi Zhu, Ting Cao, Jilong Xue, Yuqing Yang, and Fan Yang. Efficient GPU Kernels for N:M-Sparse Weights in Deep Learning. Proceedings of Machine Learning and Systems, volume 5, 2023.
Comment (1/4) for Reviewer M47n
Dear Reviewer, thank you for this valuable feedback. In the revised manuscript, we will reorganize the paper to improve readability and balance with a streamlined Methodology section, an expanded Experimental Setup section, and new figures produced during this rebuttal that illustrate different aspects of CHTsNM's performance in comparison to existing methods.
C1: We agree that these claims need distinct accounting. The 75% reduction applies to the first- and second-moment buffers of eligible matrices at 1:4 density, not total training memory. Likewise, sparse connectivity does not by itself imply faster hardware execution. We now distinguish packed-state storage (Q1), the training implementation and observed runtime (Q6), and measured 2:4 versus estimated 1:4 inference acceleration (Q14). This separates the demonstrated benefits from hardware-dependent projections.
C2. In response to this valuable feedback, we have strengthened evidence with equal-budget learning-rate sweeps, a longer SMET-style comparison with measured peak memory, and additional billion-parameter 1:4 experiments. The revisions distinguish performance at a fixed training budget from convergence, and comparable measured memory from an exactly matched memory constraint. (Q9) reports the SMET comparisons and the remaining GaLore limitation; (Q10) gives the training-token budgets; (Q13) presents the new 1B and 7B evidence. These additions support the practical value of CHTsNM.
Q1. Yes. Apologies for the confusion, there is no reduction in total GPU memory; the 75% saving follows directly from storing only active-coordinate moments. For an eligible matrix with D coordinates, two dense FP32 moment buffers occupy 8D bytes. Physically packing moments for the D4 active coordinates reduces this to 2D bytes: a 75% reduction in those moment buffers. We measure this from allocated tensor sizes; masking dense buffers alone would not achieve it. This saving follows from the density and is not claimed as an independent methodological innovation.
The 75% excludes coordinate metadata, weights, gradients and activations. Table 4 separately includes uint8 coordinate metadata and the moments of other parameters, giving a 49.5% reduction in the reported optimizer-state-plus-metadata allocation. Activations remain dense; neither percentage is a claim about total GPU memory.
Q2: We will define the Cannistraci–Hebb family at first mention in our revision. These dynamic sparse training methods learn network connectivity by periodically removing connections and regrowing others using scores derived from the local connectivity structure. In CHTsNM, soft magnitude-based removal and Cannistraci–Hebb-guided regrowth operate within legal N:M sparsity patterns, a topology mechanism inherited from CHTs24.
Q3: CoMoLoRA jointly trains the sparse backbone and an additive low-rank branch. The backbone is neither frozen nor replaced by a low-rank matrix. Unlike standard LoRA fine-tuning, it supplies additional capacity during sparse pretraining, with input-dependent modulation of the low-rank features. Its parameters and computation are additional to the N:M sparse backbone.
Q4: Interpreting “clockwise” as “block-wise”, it means four active entries out of sixteen. Specifically, in each 4 x 4 block, there are 4 active entries—with exactly one in every row and every column—resulting in a density of 25% (75% sparsity). We will make this explicit in the pattern definition.
Q5: We agree and will add a short definition and implementation reference for each baseline in the main text. STE (straight-through estimator) applies an N:M mask during the forward pass and uses a surrogate gradient through the discrete masking operation during backpropagation. We will also distinguish mask-projection methods from explicit topology-regrowth methods, and label our SMET-style adaptations separately from the original published method.