acceptodds
Under review as a conference paper at ICLR 2027

Does Energy-Aware Sparsity Allocation Help Dynamic Sparse Training? Leverage, Execution Back Ends, and Stable Regrowth

Abstract

Dynamic sparse training (DST) is motivated by lower training cost, and a natural refinement is to allocate sparsity by energy, pruning hardest where computation is most expensive. We test whether this helps, measuring GPU energy with hardware counters in randomized blocks on three execution back ends. An elementary identity bounds what any layer-wise allocation can save at a fixed number of weights: the saving is the drop in the average cost per weight when weights move to cheaper layers. This leverage is zero for masked dense kernels, as used by most DST code, and for the linear layers of Transformers, and it is large in CNNs, whose early layers cost 100-200x more per weight than their late layers. A score based on each layer's total energy captures it only on AlexNet; allocating by energy per weight captures about half of it. The measurements show why this leverage still does not pay off. Under cuSPARSE kernels, allocation effects are real and predicted within 2.8 points by per-layer costs, but energy is not monotone in density: a uniform network with twice the weights uses 44% less energy than one at the common density of 0.1 and is more accurate. Under channel-structured execution, sparse training finally costs less than dense training (0.40x on VGG-16), but at equal measured energy neither energy-aware allocation nor its reverse beats simply lowering the density of a uniform allocation. Energy-aware regrowth fares worse: it follows a replicator dynamic that collapses the network into one layer (-9.8 points on AlexNet) unless it is anchored, which we prove and confirm fixes it. On the hardware we measured, the execution back end and the density set the energy of sparse training, not the layer-wise allocation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Deep learning, DST, Energy-aware training, Model initialization, Extreme sparsity, Training, Energy consumption reduction, Emission reduction, Sustainable AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_B1LtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper investigates whether layer-wise sparsity allocation can reduce actual training energy. It combines theoretical analysis, GPU energy measurements across three computational implementations, and an analysis of regrowth stability. In the tested settings, energy-aware allocation does not outperform uniform allocation at an appropriate density when compared at equal energy.

Critical strengths
  1. Practically relevant evaluation. Direct GPU energy measurements and accuracy comparisons at equal energy provide useful evidence about the practical benefits of sparsity.
  2. Informative analysis. The analysis of per-weight costs and regrowth imbalance helps explain both the potential savings and the observed failure modes.
Critical weaknesses

Limited generalizability of the empirical findings. The controlled experiments cover one laptop GPU, two CNNs, and CIFAR-10 with short training schedules. Given the dependence on computational implementation, it remains unclear whether the same energy–accuracy relationship between energy-aware and uniform allocation holds after longer training or on different hardware.

Questions for authors

Does the relative accuracy of energy-aware and uniform allocation at equal energy remain consistent under longer training schedules, particularly for the channel-structured experiments? Could the authors provide supporting evidence?

4 comments

Thank you for your positive review and for highlighting our equal-energy comparisons and regrowth analysis. (L. gives line numbers in the initial submission; "new" marks material added in the revised PDF.)

Q. Yes. In the initial submission, the masked and CSR back ends already used the full schedules (30 epochs for AlexNet, 20 for VGG-16), and only the channel-structured comparison was shortened to 8/5 epochs to fit nine configurations per block (L. 274–276). We have now rerun the channel-structured comparison at the full 30/20-epoch schedules across five randomized blocks (up from three), each including uniform networks at seven densities (0.035–0.3) so that every allocation is evaluated against uniform networks at matched measured energy (General response, Table A). While longer training raises absolute test accuracy across the board (for uniform at d = 0.1, AlexNet rises from 69.8% to 78.9% and VGG-16 from 76.4% to 86.2%), the relative conclusion is unchanged: no energy-aware allocation beats the uniform trade-off curve (every 95% CI includes zero or lies below it). Below is the equal-energy gain G (in percentage points at d = 0.1) comparing the initial short schedules (L. 381–384) to the new full schedules:

AllocationAlexNet (8 → 30 epochs)VGG-16 (5 → 20 epochs)
EA+ (α = 1)−0.83 → −0.87 [−1.41, −0.33]−0.37 → +0.04 [−0.58, +0.66]
EPW (ν = 0.25)−0.14 → −0.77 [−1.91, +0.37]−0.63 → −1.19 [−1.81, −0.58]

On three data-center GPUs of an HPC cluster, the same full schedules lead to the same conclusion (see W below).

W. The same energy–accuracy relationship holds after longer training and on different hardware, and two further results cover other models and datasets:

  • Short training schedules: At the full 30/20-epoch schedules, the channel-structured runs reach the same conclusion: no energy-aware allocation beats uniform allocation at equal energy (Q above). The other back ends already used these schedules.
  • One laptop GPU: On an HPC cluster, we repeated the full-schedule channel-structured experiments on three data-center GPUs with fixed power limits: an NVIDIA A40 and an L40 at 300 W and an RTX PRO 6000 at 600 W (3 blocks each). On VGG-16, both EA+ and EPW lie on or below the uniform curve on all three (G = −0.49 to −0.78 and −1.18 to −2.74). On AlexNet, the thin sparse models are host-bound on these GPUs, and neither allocation improves over uniform at equal density. Additionally, on the CSR back end, fixing a single SDDMM operand layout made CSR cost monotone in density and allowed EA+ to beat uniform allocation on AlexNet, on the laptop (+1.02 points) and on all three HPC GPUs (+0.65 to +0.75), matching the leverage identity's prediction (General response). Finally, our theoretical results (the leverage identity and regrowth replicator dynamics) are hardware-independent.
  • Two CNNs and CIFAR-10: For other models, the leverage identity (Prop. 4.1) bounds what any allocation can save from one forward pass, before training; for other datasets, our legacy benchmark trained six architectures (including ViT, ConvNeXt, and MobileNetV2) on CIFAR-10/100 and Tiny-ImageNet (L. 792–794). The identity matches our measurements: EA+ gains at equal energy on AlexNet (above), the one model where Tab. 1 shows EA+ capturing leverage. Across the eight models of Tab. 1, it shows that reallocating weights in Transformer linear layers changes compute by at most 1% on ViT and 0.7% on GPT-2 small (L. 193–204). In the benchmark, under masked execution, energy-aware initialization matched uniform allocation (−0.21% whole-device energy and +0.04 points in accuracy, L. 899). The revised abstract now states the setting of our hardware-counter measurements.

All new full-schedule, HPC-GPU, and corrected CSR results are integrated into the revised Sec. 6.2 and Sec. 6.3 (new Tables 2 and 3) and App. E. We are happy to answer any further questions.

AuthorsOpenReview ↗

Follow-up for Reviewer B1Lt

Our reply answers your question and weakness with new runs: at the full 30/20-epoch schedules (five randomized blocks) and again on three data-center GPUs with fixed power limits, the channel-structured comparison shows the same energy–accuracy relationship between energy-aware and uniform allocation (General response, Table A). The leverage identity and our legacy benchmark (six architectures on CIFAR-10/100 and Tiny-ImageNet) cover other models and datasets. We are happy to answer any question before the discussion closes and would be glad to know whether our reply and the revised PDF address your concerns. If they do, we would be grateful if you would consider updating your score.

Thank you for the detailed rebuttal. The full-schedule runs, the additional GPUs, and the corrected CSR results address most of my concerns, and I have raised Soundness and Presentation to 3. However, the controlled experiments still cover only two CNNs on CIFAR-10, and the main positive result for energy-aware allocation (CSR on AlexNet) comes from configurations that cost more than dense training. I therefore maintain my rating of 3 (Weak Acceptance).

Thank you for your constructive feedback, which helped us strengthen the paper, and for raising Soundness and Presentation; we are glad that our responses and new results address most of your concerns. The paper states both remaining points as limitations, and in the final version, we will add to the main text our estimate of the AlexNet gain on an ideal skip-zero accelerator, on which sparse training costs less than dense training.

Reviewer_xxxQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether energy-aware layer-wise sparsity allocation improves the measured energy–accuracy trade-off of dynamic sparse training. It studies an allocation based on estimated layer energy, an alternative based on energy per weight, and energy-biased regrowth. Under a linear additive cost model, the paper derives an identity explaining when reallocating a fixed number of active weights can reduce computation. It then measures GPU energy using masked dense, cuSPARSE CSR, and channel-structured execution. The controlled experiments find that the tested energy-aware allocations do not demonstrate an advantage over choosing an appropriate uniform density. The paper also diagnoses density drift and layer concentration under multiplicatively biased regrowth, derives the corresponding fixed-factor dynamics, and evaluates an anchored correction. Controlled measurements use AlexNet and VGG-16 on CIFAR-10, supported by an audited legacy benchmark.

Critical strengths
  1. The paper connects observed failures to mechanisms. It separates two questions that usually get mixed up: a) whether a model leaves any room for energy-aware allocation and b) whether a given rule uses that room. It explains why an energy-share score does nothing on VGG-16, ResNet-34, and MobileNetV2. The reason is that the score ranks layers by total cost, not by cost per weight. The regrowth analysis is useful. It implies that any DST rule that reweights regrowth across layers by fixed factors will concentrate under a global budget. The anchored variant shows how to prevent this.
  2. The study uses hardware counters, randomized blocks with an in-block dense reference, and paired tests with Holm correction. It also compares against uniform allocations at other densities. All of the measurements are clear.
Critical weaknesses
  1. The studies are too narrow. All controlled measurements come from one laptop GPU. Its platform-managed power limit (35–45 W) cannot be fixed and moves with the run’s load. The study only counts GPU energy. Also, the models are two small CIFAR-10 CNNs.
  2. The channel-structured back end is the only one where sparse training beats dense, so it is the only one where the allocation question has practical influence.
  3. Appendix C and Table 18 show that changing the SDDMM operand from a contiguous transposed copy to a transposed view makes that operation approximately 8× cheaper at density 0.1 and 20× cheaper at density 0.05 for AlexNet’s second convolution. But the paper does not include rerunning the end-to-end CSR comparisons with this layout. Because the striking nonmonotonic density curve and allocation savings are closely tied to this slow regime, the current results establish the behavior of the released implementation but leave the robustness of the practical CSR conclusions unresolved.
Questions for authors
  1. Do the channel-structured conclusions still hold under the longer 30/20-epoch schedules? Can the authors directly evaluate uniform configurations near the energy budgets of EA+ and EPW, propagate uncertainty in both energy and accuracy, and state what practically meaningful improvement the results can exclude?
  2. Can this be done on a GPU where the power limit can be fixed?
  3. Is the laptop’s 35–45 W limit coupled to CPU load?
  4. Have you tried kernels whose cost is close to linear in kept weights, such as block-sparse kernels? 2:4 sparsity on Ampere is another option, where allocation reduces to choosing which layers are 2:4.
  5. How does uniform channel-level DST compare with a dense network of the same reduced width trained without topology updates?
  6. Could the method be extended to other tasks other than CIFAR-10 and model architectures beyond CNNs?
5 comments

Thank you for the thorough review. Your observation on the CSR operand layout was spot on, and acting on it confirmed the leverage identity's prediction, so we start there. (L. gives line numbers in the initial submission; "new" marks material added in the revised PDF.)

W3. We agree, and the rerun strengthens the paper. Passing the SDDMM operand as a transposed view is a one-line change that leaves the weight gradients unchanged (within 1.5·10⁻⁹), so no accuracy result changes. CSR cost now rises with density: one AlexNet step takes 38/57/67/83 ms at d = 0.05/0.1/0.2/0.3 (copy layout: 136/197/75/92 ms). In six new energy blocks (5 accuracy seeds per point, 95% bootstrap CIs), EA+ now beats uniform allocation at equal energy on AlexNet, as the leverage identity predicts:

AlexNet, d = 0.1Energy vs. uniformG (points)
EA+ (α = 0.5)−9%+1.02 [+0.60, +1.44]
EA+ (α = 1)−20%+0.90 [+0.62, +1.19]
FLOP-share−5%+0.27 [−0.25, +0.75]
ERK−23%−0.58 [−0.84, −0.29]

On VGG-16, where the identity predicts no leverage, EA+ changes neither energy (+0.4% to +0.8%, CIs include 0) nor accuracy (≤0.07 points). The gain holds with CPU-package energy (G = +0.90, +0.95) and without the new uniform point d = 0.07 (+0.70, +0.58). On three data-center GPUs of an HPC cluster (A40, L40, RTX PRO 6000; fixed power limits, same blocks), EA+ again gains: G = +0.65/+0.70/+0.75 (α = 0.5) and +0.52/+0.79/+0.12 (α = 1), with four of six CIs excluding zero (App. E, new). The fix corrects one energy claim of the initial submission (abstract, L. 23–26): a denser uniform network no longer costs less. At these densities, cuSPARSE itself remains slower than dense cuDNN training (2.4–5.6× its energy on all four GPUs), so CSR shows where allocation pays, while the channel back end (Q1 below) is where sparse training costs less than dense. We updated the abstract, Sec. 6.2, and Fig. 2a (new); App. C keeps the released-layout results.

W1. We address the laptop GPU and its dynamic power limit in Q3 below and, with three HPC GPUs, in Q2 (Part 2), and model/dataset scope in Q6 (Part 2). Every new laptop run logs CPU-package energy via Intel RAPL; including it shifts G of the channel allocations by ≤0.29 points and barely changes the CSR gain (W3), so our conclusions stand (App. E, new).

Q1. Yes, the channel-structured conclusions hold under the full schedules: neither energy-aware nor reverse allocation beats the uniform curve (every CI includes zero or lies below it; General response, Table A). The initial submission already propagated uncertainty in both energy and accuracy (pareto_v72.py resampled both axes for each allocation and uniform reference), though the caption (L. 330) said only "over seeds". In the revision, we ran the full schedules (AlexNet 30, VGG-16 20 epochs, vs. 8/5 initially) across five laptop blocks and three blocks on each of three HPC GPUs, each pairing a dense reference with uniform runs at seven densities (0.035–0.3) chosen to bracket the allocations' energies, and added per-block intervals that pair energy and accuracy within each run.

The table below gives one-sided 95% bounds on what the data exclude (we take 1 point at equal energy, or 5% energy saved at equal accuracy, as meaningful):

Model (GPU)EA+ / EPW: gain at equal energy below (pts)Energy saving at equal accuracy below (%)
AlexNet (laptop)−0.45 / +0.11−3.2 / 0.0
VGG-16 (laptop)+0.52 / −0.72+13.5 / −7.0
VGG-16 (A40)−0.17 / −0.48−2.2 / −2.5
VGG-16 (L40)0.00 / −1.05−0.3 / +5.1
VGG-16 (RTX PRO 6000)+0.70 / −1.44+21.3 / +97.1

A 1-point accuracy gain is ruled out in every row, and a 5% energy saving on the laptop and the A40, except for EA+ on VGG-16 (laptop), where the uniform curve is nearly flat in accuracy (+0.5 points from d = 0.1 to 0.14).

Q3. Yes, they are coupled. The initial submission noted this shift (App. B, L. 1090–1092: GPU-bound runs near 44 W vs. CPU-bound runs near 35 W), and the revised Sec. 5 explains why: NVIDIA Dynamic Boost shifts the GPU limit based on the CPU's share of a shared power budget. Runs with light host work get 38–45 W, while runs with heavier host activity (sparse-tensor masking or thin AlexNet channel models) sit near 35 W. When Windows throttled the host process in one queue, the platform raised the GPU limit from 35 W to a median of 38 W (App. E, new; those runs are set aside). The initial comparisons are unaffected because:

  1. Compared runs sit in the same power regime (all AlexNet allocations at d ≤ 0.1 ran at 35.0–35.4 W, matching their uniform references).
  2. GPU power stays within 1 W of the active limit.
  3. GPU energy tracks runtime (r = 0.992 and 0.999).

(continued in Part 2)

As in Part 1, L. gives line numbers in the initial submission, and "new" marks material added in the revised PDF.

Q2. Yes. On an HPC cluster, we repeated the full-schedule channel-structured experiments on three data-center GPUs whose power limits stayed fixed in every epoch of all 78 runs per GPU (3 blocks each): an NVIDIA A40 and an L40 at 300 W, and an RTX PRO 6000 at 600 W. On VGG-16, EA+ and EPW lie on or below the uniform curve on all three (G = −0.78/−0.50/−0.49 and −1.18/−1.45/−2.74; Table A). On AlexNet, the thin sparse models are host-bound on these GPUs: a sparse epoch takes longer than a dense one at every density (A40: 2.15–2.21 s vs. 1.36 s). At equal density, EA+ changes accuracy by −1.00 to −0.07 points and EPW by −1.39 to −0.91 points; the only significant energy difference is +6.7% [2.1, 11.3] for EA+ on the A40.

W2. Our new full-schedule experiments focus on the channel-structured back end because, as you note, it alone can be cheaper than dense training on our four GPUs. The other two back ends answer complementary questions:

  1. Masked execution reflects how most published DST code measures energy. Our results show that layer-wise allocation cannot change its energy and that it costs more than dense training (Recommendation 2, L. 462–465).
  2. Skip-zero CSR tests what happens when kernel cost tracks kept weights. With the view fix, allocation pays precisely where Prop. 4.1 predicts leverage (AlexNet, all four GPUs; W3). For faster sparse kernels or hardware accelerators that do beat dense execution, Tab. 1 shows where allocation can help: on VGG-16, ResNet-34, and MobileNetV2, EA+ changes compute by ≤2%, so it cannot pay on any such platform.

Q4. We measured both ends of the block-size range on all four GPUs: whole channels (our channel back end runs cuDNN/cuBLAS kernels on active slices, L. 254) and single weights (CSR, whose cost now rises with density; W3). Block-sparse kernels sit in between, and the leverage identity predicts their energy, as our limitations proposed (L. 474–476): an allocation costs about r⋅Euniformr\cdot E_{\rm uniform} (rr from Tab. 1), and accuracy decides whether it pays. We have not run block-sparse or 2:4 kernels directly. For 2:4 sparsity (supported on our Ampere GPUs), each layer is dense or 2:4, so allocation is indeed a 0/1 knapsack problem: at global density d∈[0.5,1)d\in[0.5,1), choose 2:4 layers SS with ∑l∈SNl=2(1−d)N\sum_{l\in S}N_l=2(1-d)N to maximize the saving ∑l∈SAl(1−1/σl)\sum_{l\in S}A_l(1-1/\sigma_l), where σl\sigma_l is layer ll's 2:4 speedup. As the saving per weight is cl(1−1/σl)c_l(1-1/\sigma_l), layers with the highest cost per weight clc_l are again best when speedups are similar. EA+, which ranks layers by total cost, misses these layers on VGG-16, ResNet-34, and MobileNetV2 (added to Sec. 4 and the new App. E). Full 2:4 training also needs 2:4 kernels for transposed backward products.

Q5. Since both CNNs use batch normalization, fixing the initial channel subsets without rewiring (ζ₀ = 0) trains a narrower dense network. At d = 0.1, this static baseline is more accurate than uniform channel DST at nearly the same energy:

  • Laptop (5 blocks): +0.86 [+0.26, +1.47] points on AlexNet (+1.3% energy) and +1.01 [+0.65, +1.37] points on VGG-16 (−0.6% energy).
  • HPC GPUs (A40 / L40 / RTX PRO 6000, 3 blocks each): +1.60 / +0.81 / +1.37 points on AlexNet and +0.71 / +0.97 / +0.85 points on VGG-16, at −11% to +7% energy; all six estimates are positive, and three CIs exclude zero (App. E).

Dynamic channel rewiring thus does not pay off on these models and schedules (now noted in Sec. 6.3 and limitations). Our allocation comparisons are unaffected because every allocation uses the same rewiring rule.

Q6. The allocation score requires only one forward pass and applies to any architecture; the main restriction is empirical scope, as our controlled runs cover two CIFAR-10 CNNs (now stated in the abstract). Three results extend beyond that setting:

  1. Without training, Prop. 4.1 shows that unstructured reallocation changes the compute of Transformer linear layers by ≤1% (ViT), 0.7% (GPT-2 small), and 0.1% (LLaMA-7B shape) (L. 193–204, Tab. 1).
  2. Our legacy benchmark points the same way under masked execution, across six architectures (including ViT, ConvNeXt, and MobileNetV2) on CIFAR-100 and Tiny-ImageNet with whole-device energy (App. A.1, L. 792–794).
  3. New in the revision: applying the identity to structured Transformer pruning yields a head-to-MLP cost-per-weight ratio of 1+T/(2D)1+T/(2D) (Sec. 4).

In summary, we corrected the CSR experiments, which confirmed the leverage identity's prediction on the laptop and on three fixed-limit HPC GPUs, and expanded the channel-structured evaluation to full schedules, five laptop blocks, explicit exclusion bounds, CPU-package energy logging, and the same three HPC GPUs. We would be glad to discuss any remaining concerns.

AuthorsOpenReview ↗

Follow-up for Reviewer xxxQ

Our two-part reply answers your main points with new runs: (1) the CSR rerun with the transposed-view layout (W3), in which EA+ now beats uniform allocation at equal energy on AlexNet, as the leverage identity predicts (α = 0.5: +1.02 points [+0.60, +1.44] on the laptop, +0.65 to +0.75 on three data-center GPUs); (2) the channel-structured comparison at the full 30/20-epoch schedules, with uniform networks at seven densities chosen to bracket the allocations' energies and one-sided 95% bounds that rule out a 1-point gain at equal energy (Q1); (3) both comparisons on the three data-center GPUs, whose power limits stayed fixed (Q2); and (4) CPU-package energy in every new laptop run (W1) and the static narrow network (Q5). We are happy to answer any question before the discussion closes and would be glad to know whether our replies and the revised PDF address your concerns. If they do, we would be grateful if you would consider updating your score.

Reviewer_mgfDOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a theoretical leverage identity that limits the energy costs of any layer-wise allocation prior to training to investigate if energy aware sparsity improves energy efficiency for dynamic sparse training. The method is empirically evaluated on three execution backends, and presents their results in terms of real hardware energy measures. The authors find that measured hardware energy consumption is tied to the execution backend and global density, not the layer-wise allocation. In addition, the authors also investigate the score-weighted regrowth, showing that unanchored energy regrowth follows a replicator dynamic that collapses the network, and they instead propose a fix by anchoring the regrowth.

Critical strengths
  1. This paper moves away from measuring FLOPs and parameter counts to actual hardware energy across execution backends.The findings show that standard masked DST implementations essentially offer an illusion of energy savings while not explicitly providing any. This provides a crucial insight that I think will benefit the community, suggesting that layerwise sparsity allocation matters far less than the execution backend. This potentially provides a new standard for how we should evaluate the true cost of sparse training.
  2. Additionally, the authors introduce two theoretical frameworks that explain why these failures happen. The first one leverage identity aims to prove why simply moving unstructured sparsity around in architectures like transformers yields zero energy savings. Second, by modelling DST regrowth the authors explain why energy aware regrowth tends to collapse layers provides an anchor mechanism to fix it. These are useful insights that give tools for designing future algorithms alongside how to more appropriately measure them.
Critical weaknesses
  1. The controlled empirical measurements rely heavily on a single gpu using smallscale datasets and short training times on more outdated architectures. While the theoretical claims do not rely on the hardware, the empirical behaviours such as the severe non-monotonicity of the cuSPARSE kernels might change significantly on hardware that is typically equipped with sparse cores. While I do not expect modern hardware for acceptance, the claims need to be tempered in that they should not be so broad without the empirical evidence to back it up.
  2. The papers controlled study measures only gpu energy via nvml. While okay for gpu computation, dynamic sparse training often moves computational bottlenecks from the gpu to the cpu and memory. Therefore I could argue that by ignoring cpu and ram energy in the experimental controls, the paper misses a significant amount of the true energy cost of DST algorithms.
Questions for authors
  1. As alluded to in the weaknesses, are the negative empirical results tied to the specific hardware of the gpu used in this studty? Could the sparse kernel capabilities of other gpu accelerators change the underlying calculations against energy-aware allocation?
  2. Do the papers claims still hold for structured sparsity on transformers, or does pruning at the head or dimension level change the memory footprint enough to negate the energy savings?
4 comments

Thank you for your constructive review. (L. gives line numbers in the initial submission; "new" marks material added in the revised PDF.)

W1 / Q1. Whether an allocation can save energy depends on the network, not the GPU; how much a kernel saves depends on the hardware, as noted in our limitations (L. 474–476).

  • Schedules, datasets, and architectures: At the full schedules, the channel-structured comparison reaches the same conclusion on all four GPUs; it was the only one initially shortened (8/5 epochs vs. 30/20 elsewhere, L. 274–276). Beyond the two controlled CNNs, Tab. 1 analyzes eight models (including ConvNeXt, ViT, GPT-2 small, and a LLaMA-7B shape), and our legacy benchmark trained six architectures on CIFAR-10/100 and Tiny-ImageNet (L. 792–794).
  • A single GPU: As Reviewer xxxQ also asked, we repeated both comparisons on three data-center GPUs of an HPC cluster with fixed power limits (A40, L40, and a Blackwell RTX PRO 6000). The severe cuSPARSE non-monotonicity came from one SDDMM operand layout (our reply to Reviewer xxxQ, W3); with it fixed, CSR cost rises with density on all four GPUs, and EA+ gains at equal energy on AlexNet, the one model where Tab. 1 shows captured leverage (+1.02 points on the laptop, +0.65 to +0.75 on the HPC GPUs), while on VGG-16 it changes nothing. No channel-structured allocation beats uniform on any GPU (Q2 of that reply; General response, Table A).
  • Hardware-independent findings: Under masked kernels, no allocation changes the arithmetic (Prop. 4.1(c), L. 177). Under kernels whose cost scales with kept weights, Prop. 4.1 bounds any allocation's saving from one forward pass (Tab. 1, L. 216): ≤1% for Transformer linear layers. On VGG-16, ResNet-34, and MobileNetV2, EA+ changes compute by ≤2% as it ranks layers by total cost, not cost per weight (L. 211). Regrowth collapse (Prop. 6.1) is a property of the update rule, and test accuracy is independent of the back end.
  • Sparse hardware: NVIDIA's sparse tensor cores accelerate only 2:4 sparsity, where allocation reduces to choosing which layers run in 2:4 mode and Prop. 4.1 still governs the savings (our reply to Reviewer xxxQ, Q4). On an ideal unstructured accelerator, EA+ on AlexNet (α = 0.5) would sit +0.48 points [+0.10, +0.88] above the uniform curve at equal compute.
  • Tempering the claims: As suggested, the revised abstract now states the setting (two CNNs on CIFAR-10; one laptop and three data-center GPUs) and says where allocation matters instead of ruling it out.

W2. The legacy benchmark measured whole-system energy (CPU, GPU, and RAM via CodeCarbon, App. A, L. 809); there, energy-aware initialization changed energy by −0.21% vs. uniform (L. 899). Every new laptop run also logs CPU-package energy (Intel RAPL), now reported in the new Tab. 2 and App. E:

  • Masked sparse-tensor execution is host-bound and 1.3–2.4× slower than dense training (L. 295), so including CPU energy makes it even costlier.
  • Skip-zero CSR: The CPU package uses 0.17–0.62× the GPU energy. Including it, the equal-energy gain G of EA+ on AlexNet is +0.90 and +0.95 points (vs. +1.02 and +0.90 for GPU only).
  • Channel-structured back end: The CPU package uses 0.25–0.71× the GPU energy on AlexNet and 0.21–0.34× on VGG-16 (dense: 0.23× and 0.19×). With it, uniform allocation at d = 0.1 costs 0.57× and 0.43× dense (vs. 0.44× and 0.41× for GPU only). Across allocations, G shifts by ≤0.29 points, while the static narrow baseline's G on AlexNet rises from +0.82 to +1.76.
  • RAM energy: A constant term for unmeasured host power such as DRAM (E=EGPU+Ph tE=E_{\rm GPU}+P_h\,t) shifts the initial comparisons by ≤0.2 points in G and 0.7 percentage points in CSR energy, even at PhP_h = 45 W; the laptop exposes no DRAM counter.

Thank you for raising this point: with CPU energy counted, every conclusion holds.

Q2. Our Transformer statement (L. 193–204) concerns unstructured sparsity, where every linear layer performs TT MACs per weight, so reallocating weights across layers cannot cut compute. For structured pruning, Prop. 4.1 shows genuine leverage: an attention head of width dhd_h has 4Ddh4Dd_h weights and 4DdhT+2T2dh4Dd_hT+2T^2d_h MACs (attention scores carry no weights), whereas an MLP unit has 2D2D weights and 2DT2DT MACs. Their cost-per-weight ratio is

chead/cMLP=1+T/(2D),c_{\rm head}/c_{\rm MLP}=1+T/(2D),

or 1+T/(4D)1+T/(4D) if kernels skip the causally masked half: 1.67 (or 1.33) for GPT-2 small at T=1024T=1024. Head-level allocation thus offers leverage growing with TT. Physically removing heads or MLP units keeps the remaining tensors dense without index overhead and shrinks attention activations, so memory traffic falls with compute instead of negating the savings. We added this derivation to Sec. 4.

We would be glad to answer any further questions.

AuthorsOpenReview ↗

Follow-up for Reviewer mgfD

Our reply addresses both weaknesses and both questions: (1) on three data-center GPUs with fixed power limits, both comparisons reach the same conclusions as on the laptop (EA+ gains under CSR on AlexNet; no channel-structured allocation beats uniform), and the severe cuSPARSE non-monotonicity came from one operand layout, whose correction makes CSR cost rise with density on all four GPUs (W1/Q1); (2) with CPU-package energy, measured in every new laptop run, every conclusion holds, and a conservative bound covers DRAM (W2); (3) the revised abstract states the setting, tempering the claims as you suggested (W1); and (4) Sec. 4 now derives the head-to-MLP cost-per-weight ratio for structured Transformer pruning (Q2). We are happy to answer any question before the discussion closes and would be glad to know whether our reply and the revised PDF address your concerns. If they do, we would be grateful if you would consider updating your score.

Thank you to the authors for their detailed and thorough rebuttal.

My weaknesses have been primarily addressed with further empirical findings and an update to the manuscript which tempers the claims. In addition, the further experimentation provides additional support to the claims made, and i believe strengthens the work.

I would ask the authors to update the main manuscript with these additions.

Given my weaknesses have been primarily addressed I have raised my score.

Thank you for your constructive feedback, which made our work stronger, and for raising your score. The revised PDF brings additions into the main text, marked in blue, and in the final version we will add the DRAM bound from our reply.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful reading. All three valued our energy measurements across execution back ends and the two analyses (the leverage identity and the regrowth dynamics). Their shared concern is the evidence base, which the limitations of our initial submission already listed (L. 469–471): a single laptop GPU with a platform-managed power limit, GPU-only energy, two CIFAR-10 CNNs, and short channel-structured schedules. Reviewer xxxQ also found a valid point in our CSR back end. One new result changes a conclusion (first row), while the channel-structured conclusion holds. New and revised text is in blue in the revised PDF, and we also updated the OpenReview abstract.

A note on versions: "Initial submission" is the version the reviews refer to, and "revised PDF" is the revision uploaded with this comment. In our replies, L. gives line numbers in the initial submission, "new" marks material added in the revised PDF, and section, table, and appendix numbers refer to the revised PDF.

Concern (reviewers)What we did and foundWhere in the revised PDF
CSR operand layout (xxxQ)We reran the CSR comparison with the operand passed as a transposed view (6 blocks per GPU). CSR cost now grows with density, and EA+ beats uniform allocation at equal energy on AlexNet, as the leverage identity predicts: by +1.02 points [+0.60, +1.44] at α = 0.5 on the laptop and by +0.65 to +0.75 on three HPC GPUs. VGG-16 is unchanged.Abstract, Sec. 6.2, Tab. 2 (new), Fig. 2a (new), App. C, E
Short schedules; what the data exclude (mgfD, xxxQ, B1Lt)We reran the channel-structured comparison at the full 30/20-epoch schedules (5 laptop blocks, 3 per HPC GPU; uniform networks at 7 densities). No energy-aware or reverse allocation beats the uniform curve (Table A), and on the laptop one-sided 95% bounds rule out any gain above +0.52 points at equal energy.Sec. 6.3, Tab. 3 (new), App. E (new)
One GPU and its power limit (mgfD, xxxQ, B1Lt)We repeated both comparisons on three data-center GPUs of an HPC cluster with fixed limits (A40 and L40 at 300 W, RTX PRO 6000 at 600 W): EA+ again gains under CSR, and no channel-structured allocation beats uniform (Table A). The laptop's limit is coupled to CPU load through NVIDIA Dynamic Boost, but the compared configurations share one power regime.Sec. 5, 6.2–6.3, App. E (new)
CPU and RAM energy (mgfD, xxxQ)Every new laptop run logs CPU-package energy (RAPL). With it, channel-structured training stays cheaper than dense (0.57× and 0.43× at d = 0.1), G of the allocations changes by at most 0.29 points, and the CSR gain barely changes. With no DRAM counter, a conservative constant-power bound shifts G of the initial comparisons by at most 0.2 points.Tab. 2 (new), App. E (new)
Narrow network without rewiring (xxxQ)A static network that keeps its initial channels (ζ₀ = 0) is more accurate than channel DST at nearly equal energy (+0.86 and +1.01 points on the laptop; +0.71 to +1.60 on the HPC GPUs).Sec. 6.3
Transformers; 2:4 (mgfD, xxxQ)Prop. 4.1 gives a head-to-MLP cost-per-weight ratio of 1 + T/(2D) for structured pruning and turns 2:4 allocation into a 0/1 knapsack over layers.Sec. 4, App. E (new)
Presentation (xxxQ, B1Lt)The abstract now states the setting. A short guide to the back ends opens Sec. 6, the kernel analysis moved to App. C, and Table 10 (initially 9) and two equations now fit the page.Abstract, Sec. 6, App. A, C; Eq. 1

Table A: Channel-structured back end, full schedules: G is the accuracy minus the block's uniform curve at equal measured energy (mean over blocks; 95% t-intervals for the laptop, in App. E for the HPC GPUs).

Modeld = 0.1Laptop: energy (× dense)Acc. (%)G [95% CI]HPC GPUs: G (A40 / L40 / RTX PRO 6000)
AlexNetUniform0.4478.85——
EA+ (α = 1)0.4578.01−0.87 [−1.41, −0.33]host-bound*
EPW (ν = 0.25)0.4277.03−0.77 [−1.91, +0.37]host-bound*
Reverse EPW0.5579.28−0.17 [−0.97, +0.63]host-bound*
Static0.4579.71+0.82 [+0.15, +1.49]; +1.76 with CPU†host-bound*
VGG-16Uniform0.4186.22——
EA+ (α = 1)0.4286.38+0.04 [−0.58, +0.66]−0.78 / −0.50 / −0.49
EPW (ν = 0.25)0.3283.06−1.19 [−1.81, −0.58]−1.18 / −1.45 / −2.74
Reverse EPW0.6687.27−0.39 [−0.69, −0.09]−0.31 / −0.55 / −0.20
Static0.4187.23+1.05 [+0.67, +1.44]+0.56 / +0.09 / +0.78

*On the HPC GPUs, the thin AlexNet networks are host-bound (the uniform curve is flat), so we compare them at equal density instead (App. E, new). †The static runs draw less CPU power; at equal density, the static network is 0.86 points more accurate either way.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.