- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates whether layer-wise sparsity allocation can reduce actual training energy. It combines theoretical analysis, GPU energy measurements across three computational implementations, and an analysis of regrowth stability. In the tested settings, energy-aware allocation does not outperform uniform allocation at an appropriate density when compared at equal energy.
Critical strengths
- Practically relevant evaluation. Direct GPU energy measurements and accuracy comparisons at equal energy provide useful evidence about the practical benefits of sparsity.
- Informative analysis. The analysis of per-weight costs and regrowth imbalance helps explain both the potential savings and the observed failure modes.
Critical weaknesses
Limited generalizability of the empirical findings. The controlled experiments cover one laptop GPU, two CNNs, and CIFAR-10 with short training schedules. Given the dependence on computational implementation, it remains unclear whether the same energy–accuracy relationship between energy-aware and uniform allocation holds after longer training or on different hardware.
Questions for authors
Does the relative accuracy of energy-aware and uniform allocation at equal energy remain consistent under longer training schedules, particularly for the channel-structured experiments? Could the authors provide supporting evidence?
4 comments
Follow-up for Reviewer B1Lt
Our reply answers your question and weakness with new runs: at the full 30/20-epoch schedules (five randomized blocks) and again on three data-center GPUs with fixed power limits, the channel-structured comparison shows the same energy–accuracy relationship between energy-aware and uniform allocation (General response, Table A). The leverage identity and our legacy benchmark (six architectures on CIFAR-10/100 and Tiny-ImageNet) cover other models and datasets. We are happy to answer any question before the discussion closes and would be glad to know whether our reply and the revised PDF address your concerns. If they do, we would be grateful if you would consider updating your score.
Thank you for the detailed rebuttal. The full-schedule runs, the additional GPUs, and the corrected CSR results address most of my concerns, and I have raised Soundness and Presentation to 3. However, the controlled experiments still cover only two CNNs on CIFAR-10, and the main positive result for energy-aware allocation (CSR on AlexNet) comes from configurations that cost more than dense training. I therefore maintain my rating of 3 (Weak Acceptance).
Thank you for your constructive feedback, which helped us strengthen the paper, and for raising Soundness and Presentation; we are glad that our responses and new results address most of your concerns. The paper states both remaining points as limitations, and in the final version, we will add to the main text our estimate of the AlexNet gain on an ideal skip-zero accelerator, on which sparse training costs less than dense training.
Thank you for your positive review and for highlighting our equal-energy comparisons and regrowth analysis. (L. gives line numbers in the initial submission; "new" marks material added in the revised PDF.)
Q. Yes. In the initial submission, the masked and CSR back ends already used the full schedules (30 epochs for AlexNet, 20 for VGG-16), and only the channel-structured comparison was shortened to 8/5 epochs to fit nine configurations per block (L. 274–276). We have now rerun the channel-structured comparison at the full 30/20-epoch schedules across five randomized blocks (up from three), each including uniform networks at seven densities (0.035–0.3) so that every allocation is evaluated against uniform networks at matched measured energy (General response, Table A). While longer training raises absolute test accuracy across the board (for uniform at d = 0.1, AlexNet rises from 69.8% to 78.9% and VGG-16 from 76.4% to 86.2%), the relative conclusion is unchanged: no energy-aware allocation beats the uniform trade-off curve (every 95% CI includes zero or lies below it). Below is the equal-energy gain G (in percentage points at d = 0.1) comparing the initial short schedules (L. 381–384) to the new full schedules:
On three data-center GPUs of an HPC cluster, the same full schedules lead to the same conclusion (see W below).
W. The same energy–accuracy relationship holds after longer training and on different hardware, and two further results cover other models and datasets:
All new full-schedule, HPC-GPU, and corrected CSR results are integrated into the revised Sec. 6.2 and Sec. 6.3 (new Tables 2 and 3) and App. E. We are happy to answer any further questions.