- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces BARTER, a dynamic sparse training method for Transformers. It keeps the model physically compact during training by reallocating attention heads and feed-forward channels across layers: it prunes less useful active units and clones useful ones, guided by activation and gradient statistics. This lets it use standard dense kernels without computing gradients for inactive units. In experiments on CIFAR-100 and Tiny-ImageNet, the authors report competitive classification accuracy, faster training, and lower GPU memory use than several full-tensor sparse-training baselines. A smartphone benchmark also reports a speedup over RigL.
Critical strengths
-
A practical solution to efficient dynamic sparse training. BARTER combines decisions based only on active units with physical tensor resizing and optimizer-state remapping, allowing capacity reallocation while training a compact model with standard dense kernels. This is consequential because it translates sparsity into actual execution savings without requiring specialized sparse kernels. The measured GPU and smartphone speedups substantiate the practical value of this design.
-
Evidence that dynamic reallocation improves compact-model training. Compared with static compact models using the same initial architecture and training settings, BARTER achieves higher mean accuracy at every evaluated density on both tasks, including gains of up to 2.97 percentage points on CCT/CIFAR-100, with modest additional training overhead. This comparison strengthens the case for the proposed reallocation mechanism by demonstrating its benefit under comparable model size and computation.
Critical weaknesses
-
The efficiency evaluation does not establish an end-to-end advantage over optimized alternatives. The headline speedups compare BARTER with baselines that retain full-size tensors under dense-kernel execution. Moreover, GPU timings use replayed epoch-length segments rather than complete training runs (Appendix D.1). These results demonstrate savings in the evaluated setting, but leave the overall wall-clock cost of reaching comparable accuracy—and competitiveness with optimized sparse implementations—unresolved. This limits the evidence for the paper’s central efficiency contribution.
-
The experimental scope is too narrow to establish scalability. Evaluation is restricted to two small vision models on CIFAR-100 and Tiny-ImageNet, with fine-tuning between these same datasets. Training uses FP32, without mixed precision (Appendix B.2). Consequently, the paper does not establish whether its accuracy–efficiency tradeoff persists for larger models, higher-resolution inputs, or more optimized training configurations. This is consequential because the practical value of dynamic tensor resizing depends on how its benefits and overhead scale
Questions for authors
Would BARTER still outperform a static compact model trained from scratch using BARTER’s final learned allocation of attention heads and FFN channels, under the same training compute budget? This comparison would clarify whether the benefit comes from dynamically reallocating capacity during optimization or primarily from discovering a better fixed architecture.
4 comments
Response to Reviewer nF6Z [2/3]
W2: Larger models, higher-resolution inputs, and mixed precision
ViT-S/16 on ImageNet-1K. The full-density model has 22.05M parameters, approximately eight times the size of our Tiny-ImageNet ViT, with 224 × 224 inputs and 1,000 classes. We evaluate densities 0.1, 0.3, 0.5, and 0.7. All methods train from scratch for 100 epochs on an NVIDIA L40, with the same optimization and augmentation recipe. BARTER maintains competitive ACC and macro-F1 across all four densities, often achieving the best results among the compared DST methods, while reducing measured training time per epoch by 21–34% relative to the fastest DST baseline at each density. At density 0.3, it achieves 67.18% ACC and 66.56% macro-F1, exceeding the strongest DST baseline by 1.46 and 1.48 percentage points. Table 6 in Appendix C.1 reports all four densities.
The larger model also retains the efficiency benefits. Figure 7 reports directly measured epoch-length profiling segments and peak allocated GPU memory, with score collection and scheduled structural updates included. BARTER has the lowest measured time and peak memory at every density. At density 0.5, it uses 3.168 GiB versus 4.730–5.042 GiB for the five DST baselines. Figure 8 also reports 49.3% lower estimated training computation: 13.990 versus 27.593 GFLOPs per image. Together with the complete-run timings in W1, these results show that compact execution continues to save time and memory at this larger scale with reallocation included.
BF16 mixed precision. To test whether these benefits persist when all methods use mixed precision, Appendix C.3 evaluates CCT/CIFAR-100 at all four densities over three seeds. All methods use PyTorch BF16 automatic mixed precision. Table 10 reports classification results, and Figure 10 reports directly measured epoch-length training segments and peak memory under the protocol in Appendix D.1. Two representative densities are shown below; values are mean ± sample standard deviation.
| Density | Method | ACC (%) | Training time (s/segment) | Peak allocated memory (GiB) |
|---|---|---|---|---|
| 0.3 | RigL | 71.48 ± 0.32 | 68.43 ± 0.04 | 8.938 ± 0.000 |
| 0.3 | BARTER | 71.40 ± 0.35 | 25.37 ± 0.77 | 3.246 ± 0.046 |
| 0.5 | RigL | 73.01 ± 0.09 | 68.43 ± 0.05 | 8.938 ± 0.000 |
| 0.5 | BARTER | 73.04 ± 0.13 | 38.00 ± 0.17 | 4.825 ± 0.001 |
At density 0.3, BARTER achieves similar mean ACC to RigL with a 2.70× speedup in measured segments and 63.7% lower peak memory. At density 0.5, the corresponding gains are 1.80× and 46.0%. BARTER has the lowest measured time and peak memory at all four densities, and the highest mean ACC and macro-F1 among DST methods at 0.5 and 0.7. Thus, the compact-training gains remain when the baselines also benefit from BF16.
Within the rebuttal time and compute available, we prioritized these two evaluations. In future work, we plan to extend the evaluation to even larger Transformers, language models, longer sequences, and distributed training.
Response to Reviewer nF6Z [3/3]
Question: Training a static compact model with BARTER's final allocation
We have added the requested Learned Static experiment in Appendix C.2. BARTER and Learned Static achieve similar mean accuracy. Their shared improvement over Initial Static identifies the learned allocation as an important source of the gains.
Initial Static keeps BARTER's initial layer-wise head counts and FFN widths fixed. BARTER starts from that allocation and adjusts it during training. Learned Static takes the final allocation from the corresponding BARTER run at each density and seed, randomly initializes all weights, and trains that fixed architecture from scratch.
All three use the same capacity budgets, training recipe, and epoch counts. Table 9 reports matching parameter counts and estimated training GFLOPs per image under Appendix D.2's counting protocol. Table 8 reports measured execution costs, including reallocation. Below, ACC is mean ± sample standard deviation over three seeds; Table 7 also reports macro-F1.
| Model / dataset | Density | Initial Static | BARTER | Learned Static |
|---|---|---|---|---|
| CCT / CIFAR-100 | 0.1 | 63.16 ± 0.27 | 66.13 ± 0.40 | 66.76 ± 0.24 |
| CCT / CIFAR-100 | 0.3 | 70.43 ± 0.13 | 71.38 ± 0.36 | 71.01 ± 0.18 |
| CCT / CIFAR-100 | 0.5 | 71.80 ± 0.14 | 72.85 ± 0.16 | 72.24 ± 0.15 |
| CCT / CIFAR-100 | 0.7 | 72.99 ± 0.46 | 73.26 ± 0.28 | 73.33 ± 0.40 |
| ViT / Tiny-ImageNet | 0.1 | 42.15 ± 0.81 | 42.56 ± 0.86 | 42.85 ± 0.41 |
| ViT / Tiny-ImageNet | 0.3 | 51.08 ± 0.43 | 51.42 ± 0.39 | 51.57 ± 0.31 |
| ViT / Tiny-ImageNet | 0.5 | 52.22 ± 0.67 | 52.77 ± 0.23 | 52.54 ± 0.41 |
| ViT / Tiny-ImageNet | 0.7 | 52.37 ± 0.28 | 53.00 ± 0.28 | 52.73 ± 0.21 |
BARTER and Learned Static achieve similar mean accuracy across the evaluated settings, and both outperform Initial Static in mean ACC and macro-F1 at every tested density on both tasks. Learned Static's ACC gains over Initial Static range from 0.34 to 3.60 percentage points on CCT and from 0.32 to 0.70 points on ViT. These improvements after weight reinitialization show that the allocations discovered by BARTER provide better compact structures for training.
These results suggest that a fixed compact model can attain comparable accuracy once an effective allocation is available. That allocation still needs to be found. In our comparison, Learned Static receives the allocation discovered by a completed BARTER run. Dynamic reallocation provides this structure-discovery step, so finding a better fixed architecture is itself a contribution of BARTER's dynamic training.
BARTER performs this discovery while optimizing the model weights, directly producing a trained compact model in one run without a separate retraining stage. Its training tensors remain physically compact throughout, and the measured overhead over fixed compact training is modest. For example, at CCT density 0.1, Initial Static, BARTER, and Learned Static take 11.53, 11.90, and 11.88 seconds per measured epoch-length segment, respectively, with similar peak memory.
A gentle reminder
Dear Reviewer nF6Z,
We would like to briefly follow up on our response to your review. We added ViT-S/16 experiments on ImageNet-1K with complete-run training-loop measurements, BF16 mixed-precision experiments, and the requested Learned Static comparison (Appendices C.1–C.3). We also explained the reason for our dense-kernel comparisons. Suitable sparse kernels are not available on every device, while masking full-sized tensors in existing dense-kernel DST implementations does not reduce dense computation. BARTER addresses this gap by discovering and training compact structures on standard dense kernels, with actual computation and training-time savings.
We would be grateful to know whether these address your concerns, and would be happy to provide any further clarification. Thank you for your time and feedback.
Response to Reviewer nF6Z [1/3]
We sincerely thank you for your careful review and constructive suggestions. In response, we have added complete-training measurements on ViT-S/16 and ImageNet-1K, BF16 mixed-precision experiments, and the requested static comparison using BARTER's learned allocation. The new results are reported in Appendices C.1–C.3. We have updated the manuscript PDF, with additions and revisions highlighted in blue.
The prevailing dense-kernel DST implementation retains full-sized masked tensors, so changing the masks does not shrink the executed operations. Specialized sparse kernels can exploit sparsity, but suitable kernels are not available on every device and require adaptation to the sparsity pattern and backend.
BARTER discovers and trains compact Transformer structures directly with widely supported standard dense kernels. It uses active-unit statistics to reallocate attention heads and FFN channels, physically resizes the affected tensors, and remaps their optimizer states so training continues on the updated compact model. This allows devices without suitable sparse-kernel support to perform dynamic structure discovery while training directly on smaller dense tensors. Our reported speedups therefore compare BARTER with existing dense-kernel DST implementations and quantify the gains achieved through standard dense execution.
To evaluate whether these execution savings translate into faster attainment of comparable accuracy, we have added cumulative training-loop measurements over complete 100-epoch runs of ViT-S/16 on ImageNet-1K at density 0.5. Figure 9 in Appendix C.1 plots validation accuracy against this recorded time, including score collection, tensor resizing, and optimizer-state remapping.
BARTER reaches the shared accuracy target sooner and completes the full schedule in less training-loop time than every DST baseline in these runs. At the same density, it also achieves the highest ACC and macro-F1 on the separate test set among the compared DST methods: 71.90% and 71.43%, respectively, as reported in Table 6.