acceptodds
Under review as a conference paper at ICLR 2027

BARTER: Compact Dynamic Sparse Training with Standard Dense Kernels

Abstract

Dynamic sparse training (DST) aims to reduce training costs while maintaining accuracy. Existing DST methods mask out parameters but keep matrix dimensions unchanged. Training speedups therefore depend on specialized support, such as sparse kernels. We introduce BARTER, a structured DST framework that reduces training cost using standard dense kernels by keeping training tensors physically compact while dynamically reallocating capacity across layers. Unlike existing works that require computing gradients for all parameters, BARTER instead uses statistics collected only from active units during ordinary backpropagation to prune and clone attention heads and feed-forward channels under separate fixed global budgets. At each topology update, it resizes the affected parameter tensors and remaps their optimizer states, so subsequent training steps operate directly on the compact model. Training and fine-tuning experiments with Transformer-based vision models on two image classification benchmarks at densities from 0.1 to 0.7 show competitive Top-1 accuracy relative to full-tensor DST baselines. On an NVIDIA L40, BARTER achieves speedups of up to 6.29× in measured training segments and reduces peak allocated GPU memory by up to 81.3% relative to these implementations. An iPhone 13 Pro Max training benchmark additionally shows a 2.78× speedup over the full-tensor DST baseline.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Dynamic Sparse Training, Structured Sparsity, Efficient Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nF6ZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces BARTER, a dynamic sparse training method for Transformers. It keeps the model physically compact during training by reallocating attention heads and feed-forward channels across layers: it prunes less useful active units and clones useful ones, guided by activation and gradient statistics. This lets it use standard dense kernels without computing gradients for inactive units. In experiments on CIFAR-100 and Tiny-ImageNet, the authors report competitive classification accuracy, faster training, and lower GPU memory use than several full-tensor sparse-training baselines. A smartphone benchmark also reports a speedup over RigL.

Critical strengths
  1. A practical solution to efficient dynamic sparse training. BARTER combines decisions based only on active units with physical tensor resizing and optimizer-state remapping, allowing capacity reallocation while training a compact model with standard dense kernels. This is consequential because it translates sparsity into actual execution savings without requiring specialized sparse kernels. The measured GPU and smartphone speedups substantiate the practical value of this design.

  2. Evidence that dynamic reallocation improves compact-model training. Compared with static compact models using the same initial architecture and training settings, BARTER achieves higher mean accuracy at every evaluated density on both tasks, including gains of up to 2.97 percentage points on CCT/CIFAR-100, with modest additional training overhead. This comparison strengthens the case for the proposed reallocation mechanism by demonstrating its benefit under comparable model size and computation.

Critical weaknesses
  1. The efficiency evaluation does not establish an end-to-end advantage over optimized alternatives. The headline speedups compare BARTER with baselines that retain full-size tensors under dense-kernel execution. Moreover, GPU timings use replayed epoch-length segments rather than complete training runs (Appendix D.1). These results demonstrate savings in the evaluated setting, but leave the overall wall-clock cost of reaching comparable accuracy—and competitiveness with optimized sparse implementations—unresolved. This limits the evidence for the paper’s central efficiency contribution.

  2. The experimental scope is too narrow to establish scalability. Evaluation is restricted to two small vision models on CIFAR-100 and Tiny-ImageNet, with fine-tuning between these same datasets. Training uses FP32, without mixed precision (Appendix B.2). Consequently, the paper does not establish whether its accuracy–efficiency tradeoff persists for larger models, higher-resolution inputs, or more optimized training configurations. This is consequential because the practical value of dynamic tensor resizing depends on how its benefits and overhead scale

Questions for authors

Would BARTER still outperform a static compact model trained from scratch using BARTER’s final learned allocation of attention heads and FFN channels, under the same training compute budget? This comparison would clarify whether the benefit comes from dynamically reallocating capacity during optimization or primarily from discovering a better fixed architecture.

4 comments
AuthorsOpenReview ↗

Response to Reviewer nF6Z [1/3]

We sincerely thank you for your careful review and constructive suggestions. In response, we have added complete-training measurements on ViT-S/16 and ImageNet-1K, BF16 mixed-precision experiments, and the requested static comparison using BARTER's learned allocation. The new results are reported in Appendices C.1–C.3. We have updated the manuscript PDF, with additions and revisions highlighted in blue.

W1: Execution setting and training time to comparable accuracy

The prevailing dense-kernel DST implementation retains full-sized masked tensors, so changing the masks does not shrink the executed operations. Specialized sparse kernels can exploit sparsity, but suitable kernels are not available on every device and require adaptation to the sparsity pattern and backend.

BARTER discovers and trains compact Transformer structures directly with widely supported standard dense kernels. It uses active-unit statistics to reallocate attention heads and FFN channels, physically resizes the affected tensors, and remaps their optimizer states so training continues on the updated compact model. This allows devices without suitable sparse-kernel support to perform dynamic structure discovery while training directly on smaller dense tensors. Our reported speedups therefore compare BARTER with existing dense-kernel DST implementations and quantify the gains achieved through standard dense execution.

To evaluate whether these execution savings translate into faster attainment of comparable accuracy, we have added cumulative training-loop measurements over complete 100-epoch runs of ViT-S/16 on ImageNet-1K at density 0.5. Figure 9 in Appendix C.1 plots validation accuracy against this recorded time, including score collection, tensor resizing, and optimizer-state remapping.

MeasurementBARTER (hours)Five DST baselines (range, hours)Time reduction
First reach 70% validation ACC22.2428.00–45.7020.6–51.3%
Complete all 100 epochs34.3441.79–62.3517.8–44.9%

BARTER reaches the shared accuracy target sooner and completes the full schedule in less training-loop time than every DST baseline in these runs. At the same density, it also achieves the highest ACC and macro-F1 on the separate test set among the compared DST methods: 71.90% and 71.43%, respectively, as reported in Table 6.

AuthorsOpenReview ↗

Response to Reviewer nF6Z [2/3]

W2: Larger models, higher-resolution inputs, and mixed precision

ViT-S/16 on ImageNet-1K. The full-density model has 22.05M parameters, approximately eight times the size of our Tiny-ImageNet ViT, with 224 × 224 inputs and 1,000 classes. We evaluate densities 0.1, 0.3, 0.5, and 0.7. All methods train from scratch for 100 epochs on an NVIDIA L40, with the same optimization and augmentation recipe. BARTER maintains competitive ACC and macro-F1 across all four densities, often achieving the best results among the compared DST methods, while reducing measured training time per epoch by 21–34% relative to the fastest DST baseline at each density. At density 0.3, it achieves 67.18% ACC and 66.56% macro-F1, exceeding the strongest DST baseline by 1.46 and 1.48 percentage points. Table 6 in Appendix C.1 reports all four densities.

The larger model also retains the efficiency benefits. Figure 7 reports directly measured epoch-length profiling segments and peak allocated GPU memory, with score collection and scheduled structural updates included. BARTER has the lowest measured time and peak memory at every density. At density 0.5, it uses 3.168 GiB versus 4.730–5.042 GiB for the five DST baselines. Figure 8 also reports 49.3% lower estimated training computation: 13.990 versus 27.593 GFLOPs per image. Together with the complete-run timings in W1, these results show that compact execution continues to save time and memory at this larger scale with reallocation included.

BF16 mixed precision. To test whether these benefits persist when all methods use mixed precision, Appendix C.3 evaluates CCT/CIFAR-100 at all four densities over three seeds. All methods use PyTorch BF16 automatic mixed precision. Table 10 reports classification results, and Figure 10 reports directly measured epoch-length training segments and peak memory under the protocol in Appendix D.1. Two representative densities are shown below; values are mean ± sample standard deviation.

DensityMethodACC (%)Training time (s/segment)Peak allocated memory (GiB)
0.3RigL71.48 ± 0.3268.43 ± 0.048.938 ± 0.000
0.3BARTER71.40 ± 0.3525.37 ± 0.773.246 ± 0.046
0.5RigL73.01 ± 0.0968.43 ± 0.058.938 ± 0.000
0.5BARTER73.04 ± 0.1338.00 ± 0.174.825 ± 0.001

At density 0.3, BARTER achieves similar mean ACC to RigL with a 2.70× speedup in measured segments and 63.7% lower peak memory. At density 0.5, the corresponding gains are 1.80× and 46.0%. BARTER has the lowest measured time and peak memory at all four densities, and the highest mean ACC and macro-F1 among DST methods at 0.5 and 0.7. Thus, the compact-training gains remain when the baselines also benefit from BF16.

Within the rebuttal time and compute available, we prioritized these two evaluations. In future work, we plan to extend the evaluation to even larger Transformers, language models, longer sequences, and distributed training.

AuthorsOpenReview ↗

Response to Reviewer nF6Z [3/3]

Question: Training a static compact model with BARTER's final allocation

We have added the requested Learned Static experiment in Appendix C.2. BARTER and Learned Static achieve similar mean accuracy. Their shared improvement over Initial Static identifies the learned allocation as an important source of the gains.

Initial Static keeps BARTER's initial layer-wise head counts and FFN widths fixed. BARTER starts from that allocation and adjusts it during training. Learned Static takes the final allocation from the corresponding BARTER run at each density and seed, randomly initializes all weights, and trains that fixed architecture from scratch.

All three use the same capacity budgets, training recipe, and epoch counts. Table 9 reports matching parameter counts and estimated training GFLOPs per image under Appendix D.2's counting protocol. Table 8 reports measured execution costs, including reallocation. Below, ACC is mean ± sample standard deviation over three seeds; Table 7 also reports macro-F1.

Model / datasetDensityInitial StaticBARTERLearned Static
CCT / CIFAR-1000.163.16 ± 0.2766.13 ± 0.4066.76 ± 0.24
CCT / CIFAR-1000.370.43 ± 0.1371.38 ± 0.3671.01 ± 0.18
CCT / CIFAR-1000.571.80 ± 0.1472.85 ± 0.1672.24 ± 0.15
CCT / CIFAR-1000.772.99 ± 0.4673.26 ± 0.2873.33 ± 0.40
ViT / Tiny-ImageNet0.142.15 ± 0.8142.56 ± 0.8642.85 ± 0.41
ViT / Tiny-ImageNet0.351.08 ± 0.4351.42 ± 0.3951.57 ± 0.31
ViT / Tiny-ImageNet0.552.22 ± 0.6752.77 ± 0.2352.54 ± 0.41
ViT / Tiny-ImageNet0.752.37 ± 0.2853.00 ± 0.2852.73 ± 0.21

BARTER and Learned Static achieve similar mean accuracy across the evaluated settings, and both outperform Initial Static in mean ACC and macro-F1 at every tested density on both tasks. Learned Static's ACC gains over Initial Static range from 0.34 to 3.60 percentage points on CCT and from 0.32 to 0.70 points on ViT. These improvements after weight reinitialization show that the allocations discovered by BARTER provide better compact structures for training.

These results suggest that a fixed compact model can attain comparable accuracy once an effective allocation is available. That allocation still needs to be found. In our comparison, Learned Static receives the allocation discovered by a completed BARTER run. Dynamic reallocation provides this structure-discovery step, so finding a better fixed architecture is itself a contribution of BARTER's dynamic training.

BARTER performs this discovery while optimizing the model weights, directly producing a trained compact model in one run without a separate retraining stage. Its training tensors remain physically compact throughout, and the measured overhead over fixed compact training is modest. For example, at CCT density 0.1, Initial Static, BARTER, and Learned Static take 11.53, 11.90, and 11.88 seconds per measured epoch-length segment, respectively, with similar peak memory.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer nF6Z,

We would like to briefly follow up on our response to your review. We added ViT-S/16 experiments on ImageNet-1K with complete-run training-loop measurements, BF16 mixed-precision experiments, and the requested Learned Static comparison (Appendices C.1–C.3). We also explained the reason for our dense-kernel comparisons. Suitable sparse kernels are not available on every device, while masking full-sized tensors in existing dense-kernel DST implementations does not reduce dense computation. BARTER addresses this gap by discovering and training compact structures on standard dense kernels, with actual computation and training-time savings.

We would be grateful to know whether these address your concerns, and would be happy to provide any further clarification. Thank you for your time and feedback.

Reviewer_yM7HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper argues that existing works compute gradients for all parameters. In contrast, they use statistics collected only from active units during ordinary backpropagation to prune and clone attention heads and feed-forward channels under separate, fixed global budgets. The experiments show the advantage of the proposed method.

Critical strengths
  1. This topic is interesting, and the results are promising.
  2. It is interesting that tensor resizing and optimizer-state remapping allow training to continue directly on the compact model after each structural update.
Critical weaknesses
  1. The method seems to be a heuristic approach. The generalizability is questionable.
  2. Only the evaluation on only CIFAR-100 and Tiny-ImageNet is not comprehensive.
Questions for authors
  1. The method seems to be a heuristic approach. The generalizability is questionable.
  2. Only the evaluation on only CIFAR-100 and Tiny-ImageNet is not comprehensive.
3 comments
AuthorsOpenReview ↗

Response to Reviewer yM7H [1/2]

We sincerely thank you for your careful review and constructive suggestions. In response, we have added ViT-S/16 experiments on ImageNet-1K and expanded the compact-model comparisons with Learned Static. These results are reported in Appendices C.1 and C.2, respectively. We have updated the manuscript PDF, with additions and revisions highlighted in blue.

W1 / Q1: Selection rationale and generalizability

BARTER bases allocation decisions on the current model's activations and loss gradients. Although collected at individual heads and FFN channels, these signals reflect their role in the network: activations depend on upstream representations, while gradients carry the training objective back through downstream layers. Their product therefore relates a unit's activity to the loss being optimized. BARTER uses this relation to estimate first-order loss sensitivity for pruning, and uses mean absolute activation–gradient products as a priority signal for growth. Receiving-layer priorities also account for how much each layer has expanded or contracted, as described in Section 3.2.

This connection to the current loss is what allows the rule to respond to different models and tasks. Changing the architecture or dataset changes both the representations and the gradients, so BARTER recomputes which units to remove and where to add capacity. Deeper or wider standard Transformers retain the same head and FFN-channel units. The same selection rules can therefore use the new model's training signals to adapt its allocation.

To test whether this feedback actually improves selection, Section 4.5 compares BARTER with Random pair, which jointly randomizes donor, receiving-layer, and clone-parent selection under the same capacity budget and transfer-attempt schedule. On CCT/CIFAR-100 at density 0.5, BARTER achieves 1.65 percentage points higher mean ACC and 1.66 points higher mean macro-F1 over three seeds. Thus, the loss-dependent scores lead to better mean performance than random allocation decisions in this control.

We then test whether these decisions produce better compact structures through the newly added Learned Static comparison in Appendix C.2, Table 7. Learned Static fixes BARTER's final layer-wise head counts and FFN widths, randomly initializes all weights, and trains from scratch. Initial Static uses the same training recipe but keeps BARTER's initial allocation fixed.

Across CCT/CIFAR-100 and ViT/Tiny-ImageNet at all four densities, both BARTER and Learned Static exceed Initial Static in mean ACC and macro-F1, with three seeds per configuration. Learned Static's ACC gains range from 0.34 to 3.60 percentage points on CCT and from 0.32 to 0.70 on ViT. Because these gains persist after weight reinitialization, they show that BARTER discovers better compact structures under the same capacity budgets.

These results show that the same loss-based selection rules discover useful allocations across different model–dataset settings. Together with the rules' ability to respond to each model's own training signals, this gives us reason to expect BARTER to generalize to other standard Transformers. W2/Q2 below directly tests this expectation at a larger scale through our new ViT-S/16 experiments on ImageNet-1K.

AuthorsOpenReview ↗

Response to Reviewer yM7H [2/2]

W2 / Q2: Evaluation beyond CIFAR-100 and Tiny-ImageNet

We have added ViT-S/16 experiments on ImageNet-1K at densities 0.1, 0.3, 0.5, and 0.7, using the same scoring and reallocation rules. The full-density model has 22.05M parameters, approximately eight times the size of our Tiny-ImageNet ViT, with 224 × 224 inputs and 1,000 classes. All methods are trained from scratch for 100 epochs with the same optimization and augmentation recipe, using one training seed. The new evaluation is reported in Appendix C.1, with classification results in Table 6 and efficiency results in Figures 7 and 9.

The table below compares BARTER with the highest-accuracy DST baseline at each density, selected from SET, SNFS, RigL, SRigL, and S²ViTE. The same baseline also has the highest macro-F1 among these five methods at each density.

DensityBARTER ACC (%)BARTER Macro-F1 (%)Best baselineBaseline ACC (%)Baseline Macro-F1 (%)
0.151.4150.06RigL52.6951.45
0.367.1866.56SNFS65.7265.08
0.571.9071.43SNFS71.4270.93
0.773.5473.13RigL73.8573.45

BARTER maintains competitive ACC and macro-F1 across all four densities, often achieving the best results among the compared DST methods, while reducing measured training time per epoch by 21–34% relative to the fastest compared DST baseline at each density. At density 0.3, it exceeds the strongest baseline by 1.46 percentage points in ACC and 1.48 in macro-F1. The four-density comparison shows competitive performance across a range of capacity budgets on this larger task.

These results also retain the efficiency benefits of compact training. On an NVIDIA L40 with batch size 64, BARTER has the lowest measured training time per epoch and peak allocated GPU memory at every density. Epoch time is 21–34% lower than the fastest compared DST baseline at each matched density, with score collection and scheduled structural updates included, as reported in Figure 7. At density 0.5, its peak allocated GPU memory is 3.168 GiB, compared with 4.730–5.042 GiB for the five DST baselines.

Figure 9 further reports recorded training-loop times over complete 100-epoch runs of ViT-S/16 on ImageNet-1K at density 0.5. BARTER reaches 70% validation ACC in 22.24 hours, using 20.6–51.3% less training-loop time than the five DST baselines. It also completes all 100 epochs in 34.34 hours, reducing total training-loop time by 17.8–44.9% relative to these baselines.

Together, the original controls and the new ImageNet-1K results support generalizability across the tested Transformer architectures, dataset scales, and capacity budgets: the same mechanism discovers useful allocations on the original tasks and retains competitive accuracy with lower training costs on the larger model and dataset. Within the time and compute available during rebuttal, we prioritized this larger-scale evaluation. In future work, we will extend the study to even larger Transformers, language models, and broader datasets.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer yM7H,

We would like to briefly follow up on our response to your review. We added ViT-S/16 experiments on ImageNet-1K (Appendix C.1) and Learned Static comparisons (Appendix C.2). We also explained how BARTER uses each model’s activations and loss gradients to guide capacity allocation. These signals change with the model and task, allowing allocation decisions to adapt while retaining the same selection rules. On ImageNet-1K, these rules retain competitive accuracy with lower training costs.

We would be grateful to know whether these address your concerns, and would be happy to provide any further clarification. Thank you for your time and feedback.

Reviewer_eSCfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a DST method, BARTER, that maintains physically compact dense tensors throughout training while dynamically reallocating attention heads and feed-forward network (FFN) channels across Transformer layers under fixed global capacity budgets. Unlike other DST approaches that retain full-sized masked tensors and require gradients for inactive weights, BARTER uses activation-gradient statistics from active units to select pruning candidates and receiving layers. The authors evaluate BARTER on CCT/CIFAR-100 and ViT/Tiny-ImageNet, including fine-tuning experiments, across densities of 0.1–0.7.

Critical strengths
  1. I personally like the integration of active-only capacity reallocation with physically compact parameter tensors and optimizer-state remapping. I believe, this addresses an important practical limitation of conventional DST: reducing the number of active weights does not necessarily reduce actual training computation when full-sized tensors are retained. By operating directly on smaller dense tensors, BARTER is able to reduce training time and memory without depending on sparsity-specific hardware or kernels. The reported speedups make this contribution useful, particularly for resource-constrained training environments.

  2. The paper also includes controlled comparisons against static compact training with the same initial architecture and capacity budgets, as well as ablations of randomized donor and receiver selection, making it a well studied method. BARTER is able to improve mean classification accuracy over static compact training across both tested architectures and all four densities. On CCT/CIFAR-100, the improvement reaches 2.97 percentage points at 10% density, and the randomized ablations further support the importance of the selection mechanism. These experiments are important because they help separate the benefits of dynamic capacity allocation from the efficiency gains that arise simply from training a smaller dense network.

Critical weaknesses
  1. The reported speedups are measured against DST implementations that retain full-sized tensors and execute using standard dense kernels. As a result, a substantial portion of the efficiency advantage follows directly from executing smaller matrices rather than from the proposed capacity-reallocation algorithm. This is a legitimate practical advantage, but the comparisons do not establish that it is more efficient than other approaches that train directly on compact architectures or use optimized structured sparse execution. The static compact baseline in Appendix C.4 helps isolate the accuracy contribution of dynamic reallocation, but additional efficiency comparisons against competitive compact-training approaches would better confirm the broader efficiency claims.

  2. Secondly, the evaluation is also restricted to relatively small vision architectures (approximately 2.8M–3.8M parameters at full density) and two small-scale image classification datasets. The proposed mechanism changes attention-head and FFN widths and reconstructs parameter and optimizer-state tensors during training, yet its scalability to significantly larger Transformers, language-model training, longer sequences, and distributed training is not demonstrated. In addition, the main experiments perform relatively few head transfers (only 1–5 per training run), leaving uncertainty about how effectively the method explores and optimizes layer-wise capacity allocations in larger architectures. I believe this limitation is important to point out because the paper motivates BARTER as a broadly useful approach to efficient Transformer training, whereas its practical validation currently covers a narrower setting and hence the claims are wider than the evidence.

Questions for authors

Q1. The reported training speedups compare BARTER primarily against full-tensor DST implementations. Could you clarify which parts of the efficiency improvement are attributable to physical compactness versus the proposed dynamic capacity-reallocation mechanism? Can the authors compare against any alternative compact structured-training implementation, or provide a more explicit discussion of the scope of the claimed speedups?

Q2. How does BARTER behave on larger Transformer architectures, especially when the number of attention heads, FFN channels, and model parameters increases substantially? Given that the method repeatedly resizes parameters and remaps optimizer states, could you provide evidence or analysis of how the approach scales in runtime, memory, and accuracy?

Q3. In the matched-FLOPs experiments, BARTER receives substantially more training epochs than RigL, with a restarted learning-rate schedule. How sensitive are the reported gains to this schedule, and would the conclusions remain similar under a matched-compute comparison with appropriately tuned baseline training schedules?

4 comments
AuthorsOpenReview ↗

Response to Reviewer eSCf [1/3]

We sincerely thank you for your careful review and constructive suggestions. In response, we have added ViT-S/16 experiments on ImageNet-1K, expanded the compact-model comparisons with Learned Static, and conducted a matched-compute control without an LR restart. The ImageNet-1K and compact-model results are reported in Appendices C.1 and C.2, respectively. The no-restart control is reported below in our response to Q3. We have updated the manuscript PDF, with additions and revisions highlighted in blue.

W1 / Q1: Compact execution, dynamic reallocation, and the scope of the speedups

Source and scope of the efficiency gains. Physical compactness reduces executed computation and tensor storage; reallocation improves how the compact capacity is distributed across layers. BARTER uses active-unit statistics to prune and clone heads and FFN channels, resizing the affected parameters and optimizer states.

The prevailing dense-kernel DST implementation retains full-sized masked tensors, so changing the masks does not shrink the executed operations. Specialized sparse kernels can exploit sparsity, but suitable kernels are not available on every device and require adaptation to the sparsity pattern and backend. BARTER performs dynamic structure discovery while keeping training tensors physically compact on widely supported standard dense kernels. This allows devices without suitable sparse-kernel support to adapt the model structure while training directly on smaller dense tensors. Our reported speedups therefore compare BARTER with existing dense-kernel DST implementations and quantify the gains achieved through standard dense execution.

What the compact baselines establish. We now compare Initial Static, BARTER, and the newly added Learned Static under the same capacity budgets and training recipe. Initial Static fixes BARTER's initial layer-wise head counts and FFN widths. BARTER starts from that allocation and adjusts it during training. Learned Static takes the final allocation from the corresponding BARTER run at each density and seed, randomly initializes all weights, and trains that fixed architecture from scratch.

The example below is CCT/CIFAR-100 at density 0.1. Values are mean ± sample standard deviation over three seeds. All three methods use the same timing protocol: direct measurement of epoch-length replay segments.

MethodACC (%)Training time (s/segment)
Initial Static63.16 ± 0.2711.53 ± 0.03
BARTER66.13 ± 0.4011.90 ± 0.06
Learned Static66.76 ± 0.2411.88 ± 0.02

BARTER adds only 0.37 s per segment while improving ACC by 2.97 percentage points over Initial Static. Learned Static improves ACC by 3.60 points after weight reinitialization, showing that the allocation discovered by BARTER is a better starting architecture for fixed compact training. The three compact models have the same 0.4802M parameters and 0.750 estimated training GFLOPs per image, so these gains come from using the capacity more effectively.

This pattern extends across CCT/CIFAR-100 and ViT/Tiny-ImageNet at densities 0.1, 0.3, 0.5, and 0.7. Both BARTER and Learned Static exceed Initial Static in mean ACC and macro-F1 in all eight settings. Learned Static's ACC gains range from 0.34 to 3.60 points on CCT and 0.32 to 0.70 on ViT. BARTER obtains the allocation and trained weights together in one run, without the separate retraining stage. The revised Appendix C.2, Tables 7–9, reports all compact comparisons.

AuthorsOpenReview ↗

Response to Reviewer eSCf [2/3]

W2 / Q2: Accuracy, runtime, and memory at a larger scale

We have added a larger-scale evaluation of ViT-S/16 on ImageNet-1K: a 22.05M-parameter full-density model, approximately eight times the size of our Tiny-ImageNet ViT, with 224 × 224 inputs and 1,000 classes. Each method is trained for 100 epochs on an NVIDIA L40, with batch size 64 and one training seed.

BARTER maintains competitive ACC and macro-F1 across all four densities, often achieving the best results among the compared DST methods, while reducing measured training time per epoch by 21–34% relative to the fastest compared DST baseline at each density. It has the lowest training time per epoch and peak allocated GPU memory at all four densities. Results at density 0.5 follow; Appendix C.1, Table 6 and Figures 7–9, contains the full evaluation.

MethodACC (%)Macro-F1 (%)Training time (s/epoch)Peak allocated memory (GiB)
SET70.2769.791464.174.730
SNFS71.4270.931427.404.730
RigL71.2570.771468.964.730
SRigL69.1168.551582.634.779
S²ViTE70.4769.962194.205.042
BARTER71.9071.431121.063.168

Runtime and memory with reallocation costs included. Our timing includes score collection and scheduled structural updates, with peak allocated GPU memory recorded over the measured interval, as specified in Appendix D.1. An update rebuilds only the affected attention or FFN parameter tensors and remaps their optimizer states. Each affected module preserves its external input and output dimensions while its internal width changes. Training continues on the resized compact tensors, as described in Section 3.4.

With these operations included, BARTER reduces training time per epoch by 23.7% and peak allocated memory by 33.0% relative to RigL at density 0.5 on ViT-S/16/ImageNet-1K, while reaching higher ACC and macro-F1. Estimated training computation is also lower: 13.990 versus 27.593 GFLOPs per image, a 49.3% reduction, as reported in Figure 8. These results show that BARTER still trains faster and uses less GPU memory on ViT-S/16 with reallocation costs included.

The new time-to-accuracy results in Figure 9 report cumulative training-loop time, including reallocation and excluding validation and checkpoint saving. At density 0.5, BARTER reaches 70% validation ACC after 22.24 hours, compared with 29.60 hours for RigL, and earlier than every sparse baseline in the recorded runs.

How transfer counts scale. Scheduled head and FFN-channel transfers scale with initial active budgets, so the original 1–5 head transfers are not fixed caps. More active heads per layer and larger global head budgets generally lead to more scheduled head transfers. Time and compute limits led us to prioritize ViT-S/16 on ImageNet-1K. In future work, we will evaluate BARTER on even larger Transformers, language models, longer sequences, and distributed training. We will also study transfer ratios across different model scales.

AuthorsOpenReview ↗

Response to Reviewer eSCf [3/3]

Q3: Does the matched-compute advantage depend on restarting the learning rate?

Single-schedule control. We have added a no-restart control for Tiny-ImageNet-to-CIFAR-100 fine-tuning, using the same pretrained checkpoints and matched-compute budget. The common budget is approximately 15.276 PFLOPs under Appendix D.2's counting protocol. Because BARTER uses fewer training FLOPs per epoch, this permits 100 epochs for RigL and 906.17 and 324.28 epochs for BARTER at densities 0.1 and 0.3. BARTER uses one warmup-and-cosine schedule over its entire budget, while RigL retains its original single schedule. Both methods use the same peak learning rate, minimum learning rate, and warmup fraction, with each schedule ending at its own compute-budget endpoint. BARTER's scheduled structural-update counts and absolute update steps remain unchanged.

Results without a restart. Below we report the new control alongside the original restart and RigL results. Values are mean ± sample standard deviation over three seeds.

DensityMethod / scheduleACC (%)Macro-F1 (%)
0.1RigL, single schedule63.19 ± 0.5362.80 ± 0.56
0.1BARTER, restart65.65 ± 0.5965.33 ± 0.62
0.1BARTER, single schedule65.61 ± 0.8665.32 ± 0.86
0.3RigL, single schedule68.38 ± 0.3168.10 ± 0.34
0.3BARTER, restart70.39 ± 0.9870.18 ± 0.95
0.3BARTER, single schedule70.32 ± 1.0970.17 ± 1.12

Removing the restart changes mean ACC by at most 0.07 percentage points and macro-F1 by 0.01 points. BARTER still exceeds RigL by 2.42 and 1.94 points in ACC, and by 2.52 and 2.07 points in macro-F1. The matched-compute advantage therefore remains under this shared schedule form; the reported gains do not require a learning-rate restart.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer eSCf,

We would like to briefly follow up on our response to your review. We added ViT-S/16 experiments on ImageNet-1K (Appendix C.1), Learned Static comparisons (Appendix C.2), and a matched-compute control without an LR restart (our Q3 response). We also explained the reason for our dense-kernel comparisons. Suitable sparse kernels are not available on every device, while masking full-sized tensors in existing dense-kernel DST implementations does not reduce dense computation. BARTER addresses this gap by discovering and training compact structures on standard dense kernels, with actual computation and training-time savings.

We would be grateful to know whether these address your concerns, and would be happy to provide any further clarification. Thank you for your time and feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal additions and clarifications

Dear (Senior) Area Chair and Reviewers,

We are grateful to the reviewers for their time and constructive feedback. We also thank the Area Chair and Senior Area Chair for coordinating the review. We have made a substantial effort to address the concerns raised. This final note summarizes the contributions recognized in the reviews and the additional experiments and manuscript revisions completed in response.

  • Larger-scale training and time to accuracy (eSCf, nF6Z, yM7H). ViT-S/16 on ImageNet-1K, with profiling across four densities and complete 100-epoch training records at density 0.5.
  • BF16 mixed precision (nF6Z). CCT/CIFAR-100 experiments across four densities and three seeds.
  • Learned Static (eSCf, nF6Z, yM7H). Training BARTER’s final allocation from scratch to test the learned structures.
  • Learning-rate restart control (eSCf). Matched-compute fine-tuning with a single LR schedule over the full budget.

The new ImageNet-1K, Learned Static, and BF16 results are included in Appendices C.1-C.3. Additions and revisions are highlighted in blue. The no-restart control is reported in our response to eSCf’s Q3.

  1. Dense-kernel comparisons (eSCf, nF6Z). Masking full-sized tensors in existing dense-kernel DST implementations does not reduce dense computation. Suitable sparse kernels are not available on every device. BARTER addresses this gap by discovering and training compact structures on standard dense kernels with actual computation and training-time savings. Our comparisons evaluate these gains against existing dense-kernel DST implementations.

  2. Larger-scale evaluation and complete training (all three reviewers). We added ViT-S/16 experiments on ImageNet-1K. Its full-density parameter count is 22.05M, about eight times that of our original ViT. BARTER uses the same selection rules and achieves competitive ACC and macro-F1 across four densities. Profiling shows lower training time and peak memory than every compared DST baseline. In the recorded runs at density 0.5, BARTER uses 20.6% to 51.3% less training-loop time to reach 70% validation accuracy than the DST baselines. It completes 100 epochs with 17.8% to 44.9% less training-loop time. Appendix C.1, Table 6 and Figures 7-9.

  3. BF16 mixed precision (nF6Z). We trained all methods with BF16 on CCT/CIFAR-100 over three seeds. BARTER retains competitive accuracy and the lowest measured training time and peak memory across four densities. At density 0.3, it achieves similar mean accuracy to RigL with a 2.70× speedup and 63.7% lower peak memory. The savings therefore persist when the baselines also use mixed precision. Appendix C.3, Table 10 and Figure 10.

  4. Selection rules, generalizability, and learned structures (all three reviewers). BARTER links unit activity to the current training loss using activations and loss gradients. These signals guide capacity allocation. Larger standard Transformers have the same types of heads and FFN channels, so these rules remain applicable. ImageNet-1K results support their use on a larger model and dataset. Learned Static retrains the final allocation from a completed BARTER run with randomly initialized weights. At matched capacity and estimated training compute, its mean accuracy is similar to BARTER’s. Both improve mean ACC and macro-F1 over Initial Static’s fixed starting allocation on both original tasks at all four densities. These gains after reinitialization show that BARTER finds better compact structures. That allocation still needs to be found. Dynamic reallocation provides this step while BARTER trains the weights in the same compact run, without separate retraining. Measured execution costs remain close to those of fixed compact training. Appendix C.2, Tables 7-9.

  5. Learning-rate restart control (eSCf). We reran BARTER’s matched-compute fine-tuning with one LR schedule over its entire budget. BARTER achieves comparable mean accuracy with and without an LR restart. Without a restart, it still exceeds RigL’s original single-schedule results by 2.42 and 1.94 percentage points at densities 0.1 and 0.3, respectively. The matched-compute advantage remains in this no-restart control.

We respectfully ask that these results and clarifications be considered in the final assessment. Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.