acceptodds
Withdrawn from ICLR 2027

Dynamic Sparse Training for sustained Plasticity under Non-Stationary Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep neural networks trained on non-stationary data, while retaining previously observed samples, represent a continual learning (CL) setting in which loss of plasticity remains an important failure mode, and catastrophic forgetting is not the primary challenge. Plasticity is the ability of a network to continue learning effectively from new data; its loss shows up as warm-started networks generalizing worse than networks trained from scratch on the same data. Weight sparsity has been used in CL to reduce task interference and mitigate catastrophic forgetting, but its role in preserving plasticity remains less understood, particularly when previously observed data remains available. We study whether unstructured sparse training (static sparse training (SST), gradual magnitude pruning(GMP), and dynamic sparse training (DST)) can mitigate plasticity loss in dataincremental learning settings. Across sample-incremental and class-incremental continual visual learning on CIFAR-10, CIFAR-100, and Tiny ImageNet, DST, in particular Rigging the Lottery (RigL), at moderate sparsities outperforms dense baselines, while in continual language fine tuning on TRACE, RigL at moderate sparsity likewise improves over dense training, with lower sparsity providing no advantage. RigL also compares favorably with dense training at most fractions of retained past data. Treating DST as an alternative training procedure rather than another plasticity intervention, we combine it with other plasticity improving interventions; both perform better on top of RigL than on top of dense training in continual visual learning and continual language fine tuning. Overall, our results suggest that dynamically evolving a sparse topology can help sustain plasticity across architectures and learning domains while remaining compatible with existing interventions for plasticity loss

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, plasticity loss, dynamic sparse training, weight sparsity, non-stationary data, reinitialization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BvUKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether unstructured sparse training keeps a network plastic when new data arrive and past data are still available. Each stage warm-starts from the previous checkpoint. It compares dense training, a frozen mask, gradual magnitude pruning, SET, and RigL on incremental vision benchmarks and on TRACE instruction tuning, then adds retention, FIRE, and full reset.

Critical strengths

The regime is stated clearly. Past examples remain available, so the failure mode is weak later learning rather than classic replay-free forgetting. Held-out performance is the primary measure.

The sparse variants are distinguished in the method. A static mask never returns, gradual pruning only removes weights, and SET and RigL drop and regrow, with regrowth either random or by gradient magnitude. Reactivated weights keep a stored value instead of starting at zero, which matters once the mask is warm-started.

Where the paper supplies multi-seed vision runs, RigL is the strongest or tied-best arm on most of those settings, and gradual magnitude pruning sits below dense training. FIRE and full reset are reported to help both dense training and RigL.

Critical weaknesses

Appendix A says this draft mixes supplied tables, values reconstructed from plots, and missing results, and that reconstructed numbers are not raw seed measurements. Several headline curves are therefore not auditable as submitted.

The winning sparse settings are chosen from sweeps over sparsity, drop fraction, and update cadence, and the selection rule is not fully specified. That favors the reported RigL arm against one dense recipe. RigL also uses ERK allocation while other sparse methods use a uniform mask, so the topology update is not the only difference.

The retention claim is stronger than the figures. The text says RigL is never the worse choice across retention levels. The class-incremental Tiny ImageNet plot places it below dense at the smallest retention, and another retention level does not favor it either. Extra retained data also changes the number of optimizer steps under a fixed epoch budget.

The plasticity story has no from-scratch reference on the same accumulated data, which is how the paper itself defines the gap. The vision recipe is a plain Adam run without augmentation, so sparsity can help by regularizing that recipe. The proxy measurements are reported not to rank methods in one order, including under full reset.

The language evidence is a single seed, and sparsity is limited to gate projections. Overall performance averages unlike task scores. On the short subset, the same split is used for choosing sparsity and for reporting. Low sparsity does not beat dense training, so the cross-domain claim is thin.

Questions for authors

Please see Critical Weaknesses.

Reviewer_tmEfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether sparse training, including SST, GMP, and DST, can mitigate plasticity loss in data incremental learning settings. They conduct experiments on continual visual learning and continual language fine tuning. Their results suggest that dynamically evolving a sparse topology can help sustain plasticity across architectures and learning domains while remaining compatible with existing interventions for plasticity loss.

Critical strengths

This paper suggests that dynamically evolving a sparse topology can help sustain plasticity across architectures and learning domains. The authors verify this claim through experiments on both continual visual learning and continual language fine tuning.

Critical weaknesses
  1. The authors link the advantage of DST to dynamic topology. While binary masks can encode the connection structure, it remains unclear why the dynamic change of this structure mitigates plasticity loss. Furthermore, GMP also changes the network connection structure.

  2. Section 3.1 states that RigL uses ERK while other sparse methods use uniform allocation. Why did the authors adopt a different inter-layer sparsity allocation strategy? ERK itself may affect the network capacity distribution and final performance. Therefore, the current comparison simultaneously changes both the sparse training method and the parameter allocation strategy. It is hard to attribute the performance difference solely to RigL.

  3. Figure 1 mainly shows the test accuracy of different methods. It is not sufficient to distinguish general performance improvement from the mitigation of plasticity loss. A continuous increase in Dense accuracy does not mean plasticity loss. Similarly, RigL achieving higher accuracy does not directly prove that it maintains plasticity.

  4. The Sample Incremental setting differs from the typical continual learning setting that restricts access to historical data. The Class Incremental experiments in this paper also mainly retain all historical data. In addition, in Figure 1, the gain of RigL over dense is relatively small in the class incremental scenario, and GMP is significantly lower than dense. Therefore, is the larger gain of RigL mainly concentrated in specific sample incremental tasks?

  5. The existing work "Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning" systematically studied GMP and SET and provided evidence that sparse training improves plasticity-related metrics. SET itself is a dynamic sparse training method. Thus, the connection between dynamic sparse and plasticity already has been verified. Although this paper studies a different cumulative supervised learning setting and extends the experiments to language model fine-tuning and other combined interventions, its core method mainly follows the existing RigL. The authors need to further clarify what new mechanistic insights or explanatory empirical findings are obtained beyond cross-setting validation. If the main contribution is merely that an existing method achieves higher accuracy on some tasks, the incremental contribution of this paper remains limited.

Questions for authors

Please see Weakness.

Reviewer_hsojOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual learning on growing datasets where historical data remain available, focusing on the loss of plasticity during repeated model updates. The authors compare dense training, static sparse training, gradual magnitude pruning, and two dynamic sparse training methods, evaluating their performance in data-incremental visual learning and continual language model fine-tuning.

Critical strengths

The paper addresses a problem of practical importance. Even when historical data remain available for training, continued model updates may still be hindered by a loss of plasticity. Examining whether network connectivity can improve subsequent learning offers a useful perspective on this problem.

Critical weaknesses

The paper’s methodological contribution and the rationale for its design choices are not sufficiently clear. It mainly compares existing sparse training methods, but provides limited discussion of why these methods are suitable for preserving plasticity, how different connectivity update strategies affect subsequent learning, and what the paper adds beyond prior work. Specific choices, such as restoring previous weight values when connections regrow, also lack targeted ablations to establish their necessity and effects. The experiments primarily show performance differences between training procedures without clearly explaining which design choices account for these differences or why.

The paper is motivated by preserving plasticity, yet its central evidence is that RigL achieves better final test performance than dense training on cumulative datasets. This result does not distinguish improved plasticity from generalization benefits arising from sparse parameterization or topology selection. RigL’s strong performance after a full weight reset further suggests that its advantage may not primarily come from reducing the adverse effects of inheriting parameters from earlier training stages. To establish whether RigL mitigates plasticity loss, the authors should compare continued training with training from scratch separately for RigL and dense models under matched data and training budgets, while also separating the effects of weight resets and the retention of masks from previous stages.

Questions for authors

please refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.