- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether unstructured sparse training keeps a network plastic when new data arrive and past data are still available. Each stage warm-starts from the previous checkpoint. It compares dense training, a frozen mask, gradual magnitude pruning, SET, and RigL on incremental vision benchmarks and on TRACE instruction tuning, then adds retention, FIRE, and full reset.
Critical strengths
The regime is stated clearly. Past examples remain available, so the failure mode is weak later learning rather than classic replay-free forgetting. Held-out performance is the primary measure.
The sparse variants are distinguished in the method. A static mask never returns, gradual pruning only removes weights, and SET and RigL drop and regrow, with regrowth either random or by gradient magnitude. Reactivated weights keep a stored value instead of starting at zero, which matters once the mask is warm-started.
Where the paper supplies multi-seed vision runs, RigL is the strongest or tied-best arm on most of those settings, and gradual magnitude pruning sits below dense training. FIRE and full reset are reported to help both dense training and RigL.
Critical weaknesses
Appendix A says this draft mixes supplied tables, values reconstructed from plots, and missing results, and that reconstructed numbers are not raw seed measurements. Several headline curves are therefore not auditable as submitted.
The winning sparse settings are chosen from sweeps over sparsity, drop fraction, and update cadence, and the selection rule is not fully specified. That favors the reported RigL arm against one dense recipe. RigL also uses ERK allocation while other sparse methods use a uniform mask, so the topology update is not the only difference.
The retention claim is stronger than the figures. The text says RigL is never the worse choice across retention levels. The class-incremental Tiny ImageNet plot places it below dense at the smallest retention, and another retention level does not favor it either. Extra retained data also changes the number of optimizer steps under a fixed epoch budget.
The plasticity story has no from-scratch reference on the same accumulated data, which is how the paper itself defines the gap. The vision recipe is a plain Adam run without augmentation, so sparsity can help by regularizing that recipe. The proxy measurements are reported not to rank methods in one order, including under full reset.
The language evidence is a single seed, and sparsity is limited to gate projections. Overall performance averages unlike task scores. On the short subset, the same split is used for choosing sparsity and for reporting. Low sparsity does not beat dense training, so the cross-domain claim is thin.
Questions for authors
Please see Critical Weaknesses.