- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether early representation changes define a low-dimensional subspace that later learning continues to reuse. It constructs a “scaffold” by pooling dominant early-drift directions across training histories, then evaluates its coverage of later drift and individual optimizer updates. Paired histories with different early tasks test history dependence, while subspace penalties test the cost of restricting scaffold motion. Experiments use four pretrained visual encoders on short CIFAR-10 and partial CIFAR-100 task sequences, primarily with replay.
Critical strengths
The paired early-task branching experiment is a useful control: branches share their initial state and later training inputs, helping isolate the effect of early experience. The use of separate estimation and test histories, together with classifier-matched random subspaces, also strengthens the evidence for history-dependent alignment within the evaluated setup.
Critical weaknesses
1. The significance of the finding is unclear, and validation is too restricted to establish relevance to continual-learning practice.
The paper identifies a recurring geometric pattern, but does not sufficiently establish what this observation adds to our understanding of how existing continual-learning methods learn or forget. The connection to established methods remains largely motivational. It is unclear whether the scaffold explains important differences between methods, predicts their failure modes, or provides useful guidance for managing the stability–plasticity trade-off.
The experimental scope is considerably narrower than “four encoders and two datasets” might suggest. Appendix A uses five two-class tasks on CIFAR-10 and only forty classes arranged into four tasks on CIFAR-100, with 450 training images per class and five epochs per task. All four encoders are pretrained and use the same AdamW-based replay recipe. Multiple backbones under this recipe do not establish generality across learning methods. Validation on standard full-class benchmarks, longer task sequences, and representative modern continual-learning methods is missing.
The supplementary no-replay results further limit the generality of the observation. In Table 9, three of four DeiT-Tiny transition intervals include zero. For MLP-Mixer, the final-transition coverage advantage is negative: −1.66 percentage points, with a 95% interval of [−2.96, −0.37]. These results do not invalidate the replay experiments, but they show that persistent preferential reuse is not consistent across the tested learning regimes. The paper does not adequately explain these boundaries or establish why the observed scaffold is a consequential contribution beyond the particular setup.
2. The intervention does not sufficiently isolate the scaffold’s functional role from differences in optimization burden.
The main functional claim relies on scaffold suppression impairing new-task acquisition more than random constraints. The energy-matched control is useful, but it matches projected drift-energy reduction at the end of training, whereas acquisition is measured by averaging accuracy over all five epochs. Matching the endpoint does not match the strength or timing of the constraint throughout optimization.
The coefficient differences are substantial. Table 12 uses λ = 100 for the scaffold, while the energy-matched random constraints use λ = 0.21–0.87 for ResNet/CIFAR-10 and 0.21–1.00 for ResNet/CIFAR-100. Different coefficients are not inherently unfair, since subspaces can have different sensitivities. However, the endpoint matching does not establish comparable penalty-gradient magnitudes, interference with task gradients, or cumulative disruption of the training trajectory.
This matters because the stronger consistency appears after energy matching: the same-coefficient comparison has acquisition intervals below zero in five of eight conditions, while the energy-matched comparison has eight of eight. The paper needs stronger evidence that the additional acquisition cost reflects the scaffold’s particular functional role, rather than differences in how the constraints affect optimization over time.
Together, these concerns leave the contribution insufficiently established: the descriptive finding has limited demonstrated relevance, and the intervention does not fully resolve its functional interpretation.
Questions for authors
- What evidence establishes that the scaffold explains or predicts meaningful behavior of representative continual-learning methods beyond the evaluated replay recipe? Does the main finding persist on full-class benchmarks and longer task sequences?
- How does matching final drift-energy reduction rule out different optimization burdens during the five-epoch acquisition window? Can the authors quantify the penalty gradients and their interaction with task gradients, or provide controls that address the training trajectory?
- Appendix C selects the scaffold rank using final-transition coverage in estimation histories. Does the central result remain robust when rank is fixed independently of later drift or selected using early-task data alone? The independent test histories are valuable, but the distinction between early-fitted directions and later-calibrated dimensionality should be made explicit.