acceptodds
Withdrawn from ICLR 2027

Early Learning Shapes Later Directions of Representational Change in Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

24 views

Abstract

Representations continually change as a network learns new tasks. We ask whether early representational changes naturally form a geometric structure that continues to shape later learning. We identify a low-dimensional subspace of early representation drift, which we call a scaffold, and test whether it is reused across subsequent tasks. Across four pretrained visual encoders and two datasets, later representational changes consistently favor this early-defined subspace over matched random alternatives. This reuse is history-dependent: when networks experience different early tasks but identical later training inputs, each network preferentially reuses the scaffold induced by its own learning history. The same preference appears in individual optimizer updates, even though the network's dominant local response directions shift away from the original scaffold. Finally, constraining motion within the scaffold slows new-task acquisition more than matched random constraints, while effects on old-task retention are less consistent. In summary, these results suggest that early experience leaves a persistent geometric imprint on how neural networks adapt to future tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Representation Learning, Representational Drift, Learning Dynamics, Plasticity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gtbcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether early representation changes define a low-dimensional subspace that later learning continues to reuse. It constructs a “scaffold” by pooling dominant early-drift directions across training histories, then evaluates its coverage of later drift and individual optimizer updates. Paired histories with different early tasks test history dependence, while subspace penalties test the cost of restricting scaffold motion. Experiments use four pretrained visual encoders on short CIFAR-10 and partial CIFAR-100 task sequences, primarily with replay.

Critical strengths

The paired early-task branching experiment is a useful control: branches share their initial state and later training inputs, helping isolate the effect of early experience. The use of separate estimation and test histories, together with classifier-matched random subspaces, also strengthens the evidence for history-dependent alignment within the evaluated setup.

Critical weaknesses

1. The significance of the finding is unclear, and validation is too restricted to establish relevance to continual-learning practice.

The paper identifies a recurring geometric pattern, but does not sufficiently establish what this observation adds to our understanding of how existing continual-learning methods learn or forget. The connection to established methods remains largely motivational. It is unclear whether the scaffold explains important differences between methods, predicts their failure modes, or provides useful guidance for managing the stability–plasticity trade-off.

The experimental scope is considerably narrower than “four encoders and two datasets” might suggest. Appendix A uses five two-class tasks on CIFAR-10 and only forty classes arranged into four tasks on CIFAR-100, with 450 training images per class and five epochs per task. All four encoders are pretrained and use the same AdamW-based replay recipe. Multiple backbones under this recipe do not establish generality across learning methods. Validation on standard full-class benchmarks, longer task sequences, and representative modern continual-learning methods is missing.

The supplementary no-replay results further limit the generality of the observation. In Table 9, three of four DeiT-Tiny transition intervals include zero. For MLP-Mixer, the final-transition coverage advantage is negative: −1.66 percentage points, with a 95% interval of [−2.96, −0.37]. These results do not invalidate the replay experiments, but they show that persistent preferential reuse is not consistent across the tested learning regimes. The paper does not adequately explain these boundaries or establish why the observed scaffold is a consequential contribution beyond the particular setup.

2. The intervention does not sufficiently isolate the scaffold’s functional role from differences in optimization burden.

The main functional claim relies on scaffold suppression impairing new-task acquisition more than random constraints. The energy-matched control is useful, but it matches projected drift-energy reduction at the end of training, whereas acquisition is measured by averaging accuracy over all five epochs. Matching the endpoint does not match the strength or timing of the constraint throughout optimization.

The coefficient differences are substantial. Table 12 uses λ = 100 for the scaffold, while the energy-matched random constraints use λ = 0.21–0.87 for ResNet/CIFAR-10 and 0.21–1.00 for ResNet/CIFAR-100. Different coefficients are not inherently unfair, since subspaces can have different sensitivities. However, the endpoint matching does not establish comparable penalty-gradient magnitudes, interference with task gradients, or cumulative disruption of the training trajectory.

This matters because the stronger consistency appears after energy matching: the same-coefficient comparison has acquisition intervals below zero in five of eight conditions, while the energy-matched comparison has eight of eight. The paper needs stronger evidence that the additional acquisition cost reflects the scaffold’s particular functional role, rather than differences in how the constraints affect optimization over time.

Together, these concerns leave the contribution insufficiently established: the descriptive finding has limited demonstrated relevance, and the intervention does not fully resolve its functional interpretation.

Questions for authors
  1. What evidence establishes that the scaffold explains or predicts meaningful behavior of representative continual-learning methods beyond the evaluated replay recipe? Does the main finding persist on full-class benchmarks and longer task sequences?
  2. How does matching final drift-energy reduction rule out different optimization burdens during the five-epoch acquisition window? Can the authors quantify the penalty gradients and their interaction with task gradients, or provide controls that address the training trajectory?
  3. Appendix C selects the scaffold rank using final-transition coverage in estimation histories. Does the central result remain robust when rank is fixed independently of later drift or selected using early-task data alone? The independent test histories are valuable, but the distinction between early-fitted directions and later-calibrated dimensionality should be made explicit.
Reviewer_gFj7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether there is stability not in representations but in the directions in which they change. Networks are trained as "training histories": several runs share one pretrained encoder but differ in task order and randomness, over CIFAR-10 (five two-class tasks) and CIFAR-100 (four ten-class tasks) with ResNet-18, DeiT-Tiny, MobileNetV3-Small and MLP-Mixer-B/16. Representations of fixed first-task anchors are tracked, and centred drift between task endpoints is defined as D_{a→b} = H(Z_b − Z_a) (Eq. 1). The leading r right singular vectors of the task-1→2 drift are pooled across S = 6 estimation histories into an averaged projection matrix whose leading eigenspace defines the scaffold (Eq. 2), with r ∈ [8,24] chosen by how well the scaffold captures later drift. The scaffold is then frozen and evaluated on separate test histories through three questions: (i) do later representational changes and (ii) individual AdamW updates place disproportionate motion inside it, compared with matched random subspaces (including a classifier-matched and an energy-matched construction); and (iii) does restricting motion inside the scaffold via a quadratic penalty on old-memory features (Eq. 6) impair new-task acquisition more than the same restriction applied to random subspaces, with effects on old-task retention as a secondary outcome. A history-swap control (different early tasks, identical later inputs) tests whether reuse is history-specific.

Critical strengths
  1. A crisp, falsifiable question with a genuinely careful protocol. The scaffold is fitted on six estimation histories and evaluated on held-out test histories; random controls are matched both by penalty strength and by removed drift energy; anchors are fixed images with held-out evaluation images excluded from training and replay (L304–305). This is a much stronger design than the correlational analyses common in this area.
  2. Honest reporting. The abstract concedes that retention effects are inconsistent; Table 1 shows the scaffold constraint often worse on retention; the tension between "dominant local response directions shift away" and continued reuse is stated rather than smoothed over. The appendix goes further and reports the trade-off explicitly: Mixer/CIFAR-100 gains 3.03 points of first-task retention [0.53, 5.33] relative to unconstrained training while losing 5.61 points of acquisition [4.56, 6.61], i.e. "an acquisition cost can accompany a retention benefit".
Critical weaknesses
  1. Missing baselines in datasets and architectures. The paper lacks comparisons on ImageNet-R, ImageNet-A, and DomainNet, and does not evaluate a ViT backbone. These should be added or their omission justified.
  2. No comparison with continual learning methods. The paper does not compare against any continual learning baseline, even basic ones. Comparisons with standard baselines (e.g., EWC, LwF, replay) are necessary.
  3. Unexplained CIFAR-100 task split. The four ten-class split is not justified and differs from the common ten ten-class setting.
  4. Geometric evidence is not linked to behavior. Except through the intervention, no analysis shows that higher scaffold coverage predicts acquisition or retention across histories/conditions. Correlational analyses linking coverage to acquisition and retention are needed.
  5. Lack of ablation studies. The paper does not provide ablations to isolate the contribution of key components/hyperparameters. Ablation experiments are needed to verify which design choices drive the reported gains.
Questions for authors

refer to weaknesses.

Reviewer_XMruOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether early continual learning establishes a persistent, low-dimensional geometric subspace (called a "scaffold") that shapes future representation changes. Defining this scaffold via the dominant singular directions of representation drift between the first two tasks, the authors test 3 hypotheses across 4 pretrained vision architectures (ResNet-18, DeiT-Tiny, MobileNetV3-Small, MLP-Mixer-B/16) on the datasets CIFAR-10 and CIFAR-100 in the following manner: (1) later representational drift and individual optimizer steps preferentially reuse this subspace (2) preferred drift directions depend on early task history even when subsequent training data are identical (3) regularizing against motion within this scaffold impairs new-task acquisition more than constraining random directions. While the empirical regularities are compelling and well-documented in the replay setting, the current intervention and optimization controls do not yet isolate a history-specific causal mechanism.

Critical strengths
  1. Controlled Early-History Branching Design (Section 4.2): By branching from a shared network state onto different second tasks before exposing both branches to identical subsequent training inputs, the authors convincingly demonstrate that subsequent representational drift geometry depends on early learning history. Evaluating bilateral own-history versus cross-history contrasts rules out trivial explanations where one scaffold is simply universally superior, providing strong support for history-dependent geometric persistence across diverse model inductive biases (CNNs, ViTs, MLPs).
  2. Moving Beyond Descriptive Geometry to Functional Interventions (Section 6): The authors did not stop at subspace overlap correlations. The paper tests functional relevance by actively regularizing against motion within the identified scaffold, combined with held-out estimation/test histories and energy-matched random controls. This directly connects descriptive representation geometry to downstream continual learning plasticity and task acquisition.
Critical weaknesses
  1. Unresolved Causal and Optimization-Control Ambiguities in Functional Interventions (Sections 4 & 6): The paper does not fully establish that the scaffold is a distinctive, history-specific causal mechanism for future plasticity. And not a marker of directions generally important for optimization. First, the functional interventions in Section 6 use the pooled baseline scaffold rather than testing own-history versus cross-history scaffolds within the controlled branching setup (Section 4.2). Second, matching final projected drift energy reduction requires vastly different regularization strengths (e.g., for the scaffold vs. for random controls inλ = 100 0. 21 𝑡𝑜 0. 87 ResNet-18/CIFAR-10, Table 12), which can confound subspace identity with differences in local optimization conditioning and gradient interference. If heavy tuning was involved in finding these parameters, the generality of results would be questionable.
  2. Incomplete Isolation of Scaffold Reuse from Local Parameter Sensitivity and Future-Informed Rank Selection (Sections 3.2 & 5): The claim that optimizer updates actively favor the historical scaffold despite declining local-response alignment compares different geometric metrics: orientation overlap (Fig. 4) versus step energy coverage (Fig. 5). Without comparing actual step coverage directly against expected coverage under isotropic parameter perturbations using the full local response covariance, it remains unclear if updates genuinely favor the scaffold beyond ordinary local parameter sensitivity. Additionally, selecting scaffold dimensionality using final-task drift on estimation histories and poor persistence in no-replay regimes (e.g., DeiT and MLP-Mixer in Appendix G) weaken the claim of a purely early-defined, broadly general mechanism
Questions for authors

Critical Experiments / Clarifications (Could Materially Affect Assessment):

  1. Branch-Specific Functional Interventions: Can the authors extend the Section 6 suppression intervention to the controlled branching setup of Section 4.2? Specifically, within branch and branch , does constraining the own-history scaffold𝑋 𝑌 ( vs. ) impair new-task acquisition more than constraining the cross-history𝑈𝑋 𝑈𝑌 scaffold under matched training streams and penalty? Demonstrating a double dissociation here would directly link the strongest history-dependence evidence to the actual functional claim.
  2. Direct Control for Isotropic Local Parameter Sensitivity: In Section 5 / Appendix E, instead of comparing orientation overlap (Fig. 4) to step energy coverage (Fig. 5), can the authors directly compare the scaffold coverage of actual optimizer updates against the expected scaffold coverage under isotropic parameter perturbations, at the same checkpoints?

Suggestions for Future Work (Not required for rebuttal):

  1. Evaluate scaffold coverage on later-task and unseen-class anchor populations (beyond first-task memory anchors, which may primarily reflect old-task maintenance) and report the fraction of total displacement removed by centering.
  2. Investigate longer task horizons and training from scratch versus pretrained encoders to clarify why persistence breaks down for transformers/mixers without replay (Appendix G).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.