acceptodds
Under review as a conference paper at ICLR 2027

Neural Succession: A Mesoscopic Theory of Invasion, Coexistence, and Stabilization in Continual Learning

Abstract

Continual learning is usually studied through mechanisms that preserve old knowledge. Here we study what happens when a new task enters a representation that already supports earlier tasks. We develop Successional Learning Theory (SLT), a mesoscopic account in which the current representation is a resident community, the incoming task is an invader, forgetting is resident displacement, joint retention is coexistence, replay is resident reinforcement, and training moves from establishment toward stabilization. Its empirical coordinate is directional pre-invasion compatibility, measured on the resident model before the incoming task is learned. Across eight experiments, compatibility orders later forgetting on the 20 directed Split-CIFAR-10 transitions (three-repeat , incoming-task cluster 95% CI ; every repeat alone ), forecasts held-out forgetting with 24% lower error than a no-information baseline, and reproduces under controlled MNIST permutations and CIFAR-10 rotations (, ). The same coordinate also resolves coexistence. The three natural transitions that coexist are exactly the three most compatible (AUC ). On an 84-transition suite, compatibility separates coexistence from exclusion at every retention threshold (AUC –). Replay repairs every transition with at most 325 stored examples and is most efficient where displacement is largest. The predictive structure appears at the transition scale. Compatibility reaches , while activation, representation, Jacobian, and fixed-coefficient Lotka–Volterra specializations do not. Plasticity and feature turnover fall reliably from early to late training (15/15 and 14/15 runs). We formalize a minimum habitat-modification bound, a displacement floor, a sufficient coexistence condition, an identifiability law with a range-restriction corollary, successional stabilization, and local reinforcement. The identifiability law also predicts where the coordinate loses leverage, and the prediction matches three CIFAR-100 partitions and five optimizer regimes. SLT is a pre-adaptation diagnostic that complements replay, regularization, and projection methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, lifelong learning, catastrophic forgetting, representation learning, task compatibility, experience replay, learning dynamics, ecological succession

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sC63OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes the idea of Successional Learning Theory as an interpretation of continual learning using terminology from ecology. A representation supporting an existing task is treated as a resident system, and learning an incoming task is interpreted as an invasion that can lead to displacement of the resident, or coexistence between the two species (tasks). The authors compute a compatibility score using the resident model and examples from the invading task, and present mathematical and empirical results that claim to show when the compatibility score is informative for forgetting. The intended contribution looks to be a way to understand and diagnose learning transitions, rather than a method for preventing forgetting altogether.

My main reason for the recommendation to reject is that the description does not let the reader understand several measurements which are central to the paper’s claim well enough to evaluate the conclusions. I do not believe these are merely stylistic issues, but rather they also affect the interpretation of the experiments and their strength in justifying the main objective of the paper. At the same time, my difficulty in following the mathematical section of the paper is not evidence that the proofs themselves are wrong. I would cap my confidence in this score at medium, and a healthy amount of clarification of the definitions, procedures, and interpretations could change my assessment.

Critical strengths

1. The central question of the paper is worth investigating

  • Practical relevance: Estimating the risk of forgetting a pretrained task before adapting to a new one could help determine when protection against forgetting is necessary.

  • Appropriate acknowledgment of prior work: The authors appropriately acknowledge that their transferability score construction has precedent, rather than claiming it as a novel contribution.

2. There is meaningful empirical evidence for an association between compatibility of support and forgetting

  • Experiments E1 and E2: The authors report negative correlations across three seeds between the pre-invasion score and subsequent forgetting. This repeatable pattern suggests that forgetting is predictable to some extent from the resident model's support for the incoming task, even before exposure to that task.

  • Generalization within task families: Controlled pixel shuffles of MNIST and rotations of CIFAR-10 yield reasonable group-CV R2R^2 values, suggesting that the approach generalizes to transformations within the same task family.

Critical weaknesses

1. The central measurement object is insufficiently explained

It is difficult to understand and reconstruct the central measurement object clearly enough. Some terminology may be familiar to specialists in the field, but my concern is that central quantities in this paper are left unexplained to the point that their essential meaning is lost.

  • Equation (1): This equation is clearly central to the paper, but the explanation in lines 158–160 does not tell us exactly how the two required errors are obtained. What exactly is constrained here? How is the error measured? When the labels are shuffled, does any retraining take place? These choices seem necessary to understand what the score means, but the details are lacking.

  • Probe calculation: Section 4 and Appendix B allow me to follow the rough sequence (train on A, measure compatibility, train on B, check performance on A). Despite this, I cannot reconstruct the probe calculation from the description provided.

  • Representation compatibility: The representation compatibility score introduced in Figure 4c seems highly relevant, but it is not defined anywhere.

2. The mathematical section is too compressed to explain the proposed theory

Section 3 introduces many quantities and results in quick succession, without sufficient interpretation or explanation of why each new quantity is needed or what the assumptions mean. My concern is not that the paper contains many mathematical results; that is fine. The problem is that the paper does not elaborate on the intuition behind these results, the motivation for the assumptions, what each result contributes to the story, or how the defined quantities connect to the empirical results later. Table 2 appears to map proofs to empirical results, but it does little to connect the defined quantities for the reader.

  • Equation (2): The constrained risk is defined mathematically, but there is no explanation of what the target loss ℓB∗\ell_B^\ast represents or how it is constructed.

  • Resident-sensitive displacement (line 172): SA(ϕ)S_A(\phi) is introduced as a measure of "resident-sensitive displacement," followed immediately by an assumption about its relationship to LAL_A, which is also undefined at that point (and is later interpreted as a loss in Proposition 2). I could not identify concrete examples specifying this displacement or an explanation of why the assumption is reasonable. Such examples would help provide an interpretative guide to the theory.

  • Forgetting (line 161): FA→BF_{A\rightarrow B} appears in Hypothesis 1 before its definition in Section 4 (in terms of two other variables). Although this is not strictly a missing definition, it makes the hypothesis difficult to follow on an initial reading.

3. The experimental results lack sufficient explanation and interpretation

Section 4 provides datasets, models, and experiment counts, but I found it difficult to keep track of what each experiment is doing and measuring. Table 3 attempts to provide an overview, but the individual procedures need more explanation.

  • Experiment E8: Appendix B refers to graded transformations and held-out manipulation groups, but there is no explanation of what constitutes a group or how the experimental design is constructed.

  • Establishment bridge (lines 321–323): This paragraph is a good example of the lack of explanation. It moves from the "empirical coordinate" to the theoretical demand, "unclipped incoming probe loss," and "clipped deficit" without explaining what these quantities mean or what is being compared. Appendix C.2 helps clarify the establishment bridge somewhat, but the main text should explain why this comparison is necessary and what its outcome contributes to the story. Furthermore, the reported interval for the clipped deficit includes zero, raising the question of whether RB(ϕA)R_B(\phi_A) can fall below the target loss. It would be useful to explain whether this possibility supports or conflicts with the mathematical theory.

  • Ecological interpretation: While I generally understand the empirical claim that the score predicts forgetting, I am much less clear about what additional purpose the ecological interpretation serves. The paper explicitly acknowledges that the transfer score idea is not new, so it is unclear what the ecological framework adds to the story or the results.

4. Some experimental conclusions appear insufficiently justified

  • Experiment E2 (group-CV): Group-CV is introduced, but the definition of a group is not stated, even though it is important for understanding how this statistic is computed.

  • Figure 3a (quadratic fit): A quadratic fit is introduced that improves the group-CV R2R^2, as mentioned in lines 317–318. However, the immediately following sentence states that forgetting effectively reaches a floor once the invader is almost fully supported. If the fitted quadratic has a minimum within the support ρ∈[0,1]\rho\in[0,1], it predicts that forgetting increases again beyond that minimum, rather than approaching a floor. This appears inconsistent with the interpretation given in the text.

  • Early/late training contrasts: The introduction states that "within an invasion, plasticity and feature turnover fall from early to late training." However, Appendix B explicitly defines the reported early/late contrasts using task A training. This suggests that the 15/15 and 14/15 positive contrasts establish an early-versus-late pattern during initial resident training, but not necessarily during adaptation to the incoming task.

Questions for authors

1. Could you walk through one complete task transition?

  • Please explain exactly what is trained and what is held fixed for an incoming task, both before and after exposure to it. How are the two errors in Equation (1) obtained, and how is forgetting evaluated?

2. Could you explain one main theoretical result using a concrete empirical example?

  • Theorem 1 appears central to the paper, as it characterizes how much representations must change during invasion and how the target loss and resident-sensitive displacement behave. Could you explain the result in detail using a concrete example and clarify how at least one of your experimental observations tests its predictions?

3. Could you reconcile the E7 comparison with the natural transition experiments?

  • In particular, which result establishes early-to-late stabilization during invasion by task B, rather than during initial training on task A?
5 comments
AuthorsOpenReview ↗

Response to Reviewer sC63 [1/5]

We thank the reviewer for the careful reading and for stating what would change the assessment. We agree that the central measurements were not explained well enough to evaluate the claims. We made three changes. Every quantity is now defined in words before it is used. One complete transition is walked through with its actual numbers. Three interpretations the reviewer questioned are corrected with new analyses. Section numbers refer to the revised manuscript.

W1(a). Equation (1). The submission described eBe_B as the error of a "constrained probe", which mixed it up with the probe of Equation (2). The two are different measurements.

What is computed. Equation (1) trains no neural network or probe. The frozen resident, with its A head, classifies 256 class-balanced images from B's training split. The only free choice is which of the head's two outputs corresponds to which B label. That assignment is fitted on one half of the sample, and the 0-1 error eBe_B is measured on the other half, in both directions and over 5 random half-splits.

What is constrained. Only the label assignment is chosen. The resident's weights and head stay fixed, so eBe_B measures how well the resident's existing decision rule already separates B's classes.

The shuffled reference. eB,sfe_{B,\rm sf} repeats the same computation with B's labels randomly shuffled, 20 times, on the same halves. No network is retrained. Only the label assignment is refitted, in the same way as for the real labels. So eB,sfe_{B,\rm sf} is the error the procedure gives when the head carries no information about B.

What the score means. ρ=1−eB/eB,sf\rho = 1 - \sqrt{e_B/e_{B,\rm sf}} equals 1 when the resident head already classifies B perfectly and is near 0 when it does no better than the shuffled reference. It can be slightly negative from sampling noise. Larger ρ\rho means the resident already supports B.

W1(c). Representation compatibility SS (submitted Figure 4c, now Figure 7c). SS is a label-free comparison score. Let CTC_T be the average of ϕA(x)ϕA(x)⊤\phi_A(x)\phi_A(x)^\top over up to 2,000 images of task TT, where ϕA\phi_A are the resident's features. Then S=⟨CA,CB⟩F/(∥CA∥F∥CB∥F)S = \langle C_A, C_B \rangle_F / (\Vert C_A\Vert _F \Vert C_B\Vert _F), the cosine between the two uncentered feature second-moment matrices. It asks whether B's images excite the same feature directions as A's. Compatibility ρ\rho, which uses B's labels, has a higher observed AUC for coexistence than SS at all four thresholds of the 84-transition suite. SS is now described in Section 5.2 and defined in Appendix D.3.

We have rewritten the text around Equation (1) in Section 3.1, with the full procedure in Appendix B.

Continued in Part 2/5 below.

AuthorsOpenReview ↗

Response to Reviewer sC63 [2/5]

Q1 and W1(b). One complete transition and the probe calculation.

What is trained and what is fixed. Every task's training data are split once into 80% for training and 20% held-out validation on Split-CIFAR-10 (90% and 10% on the new collections), and test data are used only for outcomes.

StepTrainedHeld fixedMeasured
1. ResidentBackbone and A head, on A's training split (20 epochs)NothingRAAR_{AA}, A test accuracy
2. Before BAn auxiliary constrained affine readout for Equation (2), with no update to the residentThe whole residentρ\rho from 256 B images (Equation 1), and the constrained readout risk on frozen features (Equation 2)
3. B-only referenceA separate network trained only on BThe residentℓB⋆\ell^\star_B on held-out B images
4. InvasionBackbone and a new B head, on B's training split (20 epochs)The A headFeature change DBD_B, head-visible change SAS_A, resident loss LAL_A
5. After BNothingEverythingRBAR_{BA}, A test accuracy with the A head, so FA→B=RAA−RBAF_{A\to B} = R_{AA} - R_{BA}

The probe of Equation (2). This is the only trained readout in the theory. An affine readout, with weight matrix WW and a free bias, is fitted on the frozen resident features of 2,048 B training images with cross-entropy, for 600 Adam steps. After each step its weight matrix is projected back to Frobenius norm at most RBR_B, and the bias is left free. We use RB=c∥WA∥R_B = c \Vert W_A\Vert for cc from 0.25 to 16, so the radius is set relative to the resident's own head WAW_A. We estimate the constrained risk by fitting a projected affine readout at each radius and evaluating its cross-entropy on held-out B validation images. We use the lowest observed validation risk over the tested radii up to RBR_B as an empirical estimate of RB(RB)(ϕA)\mathcal R_B^{(R_B)}(\phi_A).

How forgetting is evaluated. The setting is task-incremental, with one head per task. Learning B changes the shared backbone and trains B's new head, and the A head is never trained on B. Forgetting is the drop in A's test accuracy measured through A's own head, FA→B=RAA−RBAF_{A\to B} = R_{AA} - R_{BA}.

The transition in numbers. Resident A is frog vs horse, invader B is ship vs truck (Split-CIFAR-10, seed 43).

QuantityValueMeaning
ρ\rho0.003The resident head is at chance on B
RB(RB)(ϕA)\mathcal R_B^{(R_B)}(\phi_A) at RB=∥WA∥=0.87R_B = \Vert W_A\Vert = 0.870.66Best constrained readout on frozen features
ℓB⋆\ell^\star_B0.20What a B-only network reaches
HH0.46The frozen features fall short of B's target
DBD_B after B16.5Feature change on B images
SAS_A after B6.4Change of A's logits seen through A's head
ΔLA\Delta L_A0.68Increase in the resident's A loss
RAA→RBAR_{AA} \to R_{BA}0.939 → 0.538F=0.40F = 0.40
RBBR_{BB}0.940 (B-only network 0.911)B is learned

We added this walk-through to Section 4 and the full protocol to Appendix B.

Continued in Part 3/5 below.

AuthorsOpenReview ↗

Response to Reviewer sC63 [3/5]

W2. Why each quantity is needed and how it connects to the experiments.

Motivation. Section 3 now follows one transition and introduces each quantity only when the story needs it. The table below is the new Table 2. It replaces the proof-to-result map with a quantity-to-measurement map.

QuantityQuestion it answersMeasured byResult
ρ\rho, Ω=1−ρ\Omega = 1 - \rhoDoes the resident already support B?Equation (1) before BPredicts FF, r=−0.62r = -0.62 and −0.57-0.57 on two collections
FA→BF_{A\to B}How much of A is lost?A test accuracy before and after BThe outcome
ℓB⋆\ell^\star_B, HHCan the frozen features already reach B's target?Constrained readout vs a B-only networkHH tracks FF, r=0.43r = 0.43 on CIFAR-100
DBD_BHow far must the features move?RMS feature change on B imagesTheorem 1 floor holds in every establishing run
SAS_AHow much of that move does A's head see?RMS change of A's logitsTracks FF, r=0.90r = 0.90 in the intervention study
LAL_AHow much does A's loss rise?Cross-entropy of A with A's headGrows with SAS_A at log-log slope 1.92
σ\sigma, s⋆s^\starCan B be learned with little head-visible change?Defined as infima, tested through SAS_APenalizing SAS_A keeps A at matched B accuracy

W2(b). Equation (2) and the target loss ℓB⋆\ell^\star_B. ℓB⋆\ell^\star_B is the cross-entropy that a network of the same architecture, trained only on B, reaches on held-out B validation images. It defines "B is established" by the performance B reaches without a resident. HH then measures how far the resident's frozen features fall short of it.

W2(c). SAS_A and LAL_A. LAL_A is the resident's cross-entropy on task A through A's head, and it is now defined before Assumption 1. SA(ϕ)=(EA∥WA(ϕ(x)−ϕA(x))∥2)1/2S_A(\phi) = (\mathbb E_A\Vert W_A(\phi(x) - \phi_A(x))\Vert ^2)^{1/2} is the root-mean-square change of A's logits caused by the feature change. A feature change that A's head does not read leaves SAS_A at zero, however large it is. In the transition above, SA=6.4S_A = 6.4 came with a loss increase of 0.68 and F=0.40F = 0.40. Assumption 1 relates the rise in LAL_A to SA2S_A^2. Smoothness of the cross-entropy now gives its upper side as a bound with a linear and a quadratic term in SAS_A (Section 3.2). The lower quadratic relation remains an explicit assumption, and its measured trend has a log-log slope of 1.92. Proposition 2 uses the same LAL_A.

W2(d). Forgetting. FA→B=RAA−RBAF_{A\to B} = R_{AA} - R_{BA} is now defined at the start of Section 3, before Hypothesis 1. The other quantities of Section 3 are also defined before their first use.

We revised Sections 3.1 to 3.3 and replaced Table 2.

Continued in Part 4/5 below.

Reviewer_kXzSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Successional Learning Theory (SLT), a framework that interprets continual learning through task-transition compatibility, coexistence, and forgetting. It proposes a normalized, directional compatibility score computed from a frozen resident model to predict forgetting before learning an incoming task. Theoretical results characterize representation displacement, coexistence conditions, and the identifiability of compatibility-based predictions. Experiments across Split-CIFAR-10, CIFAR-100, MNIST, and pretrained models show that compatibility predicts forgetting in several task-incremental settings and helps characterize coexistence and exclusion. However, its predictive power weakens in some settings, and compatibility-guided interventions do not demonstrate statistically significant improvements. Overall, SLT offers a promising diagnostic perspective, although its theoretical grounding, novelty relative to existing transferability measures, and generalizability require further validation.

Critical strengths
  1. Prospective diagnosis of forgetting. The score is computed from a frozen resident model before training on the incoming task. This provides a potentially useful diagnostic for identifying transitions that may require greater protection.
  2. Broad experimental coverage. The study includes natural and controlled task relationships, coexistence tests, replay, multiple optimizers, label-budget analyses, pretrained representations, and tests of feature-level alternatives.
Critical weaknesses

Major concerns (1) The theoretical framework does not adequately explain the central empirical predictor. The main theoretical results concern representation displacement, establishment deficit, and resident-sensitive loss increase. The main empirical claim concerns a normalized transferability score predicting accuracy-based forgetting. The paper does not establish the necessary connection between these quantities, and the empirical bridge to the clipped establishment deficit is inconclusive. The result is a collection of mathematically reasonable statements around a useful empirical association, rather than a sufficiently integrated theory of the observed phenomenon. Theorem 1 provides a generic continuity bound The minimum habitat-modification result establishes a lower bound on the representation change required to reduce the incoming task's constrained probe risk. The proof uses Lipschitz continuity of the loss, a bounded readout norm, and Jensen's inequality. Under the stated definitions, the argument is mathematically reasonable. However, this is essentially a generic stability bound for a constrained linear probe. Its interpretation as a theory of continual learning depends on additional connections that are not established by the theorem itself. In particular, the theorem concerns the incoming-task establishment deficit HA→BH_{A\to B} and the representation displacement DB(ϕ)D_B(\phi). The empirical hypothesis concerns the compatibility score ρA→B\rho_{A\to B} and subsequent resident-task forgetting measured by accuracy change. The theorem does not directly connect these quantities. 2) The coexistence result depends on a strong assumption Assumption 1 requires resident loss increase to be bounded above and below by quadratic functions of resident-sensitive displacement. This is a substantive assumption about the relationship between representation changes and resident-task damage, not a consequence established by the preceding analysis. Theorem 2 then derives a sufficient coexistence condition from this assumption. The argument is valid as a conditional result, but its explanatory value depends on whether the assumption holds for realistic neural networks. The paper does not empirically establish the proposed sensitivity band or demonstrate that the latent quantities σ_(A→B) and s_(A→B)^⋆ are meaningful predictors in the evaluated networks. As a result, the theory provides a useful conditional characterization, but not yet a validated mechanistic explanation of coexistence and exclusion. 3) The central empirical coordinate is not theoretically identified This is the most important theoretical limitation. The theory is formulated in terms of establishment deficit, representation displacement, and resident-sensitive displacement. In contrast, the main experiments use the normalized compatibility score ρ, which is computed from the frozen representation using an incoming-task probe. The authors acknowledge that Ω=1-ρ is an empirical coordinate and do not identify it with the theoretical establishment demand. Their bridge analysis reports a correlation of 0.692 between Ω and the clipped establishment deficit, with a 95% confidence interval of [-0.14,0.94] based on only five incoming-task clusters. This interval includes zero. The stronger association with unclipped probe loss is suggestive, but does not establish the theoretical relationship required by the central interpretation. 4) The remaining results are mathematically standard or narrowly scoped Theorem 3: The identifiability result follows from a standard linear regression model with conditionally mean-zero residuals. The range-restriction corollary is also a classical result. These results are correct in spirit but offer limited theoretical novelty. Proposition 1: The finite-feature-variation result follows from a Polyak–Łojasiewicz condition and a bound on the gradient of log-feature statistics. The paper appropriately describes this as an idealized sufficient result. However, the actual experiments use finite-duration neural-network training rather than gradient flow under a verified PL condition. Proposition 2: The replay result is a local first-order gradient calculation. It explains when increasing resident-task weight reduces the first-order change in resident loss, but does not establish a global replay-budget result or a general relationship between compatibility and the optimal replay allocation. Overall theoretical judgment: The mathematical presentation is coherent, but the paper overstates the extent to which these results establish a unified theory of the measured continual-learning phenomena. Stronger theoretical novelty or a more explicit theory-to-measurement connection is needed.

(2) The novelty relative to existing transferability and task-similarity measures is insufficiently demonstrated. The paper explicitly derives its score from a prior normalized-transfer construction. The principal distinction appears to be its interpretation and the additional outcomes it is used to predict. However, the paper does not provide sufficiently direct comparisons against the strongest relevant prior measures to establish that the new framework adds predictive value beyond existing task-similarity analyses. (3) The statistical evidence for the primary result is weaker than the headline results suggest. The central natural-task experiment contains only five task identities and 20 dependent directed transitions. The primary three-repeat task-label permutation test gives p=0.067, while the separate revalidation gives p=0.033. The small number of incoming-task clusters also limits the reliability of cluster-bootstrap intervals. The strong observed correlation is encouraging, but the inferential evidence does not yet support treating the reported effect size and uncertainty as robust across task collections. (4) The practical utility and generality of SLT are not established sufficiently. The intervention experiments do not show statistically reliable gains from compatibility-based gating with the current number of seeds. The coexistence results also indicate that domain membership can carry substantial information beyond compatibility. Finally, the predictor weakens substantially in the reported single-head class-incremental setting. Together, these results limit the claim that SLT is a general-purpose diagnostic for continual learning, even if it is effective in the evaluated task-incremental settings.

Minor concerns The label-budget robustness results are not monotonic. The reported agreement with the full compatibility estimate varies substantially across small label budgets. For example, the agreement at five labels per class is lower than at two labels per class, while ten labels per class produces a substantially higher value. This may reflect sampling variability, but it makes the practical label-efficiency claim less straightforward than the summary suggests. The authors should report variability across probe-sample draws and clarify how the incoming-task examples are sampled. (2) The coexistence thresholds need a clearer justification. The natural Split-CIFAR-10 experiment uses a forgetting threshold of 0.20, whereas the pure-domain construction uses 0.10. The larger suite evaluates several mutual-retention thresholds, which is helpful, but the rationale for the differing thresholds across constructions should be made explicit. Sensitivity to threshold selection should be distinguished from genuine changes in the underlying coexistence behavior. (3) Some ecological interpretations are stronger than the formal results warrant. Terms such as “invasion,” “habitat modification,” and “successional stages” are useful conceptual devices. However, the current results do not establish that continual learning follows a general ecological mechanism at the feature level; indeed, the paper argues that fixed feature-level ecological coefficients do not explain the observed transitions. The authors should consistently present SLT as a task-transition abstraction unless stronger mechanistic evidence is supplied.

Questions for authors
  1. What is the exact technical distinction between the proposed compatibility score and the normalized-transfer construction of Li and Hiratani (2025)? Can the authors provide direct, controlled comparisons against this measure and other established task-transferability baselines?
  2. Can the authors establish a formal or empirically robust connection between Ω_(A→B), the theoretical establishment deficit H_(A→B), and subsequent resident-task forgetting? How should the bridge analysis be interpreted given that the confidence interval for the clipped deficit includes zero?
  3. Does compatibility predict coexistence after controlling for semantic similarity and domain membership? In particular, how should the weak compatibility–forgetting correlation within the pure-domain construction be reconciled with the claim that the same coordinate resolves coexistence and exclusion?
  4. Given the weak single-head class-incremental result and the non-monotonic performance across small label budgets, what are the expected operating conditions under which SLT remains useful? Can the authors identify the minimum probe-label budget needed for reliable predictions, with uncertainty quantified across different probe-sample draws?
  5. SLT is a promising empirical perspective on pre-adaptation forgetting prediction, but the paper has not yet established that its ecological interpretation constitutes a sufficiently novel and theoretically grounded contribution beyond existing transferability and task-similarity approaches. The most important revision is to strengthen the connection between the formal theory and the measured predictor, while independently validating the main empirical result.
7 comments
AuthorsOpenReview ↗

Response to Reviewer kXzS [1/6]

We thank the reviewer for the careful reading of both the theory and the statistics. The review asks for two things above all, a stronger connection between the formal theory and the measured predictor, and an independent validation of the main result.

First, the revised theory separates the connection into formal inequalities, measured associations and a controlled intervention, each tied to a defined quantity. Second, the main result replicates on a new CIFAR-100 collection with ten task identities. All tests treat task identities as the unit, and section numbers refer to the revised manuscript.

Major 1. Connecting the theory to the empirical predictor.

We agree that the submission presented the theoretical results next to the empirical association without connecting them. The revised Section 3 organizes the quantities along one transition, from the resident's support for B to the forgetting of A, and states the kind of evidence for each link.

LinkTypeStatement and result
Resident support and deficitProved (Proposition R1, new)H≤[Lres−ℓB⋆]+H \le [L_{\rm res} - \ell^\star_B]_+ for the cross-entropy form of the score
MeasuredCIFAR-100, Ω\Omega to HH, r=0.51r = 0.51, and 0.470.47 for the cross-entropy form (both p<0.001p < 0.001)
Deficit and necessary changeProved (Theorem 1)DB≥H/(λR)D_B \ge H/(\lambda R), holds in 108/108 establishing runs
Head-visible change and resident lossUpper side proved (Lemma, new), lower side assumed (Assumption 1)ΔLA≤GASA+SA2/4\Delta L_A \le G_A S_A + S_A^2/4, measured trend close to quadratic (log-log slope 1.92)
Head-visible change and accuracy forgettingProved (margin bound, new)F≤P(0<m0≤γ)+2SA2/γ2F \le P(0 < m_0 \le \gamma) + 2S_A^2/\gamma^2, margin-relative SAS_A tracks FF at r=0.90r = 0.90, 0.740.74, 0.680.68
Which change to protectIntervention (new)Penalizing SAS_A forgets 7.0 pp less than penalizing all features at matched B accuracy

Here LresL_{\rm res} is the B risk of the resident's own head, ℓB⋆\ell^\star_B a fixed B-only target, DBD_B the feature change, SAS_A the root-mean-square change of the resident's A logits, GAG_A the root-mean-square gradient of the A loss in the logits, and m0m_0 the original margin of an A image.

Theorem 1. We agree that Theorem 1 is a continuity bound for a constrained linear readout. Its role in SLT is the link between support and change. Proposition R1 caps the deficit of a well-supported invader, and Theorem 1 turns any deficit into a necessary minimum change of the representation. Together they connect the resident's support for B to the change that establishment requires. The bound holds in all 41 establishing runs on Split-CIFAR-10 and all 108 in the new intervention study. The actual movement depends on the optimizer, and the remaining links in the table describe which part of it damages A.

Continued in Part 2/6 below.

AuthorsOpenReview ↗

Response to Reviewer kXzS [2/6]

Assumption 1 and Theorem 2. Assumption 1 is now used only where it is needed. Its upper half follows from cross-entropy smoothness. The softmax cross-entropy has a Hessian bounded by 1/2 in the logits, which gives ΔLA≤GASA+SA2/4\Delta L_A \le G_A S_A + S_A^2/4. The sufficient coexistence condition of Theorem 2 becomes GAs⋆+(s⋆)2/4≤δAG_A s^\star + (s^\star)^2/4 \le \delta_A and needs no assumption on the upper side. The lower half remains an explicit assumption, and its trend is measurable. Over 1,860 checkpoints along B training, resident loss grows with SAS_A at a log-log slope of 1.92, close to the quadratic form it assumes.

σ\sigma and s⋆s^\star. σ\sigma is the smallest fraction of the feature change that the resident head sees, and s⋆s^\star is the least SAS_A. Both are infima over all representations that establish B, so no finite experiment recovers them. We tested the quantity they are built on, SAS_A, directly. We trained B on six Split-CIFAR-10 transitions (3 seeds, 144 runs) from the same residents under two penalties, one on SAS_A and one on all feature change. Strengths were chosen on A-validation data with B accuracy within 2 pp of unconstrained training, and forgetting was measured on A-test.

Training of BB val. accuracyForgetting of ASAS_AFeature change DAD_A
Unconstrained92.5%30.0%6.321.3
Penalize all feature change91.6%6.8%3.412.0
Penalize head-visible change SAS_A91.5%−0.3-0.3%1.619.5

What this shows. At the same B accuracy, protecting SAS_A forgets 7.0 pp less than protecting all features (task-level 95% CI [1.7,16.3][1.7, 16.3], 6/6 transitions, 17/18 cells). Under the stricter formal-establishment criterion, reselecting the eligible strengths gives +4.7+4.7 pp ([2.3,7.6][2.3, 7.6]) over the 17 valid cells. Under the SAS_A penalty the features move almost as much as in unconstrained training, and A is kept. Across runs, forgetting tracks SAS_A (r=0.90r = 0.90, partial r=0.90r = 0.90 given DAD_A) and not total change (r=0.15r = 0.15). B is learned with far less resident-visible change, as the visible fraction SA/DBS_A/D_B falls from 0.37 to 0.09 while the change on B stays the same (DBD_B 17.9 and 18.5). This supports the resident-sensitive protection principle behind Theorem 2. The penalty builds on logit distillation, while SLT provides a resident-sensitive interpretation supported by the controlled intervention.

Identification of the coordinate (point 3, Q2). We do not identify Ω\Omega with HH. They are different objects. Ω\Omega reads the resident's own head, and HH uses the best constrained readout on the same features. The gap Lres−ℓB⋆L_{\rm res} - \ell^\star_B is the sum of a head-mismatch term and a feature-gap term, and HH is the clipped feature-gap term. Proposition R1 gives a formal one-sided link for the cross-entropy form of the score, and the error-based Ω\Omega is tested empirically.

The bridge interval. HH is clipped at zero whenever the frozen features already meet the target. Theorem 1 then imposes no positive lower bound on representation change, so a zero deficit is consistent with the theory. With five incoming tasks, the submitted interval was wide. With ten task identities on CIFAR-100, the links are significant. Beyond the Ω\Omega to HH association in the table above, we find the following.

LinkResult
Deficit HH and forgettingCIFAR-100 r=0.43r = 0.43 (p<0.001p < 0.001)
Head mismatch and forgettingSplit-CIFAR-10 r=0.62r = 0.62 (p=0.025p = 0.025), CIFAR-100 r=0.46r = 0.46 (p<0.001p < 0.001)
Unclipped constrained risk and forgettingSplit-CIFAR-10 r=0.69r = 0.69 (p=0.025p = 0.025), CIFAR-100 r=0.32r = 0.32 (p=0.002p = 0.002)

So on CIFAR-100 the measured establishment associations are positive at each step, and on Split-CIFAR-10 the score acts through the head-mismatch term. The definitions, Proposition R1 and the decomposition are in Section 3.1, the lemma in Section 3.2, Theorem 2 in Section 3.3, the proofs in Appendix A, and the measurements and intervention in Section 5.3 and Appendices D.2 and D.8.

Continued in Part 3/6 below.

AuthorsOpenReview ↗

Response to Reviewer kXzS [3/6]

Theorem 3. We now credit the identity as classical and restate it as a least-squares projection identity, which removes the linearity assumption, and we state the range-restriction corollary as conditional on shared slope and residual variance. Its role in SLT is diagnostic. It decomposes each observed correlation into slope, spread of ρ\rho and residual noise, and it helps distinguish a change in predictive slope from a change in the spread of ρ\rho. In our ten regimes, the smaller correlations come with a near-zero slope (plain SGD at batch 128, single-head class-IL) or a narrow range of ρ\rho (pretrained ResNet-50). This is in Section 3.4, Section 5.7 and Appendix D.4.

Proposition 1. Proposition 1 assumes gradient flow under a PL condition, and our experiments use finite stochastic training, so we present it as an idealized sufficient result. We measured the behavior it describes directly during B training (30 runs, 6 seeds).

Contrast during B trainingResult
Top-quartile plasticity, early vs late+0.48+0.48, 95% CI [0.37,0.60][0.37, 0.60], positive in 26/30 runs
Top-quartile plasticity, first vs last three epochs+0.52+0.52, [0.43,0.64][0.43, 0.64], 28/30
Feature turnover, first vs last three epochs+0.26+0.26, [0.24,0.29][0.24, 0.29], 30/30

Feature change concentrates early in B training and then settles, as the proposition describes. We describe this as stabilization within training and make no claim of an invasion-specific stage. This is in Section 3.5, Section 5.5 and Appendix D.10.

Proposition 2. Proposition 2 is a local first-order result, and we state it as local with no global replay-budget claim. Our replay experiment (E4) is consistent with its local interpretation. Part (ii) says the first replayed examples help most where displacement is large, and initial replay efficiency correlates 0.700.70 with forgetting and −0.42-0.42 with compatibility. Beyond the local result, compatibility-guided replay allocation raised class-incremental accuracy by 0.9 pp over uniform under SGD (95% CI [0.3,2.0][0.3, 2.0]), which we report as an empirical result (Section 3.5, Section 5.6, Appendix D.6).

Overall theoretical judgment. The revision states each link as proved, measured or tested by intervention, and a quantity-to-measurement table replaces Table 2.

Major 2 and Q1. Novelty and comparison with established measures.

Exact technical distinction. The normalized-transfer formula is that of Li and Hiratani (2025), and we claim neither the formula nor its one-way use as new. Our estimator reads the frozen resident head on 256 class-balanced B training images. It maps the head's outputs to B's labels with an assignment fitted on one half of the sample and scored on the other. The shuffled-label reference uses the same halves and refits the assignment for each of 20 shuffles. Then ρ=1−eB/esf\rho = 1 - \sqrt{e_B/e_{\rm sf}}. Computing the score trains no probe, does not train the resident on B and needs no B-only model. The symmetric form of their score also needs a model trained on B, and gives r=−0.49r = -0.49 on Split-CIFAR-10 against −0.62-0.62 for ours.

What the framework adds. The new elements are the question and its outcomes. The resident state matters for a fixed task pair (within-pair β=−0.31\beta = -0.31, p=0.003p = 0.003), compatibility predicts retention of both tasks (Major 4), and the theory identifies the protection target (Major 1).

Continued in Part 4/6 below.

Reviewer_zXi2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Successional Learning Theory (SLT), which explains continual learning using ecological concepts such as invasion, coexistence, and displacement. It uses a pre-training compatibility score to predict forgetting when learning new tasks. The authors provide mathematical analyses of representation changes, coexistence, and stabilization.

Critical strengths
  1. The paper introduces an ecological interpretation of continual learning. This provides a different perspective on catastrophic forgetting.
  2. The work presents mathematical results on representation changes, coexistence, and stabilization.
  3. The proposed framework predicts forgetting before training on a new task. The reported correlations and coexistence results are encouraging.
Critical weaknesses
  1. The main compatibility measure follows Li and Hiratani (2025), as acknowledged in Section 3.1. The novelty primarily lies in its ecological interpretation and extended analysis. Contribution-wise, it is limited to this submission.
  2. The theoretical results depend on establishment deficit and resident-sensitive displacement in Theorems 1–2. These quantities are not directly measured. Their relationship with the empirical compatibility score requires stronger validation.
  3. Theorem 3 assumes a linear relationship between compatibility and forgetting. Its range-restriction result also assumes unchanged regression parameters across settings. These assumptions are not established for general neural networks.
  4. The perfect AUC of 1.00 is based on only three coexistence cases among 20 transitions as shown in Section 5.2. Coexistence is also defined through forgetting thresholds. This does not independently establish a distinct coexistence phase. So, there is limited evidence for coexistence.
  5. Compatibility-based interventions do not consistently outperform existing methods. DER++ achieves higher accuracy in Table 6. The confidence intervals for the reported gating improvements also include zero.
  6. Most experiments use task-incremental learning with known task boundaries. The compatibility score performs poorly in the single-head class-incremental setting (r = −0.201, Appendix C.5). So, more generalization is needed.
Questions for authors
  1. How does SLT perform against established transferability measures under identical evaluation settings?
  2. Can the authors demonstrate reliable prediction and practical improvements in class-incremental learning?
7 comments
AuthorsOpenReview ↗

Response to Reviewer zXi2 [1/5]

We thank the reviewer for recognizing the ecological perspective, the mathematical analysis, and the prospective forgetting results. The review allowed us to clarify the contribution beyond prior transfer measures, measure the theoretical quantities directly, and test whether they lead to useful protection decisions. We address each point (C1 to C6) and both questions (Q1, Q2) below. All tests treat task identities as the unit, and section numbers refer to the revised manuscript.

C1. Contribution beyond Li and Hiratani (2025). We agree that the normalized-transfer construction follows Li and Hiratani, as acknowledged in Section 3.1. Their work also considers directional estimates and limited labels from the incoming task. Our contribution is not a new normalization. Their main question is task ordering. SLT asks what happens when a trained resident encounters B, and whether the resident's state before adaptation predicts forgetting, mutual retention, and the need for protection.

We added tests designed around these outcomes.

QuestionNew evidenceWhy it matters
Can the resident predict forgetting before learning B?Resident-only, cross-fitted score gives r = −0.622, task-permutation p = 0.025 on Split-CIFAR-10, and r = −0.573, p < 0.001 on an independent ten-task CIFAR-100 collectionThe prospective association replicates across task collections
Does the learned resident state matter for the same task pair?Across scratch and ImageNet-initialized residents trained for 5 to 40 epochs, within-pair β = −0.307, permutation p = 0.0026Compatibility reflects the learned resident, not just the identities of A and B
Does the framework extend beyond forecasting?Two-sided coexistence prediction (C4), measured displacement (C2), and a matched protection intervention (C5)It connects diagnosis to observable transition outcomes and a concrete protection target

We also evaluated the symmetric Li and Hiratani score under our protocol. It requires a B-trained model and gives r = −0.49 on Split-CIFAR-10. Our resident-only estimator needs only the frozen resident and 256 labeled B-training examples, with no B model training. The controlled comparison against established scores follows in Q1.

The ecological vocabulary gives a common interpretation to entry, displacement, coexistence, reinforcement, and stabilization. These are task-transition outcomes, not claims of literal feature-level population dynamics. We have revised the contribution list (Section 1), the positioning paragraph (Section 2) and the opening of Section 3.1, and added the resident-state test (Appendix D.7).

Continued in Part 2/5 below.

AuthorsOpenReview ↗

Response to Reviewer zXi2 [2/5]

Q1. Controlled comparison with established transferability measures. We compared 15 measures and variants under identical settings, with the same resident checkpoints, the same 256 labeled B examples, the same forgetting outcomes and the same task-held-out folds.

Independent replication. CIFAR-100 is the new collection, built after the submission, so it is the cleanest test. Lower held-out error is better.

Scorer with forgettingHeld-out error (pp)
SLT compatibility ρ−0.577.1
NCE−0.547.3
LEEP−0.537.3
GBC−0.278.2
kNN−0.338.3
LogME−0.228.4
Linear probe−0.178.5
No-information predictor8.6
H-score−0.058.6
SPOT+0.128.7

Compatibility gives the highest correlation and the lowest held-out error of all measures. In individual, unadjusted pairwise task-level bootstrap comparisons, it exceeds 12 of the other 14 measures, and LEEP and NCE are statistically tied with it. No measure is significantly stronger than compatibility on CIFAR-100, Split-CIFAR-10 or a third collection, Tiny-ImageNet.

Stability across collections. A useful diagnostic must hold when the tasks change. The second table follows each measure from Split-CIFAR-10 to CIFAR-100.

Scorer, Split-CIFAR-10r, CIFAR-100Change in |r|
SLT compatibility ρ−0.62−0.57−0.05
NCE−0.65−0.54−0.11
LEEP−0.67−0.53−0.13
GBC−0.56−0.27−0.30
H-score−0.39−0.05−0.34
kNN−0.72−0.33−0.38
LogME−0.74−0.22−0.53
Linear probe−0.74−0.17−0.57
SPOT−0.55+0.12sign reverses

Compatibility changes least. Measures of frozen-feature learnability (LogME, linear probe, kNN) correlate strongly on Split-CIFAR-10 and drop sharply on CIFAR-100. Measures that read the resident's own head (compatibility, LEEP, NCE) stay stable, and compatibility is the only measure with |r| ≥ 0.57 on both collections. This supports the paper's hypothesis that the resident's existing support for B governs forgetting. Section 5.1 reports both collections in Table 4, and Table 11 in Appendix D.1 lists all 15 measures with the pairwise tests.

Continued in Part 3/5 below.

AuthorsOpenReview ↗

Response to Reviewer zXi2 [3/5]

C2. Direct measurements of establishment demand and resident-sensitive displacement. We agree that the submission needed a clearer connection between the theorems and the quantities measured after learning B. We now measure the establishment deficit H, representation movement D_B, resident-sensitive movement S_A, resident loss, and accuracy forgetting on the same transitions. H is the positive part of the difference between the risk of the best norm-constrained B readout on the frozen resident features and a fixed B-only target. S_A is the root-mean-square change in A logits under the frozen resident classifier. It distinguishes feature changes that affect A from those the resident head does not read.

We also make the formal links explicit. Proposition R1 uses the cross-entropy resident readout as a feasible candidate, giving *H ≤ [L_res − ℓ_B]₊**. So a resident whose head already fits B well (small L_res) can have only a small deficit. Theorem 1 provides a necessary lower bound on representation movement when B reaches the target, linking establishment demand to the displacement required for adaptation. A separate margin bound connects S_A to accuracy forgetting, since a previously correct A example can be lost only when its logit change crosses the decision margin.

MeasurementResultWhat it shows
CE incompatibility with H, CIFAR-100r = 0.47, p < 0.001Incoming-task difficulty is associated with greater establishment demand
H with forgetting, CIFAR-100r = 0.43, p < 0.001Greater demand is associated with more resident damage
Theorem 1 on establishing runs108/108 satisfy its boundThe required minimum movement is observed under the theorem's conditions
Margin-relative S_A with forgettingr = 0.90, 0.74, 0.68 across three collections (all p ≤ 0.008)The impact of head-visible changes depends on the resident's margins
Resident loss against S_A on sampled training trajectoriesLog-log exponent 1.92The observed scaling is close to quadratic

We then tested the proposed protection target experimentally. In 144 runs over six Split-CIFAR-10 transitions and three seeds, all methods started from the same resident within each transition-seed cell. One penalty limits all feature change. The other limits only the change visible to A's frozen head. Strengths were selected using validation data, requiring incoming-task accuracy within 2 pp of unconstrained training. Forgetting was evaluated on held-out A-test images.

B-training methodB validation accuracyA forgettingS_ATotal feature change D_A
Ordinary training92.5%30.0%6.321.3
All-feature penalty91.6%6.8%3.412.0
Resident-sensitive penalty91.5%−0.3%1.619.5

At virtually the same B accuracy, resident-sensitive protection reduces forgetting by 7.04 percentage points compared with all-feature protection, with task-bootstrap 95% CI [1.74, 16.31]. The advantage is positive in all six transitions and 17/18 transition-seed cells. Requiring formal establishment of B gives a 4.71 pp benefit, CI [2.29, 7.58]. Across the 126 non-frozen runs, forgetting correlates with S_A at r = 0.90, including after controlling for D_A, while its correlation with D_A is 0.15. These pooled correlations are descriptive because the runs share residents.

The intervention shows that successful adaptation can involve substantial feature movement without substantial damage to A. What matters is whether that movement changes the resident's predictions. This supports the resident-sensitive protection principle behind Theorem 2, while the theorem's coexistence condition remains conditional. The formal links are in Sections 3.1 and 3.2, the measurements and intervention in a new Section 5.3, and the details in Appendices D.2 and D.8. A quantity-to-measurement table replaces Table 2.

Continued in Part 4/5 below.

Comments on the paper

2 comments
Area_Chair_v9T7OpenReview ↗

Discussion period

Dear Reviewers,

Thank you for your reviews!

The authors have now submitted their responses. Please read through their rebuttal, acknowledge their reply, and follow up with them if any of your concerns haven't been fully addressed or if you have additional questions.

Please make sure to engage in the discussion before the deadline on November 18 (AoE).

Best,

AC

AuthorsOpenReview ↗

Request for feedback

Dear Reviewers,

Thank you for your time and thoughtful evaluations. As the discussion period approaches its end, we would sincerely appreciate it if you could kindly review our responses and share any further feedback. We hope our responses have addressed your questions and concerns, and we would be grateful if you could take them into consideration when reassessing your evaluations.

Thank you once again for your valuable time and consideration.

Best Regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.