- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes the idea of Successional Learning Theory as an interpretation of continual learning using terminology from ecology. A representation supporting an existing task is treated as a resident system, and learning an incoming task is interpreted as an invasion that can lead to displacement of the resident, or coexistence between the two species (tasks). The authors compute a compatibility score using the resident model and examples from the invading task, and present mathematical and empirical results that claim to show when the compatibility score is informative for forgetting. The intended contribution looks to be a way to understand and diagnose learning transitions, rather than a method for preventing forgetting altogether.
My main reason for the recommendation to reject is that the description does not let the reader understand several measurements which are central to the paper’s claim well enough to evaluate the conclusions. I do not believe these are merely stylistic issues, but rather they also affect the interpretation of the experiments and their strength in justifying the main objective of the paper. At the same time, my difficulty in following the mathematical section of the paper is not evidence that the proofs themselves are wrong. I would cap my confidence in this score at medium, and a healthy amount of clarification of the definitions, procedures, and interpretations could change my assessment.
Critical strengths
1. The central question of the paper is worth investigating
-
Practical relevance: Estimating the risk of forgetting a pretrained task before adapting to a new one could help determine when protection against forgetting is necessary.
-
Appropriate acknowledgment of prior work: The authors appropriately acknowledge that their transferability score construction has precedent, rather than claiming it as a novel contribution.
2. There is meaningful empirical evidence for an association between compatibility of support and forgetting
-
Experiments E1 and E2: The authors report negative correlations across three seeds between the pre-invasion score and subsequent forgetting. This repeatable pattern suggests that forgetting is predictable to some extent from the resident model's support for the incoming task, even before exposure to that task.
-
Generalization within task families: Controlled pixel shuffles of MNIST and rotations of CIFAR-10 yield reasonable group-CV values, suggesting that the approach generalizes to transformations within the same task family.
Critical weaknesses
1. The central measurement object is insufficiently explained
It is difficult to understand and reconstruct the central measurement object clearly enough. Some terminology may be familiar to specialists in the field, but my concern is that central quantities in this paper are left unexplained to the point that their essential meaning is lost.
-
Equation (1): This equation is clearly central to the paper, but the explanation in lines 158–160 does not tell us exactly how the two required errors are obtained. What exactly is constrained here? How is the error measured? When the labels are shuffled, does any retraining take place? These choices seem necessary to understand what the score means, but the details are lacking.
-
Probe calculation: Section 4 and Appendix B allow me to follow the rough sequence (train on A, measure compatibility, train on B, check performance on A). Despite this, I cannot reconstruct the probe calculation from the description provided.
-
Representation compatibility: The representation compatibility score introduced in Figure 4c seems highly relevant, but it is not defined anywhere.
2. The mathematical section is too compressed to explain the proposed theory
Section 3 introduces many quantities and results in quick succession, without sufficient interpretation or explanation of why each new quantity is needed or what the assumptions mean. My concern is not that the paper contains many mathematical results; that is fine. The problem is that the paper does not elaborate on the intuition behind these results, the motivation for the assumptions, what each result contributes to the story, or how the defined quantities connect to the empirical results later. Table 2 appears to map proofs to empirical results, but it does little to connect the defined quantities for the reader.
-
Equation (2): The constrained risk is defined mathematically, but there is no explanation of what the target loss represents or how it is constructed.
-
Resident-sensitive displacement (line 172): is introduced as a measure of "resident-sensitive displacement," followed immediately by an assumption about its relationship to , which is also undefined at that point (and is later interpreted as a loss in Proposition 2). I could not identify concrete examples specifying this displacement or an explanation of why the assumption is reasonable. Such examples would help provide an interpretative guide to the theory.
-
Forgetting (line 161): appears in Hypothesis 1 before its definition in Section 4 (in terms of two other variables). Although this is not strictly a missing definition, it makes the hypothesis difficult to follow on an initial reading.
3. The experimental results lack sufficient explanation and interpretation
Section 4 provides datasets, models, and experiment counts, but I found it difficult to keep track of what each experiment is doing and measuring. Table 3 attempts to provide an overview, but the individual procedures need more explanation.
-
Experiment E8: Appendix B refers to graded transformations and held-out manipulation groups, but there is no explanation of what constitutes a group or how the experimental design is constructed.
-
Establishment bridge (lines 321–323): This paragraph is a good example of the lack of explanation. It moves from the "empirical coordinate" to the theoretical demand, "unclipped incoming probe loss," and "clipped deficit" without explaining what these quantities mean or what is being compared. Appendix C.2 helps clarify the establishment bridge somewhat, but the main text should explain why this comparison is necessary and what its outcome contributes to the story. Furthermore, the reported interval for the clipped deficit includes zero, raising the question of whether can fall below the target loss. It would be useful to explain whether this possibility supports or conflicts with the mathematical theory.
-
Ecological interpretation: While I generally understand the empirical claim that the score predicts forgetting, I am much less clear about what additional purpose the ecological interpretation serves. The paper explicitly acknowledges that the transfer score idea is not new, so it is unclear what the ecological framework adds to the story or the results.
4. Some experimental conclusions appear insufficiently justified
-
Experiment E2 (group-CV): Group-CV is introduced, but the definition of a group is not stated, even though it is important for understanding how this statistic is computed.
-
Figure 3a (quadratic fit): A quadratic fit is introduced that improves the group-CV , as mentioned in lines 317–318. However, the immediately following sentence states that forgetting effectively reaches a floor once the invader is almost fully supported. If the fitted quadratic has a minimum within the support , it predicts that forgetting increases again beyond that minimum, rather than approaching a floor. This appears inconsistent with the interpretation given in the text.
-
Early/late training contrasts: The introduction states that "within an invasion, plasticity and feature turnover fall from early to late training." However, Appendix B explicitly defines the reported early/late contrasts using task A training. This suggests that the 15/15 and 14/15 positive contrasts establish an early-versus-late pattern during initial resident training, but not necessarily during adaptation to the incoming task.
Questions for authors
1. Could you walk through one complete task transition?
- Please explain exactly what is trained and what is held fixed for an incoming task, both before and after exposure to it. How are the two errors in Equation (1) obtained, and how is forgetting evaluated?
2. Could you explain one main theoretical result using a concrete empirical example?
- Theorem 1 appears central to the paper, as it characterizes how much representations must change during invasion and how the target loss and resident-sensitive displacement behave. Could you explain the result in detail using a concrete example and clarify how at least one of your experimental observations tests its predictions?
3. Could you reconcile the E7 comparison with the natural transition experiments?
- In particular, which result establishes early-to-late stabilization during invasion by task B, rather than during initial training on task A?
5 comments
Response to Reviewer sC63 [2/5]
Q1 and W1(b). One complete transition and the probe calculation.
What is trained and what is fixed. Every task's training data are split once into 80% for training and 20% held-out validation on Split-CIFAR-10 (90% and 10% on the new collections), and test data are used only for outcomes.
| Step | Trained | Held fixed | Measured |
|---|---|---|---|
| 1. Resident | Backbone and A head, on A's training split (20 epochs) | Nothing | , A test accuracy |
| 2. Before B | An auxiliary constrained affine readout for Equation (2), with no update to the resident | The whole resident | from 256 B images (Equation 1), and the constrained readout risk on frozen features (Equation 2) |
| 3. B-only reference | A separate network trained only on B | The resident | on held-out B images |
| 4. Invasion | Backbone and a new B head, on B's training split (20 epochs) | The A head | Feature change , head-visible change , resident loss |
| 5. After B | Nothing | Everything | , A test accuracy with the A head, so |
The probe of Equation (2). This is the only trained readout in the theory. An affine readout, with weight matrix and a free bias, is fitted on the frozen resident features of 2,048 B training images with cross-entropy, for 600 Adam steps. After each step its weight matrix is projected back to Frobenius norm at most , and the bias is left free. We use for from 0.25 to 16, so the radius is set relative to the resident's own head . We estimate the constrained risk by fitting a projected affine readout at each radius and evaluating its cross-entropy on held-out B validation images. We use the lowest observed validation risk over the tested radii up to as an empirical estimate of .
How forgetting is evaluated. The setting is task-incremental, with one head per task. Learning B changes the shared backbone and trains B's new head, and the A head is never trained on B. Forgetting is the drop in A's test accuracy measured through A's own head, .
The transition in numbers. Resident A is frog vs horse, invader B is ship vs truck (Split-CIFAR-10, seed 43).
| Quantity | Value | Meaning |
|---|---|---|
| 0.003 | The resident head is at chance on B | |
| at | 0.66 | Best constrained readout on frozen features |
| 0.20 | What a B-only network reaches | |
| 0.46 | The frozen features fall short of B's target | |
| after B | 16.5 | Feature change on B images |
| after B | 6.4 | Change of A's logits seen through A's head |
| 0.68 | Increase in the resident's A loss | |
| 0.939 → 0.538 | ||
| 0.940 (B-only network 0.911) | B is learned |
We added this walk-through to Section 4 and the full protocol to Appendix B.
Continued in Part 3/5 below.
Response to Reviewer sC63 [3/5]
W2. Why each quantity is needed and how it connects to the experiments.
Motivation. Section 3 now follows one transition and introduces each quantity only when the story needs it. The table below is the new Table 2. It replaces the proof-to-result map with a quantity-to-measurement map.
| Quantity | Question it answers | Measured by | Result |
|---|---|---|---|
| , | Does the resident already support B? | Equation (1) before B | Predicts , and on two collections |
| How much of A is lost? | A test accuracy before and after B | The outcome | |
| , | Can the frozen features already reach B's target? | Constrained readout vs a B-only network | tracks , on CIFAR-100 |
| How far must the features move? | RMS feature change on B images | Theorem 1 floor holds in every establishing run | |
| How much of that move does A's head see? | RMS change of A's logits | Tracks , in the intervention study | |
| How much does A's loss rise? | Cross-entropy of A with A's head | Grows with at log-log slope 1.92 | |
| , | Can B be learned with little head-visible change? | Defined as infima, tested through | Penalizing keeps A at matched B accuracy |
W2(b). Equation (2) and the target loss . is the cross-entropy that a network of the same architecture, trained only on B, reaches on held-out B validation images. It defines "B is established" by the performance B reaches without a resident. then measures how far the resident's frozen features fall short of it.
W2(c). and . is the resident's cross-entropy on task A through A's head, and it is now defined before Assumption 1. is the root-mean-square change of A's logits caused by the feature change. A feature change that A's head does not read leaves at zero, however large it is. In the transition above, came with a loss increase of 0.68 and . Assumption 1 relates the rise in to . Smoothness of the cross-entropy now gives its upper side as a bound with a linear and a quadratic term in (Section 3.2). The lower quadratic relation remains an explicit assumption, and its measured trend has a log-log slope of 1.92. Proposition 2 uses the same .
W2(d). Forgetting. is now defined at the start of Section 3, before Hypothesis 1. The other quantities of Section 3 are also defined before their first use.
We revised Sections 3.1 to 3.3 and replaced Table 2.
Continued in Part 4/5 below.
Response to Reviewer sC63 [1/5]
We thank the reviewer for the careful reading and for stating what would change the assessment. We agree that the central measurements were not explained well enough to evaluate the claims. We made three changes. Every quantity is now defined in words before it is used. One complete transition is walked through with its actual numbers. Three interpretations the reviewer questioned are corrected with new analyses. Section numbers refer to the revised manuscript.
W1(a). Equation (1). The submission described eB as the error of a "constrained probe", which mixed it up with the probe of Equation (2). The two are different measurements.
What is computed. Equation (1) trains no neural network or probe. The frozen resident, with its A head, classifies 256 class-balanced images from B's training split. The only free choice is which of the head's two outputs corresponds to which B label. That assignment is fitted on one half of the sample, and the 0-1 error eB is measured on the other half, in both directions and over 5 random half-splits.
What is constrained. Only the label assignment is chosen. The resident's weights and head stay fixed, so eB measures how well the resident's existing decision rule already separates B's classes.
The shuffled reference. eB,sf repeats the same computation with B's labels randomly shuffled, 20 times, on the same halves. No network is retrained. Only the label assignment is refitted, in the same way as for the real labels. So eB,sf is the error the procedure gives when the head carries no information about B.
What the score means. ρ=1−eB/eB,sf equals 1 when the resident head already classifies B perfectly and is near 0 when it does no better than the shuffled reference. It can be slightly negative from sampling noise. Larger ρ means the resident already supports B.
W1(c). Representation compatibility S (submitted Figure 4c, now Figure 7c). S is a label-free comparison score. Let CT be the average of ϕA(x)ϕA(x)⊤ over up to 2,000 images of task T, where ϕA are the resident's features. Then S=⟨CA,CB⟩F/(∥CA∥F∥CB∥F), the cosine between the two uncentered feature second-moment matrices. It asks whether B's images excite the same feature directions as A's. Compatibility ρ, which uses B's labels, has a higher observed AUC for coexistence than S at all four thresholds of the 84-transition suite. S is now described in Section 5.2 and defined in Appendix D.3.
We have rewritten the text around Equation (1) in Section 3.1, with the full procedure in Appendix B.
Continued in Part 2/5 below.