- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Summary
The paper proposes FEAST, a CL method motivated by the idea that stability should be established only on the changes that alter the function to be preserved. The mathematics are clean. The mechanistic experiments provided a deeper understanding of the effects and consequences of FEAST, while the method in itself is evaluated in C-IL and T-IL for CIFAR-100 and TinyImageNet and outperforms the other non-teacher methods. However, those results are specific to the tasks and some assumptions are dependent on the setting studied.
Comment to Authors
Overall, I find the paper a positive contribution to the field. The idea is simple and motivated. The math is clean and well formulated, and I appreciate when the experiments actually go beyond endpoint accuracy and gets into a real explanation of why the method, FEAST, behaves the way it does. Sections 5.3 and 5.4 are particularly important and should be treated as such. I recommend expanding and explaining them a bit more. My main concern is around scope and generality of the final conclusions. The experiments are limited, and the benchmarks are too. I'd be interested in seeing:
- Long-horizon sequences and its consequence on functionally invariant directions.
- The impact of task-boundaries to the current formulation.
I do not expect authors to include all recommendations I made across the whole review. However, such changes will make the paper stronger and push it closer to a clearer accept. I would be open to increase my score if the rebuttal addresses some of them.
Critical strengths
Overall, the paper pays attention to the small experimental details and consequences of its design choices. It also is properly structured, and the motivation and research question are clearly stated. While the paper is not ground breaking improvement, the strengths and ideas presented on this paper have the potential to be of impact in the community and they compensate for the weaknesses.
- The main idea is simple and well motivated. The method follow directly making it easy to interpret.
- The proposed mechanism is actually evaluated in order to understand what is happening rather than just reporting final accuracy. In particular, Section 5.3 is compelling. However, some results are somewhat hard to follow (See Critical Weaknesses for minor comments).
- The main comparison with DER is informative. The study on Section 5.4 answers what DER does in opposition with FEAST. (As with Sec. 5.3 see Critical Weaknesses for minor comments).
- Appendix C.6 provides an interesting and important ablation that evaluates the final calibration step. In Section 4.5, it states that FEA is meant to preserve the old within-task class contrast. This hints at the possible concern that FEAST might not be preserving old-task structure and just benefitting from better task-wise calibration. I'd consider moving the appendix C.6 up to the main text as it answers directly to such concern.
Critical weaknesses
Major
-
The scope of the claim is larger than the setting it is evaluated on. Functional-equivalence is presented as a general principle for stability-plasticity. However, the experiments only test FEAST in one specific invariance. The conclusions should be framed more specifically. Ideally this should have been included in the paper, but the limitations are noted by the authors in Section 6, and are not required on my end for acceptance of this paper. Still, I'd flag it as a major weakness.
-
FEAST relies on the structured Class-IL setting, and task boundaries are known during training. It is limited with respect to other methods that are boundary-free, and due to this, it seems that the current problem formulation is directly dependent on this specific structure.
-
Long horizon preservation of plasticity is not studied. The main argument states that functionally invariant directions remain available for adaptation. However, the longest experiment is only 10 tasks. Studying a longer sequence or analyzing the effective rank and overlap of the subspaces that are protected could strengthen the claim directly.
-
Some figures and results are hard to parse. Section 5.3 and 5.4 contain highly relevant and really interesting questions and results around the main claim. I believe the paper would benefit from breaking them down and explaining them a bit more rather than relying on the performance results alone to carry the weight of the paper.
Minor
- Robustness to task order and composition is not evaluated. It is implied that all five seeds are using the same task partition and class order, so the experiments are susceptible to the continual-learning sequence itself. Thus, repeating the experiments over randomized class partitions would provide a clearer understanding of whether the gains are robust.
Questions for authors
- Do you expect that the available directions remain usable over longer task sequences? See Critical Weaknesses 3.
- How important is the assumption that FEAST knowns task boundaries during training? Could the method be extended to unknown task boundaries? See Critical Weaknesses 2.
- How much of FEAST's advantage comes from the FEA / FCA preserving old knowledge and how much from just correcting the relative logit score levels?
6 comments
Response to Reviewer Z4TA (2/4)
W3 & Q1 — Long-horizon plasticity and functional stability
We agree with the reviewer that evaluating whether plasticity can be sustained over longer task sequences is important, and appreciate this valuable suggestion. Accordingly, we extended our CIFAR-100 experiments from 10 tasks (10 classes/task) to 20 tasks (5 classes/task), using the same from-scratch ResNet-18 and exemplar budget of 500 over three seeds. This evaluation highlighted an aspect of long-term functional preservation that was not evident in our original benchmark.
Long-horizon evaluation and target preservation. Interestingly, the primary challenge was not the loss of new-task acquisition, but the retention of previously learned functions. FEAST maintained strong new-task acquisition even in the later stages of the sequence (Tasks 17–20), while exhibiting more pronounced degradation in the retention of previously learned functions.
In our original implementation, stored functional targets are refreshed at every task boundary to reflect the evolving model. However, this procedure can incorporate deviations from previously learned within-task functions into the refreshed targets, potentially allowing functional drift to accumulate across successive task transitions. This effect may become more pronounced over longer sequences, as more frequent target updates provide additional opportunities for such deviations to accumulate. We therefore tested a simple held-target variant, retaining each exemplar's centered-logit target from its initial inclusion in memory. Importantly, this changes only the target-management policy, leaving FEAST's functional decomposition, FEA/FCA mechanisms, reconciliation, and memory budget unchanged.
Long-horizon performance on CIFAR-100 (20 tasks × 5 classes, M=500)
| Method | Class-IL (%) | Task-IL (%) | Late acquisition (%) |
|---|---|---|---|
| DER | 27.67 ± 1.56 | 77.97 ± 0.93 | 92.05 ± 0.25 |
| SS-IL | 36.01 ± 1.09 | 88.74 ± 0.65 | 88.57 ± 0.76 |
| FEAST (original) | 24.75 ± 3.26 | 79.69 ± 4.74 | 90.95 ± 0.74 |
| FEAST (held-target) | 34.67 ± 0.37 | 86.98 ± 0.49 | 91.75 ± 1.18 |
Holding targets substantially improved long-term retention while maintaining comparable late-stage acquisition. Importantly, both versions of FEAST continued to acquire new tasks effectively even in the later stages of the 20-task sequence (Tasks 17–20), despite the accumulation of previously learned functional constraints. These results suggest that accumulating functional constraints need not prevent continued learning, at least over the task horizon examined here.
Note on target management. The two target-management policies yielded nearly identical performance in the original 10-task benchmark, differing by only 0.11 and 0.17 percentage points in Class-IL and Task-IL, respectively. An FEA×FCA intervention with held targets also reproduced the original qualitative mechanistic pattern. Although holding targets improved retention in the extended setting, refreshing targets may be beneficial in other settings by incorporating improvements to previously learned functions. We therefore retain the submitted implementation as our primary method for consistency with the original evaluations, while reporting the held-target variant separately as a targeted refinement.
Response to Reviewer Z4TA (3/4)
W3 & Q1 — Long-horizon plasticity and functional stability (continued)
Invariant freedom and accumulated functional constraints. Regarding the reviewer's question on whether invariant directions remain useful over longer task sequences, we expect their continued usefulness to depend on future learning demands, rather than being guaranteed by their availability alone. Importantly, FEAST involves two related but distinct forms of freedom. The classifier-side common-coordinate freedom exploited by FCA remains algebraically available as tasks accumulate, owing to the exact shift invariance of the within-task softmax. In contrast, feature-space directions simultaneously invisible to previously learned contrast maps may become increasingly constrained.
To examine how these constraints accumulate, we analyzed the cumulative protected contrast subspace. Each 5-class task contributes at most four independent contrast directions, bounding the cumulative rank by after tasks. In held-target FEAST, the cumulative rank reached this upper bound of 80 out of 512 feature dimensions after 20 tasks (effective rank 72.35). At Task 20, the new contrast-reader subspace showed substantial projection overlap with the previously protected subspace (41.53% versus a 14.84% rank-matched isotropic expectation), yet contributed four additional independent directions. This accumulation raises the question of whether an expanding protected subspace could eventually limit the capacity for adaptation.
However, protected-subspace rank does not directly determine FEAST's available plasticity. Importantly, FEAST does not strictly restrict encoder changes to directions invisible to previously learned contrast maps, but instead softly penalizes deviations from stored functional targets on memory exemplars. Therefore, rank and overlap alone cannot establish whether the remaining freedom is sufficient for subsequent learning. Nevertheless, our extended experiments show that FEAST continues to achieve strong new-task acquisition through 20 tasks, even as previously learned functional constraints accumulate.
W4 — Clarity of mechanistic analyses and figures
We thank the reviewer for recognizing the value of the mechanistic analyses in Sections 5.3–5.4 and for highlighting the need for clearer presentation. We have substantially revised these sections to improve the clarity and logical flow of the analyses, more explicitly connecting selective functional stabilization, invariant-coordinate adaptation, and preservation of previously learned decisions. We also redesigned Figs. 2–3 to improve readability and make the experimental comparisons easier to interpret.
Q3 — Relative contributions of FEA/FCA and reconciliation
FEA/FCA and reconciliation play complementary roles within the same functional-equivalence framework. FEA/FCA protect previously learned within-task functions while allowing functionally invariant common-score adaptation during learning. Crucially, the preserved class contrasts determine not only within-task discrimination but also the input-dependent evidence contributing to cross-block competition (Sec. 3.4). Reconciliation builds on this structure by adjusting the remaining cross-block score alignment using only one scalar offset per block, without altering the learned within-task functions.
Our ablations further clarify their respective contributions. With reconciliation held constant, disabling either FEA or FCA increases Task-IL forgetting (Figure 2c). Conversely, reconciliation substantially improves Class-IL accuracy without altering within-task decisions (Appendix C.5). However, applying the same reconciliation to DER improves its Class-IL performance but does not reproduce FEAST's advantage (Appendix C.6). Following the reviewer's suggestion, we have moved this analysis from Appendix C.6 to Figure 3e in the main text.
Together, these results highlight the importance of both functional preservation and cross-block reconciliation in FEAST. Rather than being an unrelated post-hoc correction, reconciliation naturally follows from the functional decomposition by adjusting cross-block score alignment while preserving the within-task class contrasts and input-dependent evidence maintained through FEA/FCA.
Response to Reviewer Z4TA (1/4)
We thank the reviewer for the thoughtful and constructive suggestions, which helped us identify important areas for improvement and substantially strengthen our work. We also appreciate the reviewer's encouraging assessment of our contributions. Below, we address the reviewer's comments and describe the additional experiments and manuscript revisions made in response.
W1 — Scope and generality of functional equivalence
We are grateful for the reviewer's recognition of our contribution despite this concern about generality. We agree that our experiments validate one specific instantiation of functional equivalence rather than its general applicability. Nevertheless, we believe that translating a general principle into a concrete learning rule and demonstrating its empirical benefits is a meaningful contribution in itself. In particular, our results show that even a simple yet exact invariance of within-task softmax classification can provide a principled basis for allocating stability and plasticity. Extending this approach to other forms of functional equivalence remains an important direction for future work.
W2 & Q2 — Dependence on known task boundaries
The reviewer correctly points out that the current implementation of FEAST assumes known task boundaries during training. These boundaries determine when reference contrast maps are established and functional targets are updated, while known class groupings define the blocks used for FEA, FCA, and reconciliation. However, task identity is not required at inference, where all learned classes compete within a single shared classifier.
Nevertheless, the underlying functional-equivalence principle does not inherently require externally specified task boundaries, as the softmax decomposition can be defined for any given class grouping. In principle, a boundary-free extension could construct class groups and establish reference functions online from the observed data stream. However, identifying appropriate groups and determining when to preserve them introduce additional challenges that we have not investigated. We have further clarified this previously acknowledged limitation in the revised Discussion, explicitly addressing the challenges of extending FEAST to settings without known task boundaries.