acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the stability–plasticity dilemma in continual learning through functional equivalence

Abstract

Continual learning requires models to acquire new knowledge without disrupting previously learned capabilities, creating a fundamental tension between stability and plasticity. Stability, however, is often imposed without distinguishing which internal changes are actually detrimental to retention. This paper reconceptualizes continual learning through functional equivalence: internal states that realize the same function are treated as equivalent, thereby leaving functionally invariant degrees of freedom available for adaptation. In multiclass softmax classification, this equivalence admits an exact decomposition of logits into relative class scores that determine class probabilities within a task and common score shifts that leave them unchanged. Building on this structure, we instantiate the principle in continual learning through FEAST (Functional Equivalence–guided Allocation of STability and Plasticity). FEAST yields strong Class-IL and Task-IL performance compared with methods that do not retain an additional full teacher network. We further provide a mechanistic account of how these gains arise: stability is concentrated on function-defining coordinates, while functionally invariant freedom provides an alternative path for adaptation. This division of labor improves acquisition while reducing function-defining output change and forgetting. Together, these results recast the stability–plasticity dilemma in terms of the functional consequences of change rather than change itself.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
stability-plasticity, continual learning, functional equivalence, class-incremental learning, catastrophic forgetting, representational drift

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z4TAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

The paper proposes FEAST, a CL method motivated by the idea that stability should be established only on the changes that alter the function to be preserved. The mathematics are clean. The mechanistic experiments provided a deeper understanding of the effects and consequences of FEAST, while the method in itself is evaluated in C-IL and T-IL for CIFAR-100 and TinyImageNet and outperforms the other non-teacher methods. However, those results are specific to the tasks and some assumptions are dependent on the setting studied.


Comment to Authors

Overall, I find the paper a positive contribution to the field. The idea is simple and motivated. The math is clean and well formulated, and I appreciate when the experiments actually go beyond endpoint accuracy and gets into a real explanation of why the method, FEAST, behaves the way it does. Sections 5.3 and 5.4 are particularly important and should be treated as such. I recommend expanding and explaining them a bit more. My main concern is around scope and generality of the final conclusions. The experiments are limited, and the benchmarks are too. I'd be interested in seeing:

  • Long-horizon sequences and its consequence on functionally invariant directions.
  • The impact of task-boundaries to the current formulation.

I do not expect authors to include all recommendations I made across the whole review. However, such changes will make the paper stronger and push it closer to a clearer accept. I would be open to increase my score if the rebuttal addresses some of them.

Critical strengths

Overall, the paper pays attention to the small experimental details and consequences of its design choices. It also is properly structured, and the motivation and research question are clearly stated. While the paper is not ground breaking improvement, the strengths and ideas presented on this paper have the potential to be of impact in the community and they compensate for the weaknesses.


  • The main idea is simple and well motivated. The method follow directly making it easy to interpret.
  • The proposed mechanism is actually evaluated in order to understand what is happening rather than just reporting final accuracy. In particular, Section 5.3 is compelling. However, some results are somewhat hard to follow (See Critical Weaknesses for minor comments).
  • The main comparison with DER is informative. The study on Section 5.4 answers what DER does in opposition with FEAST. (As with Sec. 5.3 see Critical Weaknesses for minor comments).
  • Appendix C.6 provides an interesting and important ablation that evaluates the final calibration step. In Section 4.5, it states that FEA is meant to preserve the old within-task class contrast. This hints at the possible concern that FEAST might not be preserving old-task structure and just benefitting from better task-wise calibration. I'd consider moving the appendix C.6 up to the main text as it answers directly to such concern.
Critical weaknesses

Major

  1. The scope of the claim is larger than the setting it is evaluated on. Functional-equivalence is presented as a general principle for stability-plasticity. However, the experiments only test FEAST in one specific invariance. The conclusions should be framed more specifically. Ideally this should have been included in the paper, but the limitations are noted by the authors in Section 6, and are not required on my end for acceptance of this paper. Still, I'd flag it as a major weakness.

  2. FEAST relies on the structured Class-IL setting, and task boundaries are known during training. It is limited with respect to other methods that are boundary-free, and due to this, it seems that the current problem formulation is directly dependent on this specific structure.

  3. Long horizon preservation of plasticity is not studied. The main argument states that functionally invariant directions remain available for adaptation. However, the longest experiment is only 10 tasks. Studying a longer sequence or analyzing the effective rank and overlap of the subspaces that are protected could strengthen the claim directly.

  4. Some figures and results are hard to parse. Section 5.3 and 5.4 contain highly relevant and really interesting questions and results around the main claim. I believe the paper would benefit from breaking them down and explaining them a bit more rather than relying on the performance results alone to carry the weight of the paper.


Minor

  1. Robustness to task order and composition is not evaluated. It is implied that all five seeds are using the same task partition and class order, so the experiments are susceptible to the continual-learning sequence itself. Thus, repeating the experiments over randomized class partitions would provide a clearer understanding of whether the gains are robust.
Questions for authors
  1. Do you expect that the available directions remain usable over longer task sequences? See Critical Weaknesses 3.
  2. How important is the assumption that FEAST knowns task boundaries during training? Could the method be extended to unknown task boundaries? See Critical Weaknesses 2.
  3. How much of FEAST's advantage comes from the FEA / FCA preserving old knowledge and how much from just correcting the relative logit score levels?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Z4TA (1/4)

We thank the reviewer for the thoughtful and constructive suggestions, which helped us identify important areas for improvement and substantially strengthen our work. We also appreciate the reviewer's encouraging assessment of our contributions. Below, we address the reviewer's comments and describe the additional experiments and manuscript revisions made in response.


W1 — Scope and generality of functional equivalence

We are grateful for the reviewer's recognition of our contribution despite this concern about generality. We agree that our experiments validate one specific instantiation of functional equivalence rather than its general applicability. Nevertheless, we believe that translating a general principle into a concrete learning rule and demonstrating its empirical benefits is a meaningful contribution in itself. In particular, our results show that even a simple yet exact invariance of within-task softmax classification can provide a principled basis for allocating stability and plasticity. Extending this approach to other forms of functional equivalence remains an important direction for future work.


W2 & Q2 — Dependence on known task boundaries

The reviewer correctly points out that the current implementation of FEAST assumes known task boundaries during training. These boundaries determine when reference contrast maps are established and functional targets are updated, while known class groupings define the blocks used for FEA, FCA, and reconciliation. However, task identity is not required at inference, where all learned classes compete within a single shared classifier.

Nevertheless, the underlying functional-equivalence principle does not inherently require externally specified task boundaries, as the softmax decomposition can be defined for any given class grouping. In principle, a boundary-free extension could construct class groups and establish reference functions online from the observed data stream. However, identifying appropriate groups and determining when to preserve them introduce additional challenges that we have not investigated. We have further clarified this previously acknowledged limitation in the revised Discussion, explicitly addressing the challenges of extending FEAST to settings without known task boundaries.

AuthorsOpenReview ↗

Response to Reviewer Z4TA (2/4)

W3 & Q1 — Long-horizon plasticity and functional stability

We agree with the reviewer that evaluating whether plasticity can be sustained over longer task sequences is important, and appreciate this valuable suggestion. Accordingly, we extended our CIFAR-100 experiments from 10 tasks (10 classes/task) to 20 tasks (5 classes/task), using the same from-scratch ResNet-18 and exemplar budget of 500 over three seeds. This evaluation highlighted an aspect of long-term functional preservation that was not evident in our original benchmark.

Long-horizon evaluation and target preservation. Interestingly, the primary challenge was not the loss of new-task acquisition, but the retention of previously learned functions. FEAST maintained strong new-task acquisition even in the later stages of the sequence (Tasks 17–20), while exhibiting more pronounced degradation in the retention of previously learned functions.

In our original implementation, stored functional targets are refreshed at every task boundary to reflect the evolving model. However, this procedure can incorporate deviations from previously learned within-task functions into the refreshed targets, potentially allowing functional drift to accumulate across successive task transitions. This effect may become more pronounced over longer sequences, as more frequent target updates provide additional opportunities for such deviations to accumulate. We therefore tested a simple held-target variant, retaining each exemplar's centered-logit target from its initial inclusion in memory. Importantly, this changes only the target-management policy, leaving FEAST's functional decomposition, FEA/FCA mechanisms, reconciliation, and memory budget unchanged.

Long-horizon performance on CIFAR-100 (20 tasks × 5 classes, M=500)

MethodClass-IL (%)Task-IL (%)Late acquisition (%)
DER27.67 ± 1.5677.97 ± 0.9392.05 ± 0.25
SS-IL36.01 ± 1.0988.74 ± 0.6588.57 ± 0.76
FEAST (original)24.75 ± 3.2679.69 ± 4.7490.95 ± 0.74
FEAST (held-target)34.67 ± 0.3786.98 ± 0.4991.75 ± 1.18

Holding targets substantially improved long-term retention while maintaining comparable late-stage acquisition. Importantly, both versions of FEAST continued to acquire new tasks effectively even in the later stages of the 20-task sequence (Tasks 17–20), despite the accumulation of previously learned functional constraints. These results suggest that accumulating functional constraints need not prevent continued learning, at least over the task horizon examined here.

Note on target management. The two target-management policies yielded nearly identical performance in the original 10-task benchmark, differing by only 0.11 and 0.17 percentage points in Class-IL and Task-IL, respectively. An FEA×FCA intervention with held targets also reproduced the original qualitative mechanistic pattern. Although holding targets improved retention in the extended setting, refreshing targets may be beneficial in other settings by incorporating improvements to previously learned functions. We therefore retain the submitted implementation as our primary method for consistency with the original evaluations, while reporting the held-target variant separately as a targeted refinement.

AuthorsOpenReview ↗

Response to Reviewer Z4TA (3/4)

W3 & Q1 — Long-horizon plasticity and functional stability (continued)

Invariant freedom and accumulated functional constraints. Regarding the reviewer's question on whether invariant directions remain useful over longer task sequences, we expect their continued usefulness to depend on future learning demands, rather than being guaranteed by their availability alone. Importantly, FEAST involves two related but distinct forms of freedom. The classifier-side common-coordinate freedom exploited by FCA remains algebraically available as tasks accumulate, owing to the exact shift invariance of the within-task softmax. In contrast, feature-space directions simultaneously invisible to previously learned contrast maps may become increasingly constrained.

To examine how these constraints accumulate, we analyzed the cumulative protected contrast subspace. Each 5-class task contributes at most four independent contrast directions, bounding the cumulative rank by 4t4t after tt tasks. In held-target FEAST, the cumulative rank reached this upper bound of 80 out of 512 feature dimensions after 20 tasks (effective rank 72.35). At Task 20, the new contrast-reader subspace showed substantial projection overlap with the previously protected subspace (41.53% versus a 14.84% rank-matched isotropic expectation), yet contributed four additional independent directions. This accumulation raises the question of whether an expanding protected subspace could eventually limit the capacity for adaptation.

However, protected-subspace rank does not directly determine FEAST's available plasticity. Importantly, FEAST does not strictly restrict encoder changes to directions invisible to previously learned contrast maps, but instead softly penalizes deviations from stored functional targets on memory exemplars. Therefore, rank and overlap alone cannot establish whether the remaining freedom is sufficient for subsequent learning. Nevertheless, our extended experiments show that FEAST continues to achieve strong new-task acquisition through 20 tasks, even as previously learned functional constraints accumulate.


W4 — Clarity of mechanistic analyses and figures

We thank the reviewer for recognizing the value of the mechanistic analyses in Sections 5.3–5.4 and for highlighting the need for clearer presentation. We have substantially revised these sections to improve the clarity and logical flow of the analyses, more explicitly connecting selective functional stabilization, invariant-coordinate adaptation, and preservation of previously learned decisions. We also redesigned Figs. 2–3 to improve readability and make the experimental comparisons easier to interpret.


Q3 — Relative contributions of FEA/FCA and reconciliation

FEA/FCA and reconciliation play complementary roles within the same functional-equivalence framework. FEA/FCA protect previously learned within-task functions while allowing functionally invariant common-score adaptation during learning. Crucially, the preserved class contrasts determine not only within-task discrimination but also the input-dependent evidence contributing to cross-block competition (Sec. 3.4). Reconciliation builds on this structure by adjusting the remaining cross-block score alignment using only one scalar offset per block, without altering the learned within-task functions.

Our ablations further clarify their respective contributions. With reconciliation held constant, disabling either FEA or FCA increases Task-IL forgetting (Figure 2c). Conversely, reconciliation substantially improves Class-IL accuracy without altering within-task decisions (Appendix C.5). However, applying the same reconciliation to DER improves its Class-IL performance but does not reproduce FEAST's advantage (Appendix C.6). Following the reviewer's suggestion, we have moved this analysis from Appendix C.6 to Figure 3e in the main text.

Together, these results highlight the importance of both functional preservation and cross-block reconciliation in FEAST. Rather than being an unrelated post-hoc correction, reconciliation naturally follows from the functional decomposition by adjusting cross-block score alignment while preserving the within-task class contrasts and input-dependent evidence maintained through FEA/FCA.

Reviewer_kbnpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper attacks the stability versus plasticity dilemma by asking what must remain stable rather than how much change to allow. Its starting point is an exact property of multiclass softmax classifiers: logits of a task block decompose into relative (within-task) class scores, which determine the class probabilities, and a common score shift that leaves them unchanged. Internal states differing only in the latter realise the same function, a property termed “functional equivalence”; therefore, the common-score direction is a degree of freedom available for adaptation while the function-defining contrast map is held fixed. FEAST instantiates this: a FEA loss penalises encoder changes that alter the function-defining output of the reference contrast map on memory exemplars (Eq. 8); FCA projects each previous classifier block's gradients onto the functionally invariant subspace and reprojects the block onto the admissible family after every step (Eqs. 9 and 10); and a cross-block reconciliation step replaces each block's learned common score with a single scalar offset fitted on class-balanced memory (Eq. 11). Evaluation: CIFAR-100 and TinyImageNet, ResNet-18 from scratch in Mammoth, ten tasks each, exemplar budgets 200/500, five seeds, plus a matched 2×2 FEA×FCA intervention and bespoke mechanism metrics.

Critical strengths
  1. The method follows mechanically from the theory. Fix the function-defining component, keep the invariant degrees of freedom, adjudicate the common score with one scalar per block. No extra teacher network, no task-specific solver, no hyper-parameter-heavy machinery.
Critical weaknesses
  1. The 2×2 intervention does not identify the mechanism the paper claims. In Table G.1, FEA alone worsens acquisition (held-out CE-AUC 0.971 → 1.127; only the combination improves it to 0.776), yet 5.3 reports only the FCA-given-FEA effect (−0.35) and presents the pair as jointly improving acquisition and retention. Worse, the FCA-off conditions show Classifier-Q = 0.000, meaning they freeze the entire old classifier block instead of removing the projection defined in 4.4, so the ablation does not ablate FCA at all. Moreover, "reallocation rather than suppression" is unidentifiable as reported, because only the classifier-visible fraction is given (39.43 → 15.21 → 15.86) and the total representational-change energy, which is the denominator, is never reported. This is the harshest point because it reduces the contribution from "a mechanism" to "a combination that happens to work".
  2. The comparison is scoped so that the methods which beat FEAST fall outside the "comparable" group. On TinyImageNet at M = 500 (Table 2), SS-IL reaches 73.37 Task-IL / 33.73 Class-IL and LUCIR 68.43 / 34.41, against FEAST's 66.72 / 25.93, a 7.8-point Class-IL deficit; on CIFAR-100 both exceed FEAST's Task-IL (85.58 and 85.55 vs 83.77). Meanwhile, the abstract's performance claim is explicitly restricted to "methods that do not retain an additional full teacher network". The defence (retained tensor memory, Figures F.3 and F.4) never appears in the main table, while FEAST itself retains per-task function-defining components and stored centred scores, so the grouping criterion is doing the persuasive work rather than a matched-memory comparison.
Questions for authors
  1. The experimental protocol is dated, even though the thesis is architecture-agnostic. The comparison set is anchored in 2019 to 2023 work (LUCIR 2019, BiC 2019, ER 2019, DER/DER++ 2020, WA 2020, SS-IL 2021, ER-ACE 2022, LODE 2023) and in a ResNet-18 trained from scratch at CIFAR scale. Since "preserve the function, not the state" makes no assumption about depth, width or pretraining, why is it tested only here? Please (a) report the result with a modern pretrained backbone (ViT-S/B, or a frozen CLIP encoder), where both representation drift and the geometry of the function-defining subspace differ materially from a from-scratch ResNet-18; (b) state which baselines remain the relevant comparison in that regime; and (c) add at least one 2025 to 2026 class-incremental baseline so that "strong Class-IL and Task-IL performance" is anchored in the current state of the art rather than in 2019 to 2023 numbers.
  2. The method is rehearsal-based at a time when the field is moving away from rehearsal. FEAST consumes an exemplar buffer, stores per-exemplar centred scores, and keeps one classifier block per task; Table 2 nevertheless groups methods by whether they retain a full teacher network rather than by retained data. Much of the current state of the art, including LoRA-based and prompt-based continual learning on ViT backbones (L2P/DualPrompt/CODA-Prompt-style methods and their successors), is rehearsal-free; it is motivated precisely by privacy, storage and data-retention constraints, and the paper cites none of it.
11 comments
AuthorsOpenReview ↗

Response to Reviewer kbnp (1/5)

We thank the reviewer for the careful assessment and constructive suggestions. In particular, the comments helped us clarify the precise intervention tested in our original 2×2 intervention and motivated additional controls and a pretrained-backbone evaluation. We respond point by point below.


W1 — Clarification of the FCA intervention and additional classifier control

Clarification of the original intervention. Our original 2×22\times2 intervention was designed to isolate the contribution of invariant-coordinate (QkQ_k) plasticity under a common constraint that preserves historical classifier contrasts (PkWk,PkbkP_kW_k, P_kb_k). As described in the Section 4 overview, FCA allows previous classifier parameters to adapt while preserving their reference contrast maps. Specifically, historical contrast preservation serves as the common constraint, while FCA enables adaptation along the remaining QkQ_k coordinates. Thus, disabling FCA freezes the previous classifier blocks, whereas enabling FCA permits old-QkQ_k updates, with historical PkP_k components fixed in all four conditions. We acknowledge that referring to the gradient projection itself as FCA in Section 4.4 made this distinction unclear. We have revised Section 4.4 to clarify the definition of FCA and Section 5.3 to explicitly specify the manipulated factor in the intervention.

Additional control: Fixed-P vs. Trainable-P. To directly address the reviewer's concern, we conducted an additional comparison between two historical classifier update policies under otherwise matched training conditions, including identical full-way CE and live FEA objectives:

  • Fixed-P (original FCA ON): The function-defining components (PkWk,PkbkP_kW_k, P_kb_k) of previous classifier blocks remain fixed, while their functionally invariant components (QkQ_k) are trainable.

  • Trainable-P: Both the function-defining (PkP_k) and functionally invariant (QkQ_k) components of previous classifier blocks are trainable, without gradient projection or post-step restoration of the reference contrast maps.

Results. CIFAR-100, 10 tasks, five matched seeds (mean ±\pm SD):

M=500M=500

MetricFixed-PTrainable-P
Acquisition CE-AUC ↓0.775 ± 0.0090.653 ± 0.009
Task-IL forgetting (pp) ↓5.26 ± 0.396.34 ± 0.37
Final Avg. Class-IL (%) ↑43.85 ± 0.7143.27 ± 0.79
Final Avg. Task-IL (%) ↑83.47 ± 0.4383.14 ± 0.33

M=200M=200

MetricFixed-PTrainable-P
Acquisition CE-AUC ↓0.775 ± 0.0080.675 ± 0.013
Task-IL forgetting (pp) ↓10.08 ± 0.4513.72 ± 0.61
Final Avg. Class-IL (%) ↑36.89 ± 0.6834.59 ± 0.31
Final Avg. Task-IL (%) ↑79.82 ± 0.4976.98 ± 0.32

Acquisition–retention trade-off under limited replay. At M=500M=500, Trainable-P facilitates acquisition, whereas Fixed-P reduces forgetting, although their final average accuracies remain comparable. Fixed-P preserves the reference contrast maps exactly regardless of memory size, whereas Trainable-P allows their function-defining components to change under exemplar-based functional-consistency supervision. We therefore extended the comparison to M=200M=200, the smaller budget evaluated in Appendix F.1, to examine whether the retention benefit of preserving the reference contrast maps becomes more pronounced with limited exemplar memory.

At M=200M=200, Fixed-P achieves higher final Class-IL (+2.31 pp, paired 95% CI [1.69, 2.92]) and Task-IL (+2.84 pp, [2.37, 3.31]) across five matched seeds, despite Trainable-P's acquisition advantage. The improvement is concentrated in previous tasks (old-task Task-IL: 78.33% vs. 75.19%). To further assess retention, we tracked test examples correctly classified by both methods immediately after learning each task. Fixed-P showed lower final within-task decision failure (14.53% vs. 18.21%), including for Task 1 (20.31% vs. 26.80%), where both methods started from identical acquisition checkpoints and predictions.

AuthorsOpenReview ↗

Response to Reviewer kbnp (2/5)

W1 — Acquisition cost of FEA and complementarity with FCA

The reviewer correctly observes that FEA alone worsens acquisition when FCA is disabled. Importantly, this result highlights the very stability–plasticity tension that our proposed allocation aims to address: imposing historical stability without retaining an alternative adaptation pathway can hinder new learning.

Our central claim in Section 5.3 is therefore not that FEA and FCA independently improve acquisition, but that their complementary roles allow stability and plasticity to coexist. Indeed, FCA provides a substantially larger acquisition benefit when FEA is enabled (CE-AUC reduction: 0.3510.351 vs. 0.0990.099 without FEA), consistent with the significant factorial interaction already reported. Furthermore, Fig. 2b shows that FCA enables common-score adaptation through functionally invariant classifier coordinates, providing an account of how acquisition can improve under the stability constraint imposed by FEA.

We have revised Section 5.3 to make this interpretation more explicit, explaining the acquisition cost of FEA alone, how FCA alleviates this cost by providing an alternative adaptation pathway, and how their complementary roles contribute to both acquisition and retention.


W1 — Selective representational stabilization and adaptation reallocation

We agree that the classifier-visible fraction alone cannot distinguish absolute redistribution of representational change from an overall reduction in its magnitude. Nevertheless, the substantial decrease in this fraction provides evidence that FEA does not uniformly suppress representational change across directions. To further clarify this distinction, we additionally measured the absolute total, classifier-visible, and classifier-null representational-change energies using the original experimental runs and aggregation procedure.

Absolute representational-change energy (M=500M=500, five seeds):

FEAFCATotalClassifier-visibleClassifier-null
OFFOFF0.56250.21950.3430
ONOFF0.32090.05140.2695
OFFON0.49140.23150.2599
ONON0.20380.03280.1710

The results show that FEA reduces overall representational change while disproportionately suppressing classifier-visible components. Specifically, enabling FEA reduces classifier-visible energy by 76.6% and 85.8% (FCA OFF/ON), compared with 21.4% and 34.2% for classifier-null energy. Thus, the results support selective stabilization of classifier-visible representational change, rather than an absolute redistribution of representational-change energy into null directions. This distinction is now reflected in the revised Section 5.3.

Crucially, the reallocation in Fig. 2b refers to how common-score adaptation is distributed between the encoder and classifier, rather than to a redistribution of representational-change energy into null directions.


W2 & Q1 & Q2 — Motivation and experimental scope

Our primary objective is to investigate functional equivalence as a principle for allocating stability and plasticity, rather than to establish state-of-the-art performance across all CL regimes. We therefore deliberately chose to focus on learning from scratch, where acquiring new representations can directly interfere with preserving previously learned functions. This setting allows us to examine the acquisition–retention tension without relying on representations obtained through large-scale pretraining.

Pretrained representations can substantially reduce the need for new feature learning. For example, Galashov et al. (2023) show that frozen pretrained representations can approach fine-tuning performance on standard CL benchmarks. Thus, from-scratch and pretrained CL pose related but distinct learning challenges, motivating our original experimental scope and baseline selection. We have clarified this rationale and the scope of our claims in the revised Discussion, including additional references to recent pretrained-model-based continual learning approaches. We address FEAST's applicability to pretrained models through additional experiments below.

AuthorsOpenReview ↗

Response to Reviewer kbnp (3/5)

Q1 — Generality under pretrained backbones

Nevertheless, we agree that pretrained-model-based continual learning is an important complementary regime, and therefore evaluate FEAST under pretrained initialization to test the broader applicability of functional-coordinate allocation.

Additional experiments with a pretrained ViT. To address this concern, we extend our evaluation to an ImageNet-21K-pretrained ViT-B/16 on CIFAR-100 (10 tasks). We include Sequential FT and DER as canonical baselines, together with LODE(DER++) and SS-IL, the strongest non-teacher- and teacher-based methods, respectively, in our original CIFAR-100 Class-IL benchmark. We additionally evaluate two recent PTM-based methods, SplitLoRA (Qiu et al., 2026) and RSIAT (Zhao et al., 2026), which address stability and plasticity through gradient-subspace partitioning and feature alignment, respectively.

Experimental protocol. For the conventional CL methods, we adapt our existing Mammoth implementations to pretrained ViT training by uniformly applying the SGD-based optimization recipe from SLCA (Zhang et al., 2023), without tuning the optimizer or learning rate. To accommodate the change in backbone while accounting for the limited computational budget during rebuttal, we conduct a small, matched hyperparameter search over each method's principal hyperparameters. For SplitLoRA and RSIAT, we follow their official implementations and recommended optimization settings without additional tuning.

Results. The table reports accuracies (%) as mean ± sample standard deviation across three random seeds.

GroupMethodFinalClass-ILAvg. incrementalClass-ILFinalTask-ILConventionalCLSeq FTDERLODE(DER++)SS-ILFEAST28.46±2.5584.31±0.5688.99±0.3490.24±0.1190.55±0.3043.71±1.2691.68±0.6293.76±0.4693.81±0.4493.86±0.6996.11±0.8097.28±0.1698.02±0.1198.45±0.2998.41±0.21PTM-basedCLSplitLoRARSIAT89.10±0.4091.62±0.0992.74±0.5894.38±0.2698.34±0.1198.45±0.19{ \def\Cell#1#2{\vphantom{\rule[-0.35em]{1pt}{1.45em}}\hspace{#1}\mathclap{#2}\hspace{#1}} \begin{array}{c} \begin{array}{ccccc} \Cell{4em}{\textbf{Group}} & \Cell{6em}{\textbf{Method}} & \Cell{4em}{\begin{array}{c}\textbf{Final}\\\\\textbf{Class-IL}\end{array}} & \Cell{4em}{\begin{array}{c}\textbf{Avg. incremental}\\\\\textbf{Class-IL}\end{array}} & \Cell{4em}{\begin{array}{c}\textbf{Final}\\\\\textbf{Task-IL}\end{array}} \end{array} \\\\[-0.5em] \kern-0.4em\rule{48.8em}{0.8pt}\kern-0.4em \\\\[-0.5em] \begin{array}{ccccc} \Cell{4em}{\begin{array}{c}\text{Conventional}\\\\\text{CL}\end{array}} & \begin{array}{c} \Cell{6em}{\text{Seq FT}}\\\\ \Cell{6em}{\text{DER}}\\\\ \Cell{6em}{\text{LODE(DER++)}}\\\\ \Cell{6em}{\text{SS-IL}}\\\\ \Cell{6em}{\textbf{FEAST}} \end{array} & \begin{array}{c} \Cell{4em}{28.46\pm2.55}\\\\ \Cell{4em}{84.31\pm0.56}\\\\ \Cell{4em}{88.99\pm0.34}\\\\ \Cell{4em}{90.24\pm0.11}\\\\ \Cell{4em}{\mathbf{90.55\pm0.30}} \end{array} & \begin{array}{c} \Cell{4em}{43.71\pm1.26}\\\\ \Cell{4em}{91.68\pm0.62}\\\\ \Cell{4em}{93.76\pm0.46}\\\\ \Cell{4em}{93.81\pm0.44}\\\\ \Cell{4em}{\mathbf{93.86\pm0.69}} \end{array} & \begin{array}{c} \Cell{4em}{96.11\pm0.80}\\\\ \Cell{4em}{97.28\pm0.16}\\\\ \Cell{4em}{98.02\pm0.11}\\\\ \Cell{4em}{\mathbf{98.45\pm0.29}}\\\\ \Cell{4em}{98.41\pm0.21} \end{array} \end{array} \\\\[-0.3em] \hline \begin{array}{ccccc} \Cell{4em}{\begin{array}{c}\text{PTM-based}\\\\\text{CL}\end{array}} & \begin{array}{c} \Cell{6em}{\text{SplitLoRA}}\\\\ \Cell{6em}{\text{RSIAT}} \end{array} & \begin{array}{c} \Cell{4em}{89.10\pm0.40}\\\\ \Cell{4em}{\mathbf{91.62\pm0.09}} \end{array} & \begin{array}{c} \Cell{4em}{92.74\pm0.58}\\\\ \Cell{4em}{\mathbf{94.38\pm0.26}} \end{array} & \begin{array}{c} \Cell{4em}{98.34\pm0.11}\\\\ \Cell{4em}{\mathbf{98.45\pm0.19}} \end{array} \end{array} \\\\[-0.3em] \hline \end{array} }

FEAST achieves the highest mean final Class-IL accuracy among the conventional CL methods and remains competitive with recent PTM-based approaches. It outperforms SplitLoRA by 1.45 percentage points and comes within 1.07 points of RSIAT, without PTM-specific architectural modifications. Its average incremental Class-IL and final Task-IL accuracies are also comparable to the strongest evaluated methods.

Scope and comparison limitations. This comparison is not fully matched in optimization or compute: conventional CL methods use the same fixed SLCA-based SGD learning rate without tuning, whereas PTM-based methods follow their native optimization procedures and training budgets. Nevertheless, FEAST remains competitive under pretrained initialization with minimal methodological changes, supporting its applicability beyond from-scratch learning. Further PTM-specific instantiations of functional equivalence remain an interesting direction for future work.

Reviewer_dsChOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes an algorithm for continual learning, where instead of masking the gradient updates in the output weights not used in the current training task, those gradient updates are projected to a rank-1 direction invariant to in-task classification. The authors applied the method to task- and class-incremental continual learning using CIFAR-100 and Tiny ImageNet and found that the algorithm outperforms previous methods, including the output-weight masking method (ER-ACE).

Critical strengths

Empirical results demonstrate a clear advantage of the proposed method over a range of algorithms, though it remains unclear why this is the case and how exactly the algorithm was implemented.

Critical weaknesses

The manuscript frames a simple algorithm in an unnecessarily convoluted manner. The functional equivalence discussed in the abstract and Section 3 is rather misleading, as the only invariance considered in the method is the invariance of the softmax function to a constant factor. The exact algorithm also remains unclear in the manuscript; for instance, it is not clear whether the ResNet backbone was trained or how Eq. 10 is applied in the experiment.

While the empirical results suggest the effectiveness of the proposed algorithm, it is unclear why it works. Section 4 provides little insight beyond a hand-wavy explanation. The algorithm employs a rank-1 modification instead of masking updates on task-irrelevant output weights, but it is difficult to understand why it significantly outperforms a masking-based method (ER-ACE), considering that a rank-1 modification shouldn’t provide a learning signal per se. Equation 10 is confusing because if the replay-based error from Eq. 8 is projected onto a rank-1 space, there is effectively no replay-based learning. The limited numerical validation and lack of clarity do not help in this regard.

Questions for authors

Is IKkI_{K_k} defined as a vector where its ii-th element is 1 if the ii-th output is included in task k and 0 if not?

A pseudo-code-style description of the algorithm may help clarify what is exactly implemented.

Eq. 11 was particularly confusing to me. Which element of βk\beta_k was trained, and at which stage was this applied? What is the loss function for fitting the class-balanced memory, and how do we know which βk\beta_k should be applied to test samples? Also, is training of bkb_k essential if βk\beta_k is trained afterwards?

9 comments
AuthorsOpenReview ↗

Response to Reviewer dsCh (1/3)

We thank the reviewer for the careful evaluation of our work and for recognizing the empirical strengths of FEAST. We appreciate the concerns regarding the conceptual framing, algorithmic clarity, and underlying mechanisms, and address them point by point below.


1. Functional equivalence and its role in FEAST

"The functional equivalence discussed in the abstract and Section 3 is rather misleading, as the only invariance considered in the method is the invariance of the softmax function to a constant factor."

We acknowledge that FEAST provides one concrete instantiation of this broader principle, based on the additive-shift invariance of within-task softmax probabilities, and that its applicability to other functions and invariances remains to be explored. Within this scope, our contribution is to demonstrate how functional equivalence can serve as a principled criterion for allocating stability and plasticity in continual learning. In particular, we derive a coherent learning design from a simple, exact mathematical property, preserving function-defining coordinates while retaining functionally invariant freedom for adaptation. We believe that this simplicity is a strength of the approach, providing a concrete demonstration of the principle's utility and a different perspective on the stability–plasticity dilemma.


2. Gradient flow and algorithmic implementation

We recognize that the algorithmic implementation could be made more explicit in the main text. We address the specific points raised below.

ResNet-18 backbone.

"[...] it is not clear whether the ResNet backbone was trained [...]"

The ResNet-18 backbone is trained end-to-end from scratch (Sec. 5.1), receiving gradients from both full-way cross-entropy (CE) and the memory-based FEA loss (Eq. 8).

FCA projection.

"[...] how Eq. 10 is applied in the experiment."

"Equation 10 is confusing because if the replay-based error from Eq. 8 is projected onto a rank-1 space, there is effectively no replay-based learning."

To clarify how Eq. 10 is applied, it is helpful to consider the encoder and previous classifier updates separately.

Encoder updates.

As specified in Secs. 4.3–4.4, Eq. 10 applies only to the gradients of previous classifier blocks, not to those of the encoder. Thus, the encoder continues to receive gradients from both full-way CE and FEA, encouraging it to remain compatible with the fixed reference contrast map.

Previous classifier updates.

For a previous task block k<tk < t, applying the FCA projection to the total objective in Eq. 8 gives

Qk∇WkLt=Qk∇WkLCEfull+λQk∇WkLFEA=Qk∇WkLCEfull.\begin{aligned} Q_k\nabla_{W_k}\mathcal{L}_t &= Q_k\nabla_{W_k}\mathcal{L}_{\mathrm{CE}}^{\mathrm{full}} +\lambda Q_k\nabla_{W_k}\mathcal{L}_{\mathrm{FEA}}\\\\ &= Q_k\nabla_{W_k}\mathcal{L}_{\mathrm{CE}}^{\mathrm{full}}. \end{aligned}

The FEA term vanishes because its classifier-side gradient lies in the PkP_k subspace and QkPk=0Q_kP_k=0. Thus, although the classifier-side FEA gradient is removed, previous classifier blocks still receive generally nonzero gradients from full-way CE along the QkQ_k direction. The same applies to their biases.

The complete update procedure, including gradient projection, optimizer steps, and restoration of the reference PkP_k components, is provided in Appendix E.4 (Algorithm 1).

AuthorsOpenReview ↗

Response to Reviewer dsCh (2/3)

3. Functionally invariant updates and comparison with ER-ACE

"The algorithm employs a rank-1 modification instead of masking updates on task-irrelevant output weights, but it is difficult to understand why it significantly outperforms a masking-based method (ER-ACE), considering that a rank-1 modification shouldn’t provide a learning signal per se."

We agree that ER-ACE is a relevant comparison, as both methods address interference between previously learned and newly introduced classes by selectively restricting updates to previous classifiers. However, FEAST is not simply a variant of ER-ACE that replaces gradient masking with a rank-one projection. Table 1 summarizes the key differences.

Table 1. Old-classifier update rules in ER-ACE and FEAST (k<tk < t).

MethodTraining objectiveOld classifierWk=PkWk+QkWkPkWkQkWkReference contrast mapBlock-common componentER-ACEMasked Lcurrent,CEFull-way Lreplay,CENo update (masked)UpdatedFEASTFull-way Lcurrent,CEfull+λLFEAExactly preservedQk-only update (FCA)Then reconciled{ \def\Cell#1#2{\vphantom{\rule[-0.35em]{1pt}{1.45em}}\hspace{#1}\mathclap{#2}\hspace{#1}} \begin{array}{c|c|c} \textbf{Method} & \Cell{6.1em}{\textbf{Training objective}} & \kern-0.5em \begin{array}{c} \Cell{10.7em}{\textbf{Old classifier}\quad W_k=P_kW_k+Q_kW_k} \\\\[-4pt] \hline \begin{array}{c|c} \Cell{5.1em}{{\textstyle P_kW_k}} & \Cell{5.1em}{{\textstyle Q_kW_k}} \\\\[-0.3em] \vphantom{\rule[-0.35em]{1pt}{1em}}{\scriptstyle\text{Reference contrast map}} & {\scriptstyle\text{Block-common component}} \end{array} \end{array} \\\\[-4pt] \hline \textbf{ER-ACE} & \kern-0.5em \begin{array}{c} \Cell{6.6em}{\text{Masked }\mathcal{L}_{\mathrm{current\\,CE}}} \\\\[0.3em] \hline \Cell{6.6em}{\text{Full-way }\mathcal{L}_{\mathrm{replay\\,CE}}} \end{array} \kern-0.5em & \kern-0.5em \begin{array}{c} \Cell{10.7em}{\text{No update (masked)}} \\\\[0.3em] \hline \Cell{10.7em}{\text{Updated}} \end{array} \\\\[-4pt] \hline \textbf{FEAST} & \Cell{6.1em}{\text{Full-way }\mathcal{L}_{\mathrm{current\\,CE}}^{\mathrm{full}}+\lambda\mathcal{L}_{\mathrm{FEA}}} & \kern-0.5em \begin{array}{c|c} \Cell{5.1em}{\textbf{Exactly preserved}} & \Cell{5.1em}{Q_k\text{-only update (FCA)}} \\\\[-0.2em] & \vphantom{\rule[-0.35em]{1pt}{1em}}{\scriptstyle\text{Then reconciled}} \end{array} \end{array} }

Note. The table describes updates to previously learned classifier blocks. In FEAST, FEA also updates the encoder, while historical classifier contrasts remain fixed. The same projection rules apply to classifier biases.

ER-ACE masks old-class logits in its current-example CE, preventing new-task examples from directly updating the previous classifier. Its replay CE, however, updates both the encoder and classifier without constraining the old classifier's contrast component. Thus, ER-ACE protects previous classifier rows from current-example updates, but does not explicitly preserve their reference contrast maps throughout subsequent learning.

FEAST instead distinguishes updates according to their functional consequences. It preserves the reference contrast map PkWkP_kW_k exactly, while allowing updates along the block-common component QkWkQ_kW_k under full-way CE. Meanwhile, FEA encourages the evolving encoder to preserve previously learned conditional functions through this fixed reference map.

"While the empirical results suggest the effectiveness of the proposed algorithm, it is unclear why it works. Section 4 provides little insight beyond a hand-wavy explanation."

The importance of the QkQ_k updates can be understood intuitively. When a new-task example receives high scores from previously learned classes, full-way CE encourages those scores to decrease. Rather than modifying the relative scores among old classes, FCA allows the previous classifier to lower the scores of the entire block together.

Although QkQ_k is rank one in class space, the permitted classifier updates take the form ΔWk=1vk⊤\Delta W_k=\mathbf{1}v_k^\top and Δbk=αk1\Delta b_k=\alpha_k\mathbf{1}, where vk∈Rdv_k\in\mathbb{R}^d. Thus, FCA retains d+1d+1 trainable degrees of freedom, enabling input-dependent common-score adaptation rather than merely a single scalar adjustment. This provides a meaningful adaptation pathway for old–new competition without directly modifying previously learned class contrasts. Note that a common-score change is invariant to within-task classification, but not to cross-task competition (Sec. 3.4).

"The limited numerical validation and lack of clarity do not help in this regard."

Sec. 5.3 (Fig. 2; Table G.1) directly examines the role of functionally invariant classifier updates through a matched FEA × FCA intervention. The results provide empirical evidence for their contribution to both acquisition and retention, complementing the algorithmic explanation above.

AuthorsOpenReview ↗

Response to Reviewer dsCh (3/3)

4. Cross-block reconciliation (Q3)

"Eq. 11 was particularly confusing to me."

We acknowledge that the concise description of reconciliation in Sec. 4.5 may have caused some confusion. The detailed fitting objective and implementation are provided in Appendix C. We moved the essential details into Sec. 4.5 to make the procedure clearer.

Parameterization and fitting.

"Which element of βk\beta_k was trained, and at which stage was this applied? What is the loss function for fitting the class-balanced memory, [...]"

Each βk\beta_k is a single scalar offset associated with task block kk, rather than an individual class or sample. After training each task, we jointly fit the offsets for all learned blocks using memory exemplars, while keeping the encoder and centered classifier fixed. Specifically, we minimize cross-entropy over task-block predictions:

Lrec(β)=−1∣Mt∣∑(xi,yi)∈Mtlog⁡exp⁡(eki(xi)+βki)∑j=1texp⁡(ej(xi)+βj),\mathcal{L}_{\mathrm{rec}}(\boldsymbol{\beta}) = -\frac{1}{|\mathcal{M}_t|} \sum_{(x_i,y_i)\in\mathcal{M}_t} \log \frac{\exp(e_{k_i}(x_i)+\beta_{k_i})} {\sum_{j=1}^{t}\exp(e_j(x_i)+\beta_j)},

where kik_i is the task block containing class yiy_i, and ek(x)=log⁡∑c∈Ckexp⁡(z~k,c(x))e_k(x)=\log\sum_{c\in\mathcal{C}_k}\exp(\widetilde{z}_{k,c}(x)) is the centered log-evidence. This is equivalent to minimizing class-level CE with respect to β\boldsymbol{\beta} (Appendix C.2). The offsets are fitted using L-BFGS (Appendix C.4), and the reconciled classifier is used for evaluation and as the initialization for the next task.

Inference.

"[...] how do we know which βk\beta_k should be applied to test samples?"

No test-time task identity is required. Every input is evaluated against all learned blocks, each with its own fixed βk\beta_k, and the final prediction is made over all observed classes. As specified in Appendix C.3 (Eq. 34), reconciliation is incorporated directly into the shared classifier:

Wkeval=PkWk,bkeval=Pkbk+βk1.W_k^{\mathrm{eval}}=P_kW_k,\qquad b_k^{\mathrm{eval}}=P_kb_k+\beta_k\mathbf{1}.

Role of the original classifier bias.

"Also, is training of bkb_k essential if βk\beta_k is trained afterwards?"

Fitting βk\beta_k does not make training bkb_k redundant. The function-defining component PkbkP_kb_k contributes to within-task discrimination and is retained after reconciliation. In contrast, the block-common components QkbkQ_kb_k and QkWkQ_kW_k provide freedom for adaptation during full-way training. Although subsequently replaced by βk\beta_k, these components can influence the learning trajectory.

Importantly, reconciliation is not an arbitrary post-hoc calibration step. It addresses the cross-block score ambiguity left unresolved by preserving within-task conditional functions, without altering the class contrasts that FEAST is designed to protect.


5. Notation and pseudocode

Notation.

"Is IKkI_{K_k} defined as a vector where its ii-th element is 1 if the ii-th output is included in task k and 0 if not?"

IKkI_{K_k} denotes the Kk×KkK_k \times K_k identity matrix, rather than a binary indicator vector specifying which outputs belong to task kk. The class set Ck\mathcal{C}_k selects the logits belonging to task kk, after which PkP_k and QkQ_k decompose the selected block into contrast and common-score components.

Pseudocode.

"A pseudo-code-style description of the algorithm may help clarify what is exactly implemented."

The complete training procedure is already provided in Appendix E.4 (Algorithm 1), including full-way CE, memory-based FEA, FCA gradient projection, post-step restoration of historical classifier contrasts, and boundary reconciliation. We have added an explicit reference to Algorithm 1 in Section 4 of the revised manuscript to make the complete implementation procedure easier to locate.


We hope that our responses and the corresponding manuscript revisions clarify the implementation and underlying rationale of FEAST and adequately address the reviewer's concerns. We thank the reviewer again for their feedback.

Comments on the paper

1 comment
Area_Chair_SH1EOpenReview ↗

Discussion deadline approaching

Dear Reviewers and Authors,

Thank you for providing detailed reviews and the rebuttal.

To Reviewer kbnp: As the author–reviewer discussion deadline approaches, could you please review the authors’ additional response and indicate whether it addresses your concerns about the rehearsal-based design?

Kind regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.