acceptodds
Under review as a conference paper at ICLR 2027

When Preservation Becomes Stale: Continual Functional Refactoring for Fixed-Capacity CLIP Continual Learning

Abstract

Continual adaptation of pretrained vision-language models under a fixed parameter budget requires acquiring new knowledge without historical replay or task-specific capacity growth. Many existing methods protect historically important structures and constrain subsequent updates around them, effectively creating persistent capacity ownership. Through longitudinal subspace interventions, we identify Historical Necessity Drift: the functional necessity of historical subspaces can weaken, strengthen, or migrate as new tasks are learned. Directions once critical can become redundant, secondary directions can become essential, and others remain necessary throughout. This drift creates stale capacity ownership, where structure remains protected after its original functional role has diminished or migrated. More strikingly, retaining only a compact, functionally selected subspace can improve historical-task accuracy over the full adapter, showing that historical performance degradation can coexist with recoverable useful function and exposing functional interference from stale organization. Motivated by this finding, we propose Continual Functional Refactoring (CFR), which summarizes accumulated behavior with a fixed-size functional state, consolidates shared functionality into a compact core while retaining recent residuals in its complement, and reopens the full adapter for subsequent learning. Across CIFAR-100 and TinyImageNet sequences spanning 5 to 50 tasks, CFR ranks first in 11 of the 12 reported Average/Last accuracy evaluations. Its advantage is particularly evident over long horizons: on CIFAR-100 with 50 tasks, CFR achieves 84.96% Average and 76.74% Last accuracy, exceeding the strongest competing Last result by 1.08 percentage points. Code will be made publicly available upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning; Knowledge Reorganization; Parameter-Efficient Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xAKzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies replay-free, fixed-capacity continual adaptation of CLIP. Through removal and keep-only interventions on adapter subspaces, it investigates how dependence on historically important directions changes during subsequent learning. It proposes Continual Functional Refactoring, which accumulates local input–output statistics, reconstructs a mapping within a compact core, retains the current complementary component, and reopens the full adapter for subsequent training. Experiments cover CIFAR-100 and TinyImageNet with sequences ranging from 5 to 50 tasks.

Critical strengths
  1. CFR operates without historical sample replay or task-specific parameter growth. The matched no-refactoring comparison is useful: on TinyImageNet with 20 tasks, CFR improves final accuracy from 73.84% to 76.67%. Component ablations and capacity sensitivity experiments provide additional evidence that the proposed procedure has practical effects.
  2. Distinguishing the effect of removing a historical subspace from its effectiveness under keep-only evaluation is useful. The reported trajectories illustrate that a historical realization can remain useful without remaining indispensable.
Critical weaknesses
  1. Prior work already studies changing parameter importance, drifting functional directions, evolving shared subspaces, and continual adapter merging, as acknowledged in Section 2.2. Furthermore, reconstructing linear mappings from activation statistics is closely related to the regression-based merging formulation of RegMean. CFR introduces a specific combination of accumulated statistics, core–complement reconstruction, and reopening, but the current comparisons do not sufficiently establish what substantive new capability or understanding this combination provides beyond these existing principles. The results support an improvement in the evaluated CLIP setting; the broader methodological significance remains less convincing.
  2. CFR accumulates statistics in representation spaces that themselves change during learning. Algorithm 1 introduces alignment of historical statistics, but the estimator and its reliability across successive tasks are insufficiently characterized. The paper does not directly quantify reconstruction fidelity on held-out historical samples or how alignment errors accumulate. In addition, the reported forgetting metric compares checkpoints with expanding candidate-class sets, so it does not isolate deterioration caused by model updates. These issues do not invalidate the reported accuracy gains, but they limit the evidence for the proposed functional-preservation explanation.
Questions for authors
  1. What is the substantive technical distinction between CFR’s accumulated regression reconstruction and a sequential RegMean-style procedure? A matched baseline using the same statistics and alignment, but reconstructing the full mapping without the core–complement split, would help establish the additional value of the proposed organization.
  2. Please specify how the alignment matrix in Algorithm 1 is estimated, which samples are used, the summary-set size, and the regularization settings. Can you quantify reconstruction fidelity on held-out historical samples and its evolution across task boundaries?
  3. For CFR and the matched no-refactoring baseline, can you report historical-class performance across checkpoints with a fixed candidate-class set? Per-class trajectories would also clarify whether average improvements conceal substantial deterioration in particular historical classes.
  4. Please clarify the differences between the main and supplementary results under apparently matching configurations. For example, CIFAR-100/50-task CFR is reported as 84.96/76.74 in Table 2, whereas Accumulated CFR is 84.99/75.86 in Table S12. Are these different configurations or sets of runs?
6 comments

W1: The novelty beyond changing parameter importance, evolving subspaces, adapter merging, and RegMean-style regression remains unclear.

Response to W1: Methodological Novelty and Distinction from RegMean

We appreciate the reviewer’s request to identify the substantive contribution beyond established principles. CFR builds on insights about changing parameter importance and regression-based reconstruction, but introduces a distinct continual-learning mechanism: preserving historical functionality without permanently preserving its original parameter directions, while keeping the entire adapter capacity reusable. We clarify this mechanism and isolate its benefits through controlled comparisons.

1. New preservation principle and capability. Section 3 shows that historical functional necessity is non-stationary: directions once essential to earlier tasks can become replaceable or interfering. CFR therefore preserves functional behavior, rather than fixed historical coordinates. Its learn–refactor–reopen cycle (i) transports and accumulates a fixed-size historical input–output state across changing representations, (ii) reconstructs accumulated functionality within a compact geometric core while retaining recently learned behavior in the complement, and (iii) reopens all dimensions for the next task. This enables repeated historical-function recovery and further adaptation within one fixed-capacity shared adapter, without replay, task-specific expansion, or permanent subspace protection.

2. Technical distinction from RegMean and a matched comparison. RegMean uses activation-statistic-weighted regression to merge separately trained mappings. CFR employs a related regression objective, but its role is different: a sequential functional-state update within a single evolving adapter. Let W⋆W^\star be the mapping reconstructed from aligned accumulated statistics, PP the core projector, and W−W^- the adapter mapping immediately before refactoring. A full-map regression recovery gives Wfull=W⋆W_{\mathrm{full}}=W^\star, whereas CFR uses WCFR=W⋆P+W−(I−P)W_{\mathrm{CFR}}=W^\star P+W^-(I-P). Thus, CFR restores historical behavior selectively without overwriting the recently adapted complement, then reopens both components for subsequent learning.

We compared these mechanisms on TinyImageNet-20 (3 seeds), using identical accumulated statistics, alignment, regression objective, and training settings:

Reconstruction MechanismAvg Acc ↑Last Acc ↑Last For ↓
CFR Core–Complement84.1776.677.87
Full-map Recovery (RegMean-style)83.5975.848.54
Norm-matched Full Update83.7475.9110.35

CFR improves Last accuracy by 0.83 pp and reduces Last Forgetting by 0.67 pp relative to matched full-map recovery. The difference isolates the benefit of selective core–complement integration beyond the shared regression reconstruction principle (Table S21, p. 27).

3. Why the gain is not generic periodic rewriting. With the refactoring procedure held fixed, replacing current-task reconstruction statistics with accumulated historical statistics improves Last accuracy by 2.10–2.25 pp and reduces Last Forgetting by 5.13–5.48 pp, regardless of the core-selection source (Table S12, p. 23). Moreover, CFR's geometry-selected core is behaviorally meaningful: removal necessity is +7.08 pp versus +2.99 pp for loss-selected directions and +0.24 pp for a random core; retaining the CFR core alone nearly matches full-adapter accuracy (−0.06 pp, Table S20, p. 27). These results identify accumulated functional reconstruction as the main contributor and geometry-guided selection as a complementary mechanism—not merely a periodic parameter rewrite.

4. Direct functional recovery and continued reuse. On CIFAR-100-50 (3 seeds), refactoring immediately recovers +3.284±0.323 pp historical accuracy at task boundaries, with positive recovery at 145/147 boundaries and no additional training (Table S11, p. 22). Full reopening further improves TinyImageNet-20 Last accuracy from 76.12% to 76.67% (Table 3, p. 9). These interventions show that historical functionality can be restored and then reused within the same capacity, rather than only yielding an aggregate accuracy increase.

Together, these controlled comparisons and task-boundary interventions establish CFR's methodological contribution: transporting and reconstructing historical functionality across changing representations, selectively integrating it with recent learning, and reopening the entire fixed-capacity adapter for continued reuse without replay.

W2: The reliability of repeated historical-statistics transport, reconstruction fidelity, and forgetting evaluation under expanding candidate-class sets remain insufficiently established.

Response to W2: Historical-State Transport Reliability and Fixed-Class Forgetting

We thank the reviewer for highlighting these issues. We provide direct evidence from multi-step alignment, held-out historical-output reconstruction, and fixed-class evaluation.

1. Alignment reliability across task transitions. CFR estimates Rt,ℓR_{t,\ell} through orthogonal Procrustes alignment of paired bottleneck activations from the same current-task samples before and after training, subject to R⊤R=IR^\top R=I. Historical statistics are transported into the current representation coordinates without replay.

On CIFAR-100-10 (3 seeds), CFR reduces single-step alignment error from 0.5180±0.0115 to 0.3928±0.0076 and multi-step error from 0.8160±0.0284 to 0.6240±0.0297. The latter approaches the historical-data reference (0.5983±0.0285), reducing multi-step error by 23.5% relative to no alignment (Table S17, p. 25).

2. Historical functional reconstruction fidelity. We directly measured normalized mean squared error (NMSE) between reconstructed adapter outputs and historical reference outputs on held-out historical samples (CIFAR-100-10, 3 seeds).

Reconstruction MethodNMSE ↓
Current-task statistics only1.3375
Historical accumulation without transport1.1199
CFR transported accumulation0.8307
Final CFR mapping0.8075

Transported accumulation reduces historical-output NMSE by 25.8% versus unaligned accumulation and 37.9% versus current-task statistics. The final CFR mapping further achieves 0.8075, confirming that selective refactoring retains the reconstruction benefits (Table S18, p. 26).

We also measured reconstruction fidelity across successive checkpoints. CFR achieves lower NMSE than both alternatives at every evaluated checkpoint (Tasks 2/4/6/8/10). At Task 10, its NMSE remains 0.8118±0.0619, versus 1.0191±0.0740 without transport and 1.2508±0.1109 using current-task statistics, demonstrating sustained reconstruction advantages after repeated transitions (Table S19, p. 26).

3. Isolating model-induced forgetting. To eliminate the influence of expanding candidate-class sets, we evaluated CIFAR-100-50 (3 seeds) using the same fixed 100-class candidate set at every checkpoint.

MetricNo RefactorCFR
Standard Last Forget ↓12.329.57
Fixed-100 Last Forget ↓7.754.73
Fixed-100 BWT ↑−5.37−2.00

Under fixed-class evaluation, CFR reduces Last Forgetting by 3.02 pp and improves backward transfer (BWT) by 3.37 pp. Moreover, task-boundary interventions immediately restore 3.284±0.323 pp historical accuracy, with positive recovery at 145/147 boundaries, without additional training or changes to the candidate-class set (Table S11, p. 22).

Together, these results demonstrate that transported historical statistics retain predictive information across task transitions, substantially improve historical functional reconstruction, and support genuine historical retention gains independently of candidate-class expansion. This provides direct mechanistic evidence for CFR's functional-state preservation beyond aggregate accuracy improvements.

Q1: What distinguishes CFR from sequential RegMean-style reconstruction? Can a matched full-map recovery baseline establish the value of core–complement organization?

Response to Q1: Distinction from RegMean-Style Reconstruction

We thank the reviewer for this suggestion. CFR shares the regression-based reconstruction principle with RegMean, but differs in how historical functionality is integrated into the evolving adapter.

Let W⋆W^\star denote the mapping reconstructed from aligned accumulated statistics, W−W^- the current adapter mapping before refactoring, and PP the core projector. The two mechanisms are:

Wfull+=W⋆⏟RegMean-style full recovery\underbrace{W_{\mathrm{full}}^{+}=W^{\star}}_{\text{RegMean-style full recovery}}

WCFR+=W⋆P+W−(I−P)⏟CFR core-complement\underbrace{W_{\mathrm{CFR}}^{+}=W^{\star}P+W^{-}(I-P)}_{\text{CFR core-complement}}

Full-map recovery overwrites the adapter; CFR restores historical functionality in the core, retains the recent complement, and reopens all dimensions.

On TinyImageNet-20 (3 seeds), both use identical accumulated statistics, alignment, regression objective, and training settings. CFR achieves 76.67% Last accuracy / 7.87% Last Forgetting, versus 75.84% / 8.54% for RegMean-style full-map recovery (+0.83/−0.67 pp, respectively; Table S21, p. 27).

This matched comparison establishes the added value of core–complement organization beyond regression reconstruction: integrating historical functionality with recent learning rather than replacing the entire adapter. See W1 for the broader distinction.

Q2: Clarify alignment estimation, sample sizes, regularization, and historical reconstruction fidelity across task boundaries.

Response to Q2: Alignment Implementation and Reconstruction Fidelity

We thank the reviewer for requesting these details. We clarify the implementation and provide checkpoint-wise reconstruction evidence.

1. Alignment estimation and regularization. For each visual adapter layer ℓ\ell, CFR extracts paired post-ReLU bottleneck activations HprevH_{\mathrm{prev}} and HcurH_{\mathrm{cur}} from the same current-task images, using checkpoints before and after task training. We solve the orthogonal Procrustes problem:

Rt,ℓ=arg⁡min⁡R⊤R=I∥HprevR−Hcur∥F2.R_{t,\ell}=\underset{R^\top R=I}{\arg\min}\|H_{\mathrm{prev}}R-H_{\mathrm{cur}}\|_F^2.

Given Hprev⊤Hcur=UΣV⊤H_{\mathrm{prev}}^\top H_{\mathrm{cur}}=U\Sigma V^\top, the solution is Rt,ℓ=UV⊤R_{t,\ell}=UV^\top, mapping previous representations into current coordinates.

We use class-stratified sampling from the current task: up to 300 images/class for functional-statistic estimation and 200 images/class for alignment, with alignment samples included in the summary set. On CIFAR-100-10, these correspond to 3,000 summary and 2,000 alignment images/task. All image tokens contribute to the statistics.

Orthogonal alignment requires no ridge regularization. Functional reconstruction uses:

\lambda_{t,\ell}^{\mathrm{bias}}=10^{-3}\lambda_{t,\ell},$$ with $r=64$ and default core rank $k=16$. Both alignment and statistic accumulation use only current-task samples, without historical replay (Appendix A.2, lines 769–780). **2. Held-out historical reconstruction.** On CIFAR-100-10 (3 seeds), we use 20 historical images/class to construct offline diagnostic references and a separate set of 30 historical test images/class for evaluation. These samples are never used for CFR training or functional-state updates. As reported in W2, transported accumulation achieves historical-output NMSE of **0.8307**, versus **1.1199** without alignment transport and **1.3375** with current-task statistics only—reductions of **25.8%** and **37.9%**, respectively. The final CFR mapping further achieves **0.8075** (Table S18, p. 26). **3. Fidelity across task boundaries.** We additionally evaluated reconstruction NMSE across checkpoints, averaging over eligible historical source tasks and 12 adapter layers (3 seeds): | Checkpoint | CFR Aligned ↓ | Unaligned Accum. ↓ | Current-task Only ↓ | |:---|---:|---:|---:| | Task 2 | **0.5006±0.0299** | 0.9264±0.0141 | 1.0276±0.0328 | | Task 4 | **0.7499±0.2118** | 1.1987±0.2474 | 1.3744±0.2918 | | Task 6 | **0.7627±0.1032** | 1.0435±0.1443 | 1.2053±0.1642 | | Task 8 | **0.9156±0.1797** | 1.1680±0.1933 | 1.4035±0.2070 | | Task 10 | **0.8118±0.0619** | 1.0191±0.0740 | 1.2508±0.1109 | CFR achieves the lowest NMSE at every evaluated checkpoint. At Task 10, it retains **20.3%/35.1% lower error** than unaligned accumulation/current-task-only reconstruction, demonstrating that its advantage persists across successive task transitions (Table S19, p. 26). The overall NMSE of 0.8307 aggregates historical-source/checkpoint/layer evaluations across Tasks 2–10, whereas 0.8118 refers specifically to Task 10. **Together, these results demonstrate that CFR's current-task-based alignment reliably transports accumulated functional statistics across changing representations, enabling consistently more accurate historical reconstruction throughout continual learning.**
Reviewer_mn2kOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores class-incremental CLIP adaptation within one fixed adapter budget, without replay and without new modules. Remove and keep tests ask whether an old functional subspace is still necessary. Continual Functional Refactoring then stores a fixed-size summary, rebuilds a compact core plus a recent residual, and reopens the visual adapter for the next task.

Critical strengths

After each task the visual adapter is summarized, rewritten into a core and a residual, and fully reopened. Capacity stays fixed, and there is no replay buffer.

Necessity and keep-only gain are defined by removal and retention, and they are kept separate from subspace overlap. The drift claim is therefore about behavior, not only about geometry. Algorithm 1 and the appendix state that the rewrite hits the visual branch.

On CIFAR-100 and TinyImageNet, across several task lengths, CFR is usually first on average and last accuracy, and the margin is clearer on the longest streams. Turning refactoring off hurts more than replacing the functional core with a random core.

Critical weaknesses

The drift evidence is thin. The trajectories follow three source tasks, and the section that reports them does not name which benchmark produced them. Those measurements use held-out samples from past tasks. Training does not replay those samples, but the motivating analysis still depends on stored past data.

The operator that is trained is not the operator that was diagnosed. Section 3 ranks directions by how removal changes the task loss. CFR fits a core by least squares on accumulated activation statistics. A random core already takes most of the gain over no refactoring, and changing the core width barely moves average accuracy. The benefit looks more like periodic low-rank rewriting than like function-aware selection.

Refactoring is visual only. Text adapters keep training without the same rewrite, so the method is not a uniform treatment of the CLIP adapter budget.

The ranking is not uniform, and the edges are small. On the medium CIFAR-100 horizon, a strong baseline is ahead on average accuracy. The main tables report three-seed means without dispersion. Several related methods named in the discussion do not appear in the comparison.

Table 3 and Table S13 disagree on the same TinyImageNet-20 setting with adapter width 64 and core rank 16. One lists the full method at 84.17 average and 76.67 last. The other lists 84.07 and 76.33. The no-refactor rows also differ. Those are the ablation numbers, so the inconsistency is material.

Although the paper targets CIL, many relevant CIL studies are missing from both the discussion and the experimental comparisons, e.g., Refs [1-4].

The evaluation appears to be limited to 50 tasks, whereas real-world scenarios may involve far more tasks. How does the proposed method perform when the number of tasks exceeds 100 or even reaches 300? [3]

Ref:

[1] Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning, ICCV 2025;
[2] MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning, AAAI 2025;
[3] Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts, ICML 2026;
[4] GR-LoRA: Gradient-Recycling Low-Rank Adaptation for Class-Incremental Learning, ICML 2026.

Questions for authors

Please refer to the weaknesses.

8 comments

W1: The drift analysis covers only three source tasks, lacks benchmark details, and relies on historical samples despite the replay-free setting.

Response to W1: Generality of Historical Necessity Drift and Replay-Free Learning

We thank the reviewer for these concerns and clarify the diagnostic protocol and its generality.

1. Diagnostic setup. Section 3 uses CIFAR-100 with 10 class-incremental tasks (10 classes/task), a frozen CLIP ViT-B/16 backbone, and replay-free shared-adapter training (4 epochs/task). The first three source tasks illustrate necessity reversal, persistent necessity, and late-stage weakening, respectively. These are illustrative examples rather than evidence of prevalence.

2. Generality beyond three tasks. We extended the analysis to all eligible historical source-task trajectories (3 seeds) under standard seen-class evaluation. Historical necessity is measured as Ns(t)=Asfull(t)−Asremove(t)N_s^{(t)}=A_s^{\mathrm{full}}(t)-A_s^{\mathrm{remove}}(t). Among initially necessary trajectories (Ns(s)>0N_s^{(s)}>0), weakening denotes a decrease of at least 1 pp by the final checkpoint.

Under standard shared-adapter learning, 54.2% of eligible trajectories exhibit weakening, versus 20.6% for CFR, a 62.0% relative reduction. These results demonstrate that historical necessity drift extends beyond the three illustrative tasks and that CFR more effectively maintains functionally relevant historical representations through accumulated reconstruction and reorganization. See Appendix B.5 (lines 1087–1099) in the revised manuscript.

3. Offline diagnosis versus replay-free training. Historical samples are used exclusively for offline mechanistic analysis: 500 training images per source task (50/class) for loss-based direction selection, and the independent test set for removal/keep-only evaluation. Neither these samples nor the selected diagnostic directions are used for CFR training or functional-state updates. CFR operates solely on current-task samples and fixed-size accumulated statistics, without historical replay. This distinction is clarified in Appendix A.3 (lines 784–801).

Thus, the broader diagnostic evidence supports the generality of Historical Necessity Drift, while the reliance on historical samples is confined to offline analysis and does not compromise CFR's replay-free learning mechanism.

W2: The mismatch between behavioral diagnostics and CFR's geometric cores, together with random-core gains and core-width insensitivity, raises doubts about whether CFR goes beyond periodic low-rank rewriting.

Response to W2: Functional Reconstruction Beyond Periodic Rewriting

We thank the reviewer for this concern. Section 3 uses historical samples and loss-based interventions for offline diagnosis, whereas CFR must operate without historical replay. CFR translates this insight into a practical mechanism: accumulated activation geometry selects the compact core, accumulated input–output statistics reconstruct historical functionality, the recent complement is preserved, and all dimensions are reopened. We provide four complementary analyses.

1. Beyond random-core refactoring. On TinyImageNet-20 (3 seeds), random-core refactoring improves Last accuracy from 73.84% (No Refactor) to 75.38%, but CFR further achieves 76.67%, reducing Last Forgetting from 12.08% to 7.87%. Thus, CFR improves over random-core refactoring by 1.29 pp in Last accuracy and 4.21 pp in Last Forgetting. Generic periodic rewriting alone cannot explain these gains (Table 3, p. 9).

2. Contribution of historical reconstruction. Holding the refactoring procedure fixed, replacing current-task reconstruction statistics with accumulated historical statistics improves Last accuracy by 2.10–2.25 pp and reduces Last Forgetting by 5.13–5.48 pp, regardless of core-selection geometry. This identifies accumulated functional-state reconstruction as a major contributor beyond periodic rewriting. See Table S12 (p. 23).

3. Behavioral relevance of geometric cores. We compared equal-rank (k=16k=16) random, historical loss-selected, and CFR geometry-selected cores using identical removal and keep-only interventions. Removal necessity is N=Afull−AremoveN=A_{\mathrm{full}}-A_{\mathrm{remove}}, and keep-only gain is K=Akeep−AfullK=A_{\mathrm{keep}}-A_{\mathrm{full}}.

Core SelectionNecessity N ↑Keep-only Gain K ↑
Random Core+0.24−3.25
Historical Loss-selected Core+2.99−1.23
CFR Geometry-selected Core+7.08−0.06

CFR's core shows substantially greater functional necessity and independent utility; keeping only this core nearly matches the full adapter (K=−0.06K=-0.06 pp). Its overlap with loss-selected directions increases from 0.402 to 0.472 after alignment, exceeding the random reference (0.250). Thus, although geometric selection does not directly reproduce loss-based rankings, it identifies behaviorally meaningful subspaces without historical replay (Table S20, p. 27).

4. Core-width robustness and selective reconstruction. Similar Average accuracies at k=8,16,32k=8,16,32 (84.19%, 84.17%, 84.20%) demonstrate robustness to core width, not that selective refactoring is unnecessary. At k=64k=64, the core spans the full bottleneck and reduces to full-map recovery (P=IP=I). Under matched reconstruction settings, CFR outperforms full-map recovery in Last accuracy (76.67% vs. 75.84%) and Last Forgetting (7.87% vs. 8.54%). See Table S21 (p. 27).

Together, these results demonstrate that CFR's gains extend beyond generic periodic low-rank rewriting. Its central contribution is reconstructing accumulated historical functionality within behaviorally meaningful cores and selectively integrating it with recent complementary representations, without historical replay or permanent subspace protection.

W3: CFR refactors only visual adapters, leaving text adapters without equivalent functional reorganization.

Response to W3: Motivation for Visual-Only Refactoring

We thank the reviewer for highlighting this asymmetry. This is a deliberate design choice motivated by the different adaptation dynamics of visual and textual representations.

Prior studies show that visual representations undergo greater variation during class-incremental learning, whereas textual representations remain relatively stable due to the limited variability of class-name-based prompts. Specifically, DMNSP (ICCV 2025) identifies this asymmetry and applies multi-layer null-space gradient projection exclusively to the visual branch, while updating text adapters through standard optimization. CCA-CL (CVPR 2026) further highlights asymmetric visual–textual drift and addresses it through cross-modal subspace alignment.

Motivated by these findings, CFR focuses functional-state reconstruction on visual adapters, where greater representational changes make historical functional reorganization particularly important. Meanwhile, text adapters remain trainable through standard gradient-based optimization to accommodate newly introduced class semantics.

Thus, CFR retains joint visual–textual adaptation while selectively applying functional refactoring to the modality exhibiting stronger representational changes.

W4: CFR does not consistently rank first, some gains are marginal and lack variability estimates, and several related CIL methods are absent from the comparisons.

Response to W4: Performance Robustness and Comparison Coverage

We thank the reviewer for these concerns. We acknowledge that CFR does not dominate every setting and have added three-seed means ± standard deviations for CFR and five representative baselines across all task partitions (Table S22, p. 27).

1. Performance consistency and statistical variability. The principal CIFAR-100 comparisons are:

TasksMethodAvg Acc ↑Last Acc ↑
10CCA-CL w/ FTT88.05±0.04181.42±0.158
CFR88.30±0.12681.72±0.355
20CCA-CL w/ FTT87.22±0.06979.11±0.072
CFR86.46±0.12079.21±0.320
50CCA-CL w/ FTT82.36±0.02673.33±0.271
CFR84.96±0.07476.74±0.356

On CIFAR-100-20, CFR trails CCA-CL w/ FTT by 0.76 pp in Average accuracy, while its 0.10 pp Last-accuracy advantage is modest relative to seed variability. However, at 50 tasks, CFR improves Average/Last accuracy by 2.60/3.41 pp over CCA-CL w/ FTT and achieves 2.01 pp higher Last accuracy than MG-CLIP. Thus, CFR's advantage becomes more pronounced over longer task horizons rather than relying on marginal short-horizon gains.

2. Additional cross-dataset evidence. We extended evaluation to ImageNet-R (10/20/50 tasks), TinyImageNet-100, and ImageNet-R-100 (3 seeds). On ImageNet-R-50, CFR achieves 85.37±0.20% Avg / 80.07±0.51% Last, versus CCA-CL w/ FTT's 83.20±0.14% / 75.69±0.13%, yielding +2.17/+4.38 pp. On TinyImageNet-100 and ImageNet-R-100, CFR further improves Average/Last accuracy over CCA-CL by 2.92/3.27 pp and 5.05/6.95 pp, respectively (Table 4, p. 10).

3. Coverage of related CIL methods. We have incorporated TUNA, MOS, CaRE, and GR-LoRA into the revised Related Work. Their published experiments primarily use ImageNet-pretrained visual backbones and different adaptation or inference protocols, making direct numerical comparisons with CLIP-based CFR potentially confounded. We therefore retain quantitative comparisons under compatible CLIP protocols while clarifying these methodological distinctions. A detailed comparison is provided in our response to W6.

Together, these results acknowledge CFR's non-uniform short-horizon ranking while demonstrating more substantial advantages across longer task sequences and additional datasets. The revised statistical results and expanded related-work discussion provide a more transparent and appropriately controlled evaluation of CFR's performance and contribution.

W5: Table 3 and Table S13 report inconsistent CFR and No-Refactor results under the same TinyImageNet-20 configuration.

Response to W5: Correction of Table S13

We thank the reviewer for identifying this inconsistency and apologize for the error. The discrepancy resulted from a transcription error during supplementary-table preparation, rather than different experimental configurations. The three-seed averages in Table 3 are correct.

We have corrected the CFR and No-Refactor entries in Table S13 under the same setting (r=64r=64, k=16k=16), including the corresponding performance differences. The verified TinyImageNet-20 results are:

MethodAvg Acc ↑Last Acc ↑Avg For ↓Last For ↓
No Refactor82.9673.849.4413.74
CFR84.1776.675.517.87

The corrected results confirm that CFR improves Average/Last accuracy by 1.21/2.83 pp and reduces Average/Last Forgetting by 3.93/5.87 pp relative to No Refactor.

The corrected Table S13 (p. 24) is consistent with Table 3 (p. 9) in the revised manuscript. We also plan to release our implementation and experimental configurations to facilitate reproducibility.

Reviewer_N17UOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper studies replay-free continual adaptation of CLIP under a fixed adapter capacity. Through longitudinal removal and keep-only interventions, the authors investigate how the behavioral importance of historical adapter subspaces changes during subsequent learning. They propose Continual Functional Refactoring (CFR), which maintains fixed-size input–output statistics, reconstructs accumulated behavior in a compact core, retains the recent complementary residual, and allows the full adapter to remain trainable for subsequent tasks. Experiments on CIFAR-100 and TinyImageNet cover six task configurations. CFR achieves the highest mean accuracy in 11 of the 12 reported Average/Last accuracy comparisons, with additional analyses examining forgetting, refactoring boundaries, and adapter capacity.

Critical strengths
  1. The paper asks a meaningful question beyond conventional importance preservation. Distinguishing whether a historical subspace remains necessary from whether it remains useful provides an informative perspective on continual learning. The removal and keep-only interventions offer behavioral evidence that complements representation-similarity measurements.
  2. CFR provides a concrete approach to reorganizing accumulated knowledge within a fixed adapter architecture. The combination of accumulated functional statistics, core–complement reconstruction, and unrestricted subsequent training is technically interesting. The state size remains independent of the number of tasks, and historical examples are not required by the training procedure.
  3. The empirical results are consistently competitive. Tables 1–2 show strong performance across both datasets and multiple task granularities. On CIFAR-100 with 50 tasks, CFR improves Last accuracy by 1.08 percentage points over the strongest competing result. Table 3 also shows a meaningful improvement over the matched no-refactoring baseline.
Critical weaknesses
  1. The accuracy–efficiency trade-off is not sufficiently discussed. Figure 4(a) shows that CCA-CL w/ FTT requires substantially less training time than CFR on TinyImageNet-20, while CFR improves Average/Last accuracy by 1.81/2.50 percentage points. Although these gains are meaningful, the additional runtime makes the practical advantage less clear. A comparison under matched training-time budgets would help determine whether CFR offers a favorable accuracy–efficiency trade-off.
  2. The transport of historical functional statistics is insufficiently specified. Algorithm 1 introduces an alignment matrix R_{t,ℓ}, but its estimation objective, constraints, and mapping convention are not clearly defined. Because bottleneck representations change during training, this alignment is central to interpreting the accumulated statistics as historical behavior in the current coordinates. Equation (7) is a valid quadratic reconstruction objective for the represented statistics, but its accuracy after repeated transport remains unverified.
  3. The evidence supports changing historical dependence more directly than the stronger stale-preservation explanation. The interventions demonstrate that historical directions can become less necessary or interfering. However, the analyses primarily examine three source tasks and do not directly isolate persistent protection as the cause of the observed degradation. Moreover, the diagnostic directions are behaviorally selected, whereas CFR constructs its core from accumulated activation geometry. The connection between these two procedures is plausible but not fully established.
  4. The main tables report three-seed means without standard deviations, although several improvements are only 0.10–0.30 percentage points, making their robustness difficult to assess. The CIFAR-100-50 CFR result in Table S12 also differs from Table 2, particularly in Last accuracy: 75.86 versus 76.74; the authors should clarify whether these results use different configurations. Furthermore, Table S12 shows that accumulating historical statistics improves Average accuracy and forgetting but reduces Last accuracy relative to current-task-only statistics. This suggests that the benefits of accumulation are metric-dependent and should be discussed accordingly.
  5. The distinction from closely related reorganization approaches could be sharper. Section 2.2 discusses dynamic functional directions, shared subspaces, and continual adapter merging, but the experiments do not directly establish which advantages are specific to CFR’s functional-state reconstruction. A clearer comparison with these mechanisms would strengthen the methodological contribution.
Questions for authors
  1. How is R_{t,ℓ} estimated, on which samples, and under what constraints? Can the authors provide evidence that the transported statistics remain predictive of historical adapter behavior after multiple task transitions?
  2. Which dataset, task partition, and training method produce the diagnostic trajectories in Section 3? How were the three source tasks selected, and are direction selection and intervention evaluation performed on separate samples?
  3. Can the authors explain the configuration difference between Table 2 and Table S12 and provide per-seed results or standard deviations for the principal comparisons?
  4. How should the lower Last accuracy of accumulated CFR relative to the current-task-only variant in Table S12 be reconciled with the proposed role of accumulated functional state?
6 comments

W1: CFR's accuracy–efficiency trade-off remains unclear without matched training-time comparisons.

Response to W1: Accuracy–Efficiency Trade-off

We thank the reviewer for this concern. Unlike CCA-CL, which freezes CLIP encoders after first-task fine-tuning and uses closed-form alignment, CFR continuously optimizes shared adapters and refactors historical functionality. We conducted matched-budget and long-horizon experiments to assess this trade-off.

1. Nearly matched training-time comparison. We reduced CFR's training epochs from four to one or two on TinyImageNet-20 (3 seeds), following the setting of Figure 4(a).

MethodAvg Acc ↑Last Acc ↑Avg For ↓Last For ↓Time ↓
CCA-CL82.36±0.08774.17±0.0796.51±0.1529.21±0.1458.7 min
CFR (1 epoch)82.70±0.21774.99±0.4445.69±0.1838.09±0.3779.2 min
CFR (2 epochs)83.24±0.18275.73±0.2835.47±0.2697.95±0.15915.5 min
CFR (4 epochs)84.17±0.27176.67±0.1465.51±0.1027.87±0.06825.4 min

At nearly matched runtime (9.2 vs. 8.7 min), CFR improves Average/Last accuracy by 0.34/0.82 pp and reduces final forgetting by 1.12 pp. Additional epochs further improve accuracy. Please refer to Table S25 (p. 28) in the revised manuscript.

2. Long-horizon efficiency. We further evaluated 100-task sequences on TinyImageNet and ImageNet-R (3 seeds).

DatasetMethodAvg Acc ↑Last Acc ↑Avg For ↓Last For ↓Time ↓
TinyImageNet-100CCA-CL77.33±0.01968.09±0.2558.17±0.03611.09±0.24351.92 min
CFR80.25±0.30071.36±0.5017.09±0.22510.32±0.46137.67 min
ImageNet-R-100CCA-CL78.07±0.12671.68±0.0258.10±0.1879.75±0.13557.44 min
CFR83.12±0.04678.63±0.0766.84±0.1908.10±0.11419.48 min

CFR improves Average/Last accuracy by 2.92/3.27 pp on TinyImageNet-100 and 5.05/6.95 pp on ImageNet-R-100, while reducing runtime by 27.4% and 66.1%, respectively. Both forgetting metrics also improve. Please refer to Table 4 (p. 10) in the revised manuscript.

These results demonstrate that CFR's additional optimization cost in shorter sequences does not translate into an inherent efficiency disadvantage. Instead, CFR achieves a favorable accuracy-efficiency trade-off under matched short-horizon budgets and simultaneously improves accuracy, retention, and runtime in the evaluated 100-task settings.

W2: Clarify alignment estimation and validate multi-step transport and historical reconstruction reliability.

Response to W2: Alignment and Historical Reconstruction We clarify the alignment formulation and provide two experiments on CIFAR-100 (10 tasks, 3 seeds) to validate transport reliability and reconstruction fidelity.

1. Alignment formulation. We estimate Rt,ℓR_{t,\ell} via orthogonal Procrustes alignment using paired bottleneck activations from the same current-task samples before and after training:

Rt,ℓ=arg⁡min⁡R⊤R=I∣HprevR−Hcur∣F2.R_{t,\ell}=\underset{R^\top R=I}{\arg\min}|H_{\mathrm{prev}}R-H_{\mathrm{cur}}|_F^2.

This maps previous representations into current coordinates. With Rˉ=diag⁡(R,1)\bar R=\operatorname{diag}(R,1) incorporating the bias coordinate, historical statistics are transported as

G←Rˉ⊤GRˉ,B←BRˉ.G\leftarrow\bar R^\top G\bar R,\qquad B\leftarrow B\bar R.

Alignment requires only current-task samples, without historical replay. Please refer to Appendix A.2 (lines 769–780) in the revised manuscript.

2. E2-A: Activation alignment. We measure normalized error as

Ealign=∣HoldR−Hnew∣F∣Hnew∣F+ϵ.E_{\mathrm{align}}=\frac{|H_{\mathrm{old}}R-H_{\mathrm{new}}|F}{|H{\mathrm{new}}|_F+\epsilon}.

**Activation Alignment **

Alignment MethodSingle-step Error ↓Multi-step Error ↓
No Alignment0.5180 ± 0.01150.8160 ± 0.0284
CFR (Replay-free)0.3928 ± 0.00760.6240 ± 0.0297
Historical-data Reference0.3863 ± 0.00740.5983 ± 0.0285

CFR reduces single-/multi-step errors by 24.2%/23.5% relative to no alignment, approaching the historical-data reference. The latter uses historical samples solely for diagnostic alignment estimation, not CFR training.

3. E2-B: Historical functional reconstruction. We evaluate historical-output normalized mean squared error (NMSE): NMSE=Avg⁡ℓ∑x∣z^ℓ(x)−zℓ(x)∣22∑x∣zℓ(x)∣22+ϵ.\mathrm{NMSE}=\operatorname{Avg}{\ell}\frac{\sum_x|\hat z\ell(x)-z_\ell(x)|2^2}{\sum_x|z\ell(x)|_2^2+\epsilon}.

Historical Functional Reconstruction**

Statistics / Reconstruction MethodNMSE ↓
Current-task Statistics Only1.3375
Historical Accumulation without Transport1.1199
CFR Transported Accumulation0.8307
Final CFR Mapping0.8075

Together, these results support the reliability of repeated historical-statistics transport and the functional reconstruction formulated in Eqs. (8) and (11), without historical replay.

W3: The stale-preservation explanation lacks direct evidence, the analysis covers only three source tasks, and the connection between behavioral diagnostics and CFR's geometry-based cores remains unclear.

Response to W3: Historical Necessity Drift and Functional Refactoring

We thank the reviewer for this important observation. We agree that the original diagnostics establish changing historical necessity but require further evidence for stale preservation and its connection to CFR.

Methodological connection. Section 3 uses historical samples and loss-based interventions solely for offline diagnosis. Applying this procedure during training would violate the replay-free setting. CFR instead uses accumulated activation geometry to select compact cores and input–output statistics to reconstruct historical functionality without replay. We conducted three complementary experiments to validate this connection.

1. E3-A: Generality of Historical Necessity Drift. We extended the analysis to all eligible historical source tasks on CIFAR-100 (10 tasks, 3 seeds), under standard seen-class evaluation. Historical necessity is measured as N=Afull−AremoveN=A_{\mathrm{full}}-A_{\mathrm{remove}}. Among initially necessary directions (N>0N>0), weakening denotes a decrease of at least 1 percentage point from initial necessity.

MethodHistorical Necessity Weakening ↓
Shared Adapter54.2%
CFR20.6%

CFR reduces historical necessity weakening by 62.0% relative to the shared adapter, demonstrating its effectiveness in maintaining functionally relevant historical representations through continual reorganization. Please refer to Appendix B.5 (lines 1092–1095) in the revised manuscript.

2. E3-B: Direct Intervention and Long-Term Preservation. On CIFAR-100 (50 tasks, 3 seeds), under a fixed 100-class evaluation protocol, CFR's refactoring immediately restores +3.284 ± 0.323 pp historical accuracy on average, with positive recovery at 145/147 task boundaries, without additional training. Please refer to Table S11 (p. 22) in the revised manuscript.

Repeated refactoring also improves long-term retention:

MetricNo RefactorCFR
Average Accuracy ↑84.3784.96
Final Forgetting ↓12.329.57
Fixed-100 Final Forgetting ↓7.754.73
Fixed-100 BWT ↑−5.37−2.00

Across task-boundary interventions, 64.6% of historical class-level observations improve, versus 23.9% that decline. These results demonstrate that functional refactoring directly recovers historical performance and reduces long-term forgetting, supporting the stale-preservation interpretation beyond observational necessity drift.

3. E3-C: Connection Between Diagnostic Directions and CFR Cores. We compared three equal-rank k=16 subspaces: historical loss-selected directions, CFR geometry-selected cores, and random cores, using identical removal and keep-only interventions under seen-class evaluation.

We measure removal necessity as N=Afull−AremoveN=A_{\mathrm{full}}-A_{\mathrm{remove}}, keep-only gain as K=Akeep−AfullK=A_{\mathrm{keep}}-A_{\mathrm{full}}, and normalized subspace overlap as

S(U,V)=1k∥U⊤V∥F2,S(U,V)=\frac{1}{k}\|U^\top V\|_F^2,

where U,VU,V are orthonormal subspace bases.

Functional importance of different cores (%):

Core SelectionNecessity N ↑Keep-only Gain K ↑
Random Core+0.24−3.25
Historical Loss-selected Core+2.99−1.23
CFR Geometry-selected Core+7.08−0.06

Subspace correspondence:

Subspace ComparisonOverlap S ↑
Loss-selected vs. Random0.250
Loss-selected vs. CFR (unaligned)0.402
Loss-selected vs. CFR (aligned)0.472

CFR's geometry-selected core exhibits substantially greater functional necessity and keep-only utility than both alternatives. Retaining only the CFR core nearly matches the full adapter's accuracy (K = −0.06 pp). Its overlap with behaviorally selected directions increases from 0.402 to 0.472 after alignment, exceeding the random reference (0.250). Please refer to Table S20 (p. 27) in the revised manuscript.

Together, these findings provide complementary evidence for CFR's underlying mechanism: historical necessity drift extends beyond selected examples; functional refactoring directly restores historical performance and reduces forgetting; and CFR's geometry-selected cores correspond to behaviorally important directions. Thus, CFR translates offline behavioral insights into replay-free reconstruction and reorganization of functionally meaningful historical representations, supporting the stale-preservation interpretation without requiring historical replay during training.

W4: Report statistical variability, resolve the Table 2/S12 discrepancy, and clarify the apparent Last-accuracy trade-off.

Response to W4: Statistical Robustness and Consistent Three-Seed Reporting

We rechecked the experimental records and provide additional statistical evidence and corrected ablations.

1. Statistical robustness. We report three-seed means ± standard deviations for CFR and representative baselines across all task partitions in the revised supplement. We also evaluated ImageNet-R (10/20/50 tasks), TinyImageNet-100, and ImageNet-R-100 (3 seeds). On ImageNet-R-50, CFR achieves 85.37±0.20% Avg / 80.07±0.51% Last, versus CCA-CL w/ FTT's 83.20±0.14% / 75.69±0.13% (+2.17/+4.38 pp). The 100-task results further support long-horizon effectiveness (Table 4, p. 10).

2. Table S12 correction. The original Table S12 included preliminary single-run results, whereas Table 2 reported the finalized three-seed averages. The discrepancy arose from different stages of experimental reporting. We have now standardized Table S12 to the verified three-seed results.

SeedAvg Acc ↑Last Acc ↑Avg For ↓Last For ↓
3284.93676.406.1109.550
3385.04377.115.7999.624
3484.90176.715.7919.541
Mean±Std84.96±0.07476.74±0.3565.90±0.1829.57±0.046

We corrected Table S12 and will release code and experimental configurations to facilitate reproducibility.

3. Historical accumulation. Holding refactoring fixed, we independently varied core-selection geometry and reconstruction statistics (CIFAR-100-50, 3 seeds):

Core SelectionReconstruction StatisticsAvg Acc ↑Last Acc ↑Avg For ↓Last For ↓
Accumulated (CFR)Accumulated84.96±0.07476.74±0.3565.90±0.1829.57±0.046
Current-taskAccumulated84.82±0.13576.67±0.4575.98±0.2459.51±0.362
AccumulatedCurrent-task84.38±0.33176.22±0.4488.84±0.16612.52±0.496
Current-taskCurrent-task84.34±0.30776.28±0.2028.77±0.23112.33±0.419

With core selection fixed, accumulated reconstruction improves Last accuracy by 0.39–0.52 pp and reduces Last Forgetting by 2.82–2.95 pp. On TinyImageNet-20, the corresponding improvements are 2.10–2.25 pp and 5.13–5.48 pp. Full CFR achieves 84.17±0.271% Avg / 76.67±0.146% Last, with 7.87±0.068% Last Forgetting. See Table S12 (p. 23) for both four-way ablations.

These corrected results demonstrate that accumulated functional reconstruction improves both final accuracy and historical retention across the tested core-selection settings, rather than introducing the metric-dependent trade-off suggested by the erroneous table.

W5: Distinguish CFR from existing reorganization/merging methods and isolate the benefits of functional-state reconstruction.

Response to W5: Distinction from Existing Methods

Unlike functional-direction tracking, shared-subspace methods, or adapter merging, CFR reconstructs historical functionality through transported input–output statistics within a compact core, retains the recent complement, and fully reopens the adapter. Three controlled comparisons on TinyImageNet-20 (3 seeds) isolate these contributions.

1. E5-A: Historical functional-state reconstruction. The W4 ablation shows that accumulated reconstruction improves Last accuracy by 2.10–2.25 pp and reduces Last Forgetting by 5.13–5.48 pp, regardless of core-selection source. Combining accumulated reconstruction with geometric core selection achieves 84.17% Avg / 76.67% Last. Thus, historical functional-state reconstruction is the primary contributor, while geometry-guided core selection provides complementary benefits beyond periodic reorganization (Table S12, p. 23).

2. E5-B: Selective versus full-map reconstruction. Using the same reconstructed mapping:

Reconstruction MechanismAvg Acc ↑Last Acc ↑Last For ↓
CFR Core–Complement84.1776.677.87
Full-map Recovery83.5975.848.54
Norm-matched Full Update83.7475.9110.35

CFR improves Last accuracy by 0.83/0.76 pp, demonstrating the advantage of selectively reconstructing historical functionality while preserving recent complementary representations instead of replacing the full mapping (Table S21, p. 27).

3. E5-C: Beyond periodic reorganization. Random-core refactoring improves Last accuracy over no refactoring (75.38% vs. 73.84%) but trails CFR (76.67%). Thus, periodic reorganization helps, while statistically guided functional organization yields additional gains (Table 3, p. 9).

These experiments show CFR surpasses periodic subspace rearrangement or adapter merging: it reconstructs historical functional states and selectively integrates them with recent representations in a fixed-capacity, reusable adapter.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response for Area Chair and Reviewers

Additional Evidence and Invitation for Further Discussion

We sincerely thank the Area Chair and Reviewers N17U, mn2k, and xAKz for their valuable feedback. In response, we have conducted extensive additional experiments, including matched-budget comparisons, mechanistic validations, new baselines, and evaluations extending to 300 tasks, substantially beyond commonly adopted CIL evaluation settings. We also updated the supplementary material with three-seed results.

As the discussion approaches its conclusion, we briefly summarize the key evidence and welcome further feedback.

1. Reviewer N17U: Main Contributions

We appreciate the reviewer's positive assessment and recognition of our two main contributions: Historical Necessity Drift and Continual Functional Refactoring (CFR). Additional experiments further validate CFR's historical functional recovery, reliability, and efficiency.

2. Reviewer mn2k: Fair Comparisons and Scalability

  • Fair comparisons: Our original experiments followed established practice by comparing methods under compatible pretrained backbones. Nevertheless, we additionally evaluated other CLIP-based methods (ENGINE, BOFA, MoE-Adapters++) as well as CaRE and EASE adapted to CLIP ViT-B/16. CFR outperforms these CLIP-based baselines in accuracy while achieving substantial efficiency advantages, including 33.6×–40.3× speedups over CaRE.

  • Scalability: Our original evaluation followed widely adopted CIL benchmarks and standard evaluation protocols, consistent with representative recent studies. Nevertheless, to directly address the reviewer's concern, we conducted extensive additional 100-task and 300-task evaluations, substantially extending beyond common settings. Over 300 tasks, CFR improves Average/Last accuracy over CCA-CL by +15.78/+4.32 pp, with 51.2% less runtime. Importantly, this involves 600-way task-agnostic classification, not repeated binary classification. The earlier 100-task bound was dataset-related, not a limitation of CFR.

3. Reviewer xAKz: Consistency and Novelty

  • Consistency: The original supplementary Table S12 contained single-run results, and the main-paper Table 2 reported three-seed averages. To ensure consitency, we updated S12 to include three-seed results in the supplementary, the main-paper results remain unchanged.
  • Novelty: Controlled ablations distinguish CFR's core-complement functional refactoring from RegMean-style full-space recovery, demonstrating +2.10–2.25 pp Last accuracy and reduced forgetting. Furthermore, CFR immediately restores +3.28 pp historical accuracy at 145/147 task boundaries without gradient updates.

We believe these extensive additional experiments and clarifications have comprehensively addressed the concerns raised in the reviews. We sincerely welcome any further questions or discussions.

As some of these substantial additions have not yet received follow-up feedback, we respectfully hope the Area Chair will take the complete updated experimental evidence and technical clarifications into account when making the final assessment.

We sincerely thank the Area Chair and all reviewers for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.