- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies replay-free, fixed-capacity continual adaptation of CLIP. Through removal and keep-only interventions on adapter subspaces, it investigates how dependence on historically important directions changes during subsequent learning. It proposes Continual Functional Refactoring, which accumulates local input–output statistics, reconstructs a mapping within a compact core, retains the current complementary component, and reopens the full adapter for subsequent training. Experiments cover CIFAR-100 and TinyImageNet with sequences ranging from 5 to 50 tasks.
Critical strengths
- CFR operates without historical sample replay or task-specific parameter growth. The matched no-refactoring comparison is useful: on TinyImageNet with 20 tasks, CFR improves final accuracy from 73.84% to 76.67%. Component ablations and capacity sensitivity experiments provide additional evidence that the proposed procedure has practical effects.
- Distinguishing the effect of removing a historical subspace from its effectiveness under keep-only evaluation is useful. The reported trajectories illustrate that a historical realization can remain useful without remaining indispensable.
Critical weaknesses
- Prior work already studies changing parameter importance, drifting functional directions, evolving shared subspaces, and continual adapter merging, as acknowledged in Section 2.2. Furthermore, reconstructing linear mappings from activation statistics is closely related to the regression-based merging formulation of RegMean. CFR introduces a specific combination of accumulated statistics, core–complement reconstruction, and reopening, but the current comparisons do not sufficiently establish what substantive new capability or understanding this combination provides beyond these existing principles. The results support an improvement in the evaluated CLIP setting; the broader methodological significance remains less convincing.
- CFR accumulates statistics in representation spaces that themselves change during learning. Algorithm 1 introduces alignment of historical statistics, but the estimator and its reliability across successive tasks are insufficiently characterized. The paper does not directly quantify reconstruction fidelity on held-out historical samples or how alignment errors accumulate. In addition, the reported forgetting metric compares checkpoints with expanding candidate-class sets, so it does not isolate deterioration caused by model updates. These issues do not invalidate the reported accuracy gains, but they limit the evidence for the proposed functional-preservation explanation.
Questions for authors
- What is the substantive technical distinction between CFR’s accumulated regression reconstruction and a sequential RegMean-style procedure? A matched baseline using the same statistics and alignment, but reconstructing the full mapping without the core–complement split, would help establish the additional value of the proposed organization.
- Please specify how the alignment matrix in Algorithm 1 is estimated, which samples are used, the summary-set size, and the regularization settings. Can you quantify reconstruction fidelity on held-out historical samples and its evolution across task boundaries?
- For CFR and the matched no-refactoring baseline, can you report historical-class performance across checkpoints with a fixed candidate-class set? Per-class trajectories would also clarify whether average improvements conceal substantial deterioration in particular historical classes.
- Please clarify the differences between the main and supplementary results under apparently matching configurations. For example, CIFAR-100/50-task CFR is reported as 84.96/76.74 in Table 2, whereas Accumulated CFR is 84.99/75.86 in Table S12. Are these different configurations or sets of runs?
6 comments
W2: The reliability of repeated historical-statistics transport, reconstruction fidelity, and forgetting evaluation under expanding candidate-class sets remain insufficiently established.
Response to W2: Historical-State Transport Reliability and Fixed-Class Forgetting
We thank the reviewer for highlighting these issues. We provide direct evidence from multi-step alignment, held-out historical-output reconstruction, and fixed-class evaluation.
1. Alignment reliability across task transitions. CFR estimates through orthogonal Procrustes alignment of paired bottleneck activations from the same current-task samples before and after training, subject to . Historical statistics are transported into the current representation coordinates without replay.
On CIFAR-100-10 (3 seeds), CFR reduces single-step alignment error from 0.5180±0.0115 to 0.3928±0.0076 and multi-step error from 0.8160±0.0284 to 0.6240±0.0297. The latter approaches the historical-data reference (0.5983±0.0285), reducing multi-step error by 23.5% relative to no alignment (Table S17, p. 25).
2. Historical functional reconstruction fidelity. We directly measured normalized mean squared error (NMSE) between reconstructed adapter outputs and historical reference outputs on held-out historical samples (CIFAR-100-10, 3 seeds).
| Reconstruction Method | NMSE ↓ |
|---|---|
| Current-task statistics only | 1.3375 |
| Historical accumulation without transport | 1.1199 |
| CFR transported accumulation | 0.8307 |
| Final CFR mapping | 0.8075 |
Transported accumulation reduces historical-output NMSE by 25.8% versus unaligned accumulation and 37.9% versus current-task statistics. The final CFR mapping further achieves 0.8075, confirming that selective refactoring retains the reconstruction benefits (Table S18, p. 26).
We also measured reconstruction fidelity across successive checkpoints. CFR achieves lower NMSE than both alternatives at every evaluated checkpoint (Tasks 2/4/6/8/10). At Task 10, its NMSE remains 0.8118±0.0619, versus 1.0191±0.0740 without transport and 1.2508±0.1109 using current-task statistics, demonstrating sustained reconstruction advantages after repeated transitions (Table S19, p. 26).
3. Isolating model-induced forgetting. To eliminate the influence of expanding candidate-class sets, we evaluated CIFAR-100-50 (3 seeds) using the same fixed 100-class candidate set at every checkpoint.
| Metric | No Refactor | CFR |
|---|---|---|
| Standard Last Forget ↓ | 12.32 | 9.57 |
| Fixed-100 Last Forget ↓ | 7.75 | 4.73 |
| Fixed-100 BWT ↑ | −5.37 | −2.00 |
Under fixed-class evaluation, CFR reduces Last Forgetting by 3.02 pp and improves backward transfer (BWT) by 3.37 pp. Moreover, task-boundary interventions immediately restore 3.284±0.323 pp historical accuracy, with positive recovery at 145/147 boundaries, without additional training or changes to the candidate-class set (Table S11, p. 22).
Together, these results demonstrate that transported historical statistics retain predictive information across task transitions, substantially improve historical functional reconstruction, and support genuine historical retention gains independently of candidate-class expansion. This provides direct mechanistic evidence for CFR's functional-state preservation beyond aggregate accuracy improvements.
Q1: What distinguishes CFR from sequential RegMean-style reconstruction? Can a matched full-map recovery baseline establish the value of core–complement organization?
Response to Q1: Distinction from RegMean-Style Reconstruction
We thank the reviewer for this suggestion. CFR shares the regression-based reconstruction principle with RegMean, but differs in how historical functionality is integrated into the evolving adapter.
Let denote the mapping reconstructed from aligned accumulated statistics, the current adapter mapping before refactoring, and the core projector. The two mechanisms are:
Full-map recovery overwrites the adapter; CFR restores historical functionality in the core, retains the recent complement, and reopens all dimensions.
On TinyImageNet-20 (3 seeds), both use identical accumulated statistics, alignment, regression objective, and training settings. CFR achieves 76.67% Last accuracy / 7.87% Last Forgetting, versus 75.84% / 8.54% for RegMean-style full-map recovery (+0.83/−0.67 pp, respectively; Table S21, p. 27).
This matched comparison establishes the added value of core–complement organization beyond regression reconstruction: integrating historical functionality with recent learning rather than replacing the entire adapter. See W1 for the broader distinction.
Q2: Clarify alignment estimation, sample sizes, regularization, and historical reconstruction fidelity across task boundaries.
Response to Q2: Alignment Implementation and Reconstruction Fidelity
We thank the reviewer for requesting these details. We clarify the implementation and provide checkpoint-wise reconstruction evidence.
1. Alignment estimation and regularization. For each visual adapter layer , CFR extracts paired post-ReLU bottleneck activations and from the same current-task images, using checkpoints before and after task training. We solve the orthogonal Procrustes problem:
Given , the solution is , mapping previous representations into current coordinates.
We use class-stratified sampling from the current task: up to 300 images/class for functional-statistic estimation and 200 images/class for alignment, with alignment samples included in the summary set. On CIFAR-100-10, these correspond to 3,000 summary and 2,000 alignment images/task. All image tokens contribute to the statistics.
Orthogonal alignment requires no ridge regularization. Functional reconstruction uses:
\lambda_{t,\ell}^{\mathrm{bias}}=10^{-3}\lambda_{t,\ell},$$ with $r=64$ and default core rank $k=16$. Both alignment and statistic accumulation use only current-task samples, without historical replay (Appendix A.2, lines 769–780). **2. Held-out historical reconstruction.** On CIFAR-100-10 (3 seeds), we use 20 historical images/class to construct offline diagnostic references and a separate set of 30 historical test images/class for evaluation. These samples are never used for CFR training or functional-state updates. As reported in W2, transported accumulation achieves historical-output NMSE of **0.8307**, versus **1.1199** without alignment transport and **1.3375** with current-task statistics only—reductions of **25.8%** and **37.9%**, respectively. The final CFR mapping further achieves **0.8075** (Table S18, p. 26). **3. Fidelity across task boundaries.** We additionally evaluated reconstruction NMSE across checkpoints, averaging over eligible historical source tasks and 12 adapter layers (3 seeds): | Checkpoint | CFR Aligned ↓ | Unaligned Accum. ↓ | Current-task Only ↓ | |:---|---:|---:|---:| | Task 2 | **0.5006±0.0299** | 0.9264±0.0141 | 1.0276±0.0328 | | Task 4 | **0.7499±0.2118** | 1.1987±0.2474 | 1.3744±0.2918 | | Task 6 | **0.7627±0.1032** | 1.0435±0.1443 | 1.2053±0.1642 | | Task 8 | **0.9156±0.1797** | 1.1680±0.1933 | 1.4035±0.2070 | | Task 10 | **0.8118±0.0619** | 1.0191±0.0740 | 1.2508±0.1109 | CFR achieves the lowest NMSE at every evaluated checkpoint. At Task 10, it retains **20.3%/35.1% lower error** than unaligned accumulation/current-task-only reconstruction, demonstrating that its advantage persists across successive task transitions (Table S19, p. 26). The overall NMSE of 0.8307 aggregates historical-source/checkpoint/layer evaluations across Tasks 2–10, whereas 0.8118 refers specifically to Task 10. **Together, these results demonstrate that CFR's current-task-based alignment reliably transports accumulated functional statistics across changing representations, enabling consistently more accurate historical reconstruction throughout continual learning.**
Response to W1: Methodological Novelty and Distinction from RegMean
We appreciate the reviewer’s request to identify the substantive contribution beyond established principles. CFR builds on insights about changing parameter importance and regression-based reconstruction, but introduces a distinct continual-learning mechanism: preserving historical functionality without permanently preserving its original parameter directions, while keeping the entire adapter capacity reusable. We clarify this mechanism and isolate its benefits through controlled comparisons.
1. New preservation principle and capability. Section 3 shows that historical functional necessity is non-stationary: directions once essential to earlier tasks can become replaceable or interfering. CFR therefore preserves functional behavior, rather than fixed historical coordinates. Its learn–refactor–reopen cycle (i) transports and accumulates a fixed-size historical input–output state across changing representations, (ii) reconstructs accumulated functionality within a compact geometric core while retaining recently learned behavior in the complement, and (iii) reopens all dimensions for the next task. This enables repeated historical-function recovery and further adaptation within one fixed-capacity shared adapter, without replay, task-specific expansion, or permanent subspace protection.
2. Technical distinction from RegMean and a matched comparison. RegMean uses activation-statistic-weighted regression to merge separately trained mappings. CFR employs a related regression objective, but its role is different: a sequential functional-state update within a single evolving adapter. Let W⋆ be the mapping reconstructed from aligned accumulated statistics, P the core projector, and W− the adapter mapping immediately before refactoring. A full-map regression recovery gives Wfull=W⋆, whereas CFR uses WCFR=W⋆P+W−(I−P). Thus, CFR restores historical behavior selectively without overwriting the recently adapted complement, then reopens both components for subsequent learning.
We compared these mechanisms on TinyImageNet-20 (3 seeds), using identical accumulated statistics, alignment, regression objective, and training settings:
CFR improves Last accuracy by 0.83 pp and reduces Last Forgetting by 0.67 pp relative to matched full-map recovery. The difference isolates the benefit of selective core–complement integration beyond the shared regression reconstruction principle (Table S21, p. 27).
3. Why the gain is not generic periodic rewriting. With the refactoring procedure held fixed, replacing current-task reconstruction statistics with accumulated historical statistics improves Last accuracy by 2.10–2.25 pp and reduces Last Forgetting by 5.13–5.48 pp, regardless of the core-selection source (Table S12, p. 23). Moreover, CFR's geometry-selected core is behaviorally meaningful: removal necessity is +7.08 pp versus +2.99 pp for loss-selected directions and +0.24 pp for a random core; retaining the CFR core alone nearly matches full-adapter accuracy (−0.06 pp, Table S20, p. 27). These results identify accumulated functional reconstruction as the main contributor and geometry-guided selection as a complementary mechanism—not merely a periodic parameter rewrite.
4. Direct functional recovery and continued reuse. On CIFAR-100-50 (3 seeds), refactoring immediately recovers +3.284±0.323 pp historical accuracy at task boundaries, with positive recovery at 145/147 boundaries and no additional training (Table S11, p. 22). Full reopening further improves TinyImageNet-20 Last accuracy from 76.12% to 76.67% (Table 3, p. 9). These interventions show that historical functionality can be restored and then reused within the same capacity, rather than only yielding an aggregate accuracy increase.
Together, these controlled comparisons and task-boundary interventions establish CFR's methodological contribution: transporting and reconstructing historical functionality across changing representations, selectively integrating it with recent learning, and reopening the entire fixed-capacity adapter for continued reuse without replay.