When Preservation Becomes Stale: Continual Functional Refactoring for Fixed-Capacity CLIP Continual Learning

Continual adaptation of pretrained vision-language models under a fixed parameter budget requires acquiring new knowledge without historical replay or task-specific capacity growth. Many existing methods protect historically important structures and constrain subsequent updates around them, effectively creating persistent capacity ownership. Through longitudinal subspace interventions, we identify Historical Necessity Drift: the functional necessity of historical subspaces can weaken, strengthen, or migrate as new tasks are learned. Directions once critical can become redundant, secondary directions can become essential, and others remain necessary throughout. This drift creates stale capacity ownership, where structure remains protected after its original functional role has diminished or migrated. More strikingly, retaining only a compact, functionally selected subspace can improve historical-task accuracy over the full adapter, showing that historical performance degradation can coexist with recoverable useful function and exposing functional interference from stale organization. Motivated by this finding, we propose Continual Functional Refactoring (CFR), which summarizes accumulated behavior with a fixed-size functional state, consolidates shared functionality into a compact core while retaining recent residuals in its complement, and reopens the full adapter for subsequent learning. Across CIFAR-100 and TinyImageNet sequences spanning 5 to 50 tasks, CFR ranks first in 11 of the 12 reported Average/Last accuracy evaluations. Its advantage is particularly evident over long horizons: on CIFAR-100 with 50 tasks, CFR achieves 84.96% Average and 76.74% Last accuracy, exceeding the strongest competing Last result by 1.08 percentage points. Code will be made publicly available upon acceptance.