One-Step Curvature Probes Miss the Fitting Operator: Retained Capacity and Terminal Null-Space Correction for Continual Learning
Abstract
A one-step curvature probe evaluates an initial direction, whereas continual learners are judged after reaching comparable new-task fit. These quantities need not agree. In an overparameterized linearization, projected gradient descent converges to , and its squared-displacement inflation is exactly the reciprocal of the direction-specific retained fitting capacity . More generally, the terminal old-task quadratic ratio factorizes as , where compares curvature along the normalized endpoint directions. In the rank-one case, equals the one-step probe gain; for multiple outputs, the two gains need not agree. The terminal quadratic further decomposes into a common curvature-optimal fitting floor and an algorithm-dependent null-space excess. This geometry identifies the variable omitted by a matched-norm probe and yields terminal null-space correction, a post-hoc operation that preserves linearized new-task outputs on its Jacobian batch. In separate controlled checks, the local quadratic has slope (); projection reduces the matched-norm quadratic by while terminal forgetting changes by only . Across five common-threshold configurations, projection produces the larger signed old-task loss change in matched pairs. In the eligible-subset probe study, median gain ranges from about to (eight pairs at the upper end). The common-threshold rank sweep shows median retained fitting capacity falling from to . A fixed-rank comparison separates terminal forgetting when probe gain is approximately matched, whereas the capacity-matched contrast remains compatible with effects of either sign. On Permuted MNIST and Split CIFAR-100, terminal correction decreases signed old-task loss in method–dataset–seed pairs under the all-seed intention-to-correct analysis. Because acceptance and outcome reporting use the same held-out split, this result is descriptive and test-conditioned. Together, the results replace gain-only interpretation with a matched-fit account: in the fixed-Jacobian quadratic model, terminal cost depends jointly on retained fitting capacity, endpoint-direction curvature, and the null-space component selected by optimization.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.