Continually Learning a Functional Memory from Task Experience

Trained low-rank adapters (LoRAs) record what a model has learned from task experience. We ask how a stream of these adaptations can form a shared memory that preserves past solutions and supports new ones. The core question is which weight directions each adapter should contribute. We find that projecting an adapter into existing directions can reduce its update magnitude, and restoring that magnitude often recovers much of the lost task performance. A small number of task-dependent residual additions can recover further behaviour. Retained weight energy alone therefore does not determine behavioural recovery. These findings motivate Functional Memory of Low-rank subspaces (FML), which uses adapter predictions on unlabelled task inputs to guide the growth of a shared basis after correcting update scale. The admitted task is stored through coefficients and a global scale, allowing the original adapter and task inputs to be discarded. As the shared basis expands, previously admitted reconstructed updates remain unchanged. We evaluate continual retention and basis growth over a large SNI adapter stream and several ViT image-classification benchmarks, and further demonstrate downstream reuse through target-data-free LoRA initialization for subsequent fine-tuning and description-conditioned LoRA generation and adaptation within the functional basis, with evaluations on unseen SNI tasks and seven language benchmarks. Our findings demonstrate how structure retained from completed learning can support subsequent adaptation.