Spectral Decomposition of Prompt Knowledge For Federated Continual Learning

Federated continual learning (FCL) requires clients to continuously learn new tasks while leveraging knowledge distributed across clients and historical tasks. Prompts provide a compact means of representing and aggregating such knowledge without repeatedly updating the entire model. However, existing methods typically select and aggregate historical prompts as a whole, overlooking that different structural directions may play distinct roles: some capture knowledge shared across tasks, while others may interfere with current knowledge integration. To address this issue, we propose SDPrompt, which organizes and exploits historical prompt knowledge from a spectral perspective. At the server, SDPrompt extracts a dominant spectral subspace from accumulated global prompts to preserve historical knowledge. At each client, it softly rectifies weak spectral components of historical prompts and extracts a task-conditioned subspace to facilitate relevant knowledge transfer. We further theoretically characterize the spectral concentration of shared knowledge and bound the effect of weak-spectrum rectification under explicit spectral separation conditions. Experiments on ImageNet-R, DomainNet, Cars196, and CIFAR-100 show that SDPrompt outperforms the compared methods in both Average Accuracy (Avg) and Average Incremental Accuracy (AIA), achieving 89.83% Avg on ImageNet-R with a 2.63% improvement. These results demonstrate the effectiveness of exploiting prompt knowledge at the spectral level for knowledge preservation and transfer in FCL.Code is available at https://anonymous.4open.science/r/paper_-C76A/.