acceptodds
Under review as a conference paper at ICLR 2027

Spectral Decomposition of Prompt Knowledge For Federated Continual Learning

Abstract

Federated continual learning (FCL) requires clients to continuously learn new tasks while leveraging knowledge distributed across clients and historical tasks. Prompts provide a compact means of representing and aggregating such knowledge without repeatedly updating the entire model. However, existing methods typically select and aggregate historical prompts as a whole, overlooking that different structural directions may play distinct roles: some capture knowledge shared across tasks, while others may interfere with current knowledge integration. To address this issue, we propose SDPrompt, which organizes and exploits historical prompt knowledge from a spectral perspective. At the server, SDPrompt extracts a dominant spectral subspace from accumulated global prompts to preserve historical knowledge. At each client, it softly rectifies weak spectral components of historical prompts and extracts a task-conditioned subspace to facilitate relevant knowledge transfer. We further theoretically characterize the spectral concentration of shared knowledge and bound the effect of weak-spectrum rectification under explicit spectral separation conditions. Experiments on ImageNet-R, DomainNet, Cars196, and CIFAR-100 show that SDPrompt outperforms the compared methods in both Average Accuracy (Avg) and Average Incremental Accuracy (AIA), achieving 89.83% Avg on ImageNet-R with a 2.63% improvement. These results demonstrate the effectiveness of exploiting prompt knowledge at the spectral level for knowledge preservation and transfer in FCL.Code is available at https://anonymous.4open.science/r/paper_-C76A/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
FEDERATED CONTINUAL LEARNING

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CaEqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Federated Continual Learning (FCL) and identifies a limitation of existing prompt-based FCL methods: they select and aggregate historical prompts as a whole, overlooking the fact that different directions may play distinct roles, with some capturing shared knowledge and others introducing interference during knowledge integration. To address this issue, the authors propose SDPrompt, which decomposes historical prompts from a spectral perspective to selectively preserve useful knowledge while suppressing interference, thereby improving federated continual learning performance. Experiments conducted on ImageNet-R, DomainNet, Cars196, and CIFAR-100 demonstrate that the proposed method outperforms the compared baselines.

Critical strengths
  1. This paper identifies a specific limitation of existing prompt-based federated continual learning methods: historical prompts are typically selected and aggregated as a whole, overlooking the fact that different structural directions may serve distinct roles. Some directions capture knowledge shared across tasks, while others may interfere with the integration of current knowledge. This is a concrete issue in FCL, and addressing it could significantly improve the reuse of historical knowledge across tasks and clients.

  2. The proposed SDPrompt method is novel. It leverages historical prompt knowledge from a spectral perspective. Specifically, the server extracts a dominant spectral subspace to preserve knowledge, while each client applies soft correction to weak spectral components and extracts a task-conditioned subspace. This decomposition goes beyond a simple combination of existing techniques and may inspire future research on spectral- or subspace-based prompt aggregation in federated continual learning.

Critical weaknesses
  1. The performance improvements on some datasets are relatively small. However, the paper does not explicitly report whether repeated experiments were conducted using multiple random seeds, making it difficult to assess the stability of these improvements. I suggest that the authors conduct additional experiments with multiple random seeds and report the mean and standard deviation to strengthen the reliability of the experimental results.
  2. Although the authors compare the overall performance across four datasets, the ablation studies of the three core modules (G/T/D) are conducted only on ImageNet-R. I suggest extending the ablation studies to the other datasets to provide a more comprehensive evaluation.
  3. This paper improves federated continual learning performance by introducing spectral decomposition, spectral knowledge correction, and dual-subspace constraints. However, it lacks a comparison of computational costs and other overheads with existing methods, making it difficult to comprehensively evaluate the trade-off between performance gains and additional resource consumption. I suggest that the authors provide an analysis of these overheads to further demonstrate the practicality of the proposed method.
Questions for authors
  1. The performance improvements on some datasets are relatively small. Have the authors conducted independent repeated experiments using multiple random seeds? If so, could the authors provide the corresponding mean and standard deviation to further demonstrate the stability of these improvements?
  2. The ablation studies of the G/T/D modules are conducted only on ImageNet-R. Could the authors clarify whether each of these three modules also provides performance improvements on DomainNet, Cars196, and CIFAR-100?
  3. SDPrompt introduces additional spectral decomposition, spectral knowledge correction, and dual-subspace constraints. How much additional computational and communication overhead do these operations introduce? Compared with baseline methods such as C2Prompt, how do the actual training time and resource consumption change? Flag For Ethics Review
Reviewer_U45cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SDPrompt extends C2Prompt with spectral processing of historical prompts. The server extracts a dominant global prompt subspace, while clients construct input-conditioned historical-prompt matrices. Selected weak singular directions are attenuated before fusion, and the current prompt is regularized toward both the global and task-conditioned dominant subspaces. The analysis gives common-versus-specific spectral energy bounds under separation assumptions and bounds the perturbation to a common component after rectification. Experiments report improvements on four FCL datasets and an additional disjoint-class FCIL evaluation on ImageNet-R, with component ablations and subspace diagnostics.

Critical strengths
  1. The empirical coverage is broader than a single FCL protocol: four datasets are supplemented by disjoint-class FCIL with three heterogeneity levels.

  2. The global and input-conditioned views provide a concrete, implementable way to organize historical prompts. The theoretical assumptions are stated explicitly, and the paper includes a non-orthogonal perturbation treatment rather than relying exclusively on exact orthogonality.

Critical weaknesses
  1. Propositions 4.2 and 4.4 compare energies of assumed common and specific components under shared-subspace, per-task-energy, and orthogonality conditions. Appendix A extends the analysis to perturbations smaller than a spectral gap, but the experiments do not establish that this separation holds for the learned prompts. In particular, low-energy directions may encode rare but useful classes or tasks. Proposition 4.7 bounds the disturbance by eta^2 times rho_T, where rho_T is the unknown common-component energy outside V_task. Without bounding or estimating this quantity in the evaluated setting, it need not be a small or informative guarantee. Nor does a parameter-energy bound directly bound old-task prediction error. 
  2. Reporting and evaluation gaps prevent a reliable assessment of robustness and practical cost. Tables 1-3 report point estimates without seed counts or variability, even though several gains over the strongest baseline are modest. The paper specifies tau_G = tau_T = 0.7 but does not give the numeric weak-band thresholds tau_s/tau_e, the individual-energy threshold delta, or the attenuation eta used for the main results. The method constructs a relevance-weighted Q_T for each input and applies SVD, while retaining historical prompts and increasing retrieval k over stages. There is no measured training/inference latency, peak memory, or communication overhead relative to C2Prompt.
Questions for authors
  1. Do weak spectral directions empirically cause interference? Please compare weak-band attenuation with norm-matched shrinkage, random subspaces, and equal-energy attenuation elsewhere, and report rare-class/old-task retention as well as aggregate accuracy. 
  2. Please provide all rectification hyperparameters, seed counts and mean/dispersion results, the exact inference class-mask protocol, and the normalization in Eq. (17). How does the implementation guarantee that the weak subspace excludes the retained task-subspace indices? 
  3. How often is the input-conditioned SVD computed, what is cached, and what are the measured training time, inference latency, memory, and communication overhead relative to C2Prompt as the task pool grows?
Reviewer_1CdtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a prompt-based method for Federated Continual Learning. The paper argues that historical prompts should not be reused as a whole, so it decomposes them with SVD. The paper propose make the server to extract a global knowledge subspace from the accumulated prompts. Each client extracts a task-conditioned subspace from the retrieved historical prompts and removes a band of weak spectral directions from them before fusion. Then, the current-task prompt is then regularized toward both subspaces.

Critical strengths
  1. The paper investigate an interesting problem for prompt-based FCL that how to reuse historical prompt knowledge at a finer level than whole prompts.

  2. The proposed modules are simple and lightweight, and they can be added on top of an existing pipeline.

Critical weaknesses
  1. The author need to check the related works and scope the proposed method carefully, as some components it using have been inverstigated by other works. Using SVD-based subspaces to keep or remove parameter components is the main mechanism of gradient projection methods in continual learning, such as GPM [1], Adam-NSCL [2], TRGP [3] and SGP [4]. PGP [5] applies it directly to prompts, InfLoRA [6] to LoRA, and FOT [7] to federated continual learning. Splitting shared and task-specific directions with SVD is also common in model merging [8, 9, 10], and the theory in §4.5 follows a model merging work [11]. The paper discusses none of the projection-based methods. Can the authors give a clear discussion of these works and clarify where SDPrompt stands out ?

  2. The writing now is to hard to understand and require significant improvement, in my opinion. For example, abstract mentions "structural directions may play distinct roles" without clarification about what directions here mean. "indivisible unit of knowledge" in the introduction is hard to understand. Related works section have plenty of FL works, but missing comprehensive FCL literature review. The author may check [12] for reference. In preliminaries section, does Local Optimization is an preliminaries ? seems this part is too verbose. and not bring the meaningful insight to the reader. Meanwhile, i think the prompt learning is the preliminaries that can support the reader with the concept is missing.

  3. The relation between this work and the series of Federated Orthogonal Training or Gradient Projection in Federated Learning should be discussed clearly. For example, equation 6 is similar to orthogonal training/ gradient projection.

  4. It is unclear to me about what significant insights that the theoretical analysis bring about how the method work well? It would be beneficial if the authors can clear this point.

  5. It is unclear to me what Eq. 8 actually protects. The historical prompts are kept in the server pool and do not seem to be updated by the current task. Which component causes forgetting and how does pulling the new prompt toward old directions prevent it?

  6. The experimental evaluation is not convincing enough with quite small gain. Are there any special point of the proposed method that other works do not have, except the performance ? Also, the authors should provide standard deviations across seeds to show variance of runs. Additional experiments should be added to verify that weak spectral directions cause interference.

[1] Gradient Projection Memory for Continual Learning - ICLR 2021

[2] Training Networks in Null Space of Feature Covariance for Continual Learning - CVPR 2021

[3] TRGP: Trust Region Gradient Projection for Continual Learning - ICLR 2022

[4] Continual Learning with Scaled Gradient Projection - AAAI 2023

[5] Prompt Gradient Projection for Continual Learning - ICLR 2024

[6] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning - CVPR 2024

[7] Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning - ICLR 2024

[8] Task Singular Vectors: Reducing Task Interference in Model Merging - CVPR 2025

[9] Model Merging with SVD to Tie the KnOTS - ICLR 2025

[10] No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces - ICML 2025

[11] Subspace-Boosted Model Merging - https://arxiv.org/abs/2506.16506

[12] HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning - NeurIPS 2026

Questions for authors

Please refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.