acceptodds
Under review as a conference paper at ICLR 2027

Historical Gradient Reconstruction for Adaptive Federated Continual Learning

Abstract

Federated continual learning (FCL) must balance the acquisition of new knowledge with the retention of earlier knowledge distributed across heterogeneous clients. Gradient-based task weighting offers a way to adapt this balance, but choosing the weights requires historical gradients from data distributed across clients. Private replay buffers provide only a local view of past data, while shared historical gradients can become inaccurate as clients update their models. We propose FedGRIT (Federated Gradient Reconstruction for Incremental Training), a replay-based framework that reconstructs historical gradients to guide adaptive local updates. By aggregating task-specific gradients and compact curvature information, FedGRIT enables clients to reconstruct historical gradients at their own changing models and adapt task weights without cross-client communication at every step. Periodic refreshes and two-stage curvature compression support repeated sharing. We derive conditional historical-loss retention and weighted stationarity bounds under partial participation, separating the effects of stochasticity, local training, and client heterogeneity. Across four class- and domain-incremental benchmarks, FedGRIT achieves the highest mean final accuracy among the evaluated methods with ablations examining shared reconstruction and adaptive weighting.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
federated continual learning, historical gradient reconstruction, adaptive task weighting, stability–plasticity trade-off

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qZgMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies federated continual learning in the shared-model setting, where clients must learn new tasks while preserving knowledge from previous tasks distributed across heterogeneous participants. The proposed method, FEDGRIT, reconstructs historical task gradients during local training by combining shared replay gradients with compressed curvature information, and then adaptively balances historical and current-task gradients via task weighting. The paper also provides theoretical results on historical-loss retention and weighted stationarity under partial participation. Empirically, the method is evaluated on four class- and domain-incremental benchmarks, where it achieves the best reported mean final accuracy among the compared baselines, along with component and sensitivity studies on shared reconstruction and adaptive weighting.

Critical strengths
  1. The paper addresses a meaningful problem in federated continual learning, namely how to balance learning new tasks with retaining historical knowledge when past information is distributed across heterogeneous clients. This problem setting is practically relevant and well motivated.
  2. The method design has a reasonable degree of novelty. Rather than relying only on local replay, FEDGRIT combines shared historical gradient information with compressed curvature information to reconstruct historical gradients during local training, and further uses adaptive task weighting to coordinate historical and current-task updates. This makes the proposed approach conceptually coherent and aligned with the paper’s motivation.
  3. The empirical results are generally strong. The paper reports favorable final average accuracy across four class- and domain-incremental benchmarks, and also provides component analysis on shared reconstruction and adaptive weighting, which offers some support for the effectiveness of the proposed method.
Critical weaknesses
  1. The introduction is currently not sufficiently convincing. In particular, the literature coverage is limited and does not provide a structured overview of existing FCL approaches and their respective limitations. As a result, the motivation for focusing specifically on adaptive task weighting with reconstructed historical gradients is not developed strongly enough. The logical flow is also somewhat abrupt: the text moves quickly from the general CL/FCL setting to the proposed method, without clearly establishing why existing replay-based, gradient-sharing, or multi-objective approaches are insufficient.
  2. A key concern is the reliance on Hessian-related curvature information. Although the paper uses low-rank approximations and Hessian-vector products instead of explicitly forming the full Hessian, the practical overhead may still be substantial, especially with increasing model size, number of historical tasks, and periodic summary refreshes. The current manuscript does not make this cost sufficiently transparent. I would encourage the authors to clarify the actual computational and communication overhead, including the number of HVPs, the role of the rank parameter, and the runtime/memory trade-off relative to first-order baselines and ablated variants.
  3. The paper does include ablation-style analysis, especially the comparison in Table 2 between local replay vs. shared reconstruction and fixed vs. adaptive weighting. However, the ablation study is still somewhat limited. Since the core novelty of the method lies in curvature-based reconstruction and compressed second-order information sharing, I would appreciate more fine-grained ablations on the curvature term, the rank parameter, recompression, and the associated accuracy-efficiency trade-off.
  4. I suggest that the authors further clarify the adaptation details and fairness settings of the baselines used in the experiments. Since some methods were adapted from their original formulations to the sequential / replay / federated setting, the main paper should explicitly state whether these adaptations were conducted under comparable resource constraints, training budgets, and hyperparameter tuning efforts. This would help improve the transparency and persuasiveness of the experimental comparisons.
  5. The correspondence between the theoretical analysis and the actual algorithm is not yet sufficiently clear. The method itself relies on several key approximation steps, including historical gradient reconstruction, curvature approximation, and approximate weight optimization via Frank-Wolfe, whereas the theoretical derivations in the main text seem to adopt a more idealized update form. Therefore, I would appreciate it if the authors could further clarify whether Theorems 1-2 genuinely characterize the practical implementation of FEDGRIT, or whether they apply only to an idealized variant. This point is important, since these approximation steps are precisely core components of the proposed method.
Questions for authors
  1. The introduction is not yet fully convincing. The literature review is limited, and the motivation for focusing on adaptive weighting with reconstructed historical gradients is not developed strongly enough.
  2. A key concern is the reliance on Hessian-related curvature information. Although low-rank approximations are used, the practical computation and communication overhead remains unclear.
  3. The paper includes some ablation analysis, but it is still limited. More fine-grained ablations on the curvature term, rank choice, recompression, and the accuracy-efficiency trade-off would strengthen the paper.
  4. The fairness of the experimental comparison should be clarified. Since several baselines were adapted to the sequential / replay / federated setting, the main paper should explain whether they were compared under similar resources, training budgets, and tuning effort.
  5. The connection between the theoretical analysis and the actual algorithm is not fully clear. Since the method relies on several approximation steps, it would be helpful to clarify whether Theorems 1-2 characterize the practical FEDGRIT method or only an idealized variant.
6 comments
AuthorsOpenReview ↗

Response to Reviewer qZgM Part I

W1: Motivation and Positioning Relative to Prior Work

Response:

We appreciate the suggestion to connect the literature and our design choices more explicitly. We have revised the Introduction to explain the progression from existing retention strategies to adaptive task balancing and the historical-gradient information it requires.

Existing FCL methods retain knowledge through distillation or regularization, task-specific parameter sharing, and replay. Our motivation for adaptive weighting is that the relationships between current and historical tasks change across clients and training stages. Gradient-based weighting responds to their norms and alignment, building on Pareto-based CL, federated multi-objective optimization, and adaptive FCL methods. Avoiding manual coefficient selection is therefore not our central contribution.

The specific difficulty is obtaining informative historical gradients for these decisions throughout local training. Local replay captures only one client’s history. C-FLAG broadens this coverage through a shared, pooled historical gradient and adaptive old/current learning rates, but does not provide separate historical-task gradient estimates at subsequent local models. Federated multi-objective methods coordinate accessible objectives, while centralized past-gradient reconstruction does not address sharing historical information across separately evolving clients.

FedGRIT’s contribution is a task-specific, shared reconstruction-and-weighting protocol. Clients aggregate historical gradients and compact curvature summaries at a common model, then reconstruct historical gradients along their own local trajectories to recompute task weights. Aggregation addresses limited local coverage; curvature correction accounts for local displacement; and separate task summaries support task-specific decisions. Two-stage compression and periodic refreshes support repeated sharing without cross-client queries at every local step.

We have made this motivation and contribution explicit in Section 1, with revisions highlighted in blue. The broader literature discussion remains in Appendix A.

W2: Computational, Communication, and Memory Costs

Response:

We agree that avoiding an explicit Hessian does not, by itself, establish low practical overhead. We clarify the measured costs, isolate the contributions of curvature and server recompression, and explain how the HVP budget and retained rank affect these costs.

Measured whole-method costs. Tables 9–10 in Appendix F.4 already report runtime, communication, and peak allocated GPU memory. The CIFAR-100 results, together with final accuracy from Table 1, are:

MethodACC (%)Time/round (s)Communication (MB)Peak GPU memory (GB)
FedAvg43.2127.785.734.40
FedCMOO45.0940.3214.236.22
FedPOCL47.0986.085.734.74
C-FLAG46.8259.3257.1912.65
FedGRIT54.6664.1347.8818.52

On CIFAR-100, FedGRIT improves accuracy over FedPOCL, the strongest baseline by mean ACC, by 7.57 pp, while reducing measured per-round runtime by 25.5%. Compared with C-FLAG, it improves accuracy by 7.84 pp with 8.1% higher per-round runtime. Thus, FedGRIT delivers accuracy gains at competitive measured runtime, with additional communication and memory requirements. The extra communication carries historical gradients and curvature factors; memory also includes the curvature-estimation workspace. Once shared summaries are available, historical-gradient reconstruction uses cached factors rather than replay backpropagation at every local step.

Isolating curvature and recompression costs. We additionally compare against two controlled variants: without curvature, which retains shared historical gradients and adaptive weighting, and without server recompression, which retains client-side compression but broadcasts all uploaded curvature factors.

MethodACC (%) ↑BWT (pp) ↑Time/round (s) ↓Peak Memory Usage (GB) ↓Communication Cost (MB) ↓
No Curvature40.19-13.9436.4612.03175.89
Remove recompression56.58-7.61192.9220.231889.275
FedGRIT54.81-6.0464.1018.52347.88

Retaining curvature improves ACC by 14.62 pp and BWT by 7.90 pp over the shared-gradient, no-curvature variant, quantifying the learning benefit alongside its additional costs. Server recompression substantially reduces those costs: compared with broadcasting all uploaded factors, it reduces communication by 81.6% and measured per-round runtime by 66.8%, with 1.77 pp lower ACC and 1.57 pp higher BWT. These comparisons establish complementary roles: curvature improves the historical information used for learning, while recompression controls the cost of distributing and applying that information.

AuthorsOpenReview ↗

Response to Reviewer qZgM Part II

(Continued from our response to W2.) HVP budget, rank, and scaling. Algorithm 2 uses one HVP operator evaluation per Lanczos iteration. With the ten-iteration cap specified in Appendix F.2, this requires at most ten HVPs per historical task per participating client at a refresh, forty when four historical tasks are represented. Only participating clients perform these computations.

The retained rank controls factor size and application cost, whereas the Lanczos iteration budget controls the number of HVP evaluations. Reducing rank therefore does not necessarily reduce HVP count when that budget remains unchanged. Between curvature refreshes, local reconstruction applies cached factors without additional HVPs.

For a model with parameter dimension dd and retained rank rr, a complete historical-summary refresh requires, per participating client in each communication direction, (K−1)(dr+r+d)(K-1)(dr+r+d) scalars, excluding model transmission and model-update upload. Each historical-task curvature correction requires O(dr)\mathcal{O}(dr) operations per local step. Thus, summary size and reconstruction work grow linearly with model dimension and historical-task count for fixed rank. DSHA keeps each recipient’s curvature representation independent of the number of participants, although total traffic and server computation still depend on that number. Refresh frequency controls how often curvature estimation and exchange occur.

We have included these matched ablations and their measurement details in Appendix F.3, and make the HVP budget and cost accounting explicit in Appendix F.5, with revisions highlighted in blue.

W3: Fine-Grained Ablations and Accuracy–Efficiency Trade-offs

Response:

The additional component comparisons reported in our response to W2 isolate curvature correction and server recompression. Together with the rank study in Appendix F.3 Table 7, these experiments address the requested fine-grained ablations and quantify their accuracy–resource trade-offs.

Curvature correction. The no-curvature variant retains shared historical gradients and adaptive weighting while removing curvature estimation, transmission, and correction. Compared with this variant, FedGRIT improves ACC by 14.62 pp and BWT by 7.90 pp. Unlike the original Table 2 comparison against local replay, this experiment retains historical-gradient sharing and therefore isolates the additional benefit of curvature correction. The table in our response to W2 reports the corresponding runtime, communication, and memory costs.

Server recompression. The variant without server recompression retains client-side low-rank compression but broadcasts all uploaded curvature factors rather than reducing their aggregate to the target rank. Relative to this variant, FedGRIT reduces reported communication by 81.6% and measured per-round runtime by 66.8%, with 1.77 pp lower ACC and 1.57 pp higher BWT. This comparison demonstrates substantial resource savings from recompression together with its measured performance trade-off.

Retained rank. We evaluate ranks 1, 3, 5, and 10, reporting accuracy, BWT, runtime, and peak GPU memory. Increasing rank from 1 to 3 improves ACC by 3.51 pp and BWT by 5.09 pp. Further increases to ranks 5 and 10 incur additional costs without improving ACC or BWT in this experiment. For example, rank 10 requires 79.12 seconds per round and 37.05 GB of peak GPU memory, compared with 64.10 seconds and 18.94 GB for rank 3. Thus, rank 3 provides the strongest measured learning performance among the tested configurations while avoiding the higher costs of larger representations.

Together, these experiments distinguish the components’ complementary roles: curvature correction improves learning beyond shared gradients alone; server recompression reduces the cost of distributing and applying curvature; and retained rank controls the representation-size/performance trade-off. We have incorporated the additional ablations and resource measurements into Appendix F.3, with additions highlighted in blue.

W4: Baseline Adaptations and Fairness of Comparisons

Response:

The submitted paper reports the shared experimental setup in Section 5 and additional configurations and baseline-specific hyperparameters in Appendix F.2, including Table 4. We clarify below how the baselines were adapted, which training and replay budgets were matched, and how hyperparameters were selected. We will also make these details more explicit in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer qZgM Part III

(Continued from our response to W4.) Matched training schedules and replay budgets. All methods use the same backbone, client-participation, 50 communication rounds per task, and 3 local epochs per round. Replay-based methods use the same total capacity of 400 examples per client and the same sample-selection policy. These settings match the training schedule and retained-example budget; they do not impose identical computational or communication costs. We report measured runtime, communication, and peak GPU memory separately in Appendix Tables 9-10.

Baseline adaptations. FSMGDA and FedCMOO are adapted to sequential training by evaluating historical objectives using retained replay samples. FedPOCL and FedBECAME are our federated adaptations of centralized continual-learning methods. For the federated adaptations, each client's local training follows the original method, while the resulting models are uploaded to the server for aggregation and global evaluation. Upon task completion, FedPOCL updates clients' buffers using their private datasets, while FedBECAME locally computes θk,mGP\theta_{k,m}^{GP}, θ^k,m∗\hat{\theta}_{k,m}^*, the precision matrix, and the merged model θk,m∗\theta_{k,m}^*. These quantities remain private and are not shared with the server. Therefore, their federated adaptations primarily introduce server-side aggregation and global evaluation of locally trained models. We will document these adaptations individually in Appendix F.2.

Hyperparameter selection. Baseline-specific hyperparameters follow the original papers’ recommended settings or the values explicitly reported in Table 4, without an extensive baseline-specific search. We report the detailed results across different refresh intervals and different ranks below. The results show that the configuration with a refresh interval of 3 rounds achieves the highest ACC and best BWT, reducing runtime by 29.9% compared with the frequency-1 setting. Similarly, the result with rank 33 achieves the best ACC and BWT, while avoiding the substantial memory and runtime overhead of higher ranks. We report the configuration-selection results alongside the parameter values below.

FrequencyACC (%)BWT (pp)Time (s)Peak Memory Usage (GB)
150.66-10.2691.3818.94
354.47-5.4064.1018.94
551.57-9.2853.5018.94
1045.56-15.6251.9418.94
RankACC (%)BWT (pp)Time (s)Peak Memory Usage (GB)
150.96-10.4957.8713.37
354.47-5.4064.1018.94
551.54-9.5365.1724.50
1050.39-10.2679.1237.05

The matched training schedules do not imply identical computational or communication costs: these depend on each method’s operations and auxiliary state. Appendix Tables 9-10 report the measured runtime, communication, and peak GPU memory separately, and we expand Appendix F.2 with the adaptation and parameter-selection details, highlighting the revisions in blue.

W5: Correspondence Between Theoretical Guarantees and Practical Implementation

Response:

The analysis retains FedGRIT’s compressed-curvature correction in the reconstructed-gradient updates, and it does not assume access to exact Hessians. The numerical idealization for task weighting is stated in Section 3.2. We clarify their respective roles below.

Compressed curvature is retained in the analysis. Lemmas 1–2 establish ∥H~v∥2≤L2∥v∥2\|\widetilde{\boldsymbol H}v\|^2\le L^2\|v\|^2. The proofs use this bound together with local-model displacement to control the curvature correction. Thus, they accommodate a bounded compressed representation rather than requiring the reconstructed gradient to equal the exact historical gradient. This is an operator-bound analysis, not a rank-dependent reconstruction-accuracy analysis.

Task weighting is analyzed at the subproblem optimum. Section 3.2 writes the analyzed updates using an exact minimizer of Eq. (5), whereas the implementation approximates this solution using Frank–Wolfe. This formulation is consistent with Federated Multi-Objective Learning [1], whose Algorithm 1 likewise defines the update through the minimum-norm subproblem optimizer. The present bounds do not separately quantify finite Frank–Wolfe solver error.

Under these assumptions, Theorem 1 bounds cumulative historical-loss deterioration subject to its alignment condition, while Theorem 2 characterizes weighted stationarity and the contributions of stochasticity, local training, and client heterogeneity. These results characterize the reconstructed-gradient updates at the exact-weight level, rather than certify a particular numerical solver tolerance.

We will make this correspondence explicit at the beginning of Section 4 and clarify the distinction between analyzed and implemented weights in Section 3.2, with revisions highlighted in blue.

[1] Federated Multi-Objective Learning, NeurIPS 2023.

Reviewer_n97YOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper propose FedGRIT, which learn to solve catastrophic forgetting in Federated Continual Learning. Specifically, they assume clients share task sequences with synchronized task transitions, while their data distributions may differ, and old data is saved within limited replay buffer. Each communication round, they make each client compute gradient, then historical gradient using first-order approximation for each task. The server then aggregates these approximation and compresses the curvature for each task. During local training, each client uses a first-order Taylor approximation to reconstruct historical gradients at its changing model and adaptively selects task weights through a minimum-norm optimization.

Critical strengths
  1. The proposed approach is well-motivated by the heavily discussed problem of. The work uses an intuitive and simple way to explain the challenge.
  2. The components are well-motivated, the algorithm is clear and help understand high level flow of the work.
Critical weaknesses
  1. The work novelty may not guarenteed. Many of its component are inheritated from other paper. For example, as the author already acknowledge "Gradient-based task balancing coordinates current- and previous-task gradients" in [1,2]. Also, Using Pareto optimization to combine gradients of historical tasks have been learned in [3]. Minimum-norm gradient combinations in federated learning with local updates have been discussed in [4].
  2. The work is now assume share task sequences with with synchronized task transitions. The algorithm now heavily rely on this assumption, with "Each participating client computes and uploads replay gradients and Lanczos curvature" per task, and "Server averages the replay gradients and recompresses each task’s curvature". This raise question how will the method work in heterogeneous data setting [5]
  3. The task-dependent design as explained above also raise question about how the proposed method work under a lot of task (500 to 1000 tasks). It is valuable if the authors can provide additional experiments and discussion on this problem.
  4. The propose method using Hessian as first-order approximation instead of full Hessian. However, that still need to be calculated for each task, on each client per communication round. If more clients join the system (100 clients or more), how much time and computing it will cost, can will the communication cost raise significantly ? This raise question about stability.
  5. The paper need more analysis to check if the method works as they want. For example, does the corrected gradient actually become more accurate? Do reconstruction errors change the weights or update direction in harmful ways? What happen when shared gradients become stale?
  6. The work state "private replay buffer of 400 examples per client" without details about how the sample selected. That raise serious question if the method can works with more task with more data come.

[1] Mitigating catastrophic forgetting in online continual learning by modeling previous task interrelations via pareto optimization - ICML 2024

[2] Improving continual learning by accurate gradient reconstructions of the past - TMLR 2023

[3] Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization - ICML 2024

[4] Federated Multi-Objective Learning - Neurips 2023

[5] HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning - Neurips 2026

Questions for authors

Please refer to the Critical Weaknesses section.

7 comments
AuthorsOpenReview ↗

Response to Reviewer n97Y Part I

W1: Novelty and Distinction from Prior Work

Response:

We respectfully disagree with the novelty assessment, while recognizing that the distinction from prior work should be clearer. FedGRIT’s contribution is a distributed reconstruction-and-weighting protocol that makes shared, task-specific historical information usable at each client’s evolving local model without cross-client queries at every local step.

The cited works provide important foundations, and all are acknowledged in our submission:

  • POCL [1,3] models relationships among historical tasks through Pareto optimization in centralized CL.
  • Daxberger et al. [2] improve past-gradient reconstruction by combining replay and regularization in centralized CL.
  • Federated Multi-Objective Learning [4] aggregates objective-specific accumulated local updates and computes minimum-norm weights at the server (Algorithm 1).

The additional problem addressed by FedGRIT is supplying shared historical-gradient estimates during federated local adaptation. Local replay provides only one client’s historical information, whereas gradients shared at the round’s initial model do not track subsequent local updates. FedGRIT aggregates task-specific gradients and curvature summaries, then evaluates the resulting reconstructions along each client’s local trajectory for per-step task weighting. Client compression and server recompression keep these shared representations compact.

Table 2 supports this design: shared reconstruction improves ACC over local replay by 4.27 percentage points under fixed weighting and 7.24 points under adaptive weighting. With adaptive weighting, it also improves BWT by 6.65 points. Section 4 additionally analyzes conditional historical-loss retention and weighted stationarity under partial participation.

Our contribution therefore lies in the distributed learning protocol and its analysis and evaluation, not new minimum-norm solvers or matrix factorizations. We have clarified this distinction in the Introduction and Section 3.3, with revisions highlighted in blue.

W2: Client Heterogeneity and Task Synchronization

Response:

We appreciate the reference to HERO [5]. We would like to clarify an important distinction: a shared task sequence does not imply homogeneous client data. FedGRIT supports heterogeneous client data while assuming a shared task sequence and synchronized transitions. These are separate dimensions: as HERO [5] distinguishes, client data skew need not imply task-order mismatch.

Synchronization aligns task identities and the current-historical task split, not client distributions or curvature. Clients can hold different examples and class proportions within the same task. The server aggregates their replay gradients and curvature representations, and each client evaluates the shared reconstruction at its own evolving model. DSHA does not require a common client eigenspace. Sharing these summaries broadens the historical information available to each client beyond its own replay buffer.

The submitted experiments already evaluate this non-IID setting: Table 1 uses Dirichlet partitions with concentration 1 for the class-incremental benchmarks and 0.5 for the domain-incremental benchmarks.

Client-specific task orders raise a distinct issue: the same task may be historical, current, or unseen at different clients. Supporting that setting requires changes to task-dependent aggregation and local weighting, and it is not covered by the present protocol or guarantees.

We have clarified the role and scope of synchronization in Section 2 and discussed HERO in Appendix A.2, with revisions highlighted in blue.

W3: Scalability to Longer Task Sequences

Response:

We thank the reviewer for highlighting the importance of longer task sequences. We have extended the evaluation from five tasks to ten and twenty tasks while keeping the total replay budget fixed at 400 examples per client. Specifically, we evaluate CIFAR-100 with five tasks of twenty classes, ten tasks of ten classes, and twenty tasks of five classes, using matched data partitions and training settings across methods within each configuration.

MethodDataset SplitACC (%) ↑BWT (pp) ↑Total Time (h) ↓Communication Cost (MB) ↓
FedPOCL10×1030.01-17.9414.5385.73
FedPOCL20×522.50-20.0428.6985.73
FedGRIT10×1035.76-15.3712.291717.95
FedGRIT20×526.88-21.6424.823297.96
AuthorsOpenReview ↗

Response to Reviewer n97Y Part II

(Continued from our response to W3.) FedGRIT’s accuracy advantage persists on both longer sequences. It improves ACC over FedPOCL by 5.75 and 4.38 percentage points at ten and twenty tasks, respectively, while reducing measured total training time by 15.4% and 13.5%. BWT improves by 2.57 pp at ten tasks. At twenty tasks, FedPOCL has better BWT by 1.60 pp despite its lower final accuracy. These comparisons demonstrate that FedGRIT’s accuracy benefit extends beyond the original five-task setting, even as the same replay budget must represent more historical tasks.

Task count also affects resource requirements. For fixed model size and rank, historical-summary storage and communication per refresh, as well as reconstruction work per local step, grow linearly with the number of historical tasks. Low-rank compression controls the representation size of each task, but does not eliminate this task-count dependence. The table above reports the accompanying communication costs alongside learning performance and measured runtime.

Regarding 500–1000 tasks, CIFAR-100 and ImageNet100 each contain 100 classes. Such an evaluation would require a different dataset or task construction, rather than further partitioning these benchmarks into disjoint new-class tasks. Our empirical conclusion is therefore scoped to sequences of up to twenty tasks, with the resource scaling stated separately.

We have included the longer-sequence results, configuration details, and resource-scaling discussion in the revised Appendix F.3, with additions highlighted in blue.

W4: Computational and Communication Costs with More Clients

Response:

We would like to clarify a possible misunderstanding regarding the distinction between the total client population and the number participating in each round. Only participating clients compute and upload curvature summaries. A federation with 100 clients and eight participants per round therefore processes eight clients’ summaries, not 100. For fixed model size, task count, rank, and local replay workload, increasing the total population alone does not increase this per-round workload.

FedGRIT uses Lanczos with Hessian-vector products rather than explicitly constructing full Hessians. DSHA then recompresses the aggregated factors, keeping each recipient’s historical-summary payload independent of the number of participants. Increasing simultaneous participation increases aggregate summary traffic linearly and also increases server processing, and recompression controls the broadcast representation rather than eliminating these costs.

The submission already evaluates both client population and participation. Figures 7–8 in Appendix F.3 report loss and accuracy trajectories for populations up to 64 clients under fixed and proportional participation. Tables 9–10, discussed in Appendix F.4, report total measured costs. Under the default configuration of 16 clients and eight participants, FedGRIT takes 64.1 seconds per round and communicates 347.88 MB per selected client per round on CIFAR-100, including upload and download.

We have clarified the distinction between total population and per-round participation in Section 3.3, with the revision highlighted in blue.

W5: Reconstruction Accuracy, Effects on Adaptive Updates, and Staleness

Response:

We have added checkpoint-level diagnostics to examine historical-gradient reconstruction and the resulting learning outcomes, alongside the existing refresh-interval study.

Reconstruction accuracy. At matched intermediate local models, we compare FedGRIT’s reconstructed gradients with shared historical gradients reused without curvature correction. The reference gradients are evaluated directly at those same models using the same contributing clients’ replay samples and aggregation rule, holding historical coverage fixed. We measure cosine similarity to quantify agreement with the reference gradient direction.

Current TaskRoundNo CurvatureFedGRIT
420.2150.244
4480.5590.663
520.2470.328
5480.5430.706

Curvature correction improves directional agreement in all four reported task–round settings, with gains ranging from 0.029 to 0.163. These results support its intended role: adjusting the shared historical gradients toward their directions at the receiving clients’ evolving models.

Learning outcomes after adaptive weighting. We also compare mean historical-task loss at the corresponding task–round checkpoints. Both variants use adaptive weighting; they differ in whether the shared historical gradients receive curvature correction.

Current TaskRoundNo CurvatureFedGRIT
421.94031.8869
4481.98401.9403
522.13522.0888
5482.11202.0800

Response to Reviewer nYwD Part III

AuthorsOpenReview ↗

Response to Reviewer n97Y Part III

(Continued from our response to W5.) FedGRIT yields lower historical-task loss in all four settings, with differences of 0.0320–0.0534. These results complement the directional diagnostic. Moreover, the matched end-to-end curvature ablation improves final ACC by 14.62 pp and BWT by 7.90 pp over shared historical gradients with adaptive weighting but no curvature correction. Together, these measurements show that the observed reconstruction benefits are accompanied by improved learning outcomes, rather than being confined to gradient similarity.

Staleness. Curvature correction accounts for model movement within local training, while periodic refreshes update shared information across rounds. In the implementation, Hessian information in Q~K,it\tilde{Q}_{K,i}^t and Γ~K,it\tilde{\Gamma}_{K,i}^t would be refreshed, but the previous samples stored in clients' buffers are cached consistently. Figure 2 evaluates curvature-refresh intervals of 1, 3, 5, and 10 rounds: the ten-round interval produces pronounced loss oscillations, whereas shorter intervals yield smoother trajectories. These results support periodic refreshes rather than indefinite reuse of curvature information.

We have included the diagnostic protocols and results in the revised Appendix F.3 and clarify the refresh mechanism in Section 3.3, with additions highlighted in blue.

W6: Replay-Buffer Sampling and Rebalancing as Tasks Arrive

Response:

Each client maintains a total replay budget of 400 examples, not 400 per task, using task-balanced uniform sampling. At each task boundary, this budget is divided approximately equally among the observed tasks. Earlier-task samples are uniformly subsampled without replacement from their existing buffers, while samples from the newly completed task are uniformly selected without replacement from its local data. This procedure requires neither access to the original earlier-task datasets nor additional sample-scoring procedures.

As new tasks arrive, the existing budget is redistributed rather than expanded. Thus, additional incoming data do not increase the number of retained examples. Instead, the per-task allocations decrease to accommodate new tasks while preserving representation of earlier tasks in the evaluated sequences.

Figure 4 in Appendix F.3 evaluates total budgets of 100, 400, 700, and 1000 examples on CIFAR-100. The corresponding accuracy and resource measurements are summarized below.

Buffer SizeACC (%)BWT (pp)Time (s/round)Peak Allocated Memory (GB)
10050.59-10.1359.1318.94
40054.47-5.4064.1018.94
70055.39-4.5265.0518.94
100056.19-3.8681.0418.94

Increasing the budget from 100 to 400 improves ACC by 3.88 pp and BWT by 4.73 pp. With 400 examples, accuracy is within 1.72 pp of the 1,000-example setting while retaining 60% fewer samples and reducing measured per-round runtime from 81.04 to 64.10 seconds. These results support 400 examples as a practical replay-budget choice for the evaluated sequences; larger budgets provide further accuracy and retention gains. We have clarified the replay-buffer sampling and rebalancing procedure in Section 3.3, with the revisions highlighted in blue.

We also test this policy as the task sequence grows, without increasing the replay budget. The additional experiments reported in our response to W3 retain the same total budget of 400 examples per client for both ten- and twenty-task CIFAR-100 sequences. FedGRIT achieves ACC of 35.76% and 26.88%, respectively, exceeding FedPOCL by 5.75 and 4.38 pp under the matched configurations. These results show that its accuracy advantage persists in the tested longer sequences despite smaller replay allocations per task.

Reviewer_nYwDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

FedGRIT is a replay-based FCL method. In each round, sampled clients upload per-old-task replay gradients and rank-r Lanczos Hessian factors to the server, and the server averages the gradients and re-compresses the curvature using the DSHA algorithm. During local training, clients reconstruct each old-task gradient with a first-order Taylor correction and solve an MGDA min-norm problem via Frank-Wolfe at each local step to weight the task-wise gradients. The proposed gradient reconstruction at the client is motivated by the need for an accurate historical gradient that is relevant across all clients.

Critical strengths

This work addresses the limitations of local replay in heterogeneous federated settings and provides a principled approach to incorporating global replay information into local updates.

Critical weaknesses

The paper fails to clearly motivate the usage of curvature information in the proposed algorithm. The authors should clearly justify why a second-order gradient correction is preferable to any first-order gradient correction method. An experimental validation for this motivation is also missing. The authors should perform an experiment to isolate the curvature term by comparing FEDGRIT to its no-curvature version. Lanczos, DSHA, and extra communication, computation, and storage costs associated with the Hessian can be avoided if it is not shown to provide any evident advantages. The curvature has also not been shown to provide any theoretical advantage in the paper.

Moreover, Is the shared curvature information robust to client heterogeneity? Clients truncate to rank r before averaging at the server. So DSHA is applied to the truncated Hessians, and under non-IID data, client top-r subspaces may differ and shared directions may be lost.

Does per-client, per-step MGDA provide a valid common-descent direction? The min-norm guarantee holds locally at a client, and the FedAvg of these local min-norm directions is not a common descent direction for tasks 1..k globally. Does a server-computed weight make more sense? Also, MGDA favors small-norm old-task gradients, which can suppress current-task learning. Does this affect the plasticity?

In the current design, scaling with tasks is a weak point that has not been fully evaluated. Communication, Lanczos compute and memory all grow linearly. Experiments use only 5 tasks. At least a 10 or 20 task sequence is needed to support the claim that the method is practical.

Privacy is also weakened relative to standard-FL because, in every round, each client uploads a separate gradient and Hessian eigenvectors for each old task, computed on buffer samples. This is the regime in which gradient-inversion attacks are most effective. The same small buffer is exposed repeatedly across rounds.

The novelty of each component in the design should be clearly established. Most of the components are existing ideas; therefore, the advantage and the interplay among its components in the FL setup should be justified.

There is some doubt regarding the accuracy numbers. It seems similar to the numbers usually obtained for task incremental learning and not class incremental learning. Authors are urged to check the results carefully.

Minor comments:

In the problem setting, E is defined as local gradient steps, whereas in the experimental setup it is defined as local epochs

Questions for authors

Address the weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer nYwD Part I

W1: Motivation and justification for curvature

Response:

We agree that the additional cost of curvature must be justified against a gradient-only alternative. Curvature serves a specific purpose in FedGRIT: adjusting shared historical gradients to each client’s changing model during local training.

Sharing replay gradients at the common model broadens historical coverage beyond a client’s private buffer. However, reusing these gradients unchanged does not account for subsequent local updates. This matters because adaptive weighting depends on gradient norms and alignment: stale historical gradients can distort the relationships used to select the weights. The reconstruction in Eq. (4) uses shared curvature to estimate this change locally, allowing one shared summary to support different client trajectories without communication at every step.

A natural gradient-only alternative is to correct the shared gradient using the receiving client’s local replay-gradient change. This can track local variation, but under heterogeneous data, the local change need not match the change in the shared historical gradient. FedGRIT instead approximates the latter directly.

The reconstruction also admits a local error characterization. Let

δ=θK,mt,e−θKt,εH=∥H~K,it−∇2f^it(θKt)∥op,\delta=\theta_{K,m}^{t,e}-\theta_K^t,\qquad \varepsilon_H= \left\|\tilde{\boldsymbol H}_{K,i}^t-\nabla^2\widehat f_i^t(\theta_K^t)\right\| _{\mathrm{op}},

where the operator norm is induced by the Euclidean norm. Under an additional locally Hessian-Lipschitz condition on the shared replay loss, with constant LHL_H along the segment connecting the two models,

∥∇f~i(θK,mt,e)−∇f^it(θK,mt,e)∥≤εH∥δ∥+LH2∥δ∥2.\left\|\nabla\tilde f_i(\theta_{K,m}^{t,e})-\nabla\widehat f_i^t(\theta_{K,m}^{t,e})\right\|\le\varepsilon_H\|\delta\|+\frac{L_H}{2}\|\delta\|^2.

This follows from the Taylor remainder bound and the triangle inequality. Whereas the usual gradient-Lipschitz bound for an unchanged shared gradient is linear in displacement, exact curvature removes that leading linear term; compression introduces the additional curvature-error term above. This characterizes a potential reconstruction benefit, not a universally better convergence rate, and does not remove replay or client-sampling error.

We isolate curvature through a matched no-curvature comparison. This variant retains shared historical gradients and adaptive weighting but removes the curvature correction, Lanczos, DSHA, and curvature transmission.

On CIFAR-100, curvature correction increases ACC from 40.19% to 54.81% (+14.62 pp) and improves BWT from -13.94 to -6.04 pp (+7.90 pp). Because both variants retain shared historical gradients and adaptive weighting, these gains isolate the additional benefit of adjusting historical gradients as local models evolve. This improvement comes with additional resource use: runtime increases from 36.46 to 64.10 seconds per round, peak memory from 12.03 to 18.52 GB, and communication from 175.89 to 347.88 MB under the reported measurement protocol. Removing curvature therefore reduces resource use but substantially degrades both final accuracy and BWT in this comparison, supporting its inclusion in FedGRIT.

MethodACC (%) ↑BWT (pp) ↑Time/round (s) ↓Peak Memory Usage (GB) ↓Communication Cost (MB) ↓
No Curvature40.19-13.9436.4612.03175.89
FedGRIT54.81-6.0464.1018.52347.88

We have clarified the motivation for curvature and its role in adaptive weighting in Section 3.1, with revisions highlighted in blue.

W2: Heterogeneous client curvature and information lost through truncation

Response:

FedGRIT does not require clients to share a common eigenspace. However, the reviewer is correct that truncation before aggregation can discard directions important to the uncompressed aggregate. These are distinct issues: combining different retained subspaces and controlling the information lost through compression.

In Eq. (9), DSHA averages the matrices represented by the uploaded factors, not individual eigenvectors or matching eigenvector indices. The QR construction in Algorithm 4 of Appendix C.3 expresses this aggregate in an orthonormal basis without changing the represented matrix. The subsequent eigen-decomposition selects dominant directions of the aggregate. Thus, different client subspaces can contribute to the shared representation, although components absent from the uploaded span cannot be recovered.

AuthorsOpenReview ↗

Response to Reviewer nYwD Part II

(Continued from our response to W2.) The two compression stages admit the following error decomposition:

\left\| \tilde{\boldsymbol H} _{K,i}^t -\nabla^2\widehat f_i^t(\theta _K^t) \right\| _{\mathrm{op}} } _{\varepsilon _H} \le \varepsilon _{\mathrm{client}} + \varepsilon _{\mathrm{server}}.$$ The client term is the average operator-norm error between each uploaded approximation and its uncompressed replay Hessian. The server term measures the difference between the DSHA output and the average uploaded matrices. This triangle-inequality bound requires no common eigenspace and connects both compression stages to the reconstruction-error bound in our response to W1. **The submitted experiments support effective learning with compact curvature under the evaluated heterogeneity.** Table 1 reports the highest mean final accuracy across four benchmarks with non-IID client partitions. Figure 5 evaluates ranks 1, 3, 5, and 10 on non-IID CIFAR-100; rank 3 achieves the best plotted accuracy, and larger ranks do not consistently improve performance. These are end-to-end learning results, rather than a claim that compression preserves every globally important direction. We have clarified the subspace aggregation mechanism and the two sources of approximation error in Section 3.3, highlighted in blue. ## W3: Local MGDA, server-computed weights, and plasticity **Response:** We thank the reviewer for distinguishing local gradient coordination from global common descent. **The MGDA common-descent property is local and does not automatically survive federated aggregation.** At an exact solution of Eq. (5), a nonzero minimum-norm direction has positive inner product with every supplied gradient at that client’s current model. These inputs are reconstructed historical gradients and a sampled current-task gradient. Consequently, this property alone does not guarantee descent for every true task loss, and the averaged client update is not guaranteed to be a global MGDA direction. Section 4 instead studies conditional historical-loss retention and weighted stationarity, not monotonic improvement of every task after every round. **Client-side weighting provides responsiveness to heterogeneous data and evolving local models.** Each client combines shared historical information with its own current-task gradient and recomputes the weights as their relationships change. Server-adaptive weighting is a meaningful alternative, but a single vector held fixed throughout a round cannot respond separately to these client-specific changes. Recomputing global weights at every local step would require additional coordination. Our design prioritizes this local responsiveness, and it does not assume that server-side weighting is inherently inferior. **Regarding plasticity, small historical gradients can constrain the minimum-norm direction and slow current-task learning.** Nevertheless, weights alone are not measures of task importance or learning progress, because progress depends on the combined update. We therefore examine acquisition performance. The controlled comparison in Table 2 provides relevant evidence. With shared reconstruction unchanged, adaptive weighting increases final ACC from 39.54% to 54.81% and improves BWT from -11.84 pp to -6.04 pp. Moreover, mean accuracy immediately after learning each task satisfies $$\frac{1}{K}\sum _{j=1}^{K}A _{j,j}=\mathrm{ACC} - \frac{K-1}{K}\mathrm{BWT}.$$ For this five-task experiment, the reported results correspond to approximately 49.01% with fixed weights and 59.64% with adaptive weights. Together, these results show that, on CIFAR-100, adaptive local weighting improves both new-task acquisition and historical retention relative to the fixed-weight variant with the same shared reconstruction. We have clarified the scope of the local MGDA guarantee and the rationale for client-side weighting in Section 3.2, immediately after Eq. (5), with the revisions highlighted in blue. ## W4: Scaling to longer task sequences **Response:** We have extended the CIFAR-100 evaluation from five tasks to **10 tasks with 10 classes each and 20 tasks with 5 classes each**, keeping the total replay budget fixed at 400 examples per client. We also evaluate FedPOCL, the strongest baseline by mean ACC in Table 1, under matched training settings within each configuration. | Method | Dataset Split | ACC (%) ↑ | BWT (pp) ↑ | Total Time (h) ↓ | Communication Cost (MB) ↓ | |---|---|---:|---:|---:|---:| | FedPOCL | 10×10 | 30.01 | -17.94 | 14.53 | 85.73 | | FedPOCL | 20×5 | 22.50 | -20.04 | 28.69 | 85.73 | | FedGRIT | 10×10 | 35.76 | -15.37 | 12.29 | 1717.95 | | FedGRIT | 20×5 | 26.88 | -21.64 | 24.82 | 3297.96 |
AuthorsOpenReview ↗

Response to Reviewer nYwD Part III

(Continued from our response to W4.) FedGRIT’s accuracy advantage persists at both sequence lengths. It improves ACC over FedPOCL by 5.75 and 4.38 pp at 10 and 20 tasks, respectively, while reducing measured total training time by 15.4% and 13.5%. BWT also improves at 10 tasks; at 20 tasks, FedPOCL has better BWT despite its lower final accuracy.

These experiments retain the same total replay budget while increasing the number of historical tasks that it must represent. Historical-summary communication grows with task count, as reflected in the reported costs. Nevertheless, the comparison shows that FedGRIT’s accuracy benefit extends to the requested longer sequences, with lower measured training time than FedPOCL in these runs.

We have included these results and their configuration and measurement details in the revised Appendix F.3, with additions highlighted in blue.

W5: Privacy implications of sharing historical summaries

Response:

We agree that repeated task-specific gradient and curvature releases can introduce additional inference risk. Neither a small replay buffer nor low-rank compression is a privacy safeguard.

FedGRIT keeps raw samples local, but the server receives individual client summaries before aggregation, while participating clients receive aggregated summaries. This does not establish protection against inference from those summaries, and we do not claim privacy equivalence to standard FL or resistance to gradient-inversion attacks.

Our contribution concerns adaptive learning from distributed historical information under data-locality constraints, rather than a formal privacy mechanism. We have clarified this distinction and the visibility of transmitted information in Section 3.3, with revisions highlighted in blue.

W6: Novelty and interplay among components

Response:

FedGRIT’s contribution is a distributed reconstruction-and-weighting protocol that makes shared historical information usable throughout local adaptation. We build on established past-gradient reconstruction and minimum-norm task balancing, using standard Frank–Wolfe, Lanczos, and QR/eigendecomposition routines rather than claiming these tools as individually novel.

The additional challenge in FCL is twofold: a client’s replay buffer provides only its own historical information, while shared gradients computed at the beginning of a round do not account for subsequent local updates. FedGRIT addresses these limitations jointly through task-specific shared summaries that clients evaluate along their own local trajectories.

The components have complementary roles. Aggregation broadens historical coverage; curvature correction adjusts the shared estimates to local displacement; and separate historical-task reconstructions allow adaptive weighting to distinguish their contributions. Client compression limits uploads, while server recompression keeps each task’s broadcast representation independent of the number of participants. Together, these operations support per-step reconstruction and weighting without further cross-client queries.

Table 2 provides evidence for this interplay. Replacing local replay with shared reconstruction improves ACC by 4.27 pp under fixed weighting and by 7.24 pp under adaptive weighting. With adaptive weighting, shared reconstruction also improves BWT by 6.65 pp. In this ablation, the larger ACC gain with adaptive weighting supports the complementary roles of the two components.

Section 4 additionally studies conditional historical-loss retention and weighted stationarity for these adaptive federated updates. Our contribution is therefore the specific distributed protocol and its analysis and evaluation, rather than new versions of the individual numerical routines.

We have sharpened this distinction in the Introduction and clarified the role of the standard numerical components in Section 3.3, highlighted in blue.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions and Additional Results

We thank all three reviewers for their detailed feedback. We have posted point-by-point responses and uploaded a revised manuscript with changes highlighted in blue. To facilitate review, we summarize the principal additions addressing concerns shared across the reviews.

  • Isolating the contribution of curvature. Main-paper Table 2 now includes a no-curvature variant that retains shared historical gradients and adaptive weighting. On CIFAR-100, curvature correction improves ACC by 14.62 pp and BWT by 7.90 pp. This comparison separates the benefit of curvature correction from historical-gradient sharing.

  • Quantifying reconstruction and resource trade-offs. Appendix Table 6 provides historical-gradient diagnostics, while Tables 7–8 report sensitivity and component-level accuracy, runtime, memory, and communication measurements. Relative to broadcasting all uploaded curvature factors, server recompression reduces reported communication by 81.6% and per-round runtime by 66.8%, with 1.77 pp lower ACC and 1.57 pp higher BWT. Section 3.3 and Appendix F.5 explain the distinct roles of the Lanczos HVP budget, retained rank, and refresh schedule.

  • Evaluating longer task sequences. Appendix Table 5 extends CIFAR-100 evaluation to ten and twenty tasks, retaining the same total replay budget of 400 examples per client. FedGRIT maintains an ACC advantage over FedPOCL of 5.75 pp and 4.38 pp, respectively. The table also reports retention and resource costs.

  • Clarifying the contribution and scope. Sections 1–3 distinguish the distributed reconstruction-and-weighting protocol from its established numerical components and separate within-task client heterogeneity from task-order mismatch. Sections 3.2 and 4 clarify the local nature of the minimum-norm direction property, the exact-weight formulation used in the analysis, and the conditions underlying the retention and weighted-stationarity results.

Our reviewer-specific responses provide the detailed comparisons and explanations. We welcome further questions about any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.