- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies federated continual learning in the shared-model setting, where clients must learn new tasks while preserving knowledge from previous tasks distributed across heterogeneous participants. The proposed method, FEDGRIT, reconstructs historical task gradients during local training by combining shared replay gradients with compressed curvature information, and then adaptively balances historical and current-task gradients via task weighting. The paper also provides theoretical results on historical-loss retention and weighted stationarity under partial participation. Empirically, the method is evaluated on four class- and domain-incremental benchmarks, where it achieves the best reported mean final accuracy among the compared baselines, along with component and sensitivity studies on shared reconstruction and adaptive weighting.
Critical strengths
- The paper addresses a meaningful problem in federated continual learning, namely how to balance learning new tasks with retaining historical knowledge when past information is distributed across heterogeneous clients. This problem setting is practically relevant and well motivated.
- The method design has a reasonable degree of novelty. Rather than relying only on local replay, FEDGRIT combines shared historical gradient information with compressed curvature information to reconstruct historical gradients during local training, and further uses adaptive task weighting to coordinate historical and current-task updates. This makes the proposed approach conceptually coherent and aligned with the paper’s motivation.
- The empirical results are generally strong. The paper reports favorable final average accuracy across four class- and domain-incremental benchmarks, and also provides component analysis on shared reconstruction and adaptive weighting, which offers some support for the effectiveness of the proposed method.
Critical weaknesses
- The introduction is currently not sufficiently convincing. In particular, the literature coverage is limited and does not provide a structured overview of existing FCL approaches and their respective limitations. As a result, the motivation for focusing specifically on adaptive task weighting with reconstructed historical gradients is not developed strongly enough. The logical flow is also somewhat abrupt: the text moves quickly from the general CL/FCL setting to the proposed method, without clearly establishing why existing replay-based, gradient-sharing, or multi-objective approaches are insufficient.
- A key concern is the reliance on Hessian-related curvature information. Although the paper uses low-rank approximations and Hessian-vector products instead of explicitly forming the full Hessian, the practical overhead may still be substantial, especially with increasing model size, number of historical tasks, and periodic summary refreshes. The current manuscript does not make this cost sufficiently transparent. I would encourage the authors to clarify the actual computational and communication overhead, including the number of HVPs, the role of the rank parameter, and the runtime/memory trade-off relative to first-order baselines and ablated variants.
- The paper does include ablation-style analysis, especially the comparison in Table 2 between local replay vs. shared reconstruction and fixed vs. adaptive weighting. However, the ablation study is still somewhat limited. Since the core novelty of the method lies in curvature-based reconstruction and compressed second-order information sharing, I would appreciate more fine-grained ablations on the curvature term, the rank parameter, recompression, and the associated accuracy-efficiency trade-off.
- I suggest that the authors further clarify the adaptation details and fairness settings of the baselines used in the experiments. Since some methods were adapted from their original formulations to the sequential / replay / federated setting, the main paper should explicitly state whether these adaptations were conducted under comparable resource constraints, training budgets, and hyperparameter tuning efforts. This would help improve the transparency and persuasiveness of the experimental comparisons.
- The correspondence between the theoretical analysis and the actual algorithm is not yet sufficiently clear. The method itself relies on several key approximation steps, including historical gradient reconstruction, curvature approximation, and approximate weight optimization via Frank-Wolfe, whereas the theoretical derivations in the main text seem to adopt a more idealized update form. Therefore, I would appreciate it if the authors could further clarify whether Theorems 1-2 genuinely characterize the practical implementation of FEDGRIT, or whether they apply only to an idealized variant. This point is important, since these approximation steps are precisely core components of the proposed method.
Questions for authors
- The introduction is not yet fully convincing. The literature review is limited, and the motivation for focusing on adaptive weighting with reconstructed historical gradients is not developed strongly enough.
- A key concern is the reliance on Hessian-related curvature information. Although low-rank approximations are used, the practical computation and communication overhead remains unclear.
- The paper includes some ablation analysis, but it is still limited. More fine-grained ablations on the curvature term, rank choice, recompression, and the accuracy-efficiency trade-off would strengthen the paper.
- The fairness of the experimental comparison should be clarified. Since several baselines were adapted to the sequential / replay / federated setting, the main paper should explain whether they were compared under similar resources, training budgets, and tuning effort.
- The connection between the theoretical analysis and the actual algorithm is not fully clear. Since the method relies on several approximation steps, it would be helpful to clarify whether Theorems 1-2 characterize the practical FEDGRIT method or only an idealized variant.
6 comments
Response to Reviewer qZgM Part II
(Continued from our response to W2.) HVP budget, rank, and scaling. Algorithm 2 uses one HVP operator evaluation per Lanczos iteration. With the ten-iteration cap specified in Appendix F.2, this requires at most ten HVPs per historical task per participating client at a refresh, forty when four historical tasks are represented. Only participating clients perform these computations.
The retained rank controls factor size and application cost, whereas the Lanczos iteration budget controls the number of HVP evaluations. Reducing rank therefore does not necessarily reduce HVP count when that budget remains unchanged. Between curvature refreshes, local reconstruction applies cached factors without additional HVPs.
For a model with parameter dimension and retained rank , a complete historical-summary refresh requires, per participating client in each communication direction, scalars, excluding model transmission and model-update upload. Each historical-task curvature correction requires operations per local step. Thus, summary size and reconstruction work grow linearly with model dimension and historical-task count for fixed rank. DSHA keeps each recipient’s curvature representation independent of the number of participants, although total traffic and server computation still depend on that number. Refresh frequency controls how often curvature estimation and exchange occur.
We have included these matched ablations and their measurement details in Appendix F.3, and make the HVP budget and cost accounting explicit in Appendix F.5, with revisions highlighted in blue.
W3: Fine-Grained Ablations and Accuracy–Efficiency Trade-offs
Response:
The additional component comparisons reported in our response to W2 isolate curvature correction and server recompression. Together with the rank study in Appendix F.3 Table 7, these experiments address the requested fine-grained ablations and quantify their accuracy–resource trade-offs.
Curvature correction. The no-curvature variant retains shared historical gradients and adaptive weighting while removing curvature estimation, transmission, and correction. Compared with this variant, FedGRIT improves ACC by 14.62 pp and BWT by 7.90 pp. Unlike the original Table 2 comparison against local replay, this experiment retains historical-gradient sharing and therefore isolates the additional benefit of curvature correction. The table in our response to W2 reports the corresponding runtime, communication, and memory costs.
Server recompression. The variant without server recompression retains client-side low-rank compression but broadcasts all uploaded curvature factors rather than reducing their aggregate to the target rank. Relative to this variant, FedGRIT reduces reported communication by 81.6% and measured per-round runtime by 66.8%, with 1.77 pp lower ACC and 1.57 pp higher BWT. This comparison demonstrates substantial resource savings from recompression together with its measured performance trade-off.
Retained rank. We evaluate ranks 1, 3, 5, and 10, reporting accuracy, BWT, runtime, and peak GPU memory. Increasing rank from 1 to 3 improves ACC by 3.51 pp and BWT by 5.09 pp. Further increases to ranks 5 and 10 incur additional costs without improving ACC or BWT in this experiment. For example, rank 10 requires 79.12 seconds per round and 37.05 GB of peak GPU memory, compared with 64.10 seconds and 18.94 GB for rank 3. Thus, rank 3 provides the strongest measured learning performance among the tested configurations while avoiding the higher costs of larger representations.
Together, these experiments distinguish the components’ complementary roles: curvature correction improves learning beyond shared gradients alone; server recompression reduces the cost of distributing and applying curvature; and retained rank controls the representation-size/performance trade-off. We have incorporated the additional ablations and resource measurements into Appendix F.3, with additions highlighted in blue.
W4: Baseline Adaptations and Fairness of Comparisons
Response:
The submitted paper reports the shared experimental setup in Section 5 and additional configurations and baseline-specific hyperparameters in Appendix F.2, including Table 4. We clarify below how the baselines were adapted, which training and replay budgets were matched, and how hyperparameters were selected. We will also make these details more explicit in the revised manuscript.
Response to Reviewer qZgM Part III
(Continued from our response to W4.) Matched training schedules and replay budgets. All methods use the same backbone, client-participation, 50 communication rounds per task, and 3 local epochs per round. Replay-based methods use the same total capacity of 400 examples per client and the same sample-selection policy. These settings match the training schedule and retained-example budget; they do not impose identical computational or communication costs. We report measured runtime, communication, and peak GPU memory separately in Appendix Tables 9-10.
Baseline adaptations. FSMGDA and FedCMOO are adapted to sequential training by evaluating historical objectives using retained replay samples. FedPOCL and FedBECAME are our federated adaptations of centralized continual-learning methods. For the federated adaptations, each client's local training follows the original method, while the resulting models are uploaded to the server for aggregation and global evaluation. Upon task completion, FedPOCL updates clients' buffers using their private datasets, while FedBECAME locally computes , , the precision matrix, and the merged model . These quantities remain private and are not shared with the server. Therefore, their federated adaptations primarily introduce server-side aggregation and global evaluation of locally trained models. We will document these adaptations individually in Appendix F.2.
Hyperparameter selection. Baseline-specific hyperparameters follow the original papers’ recommended settings or the values explicitly reported in Table 4, without an extensive baseline-specific search. We report the detailed results across different refresh intervals and different ranks below. The results show that the configuration with a refresh interval of 3 rounds achieves the highest ACC and best BWT, reducing runtime by 29.9% compared with the frequency-1 setting. Similarly, the result with rank achieves the best ACC and BWT, while avoiding the substantial memory and runtime overhead of higher ranks. We report the configuration-selection results alongside the parameter values below.
| Frequency | ACC (%) | BWT (pp) | Time (s) | Peak Memory Usage (GB) |
|---|---|---|---|---|
| 1 | 50.66 | -10.26 | 91.38 | 18.94 |
| 3 | 54.47 | -5.40 | 64.10 | 18.94 |
| 5 | 51.57 | -9.28 | 53.50 | 18.94 |
| 10 | 45.56 | -15.62 | 51.94 | 18.94 |
| Rank | ACC (%) | BWT (pp) | Time (s) | Peak Memory Usage (GB) |
|---|---|---|---|---|
| 1 | 50.96 | -10.49 | 57.87 | 13.37 |
| 3 | 54.47 | -5.40 | 64.10 | 18.94 |
| 5 | 51.54 | -9.53 | 65.17 | 24.50 |
| 10 | 50.39 | -10.26 | 79.12 | 37.05 |
The matched training schedules do not imply identical computational or communication costs: these depend on each method’s operations and auxiliary state. Appendix Tables 9-10 report the measured runtime, communication, and peak GPU memory separately, and we expand Appendix F.2 with the adaptation and parameter-selection details, highlighting the revisions in blue.
W5: Correspondence Between Theoretical Guarantees and Practical Implementation
Response:
The analysis retains FedGRIT’s compressed-curvature correction in the reconstructed-gradient updates, and it does not assume access to exact Hessians. The numerical idealization for task weighting is stated in Section 3.2. We clarify their respective roles below.
Compressed curvature is retained in the analysis. Lemmas 1–2 establish . The proofs use this bound together with local-model displacement to control the curvature correction. Thus, they accommodate a bounded compressed representation rather than requiring the reconstructed gradient to equal the exact historical gradient. This is an operator-bound analysis, not a rank-dependent reconstruction-accuracy analysis.
Task weighting is analyzed at the subproblem optimum. Section 3.2 writes the analyzed updates using an exact minimizer of Eq. (5), whereas the implementation approximates this solution using Frank–Wolfe. This formulation is consistent with Federated Multi-Objective Learning [1], whose Algorithm 1 likewise defines the update through the minimum-norm subproblem optimizer. The present bounds do not separately quantify finite Frank–Wolfe solver error.
Under these assumptions, Theorem 1 bounds cumulative historical-loss deterioration subject to its alignment condition, while Theorem 2 characterizes weighted stationarity and the contributions of stochasticity, local training, and client heterogeneity. These results characterize the reconstructed-gradient updates at the exact-weight level, rather than certify a particular numerical solver tolerance.
We will make this correspondence explicit at the beginning of Section 4 and clarify the distinction between analyzed and implemented weights in Section 3.2, with revisions highlighted in blue.
[1] Federated Multi-Objective Learning, NeurIPS 2023.
Response to Reviewer qZgM Part I
W1: Motivation and Positioning Relative to Prior Work
Response:
We appreciate the suggestion to connect the literature and our design choices more explicitly. We have revised the Introduction to explain the progression from existing retention strategies to adaptive task balancing and the historical-gradient information it requires.
Existing FCL methods retain knowledge through distillation or regularization, task-specific parameter sharing, and replay. Our motivation for adaptive weighting is that the relationships between current and historical tasks change across clients and training stages. Gradient-based weighting responds to their norms and alignment, building on Pareto-based CL, federated multi-objective optimization, and adaptive FCL methods. Avoiding manual coefficient selection is therefore not our central contribution.
The specific difficulty is obtaining informative historical gradients for these decisions throughout local training. Local replay captures only one client’s history. C-FLAG broadens this coverage through a shared, pooled historical gradient and adaptive old/current learning rates, but does not provide separate historical-task gradient estimates at subsequent local models. Federated multi-objective methods coordinate accessible objectives, while centralized past-gradient reconstruction does not address sharing historical information across separately evolving clients.
FedGRIT’s contribution is a task-specific, shared reconstruction-and-weighting protocol. Clients aggregate historical gradients and compact curvature summaries at a common model, then reconstruct historical gradients along their own local trajectories to recompute task weights. Aggregation addresses limited local coverage; curvature correction accounts for local displacement; and separate task summaries support task-specific decisions. Two-stage compression and periodic refreshes support repeated sharing without cross-client queries at every local step.
We have made this motivation and contribution explicit in Section 1, with revisions highlighted in blue. The broader literature discussion remains in Appendix A.
W2: Computational, Communication, and Memory Costs
Response:
We agree that avoiding an explicit Hessian does not, by itself, establish low practical overhead. We clarify the measured costs, isolate the contributions of curvature and server recompression, and explain how the HVP budget and retained rank affect these costs.
Measured whole-method costs. Tables 9–10 in Appendix F.4 already report runtime, communication, and peak allocated GPU memory. The CIFAR-100 results, together with final accuracy from Table 1, are:
On CIFAR-100, FedGRIT improves accuracy over FedPOCL, the strongest baseline by mean ACC, by 7.57 pp, while reducing measured per-round runtime by 25.5%. Compared with C-FLAG, it improves accuracy by 7.84 pp with 8.1% higher per-round runtime. Thus, FedGRIT delivers accuracy gains at competitive measured runtime, with additional communication and memory requirements. The extra communication carries historical gradients and curvature factors; memory also includes the curvature-estimation workspace. Once shared summaries are available, historical-gradient reconstruction uses cached factors rather than replay backpropagation at every local step.
Isolating curvature and recompression costs. We additionally compare against two controlled variants: without curvature, which retains shared historical gradients and adaptive weighting, and without server recompression, which retains client-side compression but broadcasts all uploaded curvature factors.
Retaining curvature improves ACC by 14.62 pp and BWT by 7.90 pp over the shared-gradient, no-curvature variant, quantifying the learning benefit alongside its additional costs. Server recompression substantially reduces those costs: compared with broadcasting all uploaded factors, it reduces communication by 81.6% and measured per-round runtime by 66.8%, with 1.77 pp lower ACC and 1.57 pp higher BWT. These comparisons establish complementary roles: curvature improves the historical information used for learning, while recompression controls the cost of distributing and applying that information.