acceptodds
Under review as a conference paper at ICLR 2027

GECKO: A Diagnostic Benchmark for Federated Continual Learning on a Shared Graph

Abstract

Federated continual graph learning (FCGL) studies how distributed clients learn a sequence of graph prediction tasks without forgetting earlier knowledge. Evaluating FCGL is challenging because the way graph data are assigned to clients can change both the prediction examples and the graph context available locally, while task order introduces a separate source of variation over time. We introduce GECKO (Graph Evaluation of Client-Knowledge and Order Sequences), a diagnostic benchmark for FCGL on a fixed graph distributed across clients. GECKO defines matched evaluation settings across node classification, link classification, and link prediction under Task-, Class-, and Domain-Incremental Learning. It varies client allocation and task order under explicit comparison rules, combining construction checks with diagnostics of task acquisition, forgetting, and cross-client support. Across the evaluated settings, both factors substantially affect final performance and forgetting, and their effects differ across methods and prediction problems. These results show that FCGL methods should be assessed across multiple allocation and task-order conditions rather than a single partition and sequence. GECKO provides reusable protocols and diagnostics for interpreting these comparisons.

Primary area:
datasets and benchmarks
Keywords:
Federated Continual Learning, Continual Graph Learning, Federated Graph Learning, Graph Neural Networks, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5n4BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces GECKO, a diagnostic benchmark for federated continual graph learning on a fixed graph distributed across clients. It defines seven scenarios combining node classification, link classification, and link prediction with Task-, Class-, and Domain-Incremental settings (Table 2), and constructs matched paired comparisons that vary client allocation (Dirichlet alpha in {0.1, 1, 10, 100}) and task order (synchronized versus unsynchronized) while holding client data fixed. Evaluation covers four datasets (OGBN-Arxiv, OGBN-Proteins, Bitcoin-OTC, GEMSEC Facebook) with three to five paired seeds. Findings include that allocation changes affect both label proportions and local graph structure, that average performance and worst-condition performance can rank methods differently (Table 3), that allocation and task order have method-dependent and sometimes opposing effects (Figure 4), that lower forgetting can reflect weaker acquisition via the decomposition AF = Acq - P + Gamma (Section 5.3), and that same-task donor updates can help receivers at intermediate checkpoints (Table 5).

Critical strengths
  1. The matched-comparison design is the real contribution. Changing one factor at a time with paired seeds, validating the constructed client data, and reporting realized allocation, connectivity, and usable supervision separately is methodologically cleaner than how FCGL methods are usually evaluated, and Table 1 makes the coverage gap against BeGin, OpenFGL, POWER, and MOTION concrete.
  2. The diagnostics yield non-obvious, useful conclusions. The acquisition-forgetting decomposition in Section 5.3 shows low AF can hide weak learning, the LC calibration shows even All-task Training fails to beat a non-learning prior, and the donor-receiver analysis with Drop and Other-task controls (Table 5) gives a concrete mechanism-level check of cross-client support.
Critical weaknesses
  1. Coverage is too thin for a benchmark intended to steer FCGL evaluation. Only three scenarios (S2, S5, S7) receive full-grid learner evaluation, S3 is construction-only, and LP runs with only 3 clients. More importantly, FCGL-specific methods (MOTION, POWER) appear only in the node classification grid, while LC and LP are evaluated with generic continual learners, so the headline claim that FCGL methods should be assessed across allocation and order conditions is itself demonstrated mostly outside FCGL methods.
  2. The findings remain cautionary rather than actionable. Most conclusions are of the form that effects depend on the method and condition, and the paper gives limited guidance on which subset of conditions future papers should report or how results should be aggregated. Combined with the decision to keep one fixed configuration per method without retuning (Section 4), part of the observed sensitivity may reflect configuration mismatch rather than genuine protocol effects, which weakens the benchmark's prescriptive value.
Questions for authors

Refer to weaknesses.

2 comments

Thank you for highlighting matched comparisons, the acquisition - forgetting diagnostic, and the donor - receiver controls as strengths. We appreciate in particular your suggestion that GECKO should give researchers clearer reporting guidance.

Q1. Scenario coverage and native FCGL baselines

A1. Following your suggestion, we strengthened scenario coverage by adding learner-level evaluations for S3 and two additional 10-client LP datasets. All seven scenarios now have learner-level results, comprising three full grids (S2, S5, S7) and four matched-anchor evaluations (S1, S3, S4, S6). Please refer to our responses to Reviewer gKXy, Q1 (S3), and Reviewer NA3v, Q1 (additional LP datasets), for detailed results.

MOTION and POWER were originally designed for node-level continual learning. Adapting their graph coarsening, replay, and prototype mechanisms to LC/LP would require modifying their original learning objectives. We therefore evaluate their native implementations on S2, where the results directly demonstrate sensitivity to GECKO's benchmark conditions. For example, increasing α\alpha from 0.1 to 100 improves MOTION's accuracy by +24.41 pp under Sync and +21.63 pp under Unsync. POWER also exhibits allocation-dependent order effects: switching to Unsync improves accuracy by +2.92 pp at α=0.1\alpha=0.1, but reduces it by 2.51 pp at α=100\alpha=100.

These findings demonstrate the relevance of allocation - order evaluation for native FCGL methods. For LC/LP, GECKO evaluates compatible continual learners under matched federated graph conditions. Accordingly, our conclusions distinguish native FCGL results on NC from link-level results obtained with the evaluated learners, while preserving GECKO as a reusable evaluation protocol across prediction families.

Q2. Reporting guidance and configuration sensitivity

A2. Following your suggestion, we propose a two-level evaluation protocol to provide actionable guidance for future FCGL studies.

For comprehensive evaluation, we recommend the full eight-condition grid, reporting mean final performance P‾\overline{P}, worst-condition performance PworstP_{\mathrm{worst}}, and average forgetting (AF). For lower computational budgets, the four predefined anchors (Base, Alloc., Order, Joint) provide a practical alternative for measuring endpoint allocation effects, task-order effects, and their interaction. These anchors do not capture intermediate allocation levels or full-grid worst-case performance. Results should be aggregated within each seed before reporting mean ±\pm SD across paired seeds, with acquisition reported alongside AF when assessing retention.

GECKO fixes each method's configuration across conditions to measure within-method responses without introducing condition-specific retuning. To assess configuration dependence, we additionally tested alternative hyperparameters on S2 and S5 using three paired seeds. On S2, allocation effects remain strongly positive for MOTION (+23.60 → +23.22 pp) and GEM (+20.96 → +19.96 pp), with unchanged four - anchor average rankings among the tested methods. S5 exhibits greater configuration sensitivity. Please refer to our response to Reviewer NA3v, Q3, for detailed results. We will clarify that GECKO evaluates fixed-configuration sensitivity and distinguish it from condition-specific optimized performance.

We sincerely appreciate the time and care you have put into reviewing our paper. We would welcome your feedback on our response, especially on whether the explanations and additional evidence address your concerns. If any points remain unclear, we would be happy to discuss them further. Thank you again for helping us strengthen our work.

Reviewer_NA3vOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a diagnostic benchmark GECKO for federated continual graph learning. The authors argue that existing evaluations are difficult to compare because they differ in client partitioning, task construction, and task order, and because most FCGL studies focus on node classification. GECKO addresses these issues by defining a common evaluation protocol covering node classification, link classification, and link prediction under several incremental learning settings. The benchmark defines seven scenarios; three receiving full evaluation across eight allocation task order conditions. Experiments with representative continual learning and federated learning methods demonstrate that client allocation and task order can substantially affect final performance and forgetting.

Critical strengths

A strength of this paper is the systematic treatment of evaluation conditions that are often overlooked in the field of federated continual graph learning. In graph learning, client partitioning affects not only label distributions but also local connectivity and the prediction examples available to each client. GECKO explicitly accounts for these effects and separates changes in client allocation from changes in task order. Extending the evaluation beyond node classification to link classification and link prediction is also valuable. The experiments provide concrete evidence that these choices matter. It also makes sense to make benchmark results more diagnostically informative rather than simply reporting method rankings.

Critical weaknesses

My main concern is that the empirical coverage is still relatively limited for a paper proposing a general-purpose (federated continual graph learning) benchmark. Although GECKO defines seven scenarios, only three receive the complete allocation task order evaluation, and one scenario (NC-Domain) has no learner-level results. The experiments use four datasets, with the main full-grid evaluations relying on OGBN-Arxiv, Bitcoin-OTC, and GEMSEC Facebook. The link prediction experiments involve only three clients and two tasks. These choices are understandable from a computational and data construction perspective, but they limit the strength of the conclusions about federated continual graph learning methods more broadly. The paper does not establish whether its findings generalize to larger federated systems, different graph structures, more diverse task sequences, or evolving graphs.

Questions for authors
  1. How representative are the selected benchmark scenarios of practical federated continual graph learning applications? Could you provide additional evidence using more datasets, larger numbers of clients, or longer task sequences?

  2. Why is link classification performance so low, even under All-task Training? Have you investigated stronger backbones, alternative edge representations, longer training, or more extensive hyperparameter tuning?

  3. How sensitive are the method rankings and reported allocation effects to hyperparameter choices?

  4. Will the complete benchmark implementation and predefined data splits be publicly available?

5 comments

Q1. Scenario representativeness and evaluation scale

A1. Following your concern about GECKO's representativeness, we added two datasets, Wiki-CS [1] and OGBL-Collab [2], expanding empirical coverage from four to six datasets. Both new LP evaluations use 10 clients, four matched anchors, and five paired seeds, extending beyond the original three-client GEMSEC Facebook setting.

We also added learner-level evaluations for S3 (NC-Domain, OGBN-Proteins). At α=0.1\alpha=0.1, switching to Unsync decreases GEM's AUROC from 62.30% to 60.36%, but increases TWP's from 55.20% to 58.08%, demonstrating learner-dependent task-order effects. All seven scenarios now have learner-level results, comprising three full grids and four matched-anchor evaluations. Please refer to our response to Reviewer gKXy, Q1 (Table A), for the detailed S3 results.

Table D. Additional 10-client LP evaluations. Each cell reports Hits@50 (%) ↑ / AF (pp) ↓, mean ±\pm SD over five paired seeds.

DatasetMethodBaseAlloc.OrderJoint
Wiki-CSBare64.45 ±\pm 5.01 / 0.87 ±\pm 1.1055.24 ±\pm 25.98 / 0.28 ±\pm 0.1965.38 ±\pm 5.03 / 0.56 ±\pm 0.4259.64 ±\pm 9.27 / 1.53 ±\pm 3.25
EWC65.44 ±\pm 4.77 / 0.45 ±\pm 0.2055.54 ±\pm 17.50 / 0.72 ±\pm 0.8265.02 ±\pm 4.39 / 0.52 ±\pm 0.5053.48 ±\pm 24.23 / 1.33 ±\pm 1.19
LwF65.28 ±\pm 5.00 / 0.42 ±\pm 0.5459.49 ±\pm 14.18 / 0.21 ±\pm 0.1964.50 ±\pm 5.00 / 0.63 ±\pm 0.9762.17 ±\pm 10.64 / 0.38 ±\pm 0.27
GraphKeeper-LP60.71 ±\pm 4.54 / 0.58 ±\pm 0.1657.24 ±\pm 6.60 / 0.65 ±\pm 0.5359.53 ±\pm 2.97 / 1.64 ±\pm 1.2658.97 ±\pm 5.85 / 0.56 ±\pm 0.17
OGBL-CollabBare55.32 ±\pm 12.53 / 2.92 ±\pm 6.3961.47 ±\pm 4.00 / 1.47 ±\pm 1.2258.39 ±\pm 11.45 / 2.34 ±\pm 5.0961.31 ±\pm 6.76 / 0.19 ±\pm 0.30
EWC54.24 ±\pm 14.86 / 0.56 ±\pm 1.1249.90 ±\pm 9.14 / 4.03 ±\pm 4.0851.62 ±\pm 18.14 / 2.17 ±\pm 4.2155.59 ±\pm 8.76 / 0.99 ±\pm 2.21
LwF49.81 ±\pm 11.45 / 0.52 ±\pm 0.9446.79 ±\pm 16.97 / 2.82 ±\pm 3.0341.16 ±\pm 19.37 / 2.14 ±\pm 1.3158.39 ±\pm 7.24 / 0.35 ±\pm 0.65
GraphKeeper-LP61.07 ±\pm 1.22 / 0.36 ±\pm 0.1961.49 ±\pm 1.45 / 0.63 ±\pm 0.2759.75 ±\pm 1.98 / 0.25 ±\pm 0.1661.97 ±\pm 1.35 / 0.33 ±\pm 0.29

The results in Table D show that performance and responses to benchmark conditions vary across datasets and learners, reinforcing the importance of evaluation beyond a single configuration. We will specify the task definitions, data splits, graph context, and negative-candidate protocols for both new LP datasets to support reproducibility. Together, these experiments broaden GECKO's empirical coverage to six datasets, all seven scenarios, and additional 10 client LP settings. Our conclusions remain focused on the evaluated static intra-graph FCGL settings, without attributing differences across datasets solely to client scaling or graph evolution.

References

[1] Mernyei, P., & Cangea, C. (2020). Wiki-CS: A Wikipedia-based benchmark for graph neural networks.

[2] Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., ... & Leskovec, J. (2020). Open Graph Benchmark: Datasets for Machine Learning on Graphs. NeurIPS.

Q2. Low link classification performance and stronger alternatives

A2. We thank the reviewer for raising this concern, which closely overlaps with Reviewer gKXy's Q2. We conducted additional All-task diagnostics over 150 communication rounds and three paired seeds to investigate the factors contributing to low LC performance.

The results show that preserving Adam state substantially improves training accuracy (22.63% to 37.65% at Base), while expanding client-local supervision improves test accuracy by +8.79 pp at Base and +4.00 pp at Joint. In contrast, increasing GCN width from 256 to 512 or allowing global-graph message passing does not consistently resolve low performance.

These targeted diagnostics indicate that optimizer-state handling and supervision availability are important contributors to LC learnability, while simply increasing model capacity or graph context does not guarantee better performance. We therefore interpret the observed difficulty in terms of the evaluated learning configurations rather than attributing it to a single architectural or representational limitation. Please refer to our response to Reviewer gKXy, Q2 for the complete experimental setup, detailed numerical results, and additional analysis of how limited learnability affects forgetting measurements.

Q3. Hyperparameter sensitivity of effects and rankings

A3. Following your concern, we examined the sensitivity of allocation effects and method rankings to hyperparameter choices by comparing the original configuration with one alternative per method on S2 and S5, using three paired seeds. The alternatives were specified before examining test results, changing only MOTION's replay weight (2→1), GEM's margin (0.5→0.2), CaT's condensation learning rate (0.001→0.0005), and TWP's λt\lambda_t (10000→1000). All other settings, including client partitions, task orders, and training budgets, remain fixed.

Table E. Hyperparameter sensitivity of allocation effects and average final accuracy. Allocation effects are computed as Δalloc=[(Base−Alloc.)+(Order−Joint)]/2\Delta_{\mathrm{alloc}}=[(\mathrm{Base}-\mathrm{Alloc.})+(\mathrm{Order}-\mathrm{Joint})]/2. Average accuracy is computed over four anchors within each seed. Results are mean ±\pm sample SD over three paired seeds.

ScenarioMethodAllocation effect (pp): OriginalAllocation effect (pp): AlternativeAvg. accuracy (%): OriginalAvg. accuracy (%): Alternative
S2 · NC-ClassMOTION+23.60 ±\pm 2.40+23.22 ±\pm 0.5438.73 ±\pm 2.8938.46 ±\pm 1.26
GEM+20.96 ±\pm 2.95+19.96 ±\pm 4.4825.39 ±\pm 2.1723.62 ±\pm 2.15
CaT+13.83 ±\pm 2.34+12.98 ±\pm 4.1312.67 ±\pm 1.5512.64 ±\pm 1.51
S5 · LC-ClassGEM+3.01 ±\pm 1.72−0.74 ±\pm 1.0620.19 ±\pm 2.4918.21 ±\pm 2.99
TWP−0.37 ±\pm 5.30+0.93 ±\pm 4.0120.75 ±\pm 1.5221.07 ±\pm 2.35
CaT−0.64 ±\pm 4.73−1.97 ±\pm 4.6020.24 ±\pm 0.7219.77 ±\pm 1.21

As shown in Table E, on S2, all three methods maintain large positive allocation effects with similar magnitudes under both configurations. In contrast, S5 is more configuration-sensitive, with GEM's mean effect changing from +3.01 to −0.74 pp and smaller, more variable effects for TWP and CaT. The four-anchor average rankings remain unchanged between the two configurations: MOTION > GEM > CaT on S2 and TWP > CaT > GEM on S5. However, S5 exhibits small performance gaps and ranking changes at individual anchors, indicating greater sensitivity to hyperparameter choices.

Overall, the results support the stability of S2's allocation effects and average rankings under the tested hyperparameter changes, while highlighting greater configuration dependence on S5. We will report these findings with their corresponding configurations and evaluation conditions.

Q4. Public availability of code and data splits

A4. Thank you for your concern regarding reproducibility. We have made our benchmark resources available for anonymous review at https://anonymous.4open.science/r/GECKO-2E1C, including the benchmark implementation, dataset construction scripts, experimental configurations, and predefined data splits. The repository also provides instructions for rerunning experiments under the supplied configurations.

Thank you again for your careful review and helpful suggestions. We hope our response and the revised manuscript have clarified the points you raised. We would be grateful to hear your thoughts and learn whether there are any remaining questions we could address. Your further feedback would help us improve the clarity and presentation of our work.

Reviewer_gKXyOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GECKO, a diagnostic benchmark for FCGL on a shared graph partitioned across multiple clients. The benchmark investigates how client data allocation and task ordering affect model performance and catastrophic forgetting. It defines seven evaluation scenarios covering node classification, link classification, and link prediction. GECKO further introduces matched experimental conditions and diagnostic analyses to distinguish task acquisition from knowledge retention and to investigate cross-client task support. Experimental results demonstrate that client allocation and task order can substantially influence model performance, while lower forgetting does not necessarily indicate better continual learning. Overall, the paper provides a structured evaluation protocol and empirical insights into the behavior of existing FCGL methods under heterogeneous learning conditions.

Critical strengths
  1. The benchmark offers broader evaluation dimensions than existing task-specific FCGL studies. GECKO considers node classification, link classification, and link prediction across multiple incremental learning settings. Its inclusion of link-level continual learning and domain-incremental scenarios extends the evaluation scope beyond the node classification settings commonly considered in FCGL research.
  2. The experimental design improves the interpretability of evaluation results. The paper varies client allocation and task order while maintaining consistent evaluation conditions within paired comparisons. This design helps identify how different sources of heterogeneity affect learning behavior, rather than attributing all performance differences to a single factor.
Critical weaknesses
  1. Benchmark coverage limits the comprehensiveness and generalizability of GECKO. Although GECKO defines seven FCGL scenarios, only three scenarios (S2, S5, S7) receive full-grid learner evaluations. S1, S4, S6 are evaluated only under selected anchor conditions, while S3 provides construction checks without learner-level results. The absence of learner-level evaluation for node domain-incremental learning makes it difficult to determine whether the identified effects of client allocation and task order generalize across all the incremental learning settings considered. Furthermore, the experimental evaluation relies on a relatively small number of datasets. Some observed differences may reflect dataset-specific characteristics rather than general properties of FCGL.
  2. The limited learnability of certain benchmark scenarios raises concerns about the validity of the resulting continual learning comparisons. The results in Table 4 reveal that several methods exhibit very low final accuracy, particularly in the link classification scenario. Even All-task Training chieves only 21.46% and 20.08% accuracy under the two evaluated anchor conditions. The training accuracy is also relatively low, suggesting that the poor performance cannot be attributed solely to catastrophic forgetting.
Questions for authors

Except the questions raised in critical weakness, some problems the authors should consider:

  1. Benchmark coverage. Why are full-grid learner evaluations restricted to S2, S5, and S7? Can the authors extend the evaluation to the remaining scenarios, particularly S3, to demonstrate the generalizability of the benchmark findings?
  2. Task learnability. Table 4 shows that All-task Training achieves relatively low performance in the LC scenario. Can the authors clarify whether this is caused by task construction, insufficient model capacity, optimization difficulties, or limited local graph information? How does this affect the validity of forgetting measurements?
  3. Evaluation fairness. Given that different methods use different architectures, auxiliary optimization steps, and aggregation strategies, how can the authors ensure fair comparisons beyond matching communication rounds? Would the conclusions remain consistent under comparable computational budgets?
  4. Task-order robustness. Are the reported effects consistent across multiple independently sampled asynchronous task sequences, rather than the predefined permutations considered in the current experiments?
  5. Reproducibility. Will the authors provide the complete benchmark implementation, dataset construction scripts, configurations, and pretrained evaluation artifacts during the review process? Can the reported results be independently reproduced using these resources?
4 comments

Thank you for your thoughtful comments. We address each point below.

Q1. Benchmark coverage and missing S3 results

A1. Thank you for suggesting to add the learner-level S3 results for node domain-incremental learning. We have now evaluated Bare, GEM, and TWP on OGBN-Proteins at all four endpoint anchors, using five paired seeds and the original protocol of 50 communication rounds per stage with one local epoch.

Table A. Additional S3 learner evaluation. Each cell reports ROC-AUC (%) ↑ / AF (pp) ↓, as mean ±\pm sample SD over five paired seeds.

MethodBaseAlloc.OrderJoint
Bare55.72 ±\pm 2.95 / 12.56 ±\pm 3.9555.49 ±\pm 5.92 / 16.15 ±\pm 4.4461.14 ±\pm 1.72 / 5.90 ±\pm 1.8556.71 ±\pm 5.64 / 12.97 ±\pm 3.43
GEM62.78 ±\pm 0.34 / 6.32 ±\pm 0.6162.30 ±\pm 3.25 / 8.41 ±\pm 1.9463.89 ±\pm 1.30 / 4.99 ±\pm 0.5060.36 ±\pm 4.27 / 10.10 ±\pm 2.97
TWP55.99 ±\pm 4.46 / 3.37 ±\pm 0.7455.20 ±\pm 4.10 / 7.57 ±\pm 0.9156.26 ±\pm 3.80 / 3.16 ±\pm 1.0258.08 ±\pm 2.11 / 5.63 ±\pm 1.04

At α=0.1\alpha=0.1, changing from synchronized to unsynchronized order changes GEM’s mean ROC-AUC by −1.94 pp and TWP’s by +2.88 pp. These descriptive differences extend the evidence to NC-Domain and show that the observed response need not have the same direction across learners. We do not interpret differences between cell means as statistically established effects without their paired uncertainty.

We selected S2, S5, and S7 for full-grid evaluation to provide one primary setting per prediction family (NC, LC, and LP) with intermediate allocation levels as well as endpoint comparisons. S1, S4, and S6 already had four-anchor evaluations. S3 now has the same evaluation depth. Thus, all seven scenarios have learner-level results, but only three have complete eight-condition grids. The anchor studies do not establish intermediate-allocation behavior or full-grid worst-condition performance.

The additional Wiki-CS and OGBL-Collab experiments reported in our response to NA3v-Q1 broaden dataset coverage from four to six. They reduce dependence on the original dataset choices, but do not establish dataset-independent effect directions or magnitudes.

We will update Table 2 and the coverage statements accordingly. We will also specify S3's ROC-AUC aggregation and handling of undefined client-task/output scores. We will specify that allocation and task order can materially change learner behavior, while the direction and magnitude depend on the learner and condition.

Q2. Link classification learnability and forgetting

A2. Following your concern, we conducted additional experiments to identify the factors contributing to low LC performance and examine the validity of forgetting measurements.

All-task Training in Table 4 achieves only 21.46% test accuracy at Base and 20.08% at Joint, indicating that removing sequential learning alone does not resolve the learning difficulty. We therefore conducted additional All-task diagnostics on S5 over 150 communication rounds and three paired seeds:

  • Optimization: Preserving Adam state increases training accuracy from 22.63% to 37.65% at Base and from 24.64% to 35.65% at Joint.
  • Supervision: Expanding client-local supervision improves test accuracy by +8.79 pp at Base and +4.00 pp at Joint, although this may also change class/client composition and computational cost.
  • Graph information and capacity: Global-graph message passing yields mixed effects (+1.49 pp at Base; −4.23 pp at Joint), while increasing GCN width from 256 to 512 does not provide consistent improvements.

These results identify optimization and supervision availability as important contributors to learnability, although they do not establish a single dominant cause of low performance.

To examine the effect of output-space expansion in Class-IL, we evaluated identical checkpoints over five paired seeds using the original growing masks and fixed first-exposure masks, without changing training.

Table B. LC acquisition, final accuracy, and forgetting under original and fixed class masks. Results are averaged over five paired seeds, accuracy is in %, and forgetting is in percentage points (pp).

ConditionMethodAcquisition (%)Final accuracy (%)Original AF (pp)Fixed-mask AF (pp)
BaseBare64.45 ±\pm 2.2818.78 ±\pm 2.5045.67 ±\pm 1.8116.00 ±\pm 10.85
BaseReplay37.18 ±\pm 4.4226.98 ±\pm 3.6714.73 ±\pm 5.292.21 ±\pm 1.42
JointBare37.83 ±\pm 3.9622.55 ±\pm 2.6022.10 ±\pm 4.3410.32 ±\pm 5.45
JointReplay38.38 ±\pm 3.4724.33 ±\pm 3.1117.57 ±\pm 3.214.68 ±\pm 2.15

As shown in Table B, AF decreases substantially under fixed masks (e.g., Bare at Base: 45.67 → 16.00 pp), confirming that the evaluation rule materially affects measured forgetting. However, this difference is not an exact decomposition of forgetting and output expansion. Moreover, Replay achieves lower AF than Bare but also lower acquisition at Base (37.18% versus 64.45%), showing that lower AF does not necessarily indicate better retention.

Thus, AF remains a measure of performance decline under the specified Class-IL protocol, but should not be interpreted as a pure measure of catastrophic forgetting. We will report acquisition, final performance, and mask-specific AF together to distinguish initial learnability from subsequent performance decline.

Q3. Fairness and comparable compute budgets

A3. Following your concern, we clarify that GECKO matches task definitions, client data, evaluation examples, participation schedules, communication rounds (50 per stage), and local epochs (one per round), with paired randomness where applicable. Architectures, auxiliary optimization steps, aggregation strategies, and computational costs remain method-specific (Section 4, Appendix C.1–C.4). Accordingly, cross-method results compare complete learning procedures under shared benchmark conditions. Our central conclusions instead rely on within-method paired comparisons, where each learner's architecture, hyperparameters, and training procedure remain fixed. These comparisons assess sensitivity to allocation and task order without introducing between-method architectural differences.

Regarding computational budgets, our additional 10-round × 5-local-update study examines whether allocation effects persist across training rules, rather than normalizing FLOPs or wall-clock time. The median NC allocation effects under synchronized order remain positive for Local-only (+3.58 pp), FedAvg (+6.50 pp), and FedProx (+10.18 pp) (Table 29, Appendix E.2). These results support qualitative consistency across the tested training rules, without implying that absolute method rankings are invariant under equal computational budgets. We will clarify this distinction when interpreting cross-method comparisons.

Q4. Robustness across sampled task orders

A4. Following your concern, we extended the evaluation to three Unsync profiles across five paired seeds and both allocation endpoints α∈{0.1,100}\alpha\in\{0.1,100\}. S2 uses the original profile and two additional randomized profiles, sampling ten distinct permutations within each four-task block. S5 uses three newly generated profiles, cyclically assigning randomly ordered permutations of its three tasks across ten clients. All profiles are fixed before training without test-based selection. We evaluate Bare, GEM, and native MOTION on S2, and Bare, GEM, and TWP on S5. Within paired comparisons, client data, initialization, participation schedules, and training budgets remain fixed, while federated aggregation remains synchronous.

Table C. Unsync − Sync final accuracy (pp, mean ±\pm sample SD across five seed-level averages, each averaging three profiles). Parentheses indicate positive seed–profile comparisons out of 15.

ScenarioMethodα=0.1\alpha=0.1α=100\alpha=100
S2 · NC-ClassBare−0.13 ±\pm 2.60 (8/15)+4.41 ±\pm 2.82 (13/15)
GEM+5.36 ±\pm 2.92 (14/15)+11.92 ±\pm 2.69 (15/15)
MOTION+2.39 ±\pm 1.95 (13/15)+0.89 ±\pm 0.28 (14/15)
S5 · LC-ClassBare+3.83 ±\pm 2.83 (11/15)+3.09 ±\pm 1.74 (12/15)
GEM+0.94 ±\pm 1.70 (11/15)+2.55 ±\pm 2.26 (11/15)
TWP+4.49 ±\pm 4.01 (13/15)+5.29 ±\pm 2.79 (12/15)

The results in Table C provide evidence that the observed task-order effects are not solely driven by the original predefined schedule. On S2, GEM consistently improves across all 10 comparisons from the additional profiles at both allocation levels. On S5, all tested methods show positive average effects, although some results vary across profiles. These variations are consistent with GECKO's central finding that task-order effects depend on the learner and allocation conditions. We will clarify the sampling protocol and the scope of these conclusions.

Q5. Reproducibility and access to benchmark artifacts

A5. Thank you for your concern regarding reproducibility. We have made our benchmark resources available for anonymous review at https://anonymous.4open.science/r/GECKO-2E1C, including the benchmark implementation, dataset construction scripts, experimental configurations, and predefined data splits. The repository also provides instructions for rerunning experiments under the supplied configurations.

We sincerely thank you for the time and effort you have devoted to reviewing our work. We would greatly appreciate your thoughts on our response and the additional results, particularly whether any concerns still need clarification. We would be happy to discuss any remaining questions and would value your further feedback as we refine the manuscript.

Comments on the paper

1 comment

We thank reviewers gKXy, NA3v, and 5n4B for their constructive feedback. GECKO provides a controlled benchmark for static intra graph federated continual graph learning (FCGL), examining client allocation and task order heterogeneity. We summarize the main concerns and our responses below, with details in the reviewer specific replies. The main concerns fall into five aspects:

  • (1) Coverage and representativeness (gKXy, NA3v, 5n4B): Incomplete learner-level coverage, limited LP client counts, and the applicability of native FCGL methods to link-level tasks.
  • (2) Learnability and forgetting (gKXy, NA3v): Low LC accuracy and whether forgetting reflects limited acquisition or changes in the Class-IL evaluation space.
  • (3) Fairness and configuration sensitivity (gKXy, NA3v, 5n4B): Computational comparability under matched communication rounds and robustness to hyperparameter choices.
  • (4) Task-order robustness (gKXy, NA3v): Whether observed effects persist beyond the predefined unsynchronized order profiles.
  • (5) Evaluation guidance and reproducibility (gKXy, NA3v, 5n4B): Clear reporting recommendations and availability of code, splits, configurations, and evaluation artifacts.

Regarding each concern, we summarize our responses as follows:

  • (1) All seven scenarios now have learner-level evaluations, with full grids for three scenarios and matched-anchor evaluations for the remaining four. Added S3 results and ten-client LP experiments on Wiki-CS and OGBL-Collab expand coverage to six datasets. We distinguish benchmark coverage from method applicability: MOTION and POWER remain evaluated within their supported node-level formulations.
  • (2) LC diagnostics identify effects of optimizer-state preservation and client-local supervision on learning. Evaluating identical checkpoints with fixed first-exposure masks reduces Bare’s Base AF from 45.67 to 16.00 pp, demonstrating evaluation-rule sensitivity without an exact decomposition of forgetting. We therefore recommend reporting acquisition, final performance, and AF together.
  • (3) Equal communication rounds do not imply equal computation. Alternative hyperparameters retain strong positive S2 allocation effects but reveal greater S5 sensitivity. Partitioning and training-rule studies further examine protocol dependence. Our primary conclusions concern within-method responses under specified conditions, not universally stable method rankings or equal computational costs.
  • (4) Additional S2/S5 evaluations use three order profiles per seed and five paired seeds at both allocation endpoints. Positive S2 responses for GEM and MOTION persist across most comparisons, while effects remain learner-dependent. Shared-seed profiles are not independent replications, and unsynchronized task order is distinct from asynchronous federated optimization.
  • (5) We recommend the full eight condition grid for comprehensive assessment or four predefined anchors for lower cost endpoint analysis, reporting final performance, worst condition performance, and AF with paired-seed aggregation. Anchors do not establish intermediate allocation behavior or full grid worst cases. We plan to release the complete codebase, construction scripts, splits, configurations, and evaluation artifacts upon acceptance.

We will incorporate these additions and clarify their scope and limitations in the revised manuscript. We appreciate the reviewers’ suggestions and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.