- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces GECKO, a diagnostic benchmark for federated continual graph learning on a fixed graph distributed across clients. It defines seven scenarios combining node classification, link classification, and link prediction with Task-, Class-, and Domain-Incremental settings (Table 2), and constructs matched paired comparisons that vary client allocation (Dirichlet alpha in {0.1, 1, 10, 100}) and task order (synchronized versus unsynchronized) while holding client data fixed. Evaluation covers four datasets (OGBN-Arxiv, OGBN-Proteins, Bitcoin-OTC, GEMSEC Facebook) with three to five paired seeds. Findings include that allocation changes affect both label proportions and local graph structure, that average performance and worst-condition performance can rank methods differently (Table 3), that allocation and task order have method-dependent and sometimes opposing effects (Figure 4), that lower forgetting can reflect weaker acquisition via the decomposition AF = Acq - P + Gamma (Section 5.3), and that same-task donor updates can help receivers at intermediate checkpoints (Table 5).
Critical strengths
- The matched-comparison design is the real contribution. Changing one factor at a time with paired seeds, validating the constructed client data, and reporting realized allocation, connectivity, and usable supervision separately is methodologically cleaner than how FCGL methods are usually evaluated, and Table 1 makes the coverage gap against BeGin, OpenFGL, POWER, and MOTION concrete.
- The diagnostics yield non-obvious, useful conclusions. The acquisition-forgetting decomposition in Section 5.3 shows low AF can hide weak learning, the LC calibration shows even All-task Training fails to beat a non-learning prior, and the donor-receiver analysis with Drop and Other-task controls (Table 5) gives a concrete mechanism-level check of cross-client support.
Critical weaknesses
- Coverage is too thin for a benchmark intended to steer FCGL evaluation. Only three scenarios (S2, S5, S7) receive full-grid learner evaluation, S3 is construction-only, and LP runs with only 3 clients. More importantly, FCGL-specific methods (MOTION, POWER) appear only in the node classification grid, while LC and LP are evaluated with generic continual learners, so the headline claim that FCGL methods should be assessed across allocation and order conditions is itself demonstrated mostly outside FCGL methods.
- The findings remain cautionary rather than actionable. Most conclusions are of the form that effects depend on the method and condition, and the paper gives limited guidance on which subset of conditions future papers should report or how results should be aggregated. Combined with the decision to keep one fixed configuration per method without retuning (Section 4), part of the observed sensitivity may reflect configuration mismatch rather than genuine protocol effects, which weakens the benchmark's prescriptive value.
Questions for authors
Refer to weaknesses.
2 comments
We sincerely appreciate the time and care you have put into reviewing our paper. We would welcome your feedback on our response, especially on whether the explanations and additional evidence address your concerns. If any points remain unclear, we would be happy to discuss them further. Thank you again for helping us strengthen our work.
Thank you for highlighting matched comparisons, the acquisition - forgetting diagnostic, and the donor - receiver controls as strengths. We appreciate in particular your suggestion that GECKO should give researchers clearer reporting guidance.
Q1. Scenario coverage and native FCGL baselines
A1. Following your suggestion, we strengthened scenario coverage by adding learner-level evaluations for S3 and two additional 10-client LP datasets. All seven scenarios now have learner-level results, comprising three full grids (
S2, S5, S7) and four matched-anchor evaluations (S1, S3, S4, S6). Please refer to our responses to Reviewer gKXy, Q1 (S3), and Reviewer NA3v, Q1 (additional LP datasets), for detailed results.MOTION and POWER were originally designed for node-level continual learning. Adapting their graph coarsening, replay, and prototype mechanisms to
LC/LPwould require modifying their original learning objectives. We therefore evaluate their native implementations on S2, where the results directly demonstrate sensitivity to GECKO's benchmark conditions. For example, increasing α from 0.1 to 100 improvesMOTION's accuracy by +24.41 pp under Sync and +21.63 pp under Unsync.POWERalso exhibits allocation-dependent order effects: switching to Unsync improves accuracy by +2.92 pp at α=0.1, but reduces it by 2.51 pp at α=100.These findings demonstrate the relevance of allocation - order evaluation for native FCGL methods. For LC/LP, GECKO evaluates compatible continual learners under matched federated graph conditions. Accordingly, our conclusions distinguish native FCGL results on NC from link-level results obtained with the evaluated learners, while preserving GECKO as a reusable evaluation protocol across prediction families.
Q2. Reporting guidance and configuration sensitivity
A2. Following your suggestion, we propose a two-level evaluation protocol to provide actionable guidance for future FCGL studies.
For comprehensive evaluation, we recommend the full eight-condition grid, reporting mean final performance P, worst-condition performance Pworst, and average forgetting (AF). For lower computational budgets, the four predefined anchors (
Base, Alloc., Order, Joint) provide a practical alternative for measuring endpoint allocation effects, task-order effects, and their interaction. These anchors do not capture intermediate allocation levels or full-grid worst-case performance. Results should be aggregated within each seed before reporting mean ± SD across paired seeds, with acquisition reported alongside AF when assessing retention.GECKO fixes each method's configuration across conditions to measure within-method responses without introducing condition-specific retuning. To assess configuration dependence, we additionally tested alternative hyperparameters on S2 and S5 using three paired seeds. On S2, allocation effects remain strongly positive for MOTION (+23.60 → +23.22 pp) and GEM (+20.96 → +19.96 pp), with unchanged four - anchor average rankings among the tested methods. S5 exhibits greater configuration sensitivity. Please refer to our response to Reviewer NA3v, Q3, for detailed results. We will clarify that GECKO evaluates
fixed-configuration sensitivityand distinguish it from condition-specific optimized performance.