- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The Authors introduce HIDE a new protocol for graph class-incremental learning, which addresses task identity leakage in the standard CGLB evaluation protocol. Under HIDE, GCIL methods' performance drops and rankings change significantly across various GCIL models, lending itself to be a harder benchmark. The authors attribute most of the loss to task identification rather than within-task classification, and show that dataset-loader differences shift results drastically.
Critical strengths
S1 The paper's motivation is clear--CGLB leaks task identity in two ways. The authors isolate the cost of each protocol choice by varying one of four protocol choices at a time.
S2 Experiments are thorough and comprehensive, reporting is extensive.
Critical weaknesses
W1a Methods are trained without inter-task edges but scored with them under HIDE conflating the cost of inter-task edges is mixing distribution shifts with difficulty.
W1b Methods run at hyperparameters tuned for CGLB, while references and cascade are tuned on calibration data.
W2 The novelty of this for GCIL is limited, and the presentation needs some work. Three works the authors cite: LLM4GCL (Cheng et al.), BeGin(Ko et al.), and (in the appendix) AGALE (Zhao et al.) make HIDE seem more of a combination of existing choices, and the related work section does not clearly separate the paper's contributions from these papers.
W2a Regarding the presentation, while I appreciate the throughoughness of documentating all experiments and results the main text depends on a very long appendix which is very daunting as a reader or reviewer. I would encourage the authors to revise the appendix content. Throughout there are also minor presentation issues, which would need addressing before a camera-ready submission.
Questions for authors
Q1 How do the rankings under HIDE change if the methods are retuned on calibrated data under HIDE or trained with inter-task edges when possible?
Q2 What does HIDE add beyond combining LLM4GCL, BeGin, and AGALE? Positioning this stronger would help convince me of the significance of the work.
Q3 Task Identification claims are made mostly around binary classes per task, Table 24 in the appendix shows within-task accuracy dropping at larger task sizes. Can the authors include content in the main text to address this?
3 comments
Part II
W2 / Q2. Novelty
We agree that the overlap with prior work should have been explained more clearly. Now Sec. 2.2 and Table 1 now distinguish the choices shared with earlier benchmarks from our contribution: controlled measurement of individual evaluation effects and systematic diagnosis of hidden-task prediction errors.
LLM4GCL already identifies task-ID leakage in local testing and introduces global evaluation. Its written protocol excludes inter-task edges and balances class sizes, whereas HIDE retains native class sizes and includes inter-task edges during scoring. BeGin already evaluates class-incremental node classification without supplying test-task identity; its fixed graph is also available during training, whereas our primary comparison trains on successive task subgraphs. AGALE contributes graph-aware task construction and analysis of how task partitioning changes homophily, while excluding inter-task edges. As shown below:
| Setting | LLM4GCL | BeGin | HIDE |
|---|---|---|---|
| Test batch | all tasks, no task given | all tasks, no task given | all tasks, no task given |
| Scoring graph | task subgraphs, without the inter-task edges (its code keeps them) | full graph | full graph |
| Training graph | subgraph of the current task | full graph, with the unlabelled nodes of later tasks | subgraph of the current task |
| Aggregation | per node | mean of per-task accuracy | per node |
| Graph | its own loader | its own loader | one shared graph for every method |
Our study adds two linked analyses:
Separate measurement of protocol effects. We measure the effects of batch composition, scoring graph, aggregation, and data loading through controlled comparisons. This establishes which choices change the apparent performance of a method and by how much. For example, adding inter-task edges to the scoring graph lowers mean per-node accuracy in 47 of 48 main method–dataset pairs. Changing from the released loaders to the shared graph changes DeLoMe’s gap to SSM on ogbn-arxiv from 0.02 to 21.49 pp at the primary configurations.
Diagnostics distinguishing task identification from within-task classification. We report task hit alongside accuracy when predictions are restricted to the true task’s classes. For example, TPP reaches 97.14% accuracy with the true task supplied on ogbn-arxiv, but its task hit under HIDE is 32.77% and its overall accuracy is 32.13%. These measurements distinguish strong within-task discrimination from successful prediction when task identity is unavailable.
We also distinguish LLM4GCL’s written protocol from the inspected implementation, which retains connections among seen-class nodes. Table 1 follows the written protocol, and Sec. 2.2 notes this distinction. Our contribution concerns the controlled comparisons and diagnostic evidence, rather than the first use of global evaluation or a full scoring graph.
To better illustrate why HIDE is both novel and applicable in practice, we can take the example of a citation graph: after learning successive groups of subject areas, a model classifies held-out papers whose citations may cross those groups. Their unknown tasks cannot be used to select their neighborhoods. Isolated eases this problem through task-pure batches and task-filtered graphs; Mixed removes only the batching aid, while HIDE removes both. Training-graph availability depends on the application: the graph may already be observable, or task subgraphs may arrive progressively.
Our analysis separates and measures the effects of these choices, showing how they change accuracy and method rankings and identifying prediction difficulties that overall accuracy alone can obscure.
W2a. Presentation
We agree that the appendix needed clearer organization and that the main text should make the experimental settings understandable without extensive consultation of supplementary material. We reorganized the appendix and added a guide linking main-text sections to supporting results. Paired differences and their uncertainty intervals are consolidated in App. T, and task-size results are brought together in App. Q, Table 28. The main text now directly explains the information available during training and evaluation and both mechanisms through which evaluation can supply task information (Secs. 3.1–3.2). We also acknowledge the remaining minor presentation issues and will address them before the final submission.
Part III
Q1. Rankings after retuning or changing training edges
We examined both interventions under HIDE. Now Tables 4 and 13 add training-connectivity controls; Table 12 reports calibration-based reselection for the three replay methods on ogbn-arxiv.
The table below evaluates each intervention separately, holding the other methods at their primary configurations. Accuracies are mean HIDE per-node accuracies (%) over ten seeds; ranks are among the six GCIL methods, excluding the references and cascade.
| Change | Dataset | Before accuracy and rank | After accuracy and rank |
|---|---|---|---|
| Reselect DeLoMe | ogbn-arxiv | 34.59, fifth | 49.02, third |
| Train PDGNNs with inter-task edges | CoraFull | 74.44, first | 70.28, second |
| Train PDGNNs with inter-task edges | ogbn-arxiv | 63.08, first | 60.34, first |
After calibration-based reselection, DeLoMe passes TAAM and DyMoE on ogbn-arxiv. The additional reselection runs leave the CoraFull ordering unchanged and TPP last on both datasets. Training PDGNNs with inter-task edges places it below DeLoMe on CoraFull, while it remains first on ogbn-arxiv. PDGNNs is the only primary method whose released code supports this training-connectivity comparison (App. G).
In the separate BeGin suite, all seven methods are evaluated under both training conditions. On Cora, MAS moves from first without training inter-task edges (80.51%) to last with them (61.21%), while GEM becomes first (66.84%; Table 13). These results show that both configuration selection and training connectivity can affect HIDE rankings. They describe the tested interventions and hyperparameter search grid, rather than a ranking obtained after jointly optimizing every method.
Q3. Larger tasks
Thank you for highlighting the importance of task size when interpreting these diagnostics. Sec. 5.2 now discusses the larger-task results and explains how increasing the number of classes per task affects within-task classification and method rankings. We also corrected the claim that task hit improves for every method. Below are the two- and ten-class conditions under HIDE on ogbn-arxiv (mean ± SD %; App. Q, Table 28):
| Method | Classes per task | Per-node accuracy | Task hit | Accuracy given the true task |
|---|---|---|---|---|
| PDGNNs | 2 | 63.08 ± 0.24 | 66.41 ± 0.20 | 93.23 ± 0.24 |
| PDGNNs | 10 | 61.37 ± 0.19 | 71.80 ± 0.18 | 80.15 ± 0.21 |
| DeLoMe | 2 | 34.59 ± 2.22 | 38.64 ± 2.30 | 85.49 ± 1.50 |
| DeLoMe | 10 | 48.67 ± 1.25 | 58.66 ± 1.07 | 75.96 ± 0.70 |
| SSM | 2 | 56.08 ± 0.88 | 58.44 ± 0.86 | 92.87 ± 1.17 |
| SSM | 10 | 41.17 ± 6.18 | 53.05 ± 4.61 | 63.00 ± 5.49 |
| TPP | 2 | 32.13 ± 0.15 | 32.77 ± 0.16 | 97.14 ± 0.07 |
| TPP | 10 | 30.45 ± 0.07 | 36.56 ± 0.13 | 81.65 ± 0.12 |
| TAAM* | 2 | 43.28 ± 0.32 | 45.60 ± 0.29 | 94.79 ± 0.11 |
| TAAM* | 10 | 42.06 ± 0.41 | 51.68 ± 0.13 | 80.72 ± 0.55 |
| DyMoE* | 2 | 42.92 ± 2.01 | 44.30 ± 2.07 | 89.85 ± 1.15 |
| DyMoE* | 10 | 54.16 ± 0.38 | 61.96 ± 0.40 | 77.47 ± 0.44 |
Two classes: ten seeds. Ten classes: five seeds for released methods, three for TAAM*/DyMoE* (* our reimplementations). Oracle accuracy restricts predictions to the true task’s classes over all test nodes.
SSM’s oracle accuracy falls from 92.87% with two classes per task to 63.00% with ten. At ten classes, its 58.83% error comprises 46.95 percentage points of wrong-task predictions and 11.88 percentage points of incorrect class predictions within the correct task. Task identification therefore, remains a major source of error, while within-task classification also becomes substantially more difficult. The high oracle accuracy observed with two-class tasks should not be generalized to larger tasks. Rankings also depend on task size: among the six methods, SSM moves from second to fifth in mean accuracy, while DyMoE moves from fourth to second.
We hope we have addressed your concerns. Please let us know if anything requires further clarification.
Thank you for the careful review. We added training controls and clarified the comparisons with prior work; changes are marked in red in the manuscript.
W1a. Training–evaluation mismatch
We agree that a model trained without inter-task edges may lose accuracy under HIDE, partly because its scoring graph differs from its training graph. Our controls show that this mismatch does not fully explain the observed loss. Now Table 1 separates training and scoring graphs; Sec. 5.3 and App. G add shared-graph training controls.
Our reference classifiers already cover this comparison. R1 and R2 are the same joint model, trained on the task subgraphs (no inter-task edges) and on the full graph (with inter-task edges), respectively, so the difference between them measures the effect of training with these edges. Each trained model is scored under both Mixed and HIDE. Rprop is fitted separately on the graph it scores, so there is no mismatch between its fitting and scoring graphs. To extend this comparison to an incremental method, we additionally trained PDGNNs with and without inter-task edges. Below are the results (Sec. 5.3, Table 4):
Mean per-node accuracy (%) over ten seeds; loss = Mixed − HIDE in percentage points (pp), computed before rounding. Every loss is positive on all ten seeds.
Rprop loses 16.10/15.49 pp despite matched graphs; PDGNNs trained with inter-task edges still loses 14.17/6.27 pp. Thus, unseen training connectivity is not the sole cause.
Below are additional BeGin controls, both training conditions evaluated under HIDE (App. G, Table 13):
Training without minus with inter-task edges, in pp; ten seeds on Cora/CiteSeer, five on ogbn-arxiv. † Seeds disagree on the sign. Both conditions retain future-task nodes with their labels hidden.
Training with inter-task edges lowers mean HIDE accuracy in all 21 pairs. Mismatch nevertheless matters on some graphs: full-graph training reduces the joint model’s Roman-empire loss from 36.26 to 2.59 pp (Table 14). These controls support separating the protocol effect from its possible causes.
W1b. Hyperparameter selection
Thank you for raising this concern. We agree that configuration selection can affect method comparisons. The main reference classifiers use fixed hyperparameters across datasets (App. K, Table 18). Rprop and R4 use calibration labels for fitting and prediction refinement, which is distinct from hyperparameter selection. The cascade does select its routing configuration on calibration data. The separately tuned joint-reference results are identified in Table 19, and App. A acknowledges the differences in configuration selection.
To examine sensitivity to the released configurations App. F Table 12 extends calibration-based reselection to CoraFull and to TPP on both datasets; the ogbn-arxiv replay runs were already submitted. Below are changes in mean HIDE accuracy over ten seeds (selected − primary configuration):
We performed a grid search to find the best hyperparameters for the methods in the table below on learning rate {0.001; 0.005, 0.02} and weight decay {0, 0.0005}. We selected calibration nodes separately per seed, with no test labels. Other hyperparameters remain fixed.
DeLoMe improves from 34.59% to 49.02%, reducing its gap to SSM from 21.49 to 7.06 pp. Its sensitivity involves graph encoding: the primary configuration reaches 44.51% on the directed shared graph versus 34.59% after symmetrization (Sec. 5.3). The other seven changes are at most 0.13 pp. TAAM/DyMoE were not reselected; this small grid does not establish rankings after unrestricted tuning. Q1 gives the observed ranking changes.