acceptodds
Under review as a conference paper at ICLR 2027

One Graph, Every Node, Hidden Task: HIDE, a Protocol for Graph Class-Incremental Learning

Abstract

Graph class-incremental learning (GCIL) methods are compared by their accuracy under the CGLB benchmark protocol. CGLB supplies the task identity the class-incremental setting is meant to withhold, in two ways: test nodes are scored one task at a time, and each node is scored on the subgraph of its own task. We measure what each of them, and each other evaluation choice, contributes to reported accuracy. We introduce HIDE (Hidden-task IDentity Evaluation), which scores every test node in one batch, on the full graph, over every class seen so far, with accuracy averaged over nodes, and we vary each choice independently. We evaluate six methods on eight datasets, using one shared graph per dataset, alongside two reference classifiers with no continual-learning mechanism, one of them trained jointly on all tasks. Every method that does not already collapse under CGLB loses accuracyunder HIDE; the one ranked first under CGLB on seven of eight datasets ranks last under HIDE on seven of eight, with its per-node accu racy falling from 97% to 32% on ogbn-arxiv and from 87% to 12% at the widest. On ogbn-arxiv, no method beats either reference classifier. We release the shared graphs, the splits, the reference classifiers, and every per-seed result. We propose assessing novel GCIL methods on HIDE, where they are evaluated under realistic, more challenging hidden-task node-by-node class-incremental learning.

Primary area:
datasets and benchmarks
Keywords:
graph continual learning, class-incremental learning, evaluation protocol, benchmark, node classification, graph neural network

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_oxbgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The Authors introduce HIDE a new protocol for graph class-incremental learning, which addresses task identity leakage in the standard CGLB evaluation protocol. Under HIDE, GCIL methods' performance drops and rankings change significantly across various GCIL models, lending itself to be a harder benchmark. The authors attribute most of the loss to task identification rather than within-task classification, and show that dataset-loader differences shift results drastically.

Critical strengths

S1 The paper's motivation is clear--CGLB leaks task identity in two ways. The authors isolate the cost of each protocol choice by varying one of four protocol choices at a time.

S2 Experiments are thorough and comprehensive, reporting is extensive.

Critical weaknesses

W1a Methods are trained without inter-task edges but scored with them under HIDE conflating the cost of inter-task edges is mixing distribution shifts with difficulty.

W1b Methods run at hyperparameters tuned for CGLB, while references and cascade are tuned on calibration data.

W2 The novelty of this for GCIL is limited, and the presentation needs some work. Three works the authors cite: LLM4GCL (Cheng et al.), BeGin(Ko et al.), and (in the appendix) AGALE (Zhao et al.) make HIDE seem more of a combination of existing choices, and the related work section does not clearly separate the paper's contributions from these papers.

W2a Regarding the presentation, while I appreciate the throughoughness of documentating all experiments and results the main text depends on a very long appendix which is very daunting as a reader or reviewer. I would encourage the authors to revise the appendix content. Throughout there are also minor presentation issues, which would need addressing before a camera-ready submission.

Questions for authors

Q1 How do the rankings under HIDE change if the methods are retuned on calibrated data under HIDE or trained with inter-task edges when possible?

Q2 What does HIDE add beyond combining LLM4GCL, BeGin, and AGALE? Positioning this stronger would help convince me of the significance of the work.

Q3 Task Identification claims are made mostly around binary classes per task, Table 24 in the appendix shows within-task accuracy dropping at larger task sizes. Can the authors include content in the main text to address this?

3 comments

Thank you for the careful review. We added training controls and clarified the comparisons with prior work; changes are marked in red in the manuscript.

W1a. Training–evaluation mismatch

We agree that a model trained without inter-task edges may lose accuracy under HIDE, partly because its scoring graph differs from its training graph. Our controls show that this mismatch does not fully explain the observed loss. Now Table 1 separates training and scoring graphs; Sec. 5.3 and App. G add shared-graph training controls.

Our reference classifiers already cover this comparison. R1 and R2 are the same joint model, trained on the task subgraphs (no inter-task edges) and on the full graph (with inter-task edges), respectively, so the difference between them measures the effect of training with these edges. Each trained model is scored under both Mixed and HIDE. Rprop is fitted separately on the graph it scores, so there is no mismatch between its fitting and scoring graphs. To extend this comparison to an incremental method, we additionally trained PDGNNs with and without inter-task edges. Below are the results (Sec. 5.3, Table 4):

DatasetModel and training graphMixed (%)HIDE (%)Loss (pp)
CoraFullPDGNNs, without inter-task edges86.2774.4411.84
CoraFullPDGNNs, with inter-task edges84.4570.2814.17
CoraFullJoint R1, without inter-task edges86.9173.2313.69
CoraFullJoint R2, with inter-task edges83.8171.0112.81
CoraFullRprop, fitted on each scoring graph86.7470.6416.10
ogbn-arxivPDGNNs, without inter-task edges69.4863.086.40
ogbn-arxivPDGNNs, with inter-task edges66.6160.346.27
ogbn-arxivJoint R1, without inter-task edges76.7867.479.31
ogbn-arxivJoint R2, with inter-task edges76.2268.038.19
ogbn-arxivRprop, fitted on each scoring graph89.9574.4615.49

Mean per-node accuracy (%) over ten seeds; loss = Mixed − HIDE in percentage points (pp), computed before rounding. Every loss is positive on all ten seeds.

Rprop loses 16.10/15.49 pp despite matched graphs; PDGNNs trained with inter-task edges still loses 14.17/6.27 pp. Thus, unseen training connectivity is not the sole cause.

Below are additional BeGin controls, both training conditions evaluated under HIDE (App. G, Table 13):

MethodCora (pp)CiteSeer (pp)ogbn-arxiv (pp)
Fine-tuning+6.33†+4.63+0.08†
LwF+3.58†+3.64+0.37†
EWC+8.24+4.26+7.66
MAS+19.30+2.82+0.79†
TWP+7.83+3.80+5.33
GEM+5.33†+3.11+4.54
ER-GNN+4.98†+3.65+8.39

Training without minus with inter-task edges, in pp; ten seeds on Cora/CiteSeer, five on ogbn-arxiv. † Seeds disagree on the sign. Both conditions retain future-task nodes with their labels hidden.

Training with inter-task edges lowers mean HIDE accuracy in all 21 pairs. Mismatch nevertheless matters on some graphs: full-graph training reduces the joint model’s Roman-empire loss from 36.26 to 2.59 pp (Table 14). These controls support separating the protocol effect from its possible causes.

W1b. Hyperparameter selection

Thank you for raising this concern. We agree that configuration selection can affect method comparisons. The main reference classifiers use fixed hyperparameters across datasets (App. K, Table 18). Rprop and R4 use calibration labels for fitting and prediction refinement, which is distinct from hyperparameter selection. The cascade does select its routing configuration on calibration data. The separately tuned joint-reference results are identified in Table 19, and App. A acknowledges the differences in configuration selection.

To examine sensitivity to the released configurations App. F Table 12 extends calibration-based reselection to CoraFull and to TPP on both datasets; the ogbn-arxiv replay runs were already submitted. Below are changes in mean HIDE accuracy over ten seeds (selected − primary configuration):

MethodCoraFull change (pp)ogbn-arxiv change (pp)
PDGNNs−0.06+0.13
DeLoMe0.00+14.43
SSM+0.040.00
TPP+0.020.00

We performed a grid search to find the best hyperparameters for the methods in the table below on learning rate {0.001; 0.005, 0.02} and weight decay {0, 0.0005}. We selected calibration nodes separately per seed, with no test labels. Other hyperparameters remain fixed.

DeLoMe improves from 34.59% to 49.02%, reducing its gap to SSM from 21.49 to 7.06 pp. Its sensitivity involves graph encoding: the primary configuration reaches 44.51% on the directed shared graph versus 34.59% after symmetrization (Sec. 5.3). The other seven changes are at most 0.13 pp. TAAM/DyMoE were not reselected; this small grid does not establish rankings after unrestricted tuning. Q1 gives the observed ranking changes.

W2 / Q2. Novelty

We agree that the overlap with prior work should have been explained more clearly. Now Sec. 2.2 and Table 1 now distinguish the choices shared with earlier benchmarks from our contribution: controlled measurement of individual evaluation effects and systematic diagnosis of hidden-task prediction errors.

LLM4GCL already identifies task-ID leakage in local testing and introduces global evaluation. Its written protocol excludes inter-task edges and balances class sizes, whereas HIDE retains native class sizes and includes inter-task edges during scoring. BeGin already evaluates class-incremental node classification without supplying test-task identity; its fixed graph is also available during training, whereas our primary comparison trains on successive task subgraphs. AGALE contributes graph-aware task construction and analysis of how task partitioning changes homophily, while excluding inter-task edges. As shown below:

SettingLLM4GCLBeGinHIDE
Test batchall tasks, no task givenall tasks, no task givenall tasks, no task given
Scoring graphtask subgraphs, without the inter-task edges (its code keeps them)full graphfull graph
Training graphsubgraph of the current taskfull graph, with the unlabelled nodes of later taskssubgraph of the current task
Aggregationper nodemean of per-task accuracyper node
Graphits own loaderits own loaderone shared graph for every method

Our study adds two linked analyses:

Separate measurement of protocol effects. We measure the effects of batch composition, scoring graph, aggregation, and data loading through controlled comparisons. This establishes which choices change the apparent performance of a method and by how much. For example, adding inter-task edges to the scoring graph lowers mean per-node accuracy in 47 of 48 main method–dataset pairs. Changing from the released loaders to the shared graph changes DeLoMe’s gap to SSM on ogbn-arxiv from 0.02 to 21.49 pp at the primary configurations.

Diagnostics distinguishing task identification from within-task classification. We report task hit alongside accuracy when predictions are restricted to the true task’s classes. For example, TPP reaches 97.14% accuracy with the true task supplied on ogbn-arxiv, but its task hit under HIDE is 32.77% and its overall accuracy is 32.13%. These measurements distinguish strong within-task discrimination from successful prediction when task identity is unavailable.

We also distinguish LLM4GCL’s written protocol from the inspected implementation, which retains connections among seen-class nodes. Table 1 follows the written protocol, and Sec. 2.2 notes this distinction. Our contribution concerns the controlled comparisons and diagnostic evidence, rather than the first use of global evaluation or a full scoring graph.

To better illustrate why HIDE is both novel and applicable in practice, we can take the example of a citation graph: after learning successive groups of subject areas, a model classifies held-out papers whose citations may cross those groups. Their unknown tasks cannot be used to select their neighborhoods. Isolated eases this problem through task-pure batches and task-filtered graphs; Mixed removes only the batching aid, while HIDE removes both. Training-graph availability depends on the application: the graph may already be observable, or task subgraphs may arrive progressively.

Our analysis separates and measures the effects of these choices, showing how they change accuracy and method rankings and identifying prediction difficulties that overall accuracy alone can obscure.

W2a. Presentation

We agree that the appendix needed clearer organization and that the main text should make the experimental settings understandable without extensive consultation of supplementary material. We reorganized the appendix and added a guide linking main-text sections to supporting results. Paired differences and their uncertainty intervals are consolidated in App. T, and task-size results are brought together in App. Q, Table 28. The main text now directly explains the information available during training and evaluation and both mechanisms through which evaluation can supply task information (Secs. 3.1–3.2). We also acknowledge the remaining minor presentation issues and will address them before the final submission.

Q1. Rankings after retuning or changing training edges

We examined both interventions under HIDE. Now Tables 4 and 13 add training-connectivity controls; Table 12 reports calibration-based reselection for the three replay methods on ogbn-arxiv.

The table below evaluates each intervention separately, holding the other methods at their primary configurations. Accuracies are mean HIDE per-node accuracies (%) over ten seeds; ranks are among the six GCIL methods, excluding the references and cascade.

ChangeDatasetBefore accuracy and rankAfter accuracy and rank
Reselect DeLoMeogbn-arxiv34.59, fifth49.02, third
Train PDGNNs with inter-task edgesCoraFull74.44, first70.28, second
Train PDGNNs with inter-task edgesogbn-arxiv63.08, first60.34, first

After calibration-based reselection, DeLoMe passes TAAM and DyMoE on ogbn-arxiv. The additional reselection runs leave the CoraFull ordering unchanged and TPP last on both datasets. Training PDGNNs with inter-task edges places it below DeLoMe on CoraFull, while it remains first on ogbn-arxiv. PDGNNs is the only primary method whose released code supports this training-connectivity comparison (App. G).

In the separate BeGin suite, all seven methods are evaluated under both training conditions. On Cora, MAS moves from first without training inter-task edges (80.51%) to last with them (61.21%), while GEM becomes first (66.84%; Table 13). These results show that both configuration selection and training connectivity can affect HIDE rankings. They describe the tested interventions and hyperparameter search grid, rather than a ranking obtained after jointly optimizing every method.

Q3. Larger tasks

Thank you for highlighting the importance of task size when interpreting these diagnostics. Sec. 5.2 now discusses the larger-task results and explains how increasing the number of classes per task affects within-task classification and method rankings. We also corrected the claim that task hit improves for every method. Below are the two- and ten-class conditions under HIDE on ogbn-arxiv (mean ± SD %; App. Q, Table 28):

MethodClasses per taskPer-node accuracyTask hitAccuracy given the true task
PDGNNs263.08 ± 0.2466.41 ± 0.2093.23 ± 0.24
PDGNNs1061.37 ± 0.1971.80 ± 0.1880.15 ± 0.21
DeLoMe234.59 ± 2.2238.64 ± 2.3085.49 ± 1.50
DeLoMe1048.67 ± 1.2558.66 ± 1.0775.96 ± 0.70
SSM256.08 ± 0.8858.44 ± 0.8692.87 ± 1.17
SSM1041.17 ± 6.1853.05 ± 4.6163.00 ± 5.49
TPP232.13 ± 0.1532.77 ± 0.1697.14 ± 0.07
TPP1030.45 ± 0.0736.56 ± 0.1381.65 ± 0.12
TAAM*243.28 ± 0.3245.60 ± 0.2994.79 ± 0.11
TAAM*1042.06 ± 0.4151.68 ± 0.1380.72 ± 0.55
DyMoE*242.92 ± 2.0144.30 ± 2.0789.85 ± 1.15
DyMoE*1054.16 ± 0.3861.96 ± 0.4077.47 ± 0.44

Two classes: ten seeds. Ten classes: five seeds for released methods, three for TAAM*/DyMoE* (* our reimplementations). Oracle accuracy restricts predictions to the true task’s classes over all test nodes.

SSM’s oracle accuracy falls from 92.87% with two classes per task to 63.00% with ten. At ten classes, its 58.83% error comprises 46.95 percentage points of wrong-task predictions and 11.88 percentage points of incorrect class predictions within the correct task. Task identification therefore, remains a major source of error, while within-task classification also becomes substantially more difficult. The high oracle accuracy observed with two-class tasks should not be generalized to larger tasks. Rankings also depend on task size: among the six methods, SSM moves from second to fifth in mean accuracy, while DyMoE moves from fourth to second.

We hope we have addressed your concerns. Please let us know if anything requires further clarification.

Reviewer_Vv8yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Four contributions: 1.Show most GCIL methods performance degrades under HIDE on every dataset evaluated, altering even the methods ranking, for instance method PDGNNs was the best performing one under CGLB, but it loses accuracy and becomes worse than the references and the baseline. 2. Justified that the performance degradation is due to incorrect task identification with quantitative support evidence.
3. Demonstrated the different protocol choices (Figure 1) can lead to different impacts on the performance to various degrees using five to eight datasets. 4. Proposes new GCIL baselines that follow vulnerability of rankings reasoning to the evaluation of various protocol choices.

Critical strengths
  1. Justified that the performance degradation is due to incorrect task identification with quantitative support evidence.
  2. Proposes new GCIL baselines that follow vulnerability of rankings reasoning to the evaluation of various protocol choices. And demonstrated the different protocol choices (Figure 1) can lead to different impacts on the performance to various degrees using five to eight datasets.
Critical weaknesses

Datasets selected/used are of different sizes, which may not offer an unbiased comparison of different performance criteria.

Questions for authors
  1. Use similar dataset sizes.
  2. Ablation tests should be validated.
  3. Computational cost to be included for each experimental test.
2 comments

Thank you for the suggestions. We added datasets of comparable size and computational-cost reporting; changes are marked in red on the manuscript.

W1 / Q1. Dataset sizes

Thank you for raising this concern. Our main comparisons are made within each dataset: the underlying data and test nodes are held fixed across methods and evaluation settings. Differences in size between datasets, therefore, should not explain the protocol differences measured within a dataset. To further address your concern, we added three datasets with node counts comparable to existing ones (App. N). These additions provide comparisons between graphs of similar size and allow us to assess whether the observed protocol effects persist. We also extended Reddit to ten seeds.

Below are the pairings, using total source-graph node counts (App. D, Table 8; App. N). We show TPP’s results as an example; Table 25 reports all six methods under both AA and per-node accuracy over ten seeds.

Original datasetNodesAdditional datasetNodes
Amazon Photo7,650LastFMAsia7,624
CoraFull19,793Coauthor-CS18,333
Roman-empire22,662Amazon-ratings24,492

Below are TPP’s per-node accuracies for these pairs (mean ± SD %, ten seeds; App. N, Table 25):

DatasetIsolatedMixedHIDE
Amazon Photo98.20 ± 0.2773.78 ± 0.9062.82 ± 1.08
LastFMAsia96.09 ± 0.6581.17 ± 0.9574.11 ± 0.92
CoraFull96.77 ± 0.3279.88 ± 0.6955.12 ± 0.62
Coauthor-CS99.49 ± 0.1282.48 ± 0.2875.27 ± 0.50
Roman-empire87.14 ± 0.2970.89 ± 0.3212.16 ± 0.44
Amazon-ratings62.86 ± 0.4841.80 ± 0.6238.31 ± 0.57

All six methods have lower mean per-node accuracy under HIDE than under Isolated or Mixed on each of the three added datasets with comparable node counts, the same holds on Reddit. Thus, the direction of these protocol effects also holds in the added comparisons, although their magnitude and the method rankings vary. Similar node counts do not imply identical topology, class distributions, or numbers of tasks, so these comparisons assess robustness across datasets of comparable size rather than isolate the effect of size.

On the three added datasets, TPP ranks last among the four released implementations under HIDE AA, but under HIDE per-node accuracy, it ranks last only on Coauthor-CS. We therefore retain the seven-of-eight ranking claim specifically for the main datasets.

Q2. Ablation validation

We assess the ablations through controlled comparisons paired by seed, repeated runs, and controls for alternative explanations. The aggregation comparison uses the same node predictions, while batch and scoring-graph comparisons hold learned GCIL parameters fixed. We document the required implementation changes: TPP uses a per-node inference adaptation, loader comparisons require retraining, and Rprop is fitted separately on each scoring graph. For the cascade’s batch comparison, we hold the selected subspace dimension fixed and recompute routing for each batch composition.

Below are representative paired changes with 95% percentile bootstrap intervals (pp; ten seeds; App. T, Table 33):

ComparisonMethodCoraFullogbn-arxiv
Per-node minus AA, IsolatedPDGNNs+4.88 [4.60, 5.16]+3.88 [3.52, 4.22]
Mixed minus Isolated, per nodeTPP−16.89 [−17.35, −16.45]−49.00 [−49.09, −48.91]
Mixed minus Isolated, per nodeCascade+0.16 [0.11, 0.21]+1.78 [1.52, 2.02]
HIDE minus Mixed, per nodePDGNNs−11.84 [−12.07, −11.59]−6.40 [−6.58, −6.22]

All ten seeds agree on the direction of the changes shown above; elsewhere, daggers mark disagreement. We construct the intervals by resampling the ten paired per-seed differences with replacement 10,000 times and taking the 2.5th and 97.5th percentiles of the resampled means. We reran cascade routing for each batch composition, added training-connectivity controls, and consolidated interval reporting in App. T. The loss for Rprop fitted on its scoring graph, showing that inter-task edges being absent during training cannot explain every scoring-graph loss (Table 4).

Please let us know if there are more experiments that you would like to better validate our ablations.

Q3. Computational cost

Thank you for this suggestion. App. C, Table 5 now reports training-plus-scoring wall-clock times for the methods, cascade, and reference runs across the eight main datasets. Times are in minutes on one NVIDIA RTX A6000 with 48 GB of memory, for seed 0. Unmarked entries are measured runs; six entries are estimated from shorter runs. The symbols follow Table 5: ≈ denotes an extrapolated estimate, and ≥ denotes an estimated lower bound.

Method or runCoraFullarxivCoraCiteSeerPhotoComp.WikiCSRoman
PDGNNs14.4≈11.30.71.61.01.61.01.8
DeLoMe≈58.6≈60.22.12.23.15.35.85.9
SSM≈129.6≈31.61.51.62.74.95.27.2
TPP3.42.10.70.70.70.80.81.2
TAAM*4.13.00.70.70.70.80.81.2
DyMoE*≈12.55.20.90.80.91.11.12.2
Cascade12.34.90.40.70.61.21.00.8
Rprop0.30.20.10.20.10.10.20.2
R1 to R40.30.70.20.20.20.20.30.2
Three settings0.30.60.20.20.20.20.20.2

Arxiv = ogbn-arxiv; Photo/Comp. = Amazon datasets; Roman = Roman-empire. * Our reimplementations. Rprop includes its no-calibration and no-smoothing variants; “Three settings” scores Rprop, R1, and R4 across Isolated/Mixed/HIDE.

The six estimates use runs with 10 and 40 epochs per task to extrapolate runtime at the reported configuration, including fixed overhead and scoring time. Checks against completed runs show both underestimation and overestimation. For SSM, the estimates underestimate measured runtime by 4.6%–39.7%, consistent with its increasing replay workload; its two extrapolated entries are therefore marked as estimated lower bounds.

The primary GCIL comparison reuses the same trained weights across Isolated, Mixed, and HIDE, so evaluating the additional settings requires rescoring rather than additional training. Hyperparameter reselection, training-edge controls, and separately fitted references incur additional fitting costs. Table 5 covers the main comparison, App. I additionally reports retained-state counts, which are distinct from peak GPU memory usage. The cascade and references do not have equal computational budgets.

We hope we have addressed your concerns. Please let us know if anything requires further clarification.

Reviewer_Wk1kOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper discusses the inadequacy of some existing evaluation protocols and systems for graph class-incremental learning and also proposed the HIDE evaluation setting. Experiments have been conducted to evaluate some existing methods under the settings.

Critical strengths

S1: The main strength of the paper is that it identifies several issues with existing evaluation protocols for graph class-incremental learning.

S2: The paper also conducts experiments to evaluate the performance of existing methods under different settings.

Critical weaknesses

W1. Writing and clarity. The paper is not well written, and many parts are difficult to understand. This may partly be because I am not an expert in graph class-incremental learning. However, I am familiar with continual learning, including class-incremental learning, as well as node classification in graph mining. I believe the paper needs improvements in clarity and presentation to make its contributions and experimental settings more accessible to readers.

W2. Information leakage in existing evaluation protocols. I agree with the authors that some existing evaluation protocols may inadvertently leak task information during class-incremental learning evaluation. However, this important issue is not sufficiently explained or discussed for each setting in the paper. More detailed explanations are needed to clarify how task information is leaked and why the proposed evaluation protocols address this problem for both training and testing.

W3. Practical relevance and interpretation of the settings in Table 1. It is unclear which of the settings presented in Table 1 is the most realistic and natural in practical graph class-incremental learning scenarios.

For example, consider the setting involving mixed data with inter-task edges (HIDE). The presence of inter-task edges necessarily implies the existence of nodes belonging to other tasks. If these nodes are labeled and their labels are available during training, the setting would appear to violate the class-incremental learning assumption. If they are unlabeled, the paper needs to explain how tasks are separated and how these unlabeled nodes are incorporated into the graph during training.

I understand that this HIDE setting is reasonable for evaluation purposes, but it is unclear how it would arise or be implemented during class-incremental training. In particular, if HIDE includes inter-task edges, the corresponding nodes at the other endpoints of these edges must also be present. The paper should clarify how these nodes and their labels are handled.

W4. Relationship between training and testing settings. It is unclear whether the settings in Table 1 apply to training, testing, or both. My understanding is that they primarily concern testing, but this should be explicitly stated.

If the testing setting differs substantially from the training setting, the training and testing distributions may no longer be identical. In such cases, a degradation in accuracy would be expected due to the distribution shift. The paper should distinguish performance degradation caused by information leakage or evaluation protocol issues from that caused by differences between training and testing conditions.

W5. Interpretation of the experimental results. It is natural for individual algorithms to be designed under specific assumptions and evaluation settings. If an algorithm is evaluated under a substantially different setting from the one for which it was designed, poor performance is not necessarily surprising.

Therefore, the paper should explain why the alternative evaluation settings are appropriate for assessing these methods and what conclusions can reasonably be drawn from their performance differences.

W6. Most realistic settings for graph class-incremental learning. In my view, the central question is which training and testing settings most accurately reflect realistic graph class-incremental learning scenarios.

The paper would benefit from a more thorough discussion of how graphs evolve in practice, what information is available at each incremental learning stage, and which evaluation protocols best represent these real-world conditions. Without this clarification, it is difficult to assess the practical significance of the proposed settings and the experimental findings.

W7. Missing references. The paper appears to be missing citations for some of the existing methods discussed or evaluated. The authors should ensure that all relevant prior methods are properly referenced.

Questions for authors

See the weaknesses section.

5 comments

Thank you for identifying where the setup needed clarification. Changes are marked in red in the manuscript.

W1. Clarity

We agree that the contributions and experimental settings needed a clearer presentation. We have revised the explanations across the introduction, background, protocol, and methods sections. Section 1 states the contributions and the research gap more explicitly. Section 2.1 clarifies the distinction between task-incremental and class-incremental learning, while Section 2.2 explains how HIDE relates to CGLB, LLM4GCL, and BeGin.

Section 3.1 now defines batch composition, scoring graph, aggregation, and data loading, and explains how their effects are measured separately. It also distinguishes average accuracy (AA) from per-node accuracy and explains how task-pure batches and task-filtered graphs can supply task information during evaluation. Section 3.2 specifies what is held fixed across comparisons and when training, calibration, and test labels are available. Table 1 separates training and scoring graphs, and Fig. 1’s expanded caption distinguishes training from the three evaluation settings and explains the visual notation.

We also expanded the cascade description in Section 4.3 to explain its task-routing and classification steps, batch dependence, and retained information. Finally, the new appendix guide connects the main-text sections to their supporting methods, results, and limitations. These changes aim to make both the experimental design and the interpretation of the results easier to follow.

W2. Task information in evaluation

We agree that the mechanisms and the distinction between training and evaluation needed clearer explanation. Task information can enter evaluation when a test node’s true task is used to organize its batch or select its neighborhood, even without an explicit task-ID input. Now Sec. 3.1 explains both mechanisms, and Sec. 2.2 credits LLM4GCL’s prior diagnosis.

In our primary comparison, all three settings evaluate the same trained models. Training uses the current task subgraph, its training labels, and each method’s permitted retained state; future-task nodes and labels are unavailable. Knowing the current task during training is permitted in class-incremental learning. At test time, however, the model must predict among all learned classes without being supplied the queried node’s task. Below are the settings (Table 1):

SettingBatch compositionScoring graphTask information supplied by evaluation
IsolatedOne task per batchEach node’s true-task subgraphBoth batch grouping and graph filtering use the true task.
MixedTasks evaluated togetherUnion of task subgraphsGraph filtering still uses the true task.
HIDETasks evaluated togetherAvailable full graphNeither batch grouping nor graph filtering uses the true task.

A task-pure batch can help a batch-based estimator, such as TPP’s, estimate a shared task from the batch features. Task filtering supplies a different cue by using the true task partition to exclude neighbors from other tasks. Mixed removes the batch-grouping cue but retains task-based graph filtering; HIDE removes both. HIDE therefore addresses these sources of task information during evaluation while preserving the permitted training information. We analyze aggregation and data loading separately.

W3. Nodes and labels

We agree that the manuscript needed to clarify when nodes, edges, and labels are available. HIDE’s inter-task edges describe the scoring graph; they do not imply that the same graph is available throughout training. We changed Table 1 to distinguish training and scoring graphs, and Fig. 1 separates the training and evaluation phases.

We construct benchmark tasks by partitioning classes into disjoint groups, normally two classes each. Dataset annotations define this partition, but only training labels reach the learner. Primary training uses the current task subgraph, its training labels, and each method’s permitted retained state. Unlabeled nodes in that subgraph provide features and connectivity; future-task nodes and their incident edges are absent.

At final evaluation, all evaluated tasks have been learned. HIDE scores nodes on the available full graph, including inter-task edges, while withholding test labels and task identities. Nodes from excluded classes remain only as unlabeled context (App. D). Both endpoints of an inter-task edge can therefore be present without exposing their test labels. This represents training on successive task subgraphs followed by prediction on a graph connecting the learned tasks.

If the graph is already observable during training and labels arrive incrementally, our BeGin controls examine that alternative: future-task nodes remain present in both training conditions, but their labels are unavailable for supervision (Sec. 5.3; App. G).

W4. Training–testing mismatch

We agree that the relationship between training and evaluation needed clarification. In the primary comparison, Isolated, Mixed, and HIDE score the same trained GCIL weights. Changing the scoring graph can introduce a training–evaluation mismatch, but our controls show that this does not fully explain the observed losses. Table 1 now separates training and scoring graphs; Sec. 5.3 and App. G add training-connectivity controls.

The Mixed–HIDE contrast measures the overall scoring-graph effect, including any mismatch. R1 and R2 use the same joint architecture trained without and with inter-task edges, respectively; we make the same training-edge comparison for PDGNNs. Each trained model is evaluated under both settings. Rprop is fitted separately on each scoring graph, so its fitting and scoring graphs always match. Below are these comparisons (Table 4):

DatasetModel and training graphMixed (%)HIDE (%)Loss (pp)
CoraFullPDGNNs, without inter-task edges86.2774.4411.84
CoraFullPDGNNs, with inter-task edges84.4570.2814.17
CoraFullJoint R1, without inter-task edges86.9173.2313.69
CoraFullJoint R2, with inter-task edges83.8171.0112.81
CoraFullRprop, fitted on each scoring graph86.7470.6416.10
ogbn-arxivPDGNNs, without inter-task edges69.4863.086.40
ogbn-arxivPDGNNs, with inter-task edges66.6160.346.27
ogbn-arxivJoint R1, without inter-task edges76.7867.479.31
ogbn-arxivJoint R2, with inter-task edges76.2268.038.19
ogbn-arxivRprop, fitted on each scoring graph89.9574.4615.49

Mean per-node accuracy (%) over ten seeds; loss = Mixed − HIDE in percentage points (pp), computed before rounding. Every loss is positive on all ten seeds.

Rprop loses 16.10 pp on CoraFull and 15.49 pp on ogbn-arxiv despite matched fitting and scoring graphs. PDGNNs trained with inter-task edges still lose 14.17 and 6.27 pp, respectively. Thus, the absence of inter-task edges during training cannot explain all observed losses.

Training with these edges lowers PDGNNs’ HIDE accuracy on both datasets and mean HIDE accuracy for the seven BeGin implementations in all 21 method–dataset pairs (App. G, Table 13). Training connectivity nevertheless matters: on Roman-empire, full-graph training reduces the joint model’s Mixed–HIDE loss from 36.26 to 2.59 pp (Table 14). These controls do not numerically separate task-information and distribution-shift effects. We therefore report the scoring-graph effect without attributing every drop solely to removing task information.

W5–W6. Why HIDE is appropriate

We agree that the evaluation must match the learning problem. HIDE assesses the requirement these GCIL methods address: predicting among all learned classes without the input’s task. Task identity may be available during training but is withheld at test time (van de Ven et al., 2022). Keeping GCIL weights fixed tests sensitivity to the evaluation choices; W4 separately examines training-connectivity mismatch. We changed secs. 2.1 and 3.1 to clarify this rationale. App. A explains that our class-index sequences are controlled benchmarks rather than reconstructions of chronological graph growth.

An example that illustrates how HIDE is an appropriate setup is a citation graph: after learning successive groups of subject areas, a model classifies held-out papers whose citations may cross those groups. Their unknown tasks cannot be used to select their neighborhoods. Isolated eases this problem through task-pure batches and task-filtered graphs; Mixed removes only the batching aid, while HIDE removes both. Training-graph availability depends on the application: the graph may already be observable, or task subgraphs may arrive progressively.

These choices change method comparisons. Among the four released implementations, TPP ranks first on seven of eight main datasets under Isolated (AA), and last on seven under HIDE (per node). This compares complete protocols, including aggregation. Below are the HIDE diagnostics showing the prediction difficulty (mean ± SD %, ten seeds, two classes/task; App. P, Table 26):

DatasetMethodPer-node accuracyTask hitAccuracy given the true task
CoraFullPDGNNs74.44 ± 0.6475.36 ± 0.5997.71 ± 0.21
CoraFullDeLoMe71.37 ± 1.0872.30 ± 1.0197.19 ± 0.27
CoraFullSSM63.16 ± 0.8764.16 ± 0.7896.31 ± 0.33
CoraFullTPP55.12 ± 0.6256.26 ± 0.5396.90 ± 0.16
CoraFullTAAM*56.55 ± 0.8857.67 ± 0.8597.18 ± 0.26
CoraFullDyMoE*60.92 ± 1.1362.11 ± 1.1588.79 ± 0.63
ogbn-arxivPDGNNs63.08 ± 0.2466.41 ± 0.2093.23 ± 0.24
ogbn-arxivDeLoMe34.59 ± 2.2238.64 ± 2.3085.49 ± 1.50
ogbn-arxivSSM56.08 ± 0.8858.44 ± 0.8692.87 ± 1.17
ogbn-arxivTPP32.13 ± 0.1532.77 ± 0.1697.14 ± 0.07
ogbn-arxivTAAM*43.28 ± 0.3245.60 ± 0.2994.79 ± 0.11
ogbn-arxivDyMoE*42.92 ± 2.0144.30 ± 2.0789.85 ± 1.15
  • Our reimplementations.

Task hit is the share whose predicted class belongs to the true task. Oracle accuracy restricts predictions to the true task’s classes over all test nodes; it is not accuracy conditioned on correct routing.

For TPP on ogbn-arxiv, 67.23 pp of its 67.87% error rate are wrong-task predictions; only 0.64 pp are wrong-class predictions within the correct task. Strong within-task discrimination therefore coexists with poor hidden-task performance. The results describe the tested implementations and configurations; larger tasks increase within-task difficulty (Sec. 5.2). Our class-index sequences are controlled benchmarks, not observed graph histories.

W7. References

We added the missing citations. Now Sec. 1 cites all six evaluated GCIL methods; Sec. 2.2 expands LLM4GCL, BeGin, and AGALE; Table 17 cites EWC, MAS, TWP, LwF, GEM, and ER-GNN. App. H identifies the released implementations and our TAAM/DyMoE reimplementations.

We hope we have addressed your concerns. Please let us know if anything requires further clarification.

Reviewer_dNNAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HIDE, an evaluation protocol for graph class-incremental learning that removes task information provided by task-pure batches and task-induced subgraphs. Experiments on six methods across eight datasets show substantial accuracy drops and ranking changes, highlighting task identification as a major challenge.

Critical strengths
  1. The paper identifies how task-pure batches and task-induced subgraphs expose task information, revealing an important limitation of existing evaluation protocols.

  2. Experiments across eight datasets show substantial accuracy drops and ranking reversals, demonstrating that evaluation choices materially affect conclusions about method performance.

  3. The analysis separates the effects of batch composition, scoring graphs, aggregation, and data loading, and distinguishes task-identification errors from within-task classification errors.

Critical weaknesses
  1. Most methods train without inter-task edges but encounter them under HIDE. The reported drops therefore do not isolate the effect of removing task information from that of training–evaluation mismatch.

  2. Most methods use their released configurations. DeLoMe’s substantial improvement after retuning suggests that the comparisons partly reflect configurations poorly suited to HIDE.

  3. The cascade uses batch-dependent normalization, but its reported zero batch cost comes from reusing predictions across settings. This does not establish that changing batch composition has no effect.

Questions for authors
  1. Could you clarify several statements in the conclusion? The reported drop of over 15 percentage points for PDGNN conflicts with the per-node drops of 11.83 and 6.40 points in Table 2. The claim that PDGNN underperforms the references and cascade also conflicts with its stronger performance on CoraFull.

  2. Section 5.3 reports a training-edge control for PDGNNs on its own graph. Is there evidence that the main findings persist when methods are trained with inter-task edges on the shared graph?

  3. How sensitive are TPP’s results to the per-node task estimator introduced in this study? Could alternative task estimators materially change the reported ranking reversal?

2 comments

Thank you for the careful review. We addressed the potential mismatch between training and testing with extra experiments where HIDE still shows a gap in performance, so our conclusions still hold. We ran both cascade and PDGNNs on the three assessed evaluation settings and updated our claims in the paper accordingly. Changes are detailed below.

W1 / Q2. Training–evaluation mismatch

We agree that a model trained without inter-task edges may lose accuracy under HIDE partly because its scoring graph differs from its training graph. Our controls show that this mismatch does not fully explain the observed discrepancy in performance. Table 1 now separates training and scoring graphs; Sec. 5.3 and App. G add shared-graph training controls.

Our reference classifiers already cover this comparison. R1 and R2 are the same joint model, trained on the task subgraphs (no inter-task edges) and on the full graph (with inter-task edges), respectively, so the difference between them measures the effect of training with these edges. Each trained model is scored under both Mixed and HIDE. Rprop is fitted separately on the graph it scores, so there is no mismatch between its fitting and scoring graphs. To extend this comparison to an incremental method, we additionally trained PDGNNs with and without inter-task edges. Below are the results (Sec. 5.3, Table 4):

DatasetModel and training graphMixed (%)HIDE (%)Loss (pp)
CoraFullPDGNNs, without inter-task edges86.2774.4411.84
CoraFullPDGNNs, with inter-task edges84.4570.2814.17
CoraFullJoint R1, without inter-task edges86.9173.2313.69
CoraFullJoint R2, with inter-task edges83.8171.0112.81
CoraFullRprop, fitted on each scoring graph86.7470.6416.10
ogbn-arxivPDGNNs, without inter-task edges69.4863.086.40
ogbn-arxivPDGNNs, with inter-task edges66.6160.346.27
ogbn-arxivJoint R1, without inter-task edges76.7867.479.31
ogbn-arxivJoint R2, with inter-task edges76.2268.038.19
ogbn-arxivRprop, fitted on each scoring graph89.9574.4615.49

Rprop loses 16.10/15.49 pp despite matched fitting and scoring graphs. PDGNNs trained with inter-task edges still decrease 14.17/6.27 pp. Thus, task-filtered scoring retains an advantage without previously unseen training connectivity.

PDGNNs is the only primary method whose released code supports this training change (App. G). Below are additional controls using seven BeGin implementations, both training conditions evaluated under HIDE (Table 13):

MethodCora (pp)CiteSeer (pp)ogbn-arxiv (pp)
Fine-tuning+6.33†+4.63+0.08†
LwF+3.58†+3.64+0.37†
EWC+8.24+4.26+7.66
MAS+19.30+2.82+0.79†
TWP+7.83+3.80+5.33
GEM+5.33†+3.11+4.54
ER-GNN+4.98†+3.65+8.39

Training without minus with inter-task edges, in pp; ten seeds on Cora/CiteSeer, five on ogbn-arxiv. † Seeds disagree on the sign. Both conditions retain future-task nodes with their labels hidden.

Training with inter-task edges does not restore accuracy: the mean is lower in all 21 pairs. Mismatch can nevertheless matter substantially: on Roman-empire, full-graph training reduces the joint model’s loss from 36.26 to 2.59 pp (Table 14). We therefore distinguish the scoring-graph effect from its possible causes.

W2. Configuration sensitivity

We agree that configuration sensitivity affects the comparisons.

We ran some new tests and now the App. F extends calibration-based reselection to CoraFull and to TPP on both datasets. The three replay-method runs on ogbn-arxiv were already in the submission. Below are changes in mean HIDE accuracy over ten seeds (selected − primary configuration; Table 12):

MethodCoraFull change (pp)ogbn-arxiv change (pp)
PDGNNs−0.06+0.13
DeLoMe0.00+14.43
SSM+0.040.00
TPP+0.020.00

We performed a grid search to find the best hyperparameters for the methods on the table above on learning rate {0.001; 0.005, 0.02} and weight decay {0, 0.0005}. We selected calibration nodes separately per seed, with no test labels. Other hyperparameters remain fixed.

DeLoMe rises from 34.59% to 49.02% on ogbn-arxiv, moving from fifth to third among six methods. Its sensitivity also involves graph encoding: at the primary configuration, accuracy is 44.51% on the directed shared graph versus 34.59% after symmetrization (Sec. 5.3). The other seven reselection changes are at most 0.13 pp; TPP stays last on both datasets. For TPP, this hyperparameter search changes its classifiers and learned prompts, but leaves task routing unchanged. From our evaluation only DeLoMe presented considerable distinction on performance after tuning.

W3. Cascade batch dependence

You are correct: reused predictions cannot establish batch invariance.

We reran routing for each batch composition with the selected subspace dimension fixed, corrected Tables 2–3, and stated the dependence in Sec. 4.3. Below are Mixed − Isolated changes in per-node accuracy (pp; ten seeds, 95% bootstrap intervals; Tables 3 and 33):

MethodCoraFull change (pp)ogbn-arxiv change (pp)
Cascade+0.16 [0.11, 0.21]+1.78 [1.52, 2.02]
TPP−16.89 [−17.35, −16.45]−49.00 [−49.09, −48.91]
TAAM*−36.46 [−36.91, −35.88]−49.45 [−49.62, −49.27]

The cascade is batch-dependent, with smaller changes than TPP and TAAM here. On the other six datasets its change ranges from −0.11 to +0.20 pp. * Our reimplementation. TPP also uses the per-node adaptation discussed in Q3.

Q1. PDGNNs claims

You are correct on both points. We changed Secs. 1 and 6 to correct the PDGNNs claims and identify TPP’s ranking reversal. Below are the relevant per-node accuracies (mean ± SD %, ten seeds; Table 2):

ModelEvaluationCoraFullogbn-arxiv
PDGNNsIsolated, per node86.27 ± 0.4369.48 ± 0.43
PDGNNsHIDE, per node74.44 ± 0.6463.08 ± 0.24
CascadeHIDE, per node71.86 ± 0.8673.56 ± 0.28
Joint R1HIDE, per node73.20 ± 0.5867.47 ± 0.22
Propagation RpropHIDE, per node70.64 ± 0.6074.46 ± 0.23
Joint R4HIDE, per node71.27 ± 0.5673.63 ± 0.18

PDGNNs’ per-node accuracy decreases by 11.84 pp on CoraFull and 6.40 pp on ogbn-arxiv from Isolated to HIDE. These differences are computed before rounding. Subtracting the displayed CoraFull accuracies gives 11.83 pp. Under HIDE, PDGNNs score higher than R1, Rprop, R4, and the cascade on CoraFull, but lower than all four on ogbn-arxiv. The first-to-last ranking reversal refers to TPP among the four methods evaluated using released implementations (PDGNNs, DeLoMe, SSM, and TPP), comparing Isolated AA with HIDE per-node accuracy.

Q3. TPP task estimator

The estimator can change TPP’s ranking. Among the tested alternatives using TPP’s own components, it remains last on both datasets. Using the cascade router improves its rank but requires additional retained information.

We found new evidence: Under every estimate built from TPP's own components, TPP stays last of the six on CoraFull and ogbn-arxiv. These runs are reported here. App. H, Table 15 describes the paper’s per-node adaptation of the released batch-prototype rule. Below are mean per-node accuracies over ten seeds; ranks are among the six methods at their primary configurations:

Task estimatorCoraFull (%)ogbn-arxiv (%)Rank, CoraFull / arxiv
Per-node prototypes, 3 hops (paper)55.1232.136th / 6th
Prototypes, best tested hops (2 / 5)56.4333.456th / 6th
Head with highest class probability44.2610.496th / 6th
Head with largest log-sum-exp43.7610.646th / 6th
Cascade router71.8273.612nd / 1st

The hop search tests 1, 2, 5, and 10 hops alongside the three-hop baseline. Its best row is selected using test labels: an optimistic diagnostic, not a deployable selection procedure. Its last-place rank describes the mean; the CoraFull gap to TAAM is only 0.12 pp.

The cascade router uses calibration information and a propagation classifier fitted on retained training/calibration labels, which TPP does not retain (App. I). Thus, it demonstrates the potential of TPP’s task-specific classifiers with stronger routing, rather than robustness to every estimator. Learning-rate/weight-decay reselection in W2 leaves the prototype router unchanged.

We hope we have addressed your concerns. Please let us know if anything requires further clarification.

Comments on the paper

2 comments

The authors still have not addressed my concern using of same data size for more objective unbiased performance comparison. Hence I am not willing to upgrade my original overall assessment score "3-weak accept".

AuthorsOpenReview ↗

Same Data Size Issue

Thank you for the follow-up. We would like to resolve this concern, and we believe we have not yet understood precisely which comparison you would like to see, so we ask for your help in specifying it.

Dataset sizes currently in the paper. With the three datasets added in the revision, the thirteen datasets fall into groups of similar node count (App. D, Table 8; App. N):

Size groupDatasets (nodes)
~3kCora (2,708), CiteSeer (3,327)
~7.6kAmazon Photo (7,650), LastFMAsia (7,624)
~12–14kWikiCS (11,701), Amazon Computers (13,752)
~18–25kCoauthor-CS (18,333), CoraFull (19,793), Roman-empire (22,662), Amazon-ratings (24,492)
~170–230kogbn-arxiv (169,343), Reddit (232,965)
2.4Mogbn-products (2,449,029; reference classifiers only)

Every dataset on which we compare the GCIL methods therefore has at least one counterpart of similar size. Within each group, the direction of the protocol effect is the same: methods lose per-node accuracy from Isolated to HIDE (Table 25).

What is held fixed. Every comparison between methods and between evaluation settings is made within a single dataset: same graph, same split, same class-to-task assignment, same test nodes, paired by seed (Sec. 3.2). We never compare a method's accuracy on one dataset with another method's accuracy on a different dataset, so differences in size between datasets do not enter any reported difference.

Question. Could you tell us specifically which experiment would address your concern? For instance:

  1. subsampling all datasets to a common number of nodes (e.g., ~7.6k) and repeating the Isolated/Mixed/HIDE comparison;
  2. equalizing another quantity across datasets, such as the number of classes, the number of tasks, or the number of training nodes per class;
  3. reporting the main rankings separately per size group of the table above;
  4. something else, in which case a short description of the datasets, methods, and metric you have in mind would let us run it.

We note that subsampling this type of graph problem may change degree distribution, homophily and the inter-task edge fraction, so any result may confound size with topology.

If you can indicate one of these, we will do our best to run it within the discussion period and report the results here. Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.