- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Hierarchical Response Preservation (HiRP), a continual adaptation method for pretrained graph-text models under class-incremental learning. The key observation is that preserving discrimination among previously learned classes is insufficient when newly introduced classes expand the prediction space and can override old predictions. HiRP addresses this issue by preserving a hierarchical response that keeps individual probabilities for old classes while aggregating all current-task classes into a single probability mass. It further uses the corresponding KL divergence to constrain response drift and guide optimization through geometry-aware updates with tangent correction. Experiments across three class-incremental graph-learning settings show that HiRP achieves strong final and average accuracy, particularly in the more challenging ten-task setting, while also mitigating the degradation of zero-shot transfer on held-out graph domains.
Critical strengths
-
The paper identifies an important issue in class-incremental learning. New classes can override correct old-class predictions even when the model still distinguishes well among old classes.
-
The method is intuitively designed. It preserves probabilities for individual old classes while grouping new classes together, which protects old-versus-new competition without preventing learning within the new-class group.
-
The empirical results are promising. HiRP outperforms the compared baselines across three continual-learning settings and retains zero-shot transfer performance on a held-out graph domain.
Critical weaknesses
-
The theoretical guarantee seems limited. It is a sufficient condition for individual predictions, whereas the actual method constrains average KL divergence over a finite support set. It therefore cannot guarantee that forgetting is prevented everywhere.
-
The evaluation setting is relatively constrained. Task boundaries are known, graphs are transductive, and the pretrained encoders remain frozen while the method mainly adapts the readout layer.
-
The evidence for broad generalization could be stronger. The experiments use a limited set of datasets and task orders, while some baselines differ in backbone, trainable components, and training budgets.
Questions for authors
See Weakness.
3 comments
Weakness 2
W2: Task Boundaries, Transductive Graph Access, and Frozen Encoders
- Task Boundaries Known During Training, Hidden at Inference: As stated in Section 3, we explicitly acknowledge that task boundaries are known during training, following the standard class-incremental protocol where each task introduces a new disjoint class set and defines the task-entry anchor . Crucially, however, task identities are strictly hidden at inference: every test node is classified across the unified set of all classes seen so far without any oracle task mask.
- Transductive Stream Access Versus Held-Out Domain Transfer: We also acknowledge that our main class-incremental streams use transductive graph access, which is standard in graph continual learning because nodes naturally reside within an existing network structure; only current-task training labels are used in the supervised loss, and validation or test nodes never enter the feature support. Separately, to evaluate generalization to completely unseen graphs, Section 6.4 Figure 7 and Appendix Table 4 test zero-shot transfer on History, Computer, and Instagram domains that are entirely excluded from continual adaptation and support memory.
- Matched Comparisons on Frozen Encoders: Freezing the pretrained encoders and adapting only the readout layer is a deliberate choice to preserve generalizable graph–text alignment, as sequential joint fine-tuning rapidly overfits the latest task and reduces History zero-shot accuracy from 49.69% to 1.46%. All matched readout baselines share the exact same frozen GraphCLIP features. Adding HiRP's protection to the unprotected Fisher proposal improves FA by +13.66 points, AA by +11.40 points, and AF by +13.96 points, and substantially outperforms joint fine-tuning, LoRA, G2LoRA, and SGD on the same GraphCLIP initialization.
| Method on GraphCLIP | FA ↑ | AA ↑ | AF ↑ |
|---|---|---|---|
| GraphCLIP joint fine-tuning | 15.37 | 9.88 | −78.03 |
| G2LoRA on GraphCLIP | 49.05 | 51.46 | −32.43 |
| SGD on frozen GraphCLIP | 50.24 | 50.05 | −41.57 |
| LoRA on GraphCLIP | 51.18 | 51.11 | −38.60 |
| Fisher proposal without KL | 61.77 | 58.46 | −28.70 |
| HiRP | 75.43 | 69.86 | −14.74 |
Weakness 3
W3: Generalization Across Backbones, Diverse Datasets, Longer Sequences, and Task Orders
To thoroughly evaluate generalization across backbones, datasets, sequence lengths, task orders, and matched budgets, we conducted four sets of new experiments:
- 1. Generalization to Two Additional Pretrained Backbones
ADAlignerandZeroG: OnADAlignerby Liu et al., 2026, we evaluated ten methods on the 10-task stream, including nine matched readout methods sharing identical frozen features, support memory, and training budgets alongsideADAligner-LoRA. HiRP achieves the best performance across all metrics, outperforming the strongest baseline Memory-KD by +2.75 FA, +10.78 AA, and +23.89 AF, and outperformingADAligner-LoRAby +42.15 FA, +35.81 AA, and +71.99 AF.
| Method on ADAligner | FA ↑ | AA ↑ | AF ↑ |
|---|---|---|---|
| MAS / EWC / OGD | 13.27 / 16.11 / 18.51 | 9.17 / 11.82 / 11.71 | −67.48 / −64.91 / −52.67 |
| ADAligner-LoRA | 13.92 | 9.57 | −85.48 |
| SGD / Normalized SGD | 18.42 / 19.07 | 10.97 / 12.70 | −60.31 / −58.49 |
| LwF / Memory-LwF | 18.93 / 19.79 | 11.03 / 11.09 | −60.19 / −67.79 |
| Memory-KD | 53.32 | 34.60 | −37.38 |
| HiRP | 56.07 | 45.38 | −13.49 |
On a third pretrained graph–text backbone ZeroG by Li et al., 2024 across three random seeds on the 4-graph, 35-class, 10-task stream, HiRP outperforms validation-tuned SGD by +16.16 FA, +14.89 AA, and +16.30 AF, and outperforms OGD by +21.43 FA, +19.47 AA, and +20.21 AF.
| Method on ZeroG | FA ↑ | AA ↑ | AF ↑ |
|---|---|---|---|
| OGD | 44.90 ± 0.68 | 47.08 ± 0.78 | −41.78 ± 3.25 |
| Validation-tuned SGD | 50.18 ± 0.21 | 51.65 ± 0.07 | −37.86 ± 0.45 |
| HiRP | 66.33 ± 1.11 | 66.54 ± 0.83 | −21.56 ± 0.69 |
- 2. Expanding Dataset Diversity With TEG-DB by Li et al., 2024 on 7-Graph and 8-Graph, 86-Class Streams: We added three new graph datasets from TEG-DB—Children, Crime, and Apps—expanding our total evaluation to 12 distinct graph domains. Appending each dataset to the 6-dataset sequence forms three 7-graph streams, where HiRP improves AA by +7.23 to +9.51 points over LwF and +8.36 to +13.12 points over EWC, with entries reported as FA / AA / signed AF.
| Method | + Children | + Crime | + Apps |
|---|---|---|---|
| LwF | 77.11 / 60.88 / −15.36 | 79.85 / 63.34 / −14.36 | 83.68 / 64.28 / −15.31 |
| EWC | 73.78 / 57.27 / −19.95 | 75.87 / 60.83 / −17.59 | 81.91 / 63.15 / −17.05 |
| HiRP | 84.73 / 70.39 / −5.28 | 83.82 / 71.37 / −5.60 | 83.67 / 71.51 / −7.59 |
Combining Cora, CiteSeer, WikiCS, Photo, Sports-Fitness, Actor, Children, and Apps into an 8-graph, 8-task, 86-class stream across three random seeds, HiRP achieves the highest FA and AA across all compared methods.
| Method on 8 graphs | FA ↑ | AA ↑ | AF ↑ |
|---|---|---|---|
| G2LoRA, four epochs | 76.97 | 58.62 | −12.39 |
| GraphCLIP-LoRA | 79.61 | 62.30 | −11.67 |
| Validation-tuned SGD | 78.08 | 64.92 | −6.61 |
| TPP-style | 74.72 | 68.76 | −1.03 |
| HiRP | 82.88 | 69.75 | −5.77 |
- 3. Longer 23-Task Sequence and Matched Controls: Splitting the six graphs and 53 classes into 23 sequential tasks, we compared HiRP against standard continual learning baselines across three seeds and against matched optimization and target controls sharing identical task-1 checkpoints, support sets, and three-epoch budgets. On the 23-task stream, HiRP outperforms EWC by +36.49 FA, +15.86 AA, and +3.01 AF, outperforms LwF by +24.50 FA, +28.09 AA, and +35.02 AF, and outperforms the matched ordinary-gradient dual-KL procedure by +14.63 FA, +8.50 AA, and +7.02 AF.
| Stream and method | FA ↑ | AA ↑ | AF ↑ |
|---|---|---|---|
| 10 tasks: Ordinary gradient with dual KL | 61.78 | 59.41 | −18.91 |
| 10 tasks: Full-posterior KL with Fisher and tangent | 70.76 | 66.67 | −17.10 |
| 10 tasks: HiRP | 75.43 | 69.86 | −14.74 |
| 23 tasks: MAS | 23.77 | 35.78 | −57.89 |
| 23 tasks: OGD | 23.13 | 37.06 | −56.07 |
| 23 tasks: LwF | 59.61 | 41.18 | −50.40 |
| 23 tasks: SGD | 42.55 | 45.33 | −42.31 |
| 23 tasks: EWC | 47.62 | 53.41 | −18.39 |
| 23 tasks: Ordinary gradient with tangent and dual KL | 69.48 | 60.77 | −22.40 |
| 23 tasks: HiRP | 84.11 | 69.27 | −15.38 |
- 4. Five Additional Random Task Orders: Across five additional random task orders on the 10-task stream under identical dual-KL budgets, Fisher updates with radial shrinking outperform ordinary-gradient updates with radial shrinking on every single order, improving the five-order mean AA from 59.02% to 69.37%. Full HiRP with tangent correction further raises the five-order mean FA from 74.16% to 75.46%, mean AA from 69.37% to 70.96%, and mean last-task accuracy from 88.10% to 94.10%.
Weakness 1
Thank you for the encouraging comments and thoughtful suggestions. Below we clarify our theoretical guarantee coverage, evaluation protocol, and expanded generalization experiments across new backbones, datasets, sequence lengths, and task orders.
W1: Pointwise Stability Guarantee Versus Support-Averaged KL Constraint