acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Response Preservation for Continual Adaptation of Zero-Shot Graph–Text Models

Abstract

Pretrained graph–text models align graph representations with textual semantics, enabling recognition of unseen classes and transfer across graph domains. However, as graph data and classes continually arrive, models should learn from new supervision while retaining their zero-shot transfer capabilities and historical task knowledge. Two challenges arise: (i) new classes can overturn historical predictions despite preserved distinctions among historical classes, and (ii) overly strict response preservation can stall learning of new classes. To address these challenges, we propose Hierarchical Response Preservation (HiRP). HiRP represents this competition through a hierarchical response that keeps each historical-class probability and sums new-class probabilities, preserving historical distinctions and aggregate competition while allowing distinctions within the new class group to adapt. It further uses the geometry induced by this response to guide constrained updates, retaining useful adaptation directions while controlling response drift. Across three class-incremental settings, HiRP achieves absolute gains of 1.84–7.95 percentage points in average accuracy over the strongest compared baseline in each setting, while mitigating zero-shot transfer degradation. Code is available at https://anonymous.4open.science/r/HiRP-807C/.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph continual learning, class-incremental learning, graph foundation models, graph-text models, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TwCqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Hierarchical Response Preservation (HiRP), a continual adaptation method for pretrained graph-text models under class-incremental learning. The key observation is that preserving discrimination among previously learned classes is insufficient when newly introduced classes expand the prediction space and can override old predictions. HiRP addresses this issue by preserving a hierarchical response that keeps individual probabilities for old classes while aggregating all current-task classes into a single probability mass. It further uses the corresponding KL divergence to constrain response drift and guide optimization through geometry-aware updates with tangent correction. Experiments across three class-incremental graph-learning settings show that HiRP achieves strong final and average accuracy, particularly in the more challenging ten-task setting, while also mitigating the degradation of zero-shot transfer on held-out graph domains.

Critical strengths
  1. The paper identifies an important issue in class-incremental learning. New classes can override correct old-class predictions even when the model still distinguishes well among old classes.

  2. The method is intuitively designed. It preserves probabilities for individual old classes while grouping new classes together, which protects old-versus-new competition without preventing learning within the new-class group.

  3. The empirical results are promising. HiRP outperforms the compared baselines across three continual-learning settings and retains zero-shot transfer performance on a held-out graph domain.

Critical weaknesses
  1. The theoretical guarantee seems limited. It is a sufficient condition for individual predictions, whereas the actual method constrains average KL divergence over a finite support set. It therefore cannot guarantee that forgetting is prevented everywhere.

  2. The evaluation setting is relatively constrained. Task boundaries are known, graphs are transductive, and the pretrained encoders remain frozen while the method mainly adapts the readout layer.

  3. The evidence for broad generalization could be stronger. The experiments use a limited set of datasets and task orders, while some baselines differ in backbone, trainable components, and training budgets.

Questions for authors

See Weakness.

3 comments
AuthorsOpenReview ↗

Weakness 1

Thank you for the encouraging comments and thoughtful suggestions. Below we clarify our theoretical guarantee coverage, evaluation protocol, and expanded generalization experiments across new backbones, datasets, sequence lengths, and task orders.

W1: Pointwise Stability Guarantee Versus Support-Averaged KL Constraint

  • Theoretical Relationship: In Section 5.1 Theorem 1 and Appendix A.3 Equations 20 to 23, Theorem 1 provides a pointwise sufficient condition that preserves an individual historical prediction cc whenever its reference response margin δR>0\delta_R > 0 and pointwise drift satisfies DH<δR2/2D_H < \delta_R^2 / 2. Controlling the source-weighted average KL divergence E[DH]≤ε\mathbb{E}[D_H] \leq \varepsilon on the support set bounds the tail probability of large pointwise drift via Markov's inequality Pr⁡(DH≥τ)≤ε/τ\Pr(D_H \geq \tau) \leq \varepsilon / \tau.
  • Empirical Certification on Unseen Test Data: In Appendix B.5 Table 10 and Figure 9, we measured how often Theorem 1's pointwise condition holds on unseen test data across our four single-graph streams, observing 80.86% to 96.42% task-averaged coverage with 0 out of 58,297 certified test predictions forgotten. Extending this audit to our three multi-graph streams over the expanded candidate set shows 86.50%, 87.47%, and 88.96% task-averaged coverage on the 4-, 6-, and 10-task streams, where 0 out of 486,572 certified node-transition records are forgotten. Following Appendix B.5, task-averaged coverage averages the per-transition certification rates equally across tasks so large graphs do not dominate the rate, whereas the record counts pool all node-transition pairs across the sequence.
ProtocolTask-averaged coverageCertified retainedCertified forgottenUncertified retainedUncertified forgotten
4 tasks86.50%16,10902,578331
6 tasks87.47%394,282022,2193,091
10 tasks88.96%76,18108,8054,307
AuthorsOpenReview ↗

Weakness 2

W2: Task Boundaries, Transductive Graph Access, and Frozen Encoders

  • Task Boundaries Known During Training, Hidden at Inference: As stated in Section 3, we explicitly acknowledge that task boundaries are known during training, following the standard class-incremental protocol where each task introduces a new disjoint class set and defines the task-entry anchor θ0\theta_0. Crucially, however, task identities are strictly hidden at inference: every test node is classified across the unified set of all classes seen so far C≤t\mathcal{C}_{\le t} without any oracle task mask.
  • Transductive Stream Access Versus Held-Out Domain Transfer: We also acknowledge that our main class-incremental streams use transductive graph access, which is standard in graph continual learning because nodes naturally reside within an existing network structure; only current-task training labels are used in the supervised loss, and validation or test nodes never enter the feature support. Separately, to evaluate generalization to completely unseen graphs, Section 6.4 Figure 7 and Appendix Table 4 test zero-shot transfer on History, Computer, and Instagram domains that are entirely excluded from continual adaptation and support memory.
  • Matched Comparisons on Frozen Encoders: Freezing the pretrained encoders and adapting only the readout layer is a deliberate choice to preserve generalizable graph–text alignment, as sequential joint fine-tuning rapidly overfits the latest task and reduces History zero-shot accuracy from 49.69% to 1.46%. All matched readout baselines share the exact same frozen GraphCLIP features. Adding HiRP's protection to the unprotected Fisher proposal improves FA by +13.66 points, AA by +11.40 points, and AF by +13.96 points, and substantially outperforms joint fine-tuning, LoRA, G2LoRA, and SGD on the same GraphCLIP initialization.
Method on GraphCLIPFA ↑AA ↑AF ↑
GraphCLIP joint fine-tuning15.379.88−78.03
G2LoRA on GraphCLIP49.0551.46−32.43
SGD on frozen GraphCLIP50.2450.05−41.57
LoRA on GraphCLIP51.1851.11−38.60
Fisher proposal without KL61.7758.46−28.70
HiRP75.4369.86−14.74
AuthorsOpenReview ↗

Weakness 3

W3: Generalization Across Backbones, Diverse Datasets, Longer Sequences, and Task Orders

To thoroughly evaluate generalization across backbones, datasets, sequence lengths, task orders, and matched budgets, we conducted four sets of new experiments:

  • 1. Generalization to Two Additional Pretrained Backbones ADAligner and ZeroG: On ADAligner by Liu et al., 2026, we evaluated ten methods on the 10-task stream, including nine matched readout methods sharing identical frozen features, support memory, and training budgets alongside ADAligner-LoRA. HiRP achieves the best performance across all metrics, outperforming the strongest baseline Memory-KD by +2.75 FA, +10.78 AA, and +23.89 AF, and outperforming ADAligner-LoRA by +42.15 FA, +35.81 AA, and +71.99 AF.
Method on ADAlignerFA ↑AA ↑AF ↑
MAS / EWC / OGD13.27 / 16.11 / 18.519.17 / 11.82 / 11.71−67.48 / −64.91 / −52.67
ADAligner-LoRA13.929.57−85.48
SGD / Normalized SGD18.42 / 19.0710.97 / 12.70−60.31 / −58.49
LwF / Memory-LwF18.93 / 19.7911.03 / 11.09−60.19 / −67.79
Memory-KD53.3234.60−37.38
HiRP56.0745.38−13.49

On a third pretrained graph–text backbone ZeroG by Li et al., 2024 across three random seeds on the 4-graph, 35-class, 10-task stream, HiRP outperforms validation-tuned SGD by +16.16 FA, +14.89 AA, and +16.30 AF, and outperforms OGD by +21.43 FA, +19.47 AA, and +20.21 AF.

Method on ZeroGFA ↑AA ↑AF ↑
OGD44.90 ± 0.6847.08 ± 0.78−41.78 ± 3.25
Validation-tuned SGD50.18 ± 0.2151.65 ± 0.07−37.86 ± 0.45
HiRP66.33 ± 1.1166.54 ± 0.83−21.56 ± 0.69
  • 2. Expanding Dataset Diversity With TEG-DB by Li et al., 2024 on 7-Graph and 8-Graph, 86-Class Streams: We added three new graph datasets from TEG-DB—Children, Crime, and Apps—expanding our total evaluation to 12 distinct graph domains. Appending each dataset to the 6-dataset sequence forms three 7-graph streams, where HiRP improves AA by +7.23 to +9.51 points over LwF and +8.36 to +13.12 points over EWC, with entries reported as FA / AA / signed AF.
Method+ Children+ Crime+ Apps
LwF77.11 / 60.88 / −15.3679.85 / 63.34 / −14.3683.68 / 64.28 / −15.31
EWC73.78 / 57.27 / −19.9575.87 / 60.83 / −17.5981.91 / 63.15 / −17.05
HiRP84.73 / 70.39 / −5.2883.82 / 71.37 / −5.6083.67 / 71.51 / −7.59

Combining Cora, CiteSeer, WikiCS, Photo, Sports-Fitness, Actor, Children, and Apps into an 8-graph, 8-task, 86-class stream across three random seeds, HiRP achieves the highest FA and AA across all compared methods.

Method on 8 graphsFA ↑AA ↑AF ↑
G2LoRA, four epochs76.9758.62−12.39
GraphCLIP-LoRA79.6162.30−11.67
Validation-tuned SGD78.0864.92−6.61
TPP-style74.7268.76−1.03
HiRP82.8869.75−5.77
  • 3. Longer 23-Task Sequence and Matched Controls: Splitting the six graphs and 53 classes into 23 sequential tasks, we compared HiRP against standard continual learning baselines across three seeds and against matched optimization and target controls sharing identical task-1 checkpoints, support sets, and three-epoch budgets. On the 23-task stream, HiRP outperforms EWC by +36.49 FA, +15.86 AA, and +3.01 AF, outperforms LwF by +24.50 FA, +28.09 AA, and +35.02 AF, and outperforms the matched ordinary-gradient dual-KL procedure by +14.63 FA, +8.50 AA, and +7.02 AF.
Stream and methodFA ↑AA ↑AF ↑
10 tasks: Ordinary gradient with dual KL61.7859.41−18.91
10 tasks: Full-posterior KL with Fisher and tangent70.7666.67−17.10
10 tasks: HiRP75.4369.86−14.74
23 tasks: MAS23.7735.78−57.89
23 tasks: OGD23.1337.06−56.07
23 tasks: LwF59.6141.18−50.40
23 tasks: SGD42.5545.33−42.31
23 tasks: EWC47.6253.41−18.39
23 tasks: Ordinary gradient with tangent and dual KL69.4860.77−22.40
23 tasks: HiRP84.1169.27−15.38
  • 4. Five Additional Random Task Orders: Across five additional random task orders on the 10-task stream under identical dual-KL budgets, Fisher updates with radial shrinking outperform ordinary-gradient updates with radial shrinking on every single order, improving the five-order mean AA from 59.02% to 69.37%. Full HiRP with tangent correction further raises the five-order mean FA from 74.16% to 75.46%, mean AA from 69.37% to 70.96%, and mean last-task accuracy from 88.10% to 94.10%.
Reviewer_xXq8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses continual adaptation of pretrained graph–text models. The objective is to learn new classes while avoiding harm to historical knowledge and zero-shot transfer capabilities. Using individual historical-class probabilities and the aggregate probability over new classes enables learning distinctions among newly introduced classes. The method further uses KL-based constraints and Fisher-guided updates with tangent correction to limit prediction drift while adapting to new tasks.

Critical strengths
  1. The accuracy decomposition and diagnostics motivate the identified failure well that the historical classes can lose to new classes even when old classes are well-discriminated.

  2. The hierarchical response is simple and interpretable and has allowed protecting individual historical-class probabilities and the new-class mass while allowing adoption of new classes.

  3. Explicit empirical improvements in accuracy over the reported baselines in all three settings.

Critical weaknesses
  1. The implemented method constrains the average KL over a finite support rather than enforcing the theorem's pointwise conditions.

  2. HiRP introduces several heuristic design choices and interacting hyperparameters. The results don't sufficiently justify these choices for individual components of the method; for example, the improvements in tangent correction average accuracy are small, while forgetting worsens. An isolated ablation on several components is needed

  3. The evaluation uses only six distinct continual-learning datasets, with substantial overlap across the three settings, particularly only one pretrained graph–text backbone. It is unclear how well the findings extend to other domains or models.

  4. Some baselines use different representations, backbones, or adaptation mechanisms, and this avoids isolating improvements to HiRP's preservation method. At least matched-backbone comparisons would be more informative.

Questions for authors
  1. Can the authors relate the implemented support-averaged KL constraint to historical test accuracy or report how often the conditions of Theorem 1 hold in practice?

  2. Which components are necessary? Could the authors isolate the effect of these: the two KL constraints, Fisher geometry, support weighting, and tangent correction?

  3. Do the gains persist with other pretrained graph–text models, more diverse datasets, and substantially longer task sequences?

  4. What is the wall-clock training overhead relative to a matched readout baseline, and which operations dominate it as the number of classes grows?

5 comments
AuthorsOpenReview ↗

Weakness 1 & Question 1

Thank you for the insightful comments. Below we connect our support-averaged KL constraint to pointwise test stability, show how our components stem from a single unified principle with isolated ablations, expand our experiments to three pretrained backbones, twelve graph datasets, and a 23-task stream, and provide a self-contained runtime analysis.

W1 and Q1: Relating Support-Averaged KL Constraints to Pointwise Prediction Stability

  • Theoretical Link: In Section 5.1 Theorem 1 and Appendix A.3 Equations 20 to 23, a historical prediction cc is guaranteed to remain unchanged when its task-entry response margin δR=pca−max⁡{max⁡o∈O∖{c}poa,qNa}\delta_R = p_c^a - \max\{\max_{o \in \mathcal{O} \setminus \{c\}} p_o^a, q_N^a\} is positive and its pointwise drift satisfies DH<δR2/2D_H < \delta_R^2 / 2 via Pinsker's inequality. Enforcing the empirical source-weighted average E[DH]≤ε\mathbb{E}[D_H] \leq \varepsilon on the support set bounds the fraction of support inputs with large pointwise drift via Markov's inequality Pr⁡(DH≥τ)≤ε/τ\Pr(D_H \geq \tau) \leq \varepsilon / \tau.
  • Empirical Coverage on Unseen Test Data: Appendix B.5 Table 10 and Figure 9 audited Theorem 1 across all four single-graph streams, finding 80.86% to 96.42% task-averaged coverage with 0 out of 58,297 certified test predictions forgotten. Extending this audit to our three multi-graph streams over the expanded candidate set shows 86.50% to 88.96% task-averaged coverage, and all 486,572 certified test records remain 100% correct. Following Appendix B.5, task-averaged coverage averages the per-transition certification rates equally across tasks so large graphs do not dominate the rate, whereas the record counts pool all node-transition pairs across the sequence.
ProtocolTask-averaged coverageCertified retainedCertified forgottenUncertified retainedUncertified forgotten
4 tasks86.50%16,10902,578331
6 tasks87.47%394,282022,2193,091
10 tasks88.96%76,18108,8054,307
AuthorsOpenReview ↗

Weakness 2 & Question 2

W2 and Q2: Unified Design Principle and Isolated Component Ablations

  • How One Core Idea Induces the Method: Rather than assembling independent heuristics, HiRP follows a single chain:
    1. Core Finding: Forgetting arises not only from losing discrimination among historical classes, but from new-class competition overtaking historical predictions.
    2. Core Innovation: The hierarchical response retains individual historical-class probabilities and aggregates current classes into one probability mass, protecting old-versus-new competition while allowing free redistribution among current classes.
    3. Unified Optimization: This single hierarchical response directly defines the step- and task-level KL budgets, induces the local Fisher metric for step scaling, and specifies the boundary geometry for tangent correction. Source-equal support weighting simply balances domains of different graph sizes.
  • Single-Component Controlled Ablations: Isolating one factor at a time on the 10-task setting under identical initialization, support features, and three-epoch budgets confirms each role. Task-level KL (+11.25 AA) prevents cumulative drift over a task, while step-level KL (+1.79 AA) enforces local validity. Fisher geometry (+10.30 AA) guides update directions, and the hierarchical target (+3.19 AA, +2.36 AF) outperforms full-posterior KL. For tangent correction, while the original 10-task order yields +1.07 FA and +0.15 AA, across five additional random task orders it improves mean AA by +1.59 points, mean last-task accuracy by +6.00 points, and final historical accuracy RR by +1.10 points; on 8-, 12-, and 23-task streams, it improves FA, AA, and signed AF simultaneously. Around default settings in Appendix B.9 Figure 10, varying the tangent trigger threshold and CG iteration cap yields stable AA within 69.72% to 69.88%.
Isolated componentΔFAΔAAΔAF
Fisher proposal vs. ordinary gradient+12.58+10.30+6.15
Task KL in Dual KL vs. Step KL only+12.59+11.25+15.94
Step KL in Dual KL vs. Task KL only+1.91+1.79+1.52
Hierarchical target vs. full-posterior KL+4.67+3.19+2.36
Source-equal vs. sample-equal weighting+1.55+1.44+1.63
Tangent on 10 tasks across 5 new orders+1.30+1.59−0.42
Tangent on 23 tasks+1.45+1.73+0.74
AuthorsOpenReview ↗

Weakness 3 & Question 3

W3 and Q3: Additional Backbones, Diverse Datasets, and Longer Task Sequences

We expanded our evaluation to two additional pretrained graph–text backbones ADAligner and ZeroG, three new graph domains from TEG-DB, and a 23-task sequence.

  • 1. Second Pretrained Graph–Text Backbone ADAligner by Liu et al., 2026: Across ten methods on the 10-task stream sharing identical frozen ADAligner features and support memory alongside ADAligner-LoRA, standard regularization, current-input distillation, and LoRA suffer severe forgetting with AA at or below 12.70%. HiRP achieves the best performance across all metrics, outperforming Memory-KD by +2.75 FA, +10.78 AA, and +23.89 AF, and outperforming LoRA by +42.15 FA, +35.81 AA, and +71.99 AF.
Method on ADAlignerFA ↑AA ↑AF ↑
MAS on readout13.279.17−67.48
ADAligner-LoRA13.929.57−85.48
EWC on readout16.1111.82−64.91
Ordinary gradient SGD18.4210.97−60.31
OGD on readout18.5111.71−52.67
LwF18.9311.03−60.19
Normalized ordinary gradient19.0712.70−58.49
Memory-LwF19.7911.09−67.79
Memory-KD53.3234.60−37.38
HiRP56.0745.38−13.49
  • 2. Third Pretrained Graph–Text Backbone ZeroG by Li et al., 2024 Across Three Seeds: On the 4-graph, 35-class, 10-task stream across three random seeds, HiRP outperforms validation-tuned SGD by +16.16 FA, +14.89 AA, and +16.30 AF, and outperforms OGD by +21.43 FA, +19.47 AA, and +20.21 AF.
Method on ZeroGFA ↑AA ↑AF ↑
OGD44.90 ± 0.6847.08 ± 0.78−41.78 ± 3.25
Validation-tuned SGD50.18 ± 0.2151.65 ± 0.07−37.86 ± 0.45
HiRP66.33 ± 1.1166.54 ± 0.83−21.56 ± 0.69
Gain over SGD+16.16+14.89+16.30
Gain over OGD+21.43+19.47+20.21
  • 3. Three New Graph Domains From TEG-DB by Li et al., 2024 on 7-Graph and 8-Graph, 86-Class Streams: Incorporating Children, Crime, and Apps from TEG-DB expands our evaluation to 12 distinct graph datasets. Appending each domain to the 6-dataset sequence forms three 7-graph streams, where HiRP improves AA by +7.23 to +9.51 points over LwF and +8.36 to +13.12 points over EWC, reported below as FA / AA / signed AF.
Method+ Children+ Crime+ Apps
LwF77.11 / 60.88 / −15.3679.85 / 63.34 / −14.3683.68 / 64.28 / −15.31
EWC73.78 / 57.27 / −19.9575.87 / 60.83 / −17.5981.91 / 63.15 / −17.05
HiRP84.73 / 70.39 / −5.2883.82 / 71.37 / −5.6083.67 / 71.51 / −7.59

Combining Cora, CiteSeer, WikiCS, Photo, Sports-Fitness, Actor, Children, and Apps into an 8-graph, 8-task, 86-class stream across three seeds, HiRP achieves the highest FA and AA, outperforming TPP-style by +8.16 FA and +0.99 AA, validation-tuned SGD by +4.80 FA, +4.83 AA, and +0.84 AF, GraphCLIP-LoRA by +3.27 FA, +7.45 AA, and +5.90 AF, and four-epoch G2LoRA by +5.91 FA, +11.13 AA, and +6.62 AF.

Method on 8 graphsFA ↑AA ↑AF ↑
G2LoRA, four epochs76.9758.62−12.39
GraphCLIP-LoRA79.6162.30−11.67
Validation-tuned SGD78.0864.92−6.61
TPP-style74.7268.76−1.03
HiRP82.8869.75−5.77
  • 4. Substantially Longer 23-Task Sequence: Splitting the six graphs and 53 classes into 23 tasks across three seeds shows that standard baselines degrade severely, with standard deviations within 0.80%. HiRP achieves the best FA, AA, and signed AF, outperforming EWC by +36.49 FA, +15.86 AA, and +3.01 AF, outperforming LwF by +24.50 FA, +28.09 AA, and +35.02 AF, and outperforming our matched ordinary-gradient dual-KL control by +14.63 FA, +8.50 AA, and +7.02 AF.
Method on 23 tasksFA ↑AA ↑AF ↑
MAS23.7735.78−57.89
OGD23.1337.06−56.07
LwF59.6141.18−50.40
SGD42.5545.33−42.31
EWC47.6253.41−18.39
Ordinary gradient with tangent and dual KL69.4860.77−22.40
HiRP84.1169.27−15.38
Reviewer_uRxwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual adaptation of pretrained graph–text models. HiRP preserves individual historical-class probabilities and the total probability of current classes through a hierarchical response. It combines a response-induced Fisher metric, per-step and task-level KL constraints, and tangent correction to support learning new classes while retaining historical knowledge and zero-shot transfer.

Critical strengths

The same hierarchical response defines both the preservation constraints and the update geometry. Experiments cover three cross-graph class-incremental settings and single-graph scenarios, with strong overall accuracy.

Critical weaknesses
  1. The incremental value of the additional complexity is not fully established. The Fisher proposal requires matrix operations, conjugate-gradient solves, and a damping parameter. Table 4 shows that the Fisher proposal performs worse without the full constraints, but does not establish whether ordinary gradients would suffice under the same KL constraints. Per-step and task-level KL constraints serve different purposes, yet a comparison between task-level constraints alone and both constraints is missing. Tangent correction also involves a trade-off. It improves FA by about four percentage points in the four- and six-task settings, but in the ten-task setting AA increases only from 69.71% to 69.86%, while signed AF worsens by about 1.98 percentage points. Whether a simpler approach could achieve comparable performance remains unclear.

  2. The attribution of some key gains remains incomplete. Table 2 compares the hierarchical response with binary group mass, while Memory-KD in Table 12 distills probabilities over all arrived classes using a different optimization procedure. These comparisons do not fully isolate the benefit of allowing redistribution within the current classes. The zero-shot gap in Figure 7 also reflects whether the encoders are frozen. Although Table 4 provides a matched control, it uses a single run, leaving limited evidence of stability.

Questions for authors
  1. Under the same hierarchical response and KL constraints, could ordinary gradients achieve comparable performance? What are the separate benefits of the per-step and task-level constraints?

  2. With identical feature support and the same KL feasibility checks and update rules, what is the independent advantage of hierarchical response preservation over preserving the probabilities of all arrived classes?

  3. If model capacity is allowed to vary, could an approach with moderate capacity expansion achieve comparable performance to HiRP in handling competition between historical and current classes?

6 comments
AuthorsOpenReview ↗

Core Motivation and Novelty

Thank you for the constructive feedback. Below we first clarify our core motivation and unified design principle, followed by single-component controlled ablations and multi-run experiments.

Core Motivation and Novelty

  • Preventing Cross-Group Encroachment Without Stalling Plasticity: In continual adaptation of pretrained graph–text models, catastrophic forgetting occurs largely because newly introduced class boundaries encroach upon historical classes and cause new-class overfitting, even when relative discrimination among historical classes remains intact. Conversely, rigidly constraining the full posterior over all arrived classes blocks the probability redistribution required to distinguish newly introduced classes.
  • A Unified Hierarchical Framework Rather Than Ad-Hoc Modules: Our core contribution is Hierarchical Response Preservation, which protects individual historical-class probabilities alongside the aggregate probability mass of current classes. Rather than assembling independent heuristics, every component in HiRP is derived directly from this single hierarchical response: it defines the local and task-level KL drift budgets, induces the local Fisher metric used to scale parameter steps, and specifies the boundary geometry for tangent projection when cumulative drift approaches the task limit.
AuthorsOpenReview ↗

Weakness 1 & Question 1

W1 and Q1: Fisher Proposal, Dual KL Constraints

1. Could ordinary gradients achieve comparable performance under the same hierarchical response and KL constraints?

  • Theoretical Justification: Equal-length parameter steps in Euclidean space can alter protected probabilities by very different amounts. As shown in Section 5.3, the response-induced Fisher metric G=F+λIG = F + \lambda I captures this local sensitivity. For current cross-entropy gradient gg, solving the local trust-region problem yields the steepest descent direction within the response-metric ball:

d∗=arg⁡min⁡d⊤Gd≤δ2g⊤d=−δG−1gg⊤G−1gd^* = \arg\min_{d^\top G d \leq \delta^2} g^\top d = -\delta \frac{G^{-1}g}{\sqrt{g^\top G^{-1}g}}

  • Empirical Isolation: We conducted a controlled comparison on the 4-dataset, 10-task setting that changes only the proposal direction while holding the task-1 checkpoint, support set, minibatches, three-epoch budget, metric normalization, scalar shrinking, and dual KL checks identical. Replacing ordinary gradients with the Fisher proposal improves FA by +12.58 points, AA by +10.30 points, and signed AF by +6.15 points. When both variants additionally incorporate tangent correction, the Fisher proposal still improves FA by +2.50 points and AA by +3.29 points.
Proposal directionFA ↑AA ↑AF ↑
Ordinary gradient61.7859.41−18.91
Fisher with CG74.3669.71−12.76
Gain from Fisher+12.58+10.30+6.15

2. What are the separate benefits of the per-step and task-level KL constraints?

  • Distinct Functional Roles: As formulated in Section 5.2 and Equation 9, the per-step KL constraint bounds local drift between consecutive iterates so the local quadratic Fisher approximation stays accurate, whereas the task-level KL constraint bounds cumulative drift from the fixed task-entry model θ0\theta_0. Locally small updates can accumulate substantial drift across an entire task, making the task-level anchor essential.
  • Empirical Ablation: On the 10-task setting, keeping Fisher normalization and step-KL enforcement shows that task-level KL provides the primary gain of +11.25 AA and +15.94 AF. Adding exact nonlinear step-KL enforcement on top of task-KL provides a complementary gain of +1.79 AA and +1.52 AF.
KL constraint configurationFA ↑AA ↑AF ↑
Step KL only62.8458.61−30.68
Task KL only73.5268.07−16.26
Dual KL75.4369.86−14.74
Gain from Step KL+1.91+1.79+1.52
Gain from Task KL+12.59+11.25+15.94
AuthorsOpenReview ↗

Weakness 1 - Tangent Correction

3. Tangent correction trade-off and evaluation across task orders and stream lengths

  • Decomposition of Signed Forgetting: In Appendix B.6 and Table 11, signed forgetting is defined as AF=R−Q\mathrm{AF} = R - Q, where QQ is initial acquisition accuracy when a task is first learned and RR is final accuracy on historical tasks. On the original 10-task order, tangent correction increases acquisition QQ by +1.58 points and boosts last-task accuracy LA\mathrm{LA} by +5.00 points, while final historical accuracy RR changes by only −0.39 points. Because initial acquisition increases much more than historical accuracy changes, the difference R−QR - Q becomes more negative even though the model learns new tasks substantially better and achieves higher overall FA and AA.
  • Consistency Across Five Additional Task Orders: We evaluated five additional random task orders on the 10-task stream with the hierarchical response, Fisher geometry, support budget, and KL limits held fixed. Tangent correction improves AA in all five orders by +1.31 to +1.79 points and raises mean last-task accuracy LA\mathrm{LA} by +6.00 points. Notably, mean final historical accuracy RR also increases by +1.10 points; mean AF is 0.42 points lower solely because initial acquisition QQ rises even more by +1.52 points.
Update rule on 10 tasksFA ↑AA ↑AF ↑LA ↑
Hierarchical with radial74.1669.37−13.6788.10
Hierarchical with tangent75.4670.96−14.0994.10
Tangent minus radial+1.30+1.59−0.42+6.00
  • Improvements in Both AA and AF on 8-, 12-, and 23-Task Streams: When partitioning the six graphs and 53 classes into 8, 12, and 23 tasks, tangent correction improves FA, AA, and signed AF simultaneously across all three stream lengths. In the 23-task stream, among boundary updates at or above 95% of the task KL limit, radial shrinking causes 34.1% of accepted step norms to collapse below 10−310^{-3} of the raw proposal norm, whereas tangent correction reduces this collapse rate to 7.8%, confirming the boundary learning mechanism in Section 6.6 and Figure 8.
TasksUpdate ruleFA ↑AA ↑AF ↑
8Hierarchical with radial84.8771.39−7.29
8Hierarchical with tangent86.1273.01−6.72
12Hierarchical with radial84.0269.25−10.61
12Hierarchical with tangent85.4671.06−9.93
23Hierarchical with radial82.6667.54−16.12
23Hierarchical with tangent84.1169.27−15.38

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reply to All: Summary of Author Responses and Additional Evidence

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We have carefully addressed each concern through theoretical clarifications, controlled ablations, and extensive additional experiments. Below, we summarize our key responses to each reviewer.

To Reviewer uRxw — Core Novelty, Component Necessity, and Controlled Comparisons

We clarified that HiRP is built upon a unified principle: preserving individual historical-class probabilities and aggregate current-class mass to prevent cross-group interference without restricting within-current-class learning. Under matched conditions, our additional experiments demonstrate that:

  • Fisher geometry improves AA by +10.30 points over ordinary gradients under identical hierarchical KL constraints.
  • Hierarchical preservation improves AA by +3.19 points and signed AF by +2.36 points over full-posterior preservation.
  • Task-level and step-level KL constraints provide complementary AA gains of +11.25 and +1.79 points, respectively.
  • Tangent correction improves mean AA by +1.59 points across five additional task orders and improves both AA and signed AF on longer task streams.
  • Capacity-expansion comparisons show that HiRP outperforms task-specific branches with up to 19.79% additional parameters under task-hidden inference.

We further conducted three paired runs to verify zero-shot transfer stability, confirming a +2.81 ± 0.31-point improvement in the three-domain mean over the unprotected Fisher proposal.

To Reviewer xXq8 — Theoretical Validity, Generalization, and Computational Efficiency

We connected the support-averaged KL constraint to pointwise prediction stability through Markov's inequality and extended our empirical certification analysis. Across the three multi-graph streams, all 486,572 certified unseen test records retained their historical predictions, with task-averaged certification coverage ranging from 86.50% to 88.96%.

To establish broader generalization, we substantially expanded our evaluation to three pretrained graph–text backbones (GraphCLIP, ADAligner, and ZeroG), 12 distinct graph datasets, additional task orders, and sequences of up to 23 tasks. HiRP consistently demonstrates strong performance under matched comparisons, including +10.78 AA over Memory-KD on ADAligner, +14.89 AA over tuned SGD on ZeroG, and +8.50 AA over the matched ordinary-gradient dual-KL control on the 23-task stream.

We also provided detailed computational complexity, wall-clock profiling, and acceleration results. HiRP completes the full 10-task training sequence in 7.18 minutes, compared with 16.93 and 20.72 minutes for LoRA and G2LoRA, respectively, while achieving substantially higher accuracy.

To Reviewer TwCq — Evaluation Protocol, Fair Comparisons, and Generalizability

We clarified that task boundaries are available during training but task identities are strictly hidden at inference, requiring classification across all previously encountered classes. We also explained the transductive setting and our deliberate choice to freeze pretrained encoders to preserve graph–text alignment.

To isolate the contribution of HiRP, we provided matched-backbone comparisons using identical pretrained features and controlled training conditions. We further strengthened the generalization evidence through the additional backbones, datasets, longer task sequences, and randomized task orders described above.

Overall, these additional results reinforce our central contribution: hierarchical response preservation provides an effective and unified mechanism for balancing historical knowledge retention and new-class adaptation. The controlled experiments isolate the benefits of its individual components, while the expanded evaluations demonstrate its effectiveness across different backbones, domains, and continual-learning scenarios.

We greatly appreciate the reviewers' time and thoughtful feedback, and hope these clarifications and new results address their concerns. We welcome any further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.