acceptodds
Under review as a conference paper at ICLR 2027

Providing Context: Enriching Graph Learning Using Language Model Priors

Abstract

We introduce GINAT, a graph-language augmentation framework that improves message-passing GNNs by conditioning their edge messages on LM-embedded global graph descriptions. Unlike text-attributed graph methods that rely on node/edge-attached text, GINAT leverages external prompts that capture graph semantics independently of the downstream task. These prompts are injected into message passing through a cross-attention mechanism that preserves permutation equivariance, enabling global contextual guidance within standard MPNN computation. Across a broad suite of node- and graph-level benchmarks, GINAT yields consistent gains over strong GNN baselines, reaching up to 23% average relative improvement. Ablations show that the gains arise from the semantic content of prompts rather than random vectors, and an analysis of the learned context scaling reveals where and how strongly different backbones and layers rely on language-conditioned information. Overall, GINAT establishes LM-embedded prompts as a practical, interpretable graph-language signal to strengthen graph representation learning. Our code is available at https://anonymous.4open.science/r/GINAT-1312/

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph neural networks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tzRwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GINAT, which augments GNN edge messages with global domain descriptions encoded by a frozen language model. Cross-attention and a scaled residual connection integrate the textual context into message passing. The authors establish permutation equivariance and report performance improvements across multiple backbones and node- and graph-level tasks.

Critical strengths
  1. Simple and modular design. The augmentation can be integrated into different message-passing backbones, preserves permutation equivariance, and recovers the original model when ξ=0\xi=0.

  2. Broad experimental coverage and consistent improvements in point estimates. Across Tables 2, 3, 10, and 11, GINAT improves 155 of 161 backbone–dataset combinations, supporting the module’s empirical value.

Critical weaknesses
  1. The independent contribution of prompt semantics and LM priors is not sufficiently isolated. Random vectors recover part of the gains, while PROTEINS and ENZYMES receive atom-and-bond descriptions that do not match their secondary-structure-element nodes; these results leave the contributions of domain semantics, pretrained representations, and the additional architecture unresolved (Tables 6 and 12).

  2. Large relative gains rely on near-chance baselines, with inconsistent input features. On featureless datasets, the baselines use random node features while GINAT uses constant features, confounding the comparison (Appendix F.1). Matched comparisons against degree features and cardinality-preserving attention are absent, although CPA reports 93.07%/57.39% on REDDIT-BINARY/MULTI5K under a different evaluation protocol (Zhang & Xie, 2020).

  3. Some performance claims exceed the supporting results. The “over 10%” improvement in Appendix F.3 does not correspond to the aggregate means in Table 7, while Table 5 contains ties and favors Concat on Chameleon despite claiming that GINAT performs best. The absence of a vanilla baseline also obscures the gains from each fusion method.

  4. The TAG adaptation removes node-specific textual information. Poor performance after replacing node-level text representations with global prompts does not establish that TAG methods cannot support this setting; the comparison primarily evaluates the particular adaptation used.

Questions for authors

Questions for Authors

  1. Appendix K lists atom-and-bond descriptions for PROTEINS and ENZYMES, although their nodes represent secondary structure elements. Could the authors confirm which prompts were used in the experiments and explain this choice?

  2. Appendix F.1 states that the baselines use random node features on featureless graphs, while GINAT uses constant features. What motivated this difference, and were the gains also observed when both models received identical node features?

  3. Table 6 shows that random conditioning vectors recover part of the improvement. Was the same cross-attention module evaluated with learned, text-independent vectors, and if so, how did it compare with LM-encoded prompts?

4 comments

We thank you for the careful reading. We address each of your concerns and describe the new experiments and corrections in the revision (we don’t report std in the tables below for space). In order to answer your concerts seriously and thoroughly, we divide our response in two.

PART 1/2 of our rebuttal:

Regarding molecular datasets used prompts We thank you for this crucial finding. Indeed, the prompts we used for PROTEINS and ENZYMES, which are the ones reported in Appendix K, don't describe them correctly. To address this, we created a new prompts dataset following the recipe we defined. We trained GINAT using all the backbones listed in the paper, and using the same configurations as listed in the manuscript. The results are summed in the tables below. The main takeaways are improved gains on both datasets, averaged across the different MPNNs: PROTEINS +3.3% gain relative to vanilla (was +3.1%), while ENZYMES now shows +18.3% relative gains (was +8.4%)!

ENZYMES

MPNNVanillaGINAT (old)Rel. Gain (old)GINAT (corrected)Rel. Gain (corrected)
GAT0.4220.467+10.7%0.467+10.7%
GATv20.4330.456+5.3%0.461+6.5%
GCN0.3610.361+0.0%0.567+57.1%
GIN0.6720.683+1.6%0.683+1.6%
GraphGPS0.6440.694+7.8%0.683+6.1%
GraphConv0.5170.594+14.9%0.622+20.3%
GraphSAGE0.3890.461+18.5%0.489+25.7%
Mean0.4910.531+8.4%0.567+18.3%

PROTEINS

MPNNVanillaGINAT (old)Rel. Gain (old)GINAT (corrected)Rel. Gain (corrected)
GAT0.7050.717+1.7%0.735+4.3%
GATv20.7110.750+5.5%0.741+4.2%
GCN0.7380.750+1.6%0.738+0.0%
GIN0.7710.777+0.8%0.750-2.7%
GraphGPS0.6790.750+10.5%0.762+12.2%
GraphConv0.7440.750+0.8%0.768+3.2%
GraphSAGE0.7560.762+0.8%0.768+1.6%
Mean0.7290.751+3.1%0.752+3.3%

We see these results as another evidence of the method’s merits. Your finding is similar to the experiment presented in Figure 4 in our manuscript. In that experiment, we have seen that using unrelated prompts provides some of the gains; Our incorrect assignment of prompts creates a similar scenario - relevant but inaccurate prompts. Those show meaningful gain on the baseline, but provide even better gain using accurate prompts.

The prompt dataset we used is now part of the anonymized git, and the revised version of our manuscript presents it, and the updated result. For your convenience, here are 2 sample prompts from that dataset:

  • Nodes representing a protein's secondary structural elements connected by edges
  • Vertices represent secondary structures; edges link them sequentially along the backbone and structurally in space Compared to the prompts dataset used in the manuscript, the new datasets had 200 prompts (compared to 193). Words per prompts: min 10, mean 12.2, max 15 (was 10, 13, 15, respectively).

** Regarding GINAT’s gains and performance ** We believe a reuse of the term “baseline” in Appendix F.3 caused confusion. The experiment in table 5 presents our exploration of other fusion strategies. It compares different fusion approaches, not the vanilla baseline. As written in that appendix, each result in the table is an aggregation of 54 seeded training and evaluations, spanning three MPNNs, three seeds, and six depth-width configurations (and for this reason, no single MPNN baseline is provided there).
These results have nothing to do with the MPNN baselines we compare GINAT to, and they serve only to explain why this specific architecture was chosen in our research. We corrected this misleading writing in our revised version to avoid this confusion in the future, and thank you for pointing it out for us. Regarding our TAG adaptation We thank you for the careful reading of our TAG experiments. When adapting TAPE, LLM4NG, and SimTeG, we chose the straightforward adaptation: we kept each method's original architecture, training procedure, and hyperparameters, only setting the node-level text with GINAT's global prompt. This setup hinders the core benefit these models have (distinguishing nodes by their description) which results in poor performance in this setting. The goal of this experiment was to explore the existing literature when proposing a new approach and is not a major part of this work and was kept in the appendix for completeness.

We underlined this in the revised version and gave this context to said experiment.

Part 2/2 of our rebuttal follows in the next comment.

PART 2/2 of our rebuttal:

Regarding random and constant node features Our motivation for this choice is based on theoretic findings in the literature and our empirical findings that were not included in the manuscript, but will be provided here and included in the revised version. As we write in the aforementioned appendix, Random Node Initialization (RNI) has been shown to improve GNNs’ expressivity on featureless graphs [1,2]. This motivated our choice for equipping the baseline with such. We also observed this phenomenon in our experiments as provided in the table below. GINAT empirically doesn’t benefit from this addition. We keep them as a future work direction, but we hypothesize that the random noise makes it harder for cross attention components to learn meaningful relations between the structural GNN signal and the contextual LLM signal.

In the table below, the GINAT (rand) and vanilla (rand) columns present the results of our and the baselines when using RNI, and the GINAT (const) and vanilla (const) counterparts are the constant node initialization for those models. The diff columns calculate the respective model’s performance using random node features minus its performance using constant node features. Each result is a mean of 3 seeds. On average, GINAT loses 3.8 accuracy points using RNI, while the vanilla baseline gains 1, empirically affirming our choice.

datasetMPNNGINAT (rand)GINAT (const)GINAT diffvanilla (rand)vanilla (const)vanilla diff
IMDB-Bgin0.6270.733-10.70.7270.730-0.3
IMDB-Bgraphconv0.6700.753-8.30.7370.727+1.0
IMDB-Bgat0.6670.753-8.70.5070.500+0.7
IMDB-Mgin0.4760.500-2.40.4220.476-5.3
IMDB-Mgraphconv0.4780.498-2.00.4760.442+3.3
IMDB-Mgat0.4730.478-0.40.3730.327+4.7
REDDIT-Bgin0.9150.945-3.00.9350.922+1.3
REDDIT-Bgraphconv0.9000.927-2.70.9170.917+0.0
REDDIT-Bgat0.8680.923-5.50.5420.510+3.3
REDDIT-M5Kgin0.5680.568-0.10.5660.560+0.6
REDDIT-M5Kgraphconv0.5650.572-0.70.5700.562+0.9
REDDIT-M5Kgat0.5430.552-0.90.2180.198+1.9
mean-3.8+1.0

Regarding text-independent vectors preformance We thank you for this question. This isolates the remaining confound in Table 6 well. To address your concern, we have conducted another experiment with a learned text-independent conditioning input that has the same shape and the same cross-attention pathway as GINAT, so it controls for the added architecture and capacity while removing the LM and the text entirely.

We replace each LM-encoded prompt embedding pi∈Rki×dp_i \in\Bbb R ^{k_i \times d} with a free parameter tensor of the same shape (initialized N(0,1), trained jointly with the GNN, same optimizer, lr, etc.). Rest is identical to GINAT: the cross-attention module, ξ, LayerNorm, the per-forward-pass prompt sampling, the backbone configs (Table 16), and the three seeds. Conditioning vectors are learned per prompt slot, so the number of distinct conditioning inputs matches the prompt dataset. We report mean over 3 seeds.

DatasetBackboneVanillaLearned vectors (text-independent)GINAT (LM-encoded prompts)
ENZYMESGCN0.3610.3280.361
ENZYMESGAT0.4220.3670.467
PROTEINSGCN0.7380.5710.750
PROTEINSGAT0.7050.6310.717
REDDIT-BGCN0.8430.6470.883
REDDIT-BGAT0.5420.7970.923
REDDIT-M5KGCN0.5210.3990.547
REDDIT-M5KGAT0.2180.5280.552

As this table shows, together with Table 6 (fixed random vectors) and the unrelated-prompt ablation (Fig. 4), this gives a three-level decomposition of gains: Architecture alone: learned or random vectors. Large effect only where the vanilla attention backbone collapses; otherwise neutral or harmful.

  • generic LM prior: unrelated prompts. Partial gains.
  • domain semantics: proper prompts. Best in every case. Meaning, our method manages to utilize the textual information in a meaningful and positive way.

We kindly ask that you reassess your score of our work. We remain committed to addressing any concern you might have. The Authors

References

[1] Abboud et al., The Surprising Power of Graph Neural Networks with Random Node Initialization. IJCAI 2021.

[2] Eliasof et al,. Graph Positional Encoding via Random Feature Propagation. ICML 2023.

Thank you for the additional experiments. Comparing the models with the same node features and testing learned vectors without text addressed my main questions about the evaluation. I’m therefore raising my score from 1 to 2. I see value in showing that domain descriptions can improve existing GNNs, though I still view this as a modest methodological advance.

AuthorsOpenReview ↗

Official Comment by Authors

Dear tzRw,

We thank you for your engaging and meaningful review of our paper. We appreciate you raising your score and helping us improve our work.

Thanks,

The Authors

Reviewer_kj5WOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GINAT, which combines the graph domain description encoded by the language model with the message passing of GNN through cross-attention. It does not require each node to have text, but instead utilizes short global descriptions to provide background information. The authors have verified the effect on multiple datasets and backbone networks, and analyzed the influence of different prompts and language encoders.

Critical strengths
  1. The paper is well-written, with clear diagrams, making it easy to understand the research motivation and methods.
  2. The method is concise and has some general applicability, allowing for the incorporation of various GNNs.
  3. The experiments are extensive, including ablation studies on random vectors, irrelevant descriptions, and different encoders, demonstrating that the author has carefully examined the sources of benefits.
  4. The discussion on why different backbones are effective or ineffective is quite enlightening.
Critical weaknesses
  1. The paper has compared models such as Graphormer and GraphGPS, and also discussed some work from 2025. However, I still want to know: Compared with more recent strong graph models, does GINAT have sufficient performance or cost advantages? Do related works such as GOFA and LLM-BP help readers better understand the research orientation of this paper? Although the experimental settings are different, direct comparison may not be necessary, but the paper should further explain these relationships.
  2. The appendix indicates that small language encoders can also achieve good results, and larger models may not necessarily be better. Since this paper mainly trains GNN for each dataset and the domain description is relatively fixed, I wonder: Can simple learnable context vectors also achieve similar effects? What exactly is the improvement observed, mainly coming from language knowledge, or from additional model structure and parameters? This also relates to whether this method really requires language models to have strong cross-domain generalization ability.
  3. The appendix mentions that in datasets without node features, the baseline model uses random features, while GINAT uses constant features. If the same initialization method is adopted, will the significant performance improvement still exist? Adding this control experiment will make the results more convincing.
  4. Finally, regarding the significance of the language information represented by the scaling coefficient, as well as the explanation for the poor performance of the TAG method under the current input settings, the author might appropriately limit the scope of the conclusion to avoid drawing overly broad conclusions from these experiments.
Questions for authors

As the Weaknesses.

5 comments

We thank you for the positive review and constructive comments. We address each of them below.

Regarding additional recent works

We thank you for the opportunity to further place our work in the current research landscape. Both models you listed are solving a TAG task - they assume each node is attributed with a node-specific textual description. As we discuss in the main paper, this is a slightly different task than the one we suggest. Mainly - TAGs encode the text for each prompt with general O(Nkˉ2),kˉO(N\bar k^2), \bar k being the average length of a node’s description. In contrast, we only use a single prompt with length kk per graph, thus O(k2)O(k^2). However, these computational differences stem from the different tasks the models aim at solving. This makes the comparison slightly ill-posed.

We thank you for bringing this to our attention and have added discussion of these models to the Related Work section of our revised version.

Regarding the sources of observed improvements

In your comment, you list three possible sources of gains: language cues (our claim), parameter count, and the additional model structure. We share your viewpoint, and thus planned our experiments accordingly:

  • Parameter counts: We made sure to use the same parameter count for vanilla baselines, by using the same number of layers and adjusting the vanilla width to match the GINAT parameter count. In all our experiments, the vanilla parameter count was within 2% of the GINAT parameter count.
  • Isolating the contribution of the cross-attention component: This addition can have a performance contribution of its own - e.g. have better gradient flows. In order to isolate that, we experimented with an additional source of external signal: pure noise instead of LM output. The results of these experiments (Figure 4 and Table 6 in our submission), as well as the experiment we added per your request below, show that some gains are independent of the language signal, which we attribute as stemming from the added model structure.
  • Contribution of the language signal: Our experiments show that the added LM signal provides the highest gains - specifically for relevant prompts, but also irrelevant prompts show gains. This is the most evident in Figure 4. Together, all these ablations answer your questions. Last - Your comment on a learned context is insightful and intriguing, and we added it as a future work direction. We thank you for this contribution.

Regarding same-feature distribution training on featureless graphs

As we write in Appendix F.1, Random Node Initialization (RNI) has been shown to improve GNNs’ expressivity on featureless graphs [1,2]. This motivated our choice for equipping the baseline with such. We also observed this phenomenon in our experiments as provided in the table below. GINAT empirically doesn’t benefit from this addition, and we hypothesize that the random noise makes it harder for cross attention components to learn meaningful relations between the structural GNN signal and the contextual LLM signal. In the table below, the GINAT (rand) and vanilla (rand) columns present the results of our and the baselines when using RNI, and the GINAT (const) and vanilla (const) counterparts are the constant node initialization for those models. The diff columns shows the respective model’s performance using RNI, minus its performance using constant node features. Each result is an average of three seeded training and evaluation runs (std dropped for space). On average, GINAT loses 3.8 points using random node features, while the vanilla baseline gains 1, empirically affirming our choice.

datasetMPNNGINAT (rand)GINAT (const)GINAT diffvanilla (rand)vanilla (const)vanilla diff
IMDB-Bgin0.6270.733-10.70.7270.730-0.3
IMDB-Bgraphconv0.6700.753-8.30.7370.727+1.0
IMDB-Bgat0.6670.753-8.70.5070.500+0.7
IMDB-Mgin0.4760.500-2.40.4220.476-5.3
IMDB-Mgraphconv0.4780.498-2.00.4760.442+3.3
IMDB-Mgat0.4730.478-0.40.3730.327+4.7
REDDIT-Bgin0.9150.945-3.00.9350.922+1.3
REDDIT-Bgraphconv0.9000.927-2.70.9170.917+0.0
REDDIT-Bgat0.8680.923-5.50.5420.510+3.3
REDDIT-M5Kgin0.5680.568-0.10.5660.560+0.6
REDDIT-M5Kgraphconv0.5650.572-0.70.5700.562+0.9
REDDIT-M5Kgat0.5430.552-0.90.2180.198+1.9
mean-3.8+1.0

We thank you for your positive review and constructive feedback, and ready to answer any other concern you may have.

The Authors.


References

[1] - Abboud et al., The Surprising Power of Graph Neural Networks with Random Node Initialization. IJCAI 2021.

[2] - Eliasof et al., Graph Positional Encoding via Random Feature Propagation. ICML 2023.

Dear reviewer kj5W,

Thank you again for your time and effort.

We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.

We would be happy to address any remaining concerns or provide further clarification. If our responses have addressed your remaining concerns, we would greatly appreciate it if you could consider strengthening your support for the paper.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer kj5W,

We hope that our rebuttal has adequately addressed the concerns you raised. To briefly summarize, we:

  • Clarified the distinction between GINAT and recent text-attributed graph (TAG) methods, highlighting their different tasks and computational requirements. Added discussion of these works to the Related Work section.
  • Explained how our experiments disentangle the contributions of parameter count, cross-attention architecture, and LM signals. Demonstrated that language signals provide the largest gains and added learned context as a future research direction.
  • Conducted additional experiments across 4 featureless graph benchmarks and 3 MPNN backbones, showing that random node initialization hurts GINAT performance by an average of 3.8 points, while improving vanilla baselines by 1.0 point. This supports our choice of constant node initialization for GINAT.
  • Clarified that vanilla baselines are parameter-matched to GINAT, with parameter count differences within 2%.

As the discussion period is approaching its conclusion, we would greatly appreciate hearing your final thoughts.

Thanks,

The Authors

Reviewer_6J6COpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes GINAT, which adds global text context to GNN messages through cross-attention. Short descriptions are embedded by a frozen LM, and each layer learns a scale for the added signal. The paper shows that the method preserves permutation equivariance and reports improvements across graph and node tasks, with ablations on prompts, encoders, and fusion methods.

Critical strengths
  1. The method offers a simple way to add global text context without requiring text for every node or jointly training an LM. It also preserves permutation equivariance and includes the original backbone as a special case.

  2. The reported gains cover several backbones and tasks. The prompt-source and noise ablations provide useful evidence that the conditioning representation matters.

Critical weaknesses
  1. It's unclear how much the largest gains need language. On featureless graphs, even a constant extra message can restore degree information, but the language-free controls weren't run on the Reddit/IMDB cases with the largest gains.

  2. The featureless comparisons change the node inputs: baselines use random features while GINAT uses constants, affecting the inductiveness etc

  3. The evaluation mostly uses older benchmarks and established GNN backbones, and several of the largest gains come from baselines that perform poorly in this setup. I'd like to see whether the gains still hold against stronger recent methods and on larger, more challenging benchmarks.

Questions for authors
  1. A Reddit/IMDB comparison with identical node features and a language-free control would be helpful, such as a constant or learned attention memory, or cardinality-preserving attention. This would help separate the language contribution from the added degree signal.

  2. What were the baseline and GINAT widths, parameter counts, and node-feature settings for the reported runs?

  3. It'd also be helpful to provide evidence from branch-output measurements or removal tests for the interpretation of the learned scales. A change in the scale can be offset by the attention weights without changing the model's output

4 comments

We thank you for the encouraging assessment. We appreciate you noticing the clear presentation, the general applicability of GINAT across GNN backbones, and the ablations on prompt sources and encoders. We address each concern you raised below, and split our rebuttal into two comments in order to thoroughly cover each.

PART 1/2 of our rebuttal

Regarding constant vs. random node initializations We thank you for the experiment idea and the opportunity to provide more evidence supporting our proposed approach. Per your comment, we conducted the following experiment: We trained GINAT with GAT as backbone (where gains were the most significant) on the REDDIT-B datasets in two settings: Our proposed setting using language controls, and a setting where the MPNN’s signal is cross attended with random noise (similarly to the experiment presented in Figure 4 and Table 6 in our manuscript, which was done on datasets with node features). Each setting was explored using both random and constant node initialization for the REDDIT-B nodes. The results of the experiment (each is an aggregation of 3 seeded training and eval runs) are in the table below. We highlight again that the vanilla baseline had the same 3.9M parameter count as GINAT.

ModelSignalNode featuresTest score
GINATQwen3-4Bconst0.923 ± 0.013
GINATQwen3-4Brand0.868 ± 0.040
GINATnormal noiseconst0.605 ± 0.023
GINATnormal noiserand0.630 ± 0.080
vanilla—rand0.542 ± 0.053
vanilla—const0.510 ± 0.068

These results follow a similar pattern to those we have seen in the experiments reported in Table 6: random signal provides some gains compared to the baseline by 5-10%, but the proposed GINAT has far more significant gains, again showcasing that the language contribution is separated from the added degree signal.

Regarding our evaluation benchmarks and backbones Thank you for raising this point. This work proposes that the well established class of MPNNs can be improved by adding auxiliary information as encoded by LMs. In order to show that, we compare 7 different such architectures, all frequently used baselines in the graph learning literature, on over 20 different benchmarks - frequently used as well - and exhibit performance gains. Our baseline results are faithful to the known performance on these benchmarks in the literature. Regarding your additional backbones request: we’ve included all relevant MPNNs established in the literature. Most recent GNNs aren’t MPNNs (we provided Graphormer and GraphMAE as examples of such) and thus are irrelevant to our setting. We would greatly appreciate any specific suggestion you might have. Regarding additional benchmarks, we agree that additional experimenting on additional benchmarks is always welcome and enriching. However, it takes time and resources to evaluate each. We note that our experimental protocol is agnostic to the downstream task, designed exactly so that the “quality” or “novelty” of benchmarks isn’t a major factor (specifically, no information is given to the LM on the downstream task). While not discounting the importance of this suggestion, we remain confident that the results presented in our paper establish the validity of our claim. Our evaluation suite includes 24 diverse, established benchmarks, covering social, citation, and transport networks, among others, as well as molecular and biological domains. They cover classification, regression, node- and graph-level tasks. Each is evaluated on 7 different MPNNs, which result in 168 different experiments; our approach beats the vanilla baseline on 160 of them.
Lastly - regarding your statement that “larger gains come from baselines that perform poorly in this setup”, we offer another perspective. Our largest gains were obtained on GAT and GATv2, mean-aggregation based MPNN. These specific architectures, and other mean-based MPNNs were shown to struggle on featureless benchmarks [3,4], even when equipped with established expressivity enhancing techniques, such as Random Node Initialization [1,2]. Their failure has nothing to do with our setup. We see the fact that our framework allowed these architectures to successfully infer on these benchmarks a demonstration of our approach’s strengths, rather than a weakness.

Part 2/2 of our rebuttal follows in the next comment.

PART 2/2 of our rebuttal

Regarding GINAT’s hyperparameters We kindly direct your attention to Tables 16 and 17 in our submission (p. 43), which present the MPNN configuration (depth and width) used to obtain each result. For your convenience, we add some of these configurations below as well. Baselines vanilla used the same depth and set the width such that the parameter count would be as close as possible, to account for gains originating in the additional computational complexity. The exact parameter count differs based on the depth and width configuration and the embedding method used for a specific dataset. To put some concrete figures, the mean GINAT parameter count across all backbones and configurations reported is 7.4M, with the maximum being 30M and the smallest being 1.3M. All baseline parameters count are similar to their GINAT counterparts up to at most 2% difference, always in favor of the baseline. This is also reflected in our published code and can be easily reproduced.

DatasetGATGATv2GCNGINGPSGraphConvGraphSAGE
Enzymes4 / 5124 / 2564 / 2564 / 2564 / 5125 / 5123 / 256
OGBG-MolHIV5 / 2564 / 5125 / 5123 / 2563 / 5125 / 5123 / 512
Proteins3 / 2563 / 5123 / 2565 / 5123 / 2565 / 5123 / 256

Regarding interpretation of learned ζ\zeta Per your suggestion, we performed the following experiment. Given a trained GINAT model with a certain learned ζ0≠0\zeta_0\ne 0, we evaluate it on the test set of the dataset (ENZYMES and PROTEINS, respectively) it was trained on with $\zeta= c \cdot \zeta_0, \ c \in \lbrace 2,1.5,1,0.5,0.1, 0.01 \rbrace $. The results are in the table below (aggregated over 7 MPNNs). For c=1c=1 we list the mean performance, and for any other cc value we provide the relative performance w.r.t that score. We also note that the 14 aggregated models learned ζ0∈[0.09,0.11]\zeta_0\in [0.09,0.11]. The results clearly show that modifying the scales (either diminishing or amplifying the LM signal) hurts GINAT performance, and that they are meaningful and accurately capture the contribution of the LM signal.

cENZYMESPROTEINS
2-32.5%-5.1%
1.5-21.0%-2.5%
10.4760.722
0.5-16.0%-12.2%
0.25-22.0%-16.6%
0.1-25.5%-19.4%
0.01-23.5%-20.5%

We kindly ask that you reassess your score of our work. We remain committed to addressing any concern you might have. The Authors

References

[1] Abboud et al., The Surprising Power of Graph Neural Networks with Random Node Initialization. IJCAI 2021.

[2] Eliasof et al,. Graph Positional Encoding via Random Feature Propagation. ICML 2023.

[3] Zhang et al., Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservation, IJCAI 2020.

[4] Purgał, Improving Expressivity of Graph Neural Networks, IJCNN 2020.

Dear reviewer 6J6C,

Thank you again for your time and effort.

We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.

We would be happy to address any remaining concerns or provide further clarification. If our responses have resolved your concerns, we would greatly appreciate it if you could reconsider your score.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 6J6C,

We hope that our rebuttal has adequately addressed the concerns you raised. To briefly summarize, we:

  • Conducted additional experiments demonstrating the distinct contribution of language signals compared to random-noise controls under different node initializations.
  • Clarified our evaluation scope, covering 7 MPNN architectures, 24 diverse benchmarks, and 168 experiments, with improvements in 160.
  • Clarified the significance of GINAT’s gains on GAT/GATv2 and their known limitations on featureless graphs.
  • Provided additional hyperparameter configurations and demonstrated fair parameter matching between GINAT and vanilla baselines.
  • Conducted inference-time scaling experiments demonstrating the meaningful contribution of the learned LM signal.

As the discussion period is approaching its conclusion, we would greatly appreciate hearing your final thoughts.

Thanks,

The Authors

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your thoughtful reviews and the opportunity to improve our manuscript. The revision corrects the biological-dataset prompts and improves the corresponding results, and adds controls for the sources of GINAT’s gains. Below we summarize the raised concerns, our response, and where to find the relevant details in the revision.

1. Corrected PROTEINS and ENZYMES prompts (tzRw W1/Q1)

Concern: The original prompts describe atoms and bonds, although the graph nodes represent secondary structure elements.

Response: We acknowledge and correct this error. We retrain all seven backbones with corrected secondary-structure prompts and unchanged configurations. The mean per-backbone relative gain increases from 8.4% to 18.3% on ENZYMES, with a smaller improvement on PROTEINS.

Where: Table 2, lines 216–240; Appendix K.1–K.3, lines 2165–2225.

2. Contribution, positioning, and computational scope (kj5W W1; 6J6C W3)

Concern: How does GINAT relate to recent graph–language models, and what performance or cost advantages does it offer?

Response: We clarify that GINAT augments existing MPNNs using frozen LM representations of global, task-agnostic descriptions. We frame the evaluation around controlled improvements to capacity-matched backbones.

Where: Appendix E.1–E.2, lines 1244–1280; Section 2.1, lines 193–205; Section 5, lines 515–520.

3. Sources of the gains: language, architecture, and capacity (tzRw W1/Q3; 6J6C W1/Q1–Q2; kj5W W2)

Concern: Do the improvements require language, or can additional parameters and language-free conditioning explain them?

Response: We match baseline capacity by adjusting widths at fixed depth. With the same GINAT architecture, LM prompts outperform learned, text-independent conditioning in all 12 comparisons. On REDDIT-BINARY, accuracy reaches 92.3% with language, versus 60.5% with Gaussian noise and 51.0% for vanilla GAT. Together with existing ablations, these controls support language benefits beyond added capacity and architecture.

Where: Appendix B, lines 1044–1050; Section 4.2 and Table 4, lines 455–504; Appendix F.11 and Table 13, lines 1807–1872.

4. Matched inputs on featureless graphs (tzRw W2/Q2; 6J6C W2/Q1; kj5W W3)

Concern: Do the gains persist when GINAT and vanilla models receive identical node features?

Response: We add constant- and random-feature comparisons for 3 MPNNs across four featureless datasets. On average, random initialization lowers GINAT accuracy by 3.8 pp and raises vanilla accuracy by 1.0 pp, supporting our feature choices.

Where: Appendix F.1 and Table 6, lines 1350–1392.

5. Large gains from weak attention baselines and variation across backbones (tzRw W2; 6J6C W3)

Concern: Why are the largest improvements concentrated in settings where vanilla attention models perform poorly?

Response: We provide the mechanistic analysis of how backbone messages interact with global context. This analysis explains both substantial improvements and settings where GINAT contributes little.

Where: Section 3.2, lines 343–399; Appendix G, lines 1890–1936.

6. Interpretation of the learned scales (6J6C Q3; kj5W W4)

Concern: Coefficient magnitudes alone may not establish the importance of language, because attention weights can compensate for their scale.

Response: We add an inference-time perturbation experiment that holds all other parameters fixed. Across seven backbones on ENZYMES and PROTEINS, both increasing and decreasing the learned scale reduce aggregate performance. For example, halving it reduces accuracy by 16.0% and 12.2% relative, respectively. This provides a functional sensitivity check beyond inspecting coefficient magnitudes.

Where: Appendix F.5, lines 1496–1503; Appendix F.10 and Table 12, lines 1782–1804.

7. Interpretation of the fusion comparison (tzRw W3)

Concern: The presentation conflates fusion-method comparisons with vanilla-backbone improvements and overstates which method performs best.

Response: We clarify that the results presented on Table 7 evaluate architectural alternatives, and not standard baselines. We rephrase our manuscript to communicate this better.

Where: Appendix F.3 and Table 7, lines 1420–1457.

8. Scope of the TAG adaptation results (tzRw W4; kj5W W4)

Concern: Replacing node-specific textual information with global prompts limits what can be concluded about TAG methods.

Response: We detail the adaptations of TAG models and explain that replacing their textual representations with shared global descriptions removes an important source of node-specific information. The results characterize this particular adaptation to our setting.

Where: Appendix F.9, lines 1705–1727 and 1766–1781.

We hope these revisions clarify the contribution and address the concerns raised. We welcome any further questions.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.