- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces a structure-aware fine-tuning framework for decoder-only LLMs. It is motivated by a widely discussed observation: serializing a graph imposes an arbitrary linear order on it, which misaligns positions with graph relations and biases aggregation by letting later tokens see more under causal attention. To address this, it provides a) summary and readout tokens, b) a role-based RoPE variant (SR-RoPE), c) a structural attention mask with learnable biases, and d) a decorrelation loss on the readouts. Evaluated on GraphInstruct, MolecularNet, node classification, and link prediction, the proposed framework is competitive with the listed baselines.
Critical strengths
- The method does not require a separate graph encoder.
- The presented empirical results appear competitive across the evaluated tasks and included baselines.
Critical weaknesses
-
The narrative is not fluent, making the core methodologies harder to digest than they should be. As far as I can tell, the method splits the graph into isolated node blocks, like Block-Attention for RAG [1] or Set-LLM [2], with each block listing the node's incident edges to inject connectivity. These blocks do not propagate messages directly; cross-node information is combined through a single global readout step and causal generation. The remaining components, i.e., its core contributions (summary tokens, SR-RoPE, and structural mask), mostly refine how each block is encoded and read out by global tokens. As a result, the method appears more complex than its core idea requires.
-
One core technical contribution of this paper is SR-RoPE. However, it mixes role and position into a single 1D coordinate (Eq. 4, 5), which may cause positional collisions for variable-length sequences. It is also unclear why roles need to enter RoPE, rather than using simpler role embeddings or attention biases. Moreover, the global-position term lets block order back into the encoding, weakening the paper's own motivation.
-
The empirical results are not convincing. Important Graph-LLM baselines such as GOFA [3] are cited but missing from comparison. The reported Cora accuracy (51.62%) is significantly lower than GOFA's zero-shot performance (70.81%). This is possibly due to the collisions mentioned in W2 or the highly isolated masks. Also, the main baseline SLASH is a training-free method, while the proposed method is reported finetuned with LoRA (Sec. 3.1).
[1] Dongyang Ma, Yan Wang, and Tian Lan. Block-Attention for Efficient Prefilling. ICLR 2025.
[2] Beni Egressy and Jan Stühmer. Set-LLM: A Permutation-Invariant LLM. NeurIPS 2025.
[3] Lecheng Kong, Jiarui Feng, Hao Liu, Chengsong Huang, Jiaxin Huang, Yixin Chen, and Muhan Zhang. GOFA: A Generative One-For-All Model for Joint Graph Language Modeling. ICLR 2025.
Questions for authors
- Regarding balancing aggregation, one of the core motivations and design goals, why is it realized through only a single global readout?
- Can the authors provide results comparing with the simpler baseline, i.e., Block Attention + listing each node's incident edges within its block?
7 comments
Response to W2
Thank you for your suggestion. Please allow us to clarify this issue.
(1) We agree that one-dimensional coordinates are not collision-free. In fact, we introduced “role anchors” in the original manuscript to address this issue. For example, when the global coefficient is zero, an NH token at local index 2 and an ER token at local index 2 both receive local coordinate 2. Although a positional collision occurs at this point, the coordinates of the NH and ER tokens change further because we introduce “role anchors.” Specifically, the new coordinate of the NH token is ; that is, the position of the graph start token, 1, and a positive number are added to the original coordinate 2. Similarly, the new coordinate of the ER token is ; that is, a positive number is added to the last . We noticed that Equation (5) did not describe this point rigorously enough and have corrected it in the revised manuscript. We hope this explanation addresses your concern.
(2) Regarding “why roles need to be introduced into RoPE and why simpler role embeddings are not used,” we hope the following explanation resolves your confusion:
Standard RoPE directly uses global physical sequence positions. This mechanism is naturally suited to language sequences because word order and relative distances usually carry actual semantic meaning. However, different regions of a serialized graph have different ordering properties. Specifically, natural-language queries and answers have a strict semantic order, as do the contents of an individual edge record. In contrast, the multiple incident edges of the same node, different node summaries, and multiple global graph readout tokens are inherently closer to unordered sets. If all tokens use a uniform physical-position scheme, tokens that have no semantic order in the graph structure will be incorrectly assigned relative distances. For example, multiple edge summaries within a node block are inherently closer to an unordered set and do not require explicit positional attributes. Different orderings of the edge records within a node block change the positions of the edge summaries, which in turn change the attention scores assigned to them by the node summary. It is therefore necessary to reduce the differences in relative positions between a node summary and all edge summaries within the same node block.
On this basis, we further introduce “roles” into RoPE to distinguish different regions of the graph sequence. We use an explicit set of rules, based on the inherent general relationships among roles, to determine role embeddings and accommodate different “relative-position requirements.” In contrast, simple role embeddings, such as randomly initialized embeddings, may deliberately distinguish “roles” at the beginning of training, but the differences among these “roles” will become blurred as training progresses because there are no explicit rule-based constraints.
(3) Regarding the concern that “global positions weaken the paper's own motivation,” we fully agree with your perceptive observation. We must acknowledge that global positions do weaken the paper's motivation to some extent. At the same time, we would like to clarify that this weakening is relatively limited in magnitude and is controllable. On the one hand, as stated in Equation (4), only the Global role retains the full global position, and this part accounts for a relatively limited proportion of the entire graph sequence. On the other hand, in Equation (4), only NH and ER have weighted global positions, whereas ES, NS, and U, which are more closely related to reasoning, have no global positions at all.
Response to W3
Thank you for your suggestion. Please allow us to clarify this issue.
(1) First, we explain the “performance difference between our method and GOFA”:
For the first stage of large-scale pretraining, GOFA uses data from MAG240M, Arxiv, PubMed, Wikikg90mv2, Ultrachat200k, and WikiGraph. Each epoch contains 935,000 sampled training instances, and there are three epochs, giving approximately 2.805 million instance presentations in total. In contrast, our pretraining data come from seven of the following eight datasets: Wisconsin, Cornell, Citeseer, Cora, Pubmed, Amazon-Computers, Amazon-Photos, and Texas. When testing on Cora, for example, the other seven datasets, excluding Cora, serve as the training set. These eight datasets contain a total of 26,839 samples; detailed statistics are provided in Table 15. Therefore, our method and GOFA are not on the same scale in terms of pretraining. In particular, Arxiv, which is closely related to Cora, itself contains approximately 169,000 nodes, over one million edges, and as many as 40 classes.
For the second stage of instruction tuning, although GOFA does not train or fine-tune on the test dataset, it still performs extensive instruction tuning on its pretraining datasets. For example, it performs “natural-language instruction–answer” instruction tuning on 80,000 Arxiv samples. By comparison, our method follows the R-GFM setting: we do not perform “natural-language instruction–answer” instruction tuning on the pretraining datasets, and instead select only one sample per class from the target test dataset for such instruction tuning.
In summary, the performance difference between our method and GOFA on the same test dataset does not arise from the models themselves, but rather from different data splits and training settings. Nevertheless, we did overlook a comparison with GOFA, and we sincerely appreciate this constructive suggestion. In the revised manuscript, we have added a comparison with GOFA (Graph Foundation Models) under our experimental configuration. The detailed results are shown below and correspond to Tables 7, 8, and 9 in the revised manuscript:
Node classification
| Method | Wisconsin | Cornell | Citeseer | Cora |
|---|---|---|---|---|
| GOFA | 34.26 ± 2.14 | 32.07 ± 3.06 | 37.46 ± 4.10 | 43.36 ± 1.92 |
| Method | Pubmed | Computers | Photos | Texas | Avg. |
|---|---|---|---|---|---|
| GOFA | 47.24 ± 3.67 | 48.60 ± 3.76 | 58.13 ± 6.76 | 29.53 ± 1.22 | 41.33 |
Link prediction
| Method | Wisconsin | Cornell | Citeseer | Pubmed |
|---|---|---|---|---|
| GOFA | 81.45 ± 1.32 | 82.66 ± 1.09 | 89.07 ± 1.24 | 86.37 ± 0.73 |
| Method | Cora | Photos | Texas | Avg. |
|---|---|---|---|---|
| GOFA | 86.09 ± 0.64 | 80.35 ± 0.80 | 85.26 ± 1.51 | 84.46 |
(2) Second, please allow us to explain the concern that “the main baseline, SLASH, is a training-free method, whereas the proposed method is reported to have been fine-tuned with LoRA”:
As stated in RQ1 in Appendix E, “How does our method perform with decoder LLMs across different model families and parameter scales?” The main purpose of Section 3.2.1 is to verify whether our method consistently improves performance across different LLMs, whereas comparisons with other models are presented in Section 3.2.2. We fully understand your confusion. In fact, our primary purpose is to compare against the Vanilla results already reported in SLASH, while SLASH itself serves more as an additional point of comparison.
Response to Q1
Thank you for your suggestion. Please allow us to address your confusion. In fact, the “balance” we seek refers to “equality in the scope of aggregation”: each readout unit can see the complete set of node summaries but cannot see the other readout units, while each node summary can see all edge summaries within its own block but cannot see content in other node blocks. This design aims to eliminate the asymmetric prefix-access pattern inherent in LLMs. The “global readout” is actually one of the components that serve this motivation. In fact, the “global readout” in our method consists of multiple tokens, each of which performs a “global readout” with the same visibility scope.
Response to Q2
Thank you very much for this constructive suggestion. We fully agree that this is a crucial simple baseline. In fact, your suggestion also reflects Reviewer ohrD's observation that “sequential component-addition experiments measure only conditional increments under a particular ordering and cannot clearly identify the independent contributions or interactions of individual components,” especially the “Structural Mask” component. To address this issue, we fix “Hierarchical serialization” and “SR-RoPE” and design four simpler comparison baselines for the “Structural Mask” component: fully causal attention among all tokens within node blocks and across node blocks, with all other parts following the “Structural Mask” setting (denoted Causal-All); fully causal attention among all tokens within each node block, with all other parts following the “Structural Mask” setting (denoted Causal-Intra); fully causal attention across all node blocks, with all other parts following the “Structural Mask” setting (denoted Causal-Inter); and our Structural Mask. The experimental results are shown below and correspond to Section 3.3 and Appendix K.2 of the revised manuscript:
Llama-3.1-8B
| Attention mask | GI | MN | NC | LP |
|---|---|---|---|---|
| Causal-All | 0.415 | 0.717 | 38.37 | 83.39 |
| Causal-Inter | 0.434 | 0.737 | 42.11 | 84.72 |
| Causal-Intra | 0.445 | 0.740 | 42.02 | 85.66 |
| Structural Mask | 0.505 | 0.780 | 46.18 | 86.82 |
Qwen3-8B
| Attention mask | GI | MN | NC | LP |
|---|---|---|---|---|
| Causal-All | 0.336 | 0.600 | 39.71 | 84.27 |
| Causal-Inter | 0.388 | 0.634 | 43.50 | 85.06 |
| Causal-Intra | 0.385 | 0.625 | 44.48 | 86.02 |
| Structural Mask | 0.465 | 0.725 | 46.92 | 87.04 |
Response to W1
Thank you for your suggestion. To address your concern that “the narrative is not fluent,” we have rewritten the introduction and the method description. Please refer to the revised manuscript for the details; the specific changes are shown in blue.
Overall, for the introduction, we have focused on revising the description of related work to highlight the limitations of existing research. In particular, we emphasize the theoretical differences between our work and the most closely related methods, GaLA, UniGTE, and SLASH. On this basis, we outline how our method addresses these limitations. The key revised paragraphs are shown below:
For the method section, we particularly agree with your summary of our method: “Building on hierarchical graph serialization, SR-RoPE and the structural mask mainly adjust the positional relationships and reading permissions among tokens, including edge summaries, node summaries, and global readouts.” Accordingly, we have mainly rewritten the first paragraph of the method section to clearly present the overall logic of our method. The key revised paragraph is shown below: