- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper introduces a structural context tool (SCT) for node classification on text-attributed graphs. The authors observe that GAC and GNNs exhibit different error patterns, suggesting potential complementarity. Motivated by this observation, they augment GAC with GCN predictions and neighborhood summaries constructed using either k-hop neighborhoods or random walk-based neighbor selection. The authors evaluate the contributions of these components through experiments and ablation studies. While the motivation and empirical study are interesting, I have several concerns about the paper.
Critical strengths
-
The motivation and observations from the experimental results are clear and interesting. Figure 1 shows that the error-set overlap between GAC and GNNs is lower than that of GNN methods and quantifies the potential gains. These analyses provide an empirical motivation. Besides, the multi-dataset experiments and component ablations show that GCN prediction augmentation provides consistent gains over SCT-O, whereas the benefits of neighborhood aggregation and topic augmentation depend on the dataset.
-
These findings offer practical guidance on which information to expose to LLMs.
Critical weaknesses
Although I find the motivation convincing and the experiments extensive, the writing makes it hard for me to fully understand the method's details. Figure 2 looks complex (e.g. many arrows and connections), but the method section does not clearly explain how the components work together or how SCT-O, SCT-G, SCT-K, and SCT-R differ. Appendix E helps, but it is still unclear what information is kept in the "w.o Topic" and "w.o Aggregation" variants. This makes the method hard to assess, and the ablation results difficult to interpret or reproduce.
Questions for authors
-
Could the authors provide more details about SCT, especially how topic information is used in SCT-K and SCT-R? The current description makes the component analysis hard to interpret.
-
Since this work builds on GAC, could the authors explain the basis for the claim that GAC "does not guarantee the formation of effective structural representations" (line 53)? After reading the original GAC paper, I am not sure what supports this claim.
-
Could the authors clarify the added value of the proposed structural summaries? In particular, could they provide comparisons with simple GCN-LLM baselines?
-
Could the authors check the appendix references and clarify which appendix sections support which parts of the main text? The current references are hard to follow. For example, I need justify Appendix E support which part of your method.
6 comments
3. The simple GCN–LLM fusion baselines and the value of structural summaries
We add a simple combination experiment that uses the frozen GCN prediction when the confidence for the target node is strictly greater than the threshold, otherwise using SCT-O's prediction. We used only the other eight datasets to select the thresholds.
Table A. Average accuracy across the eight non-social graphs in the threshold sweep.
| Threshold | Average accuracy (%) |
|---|---|
| 0.7 | 85.12 |
| 0.8 | 85.20 |
| 0.9 | 85.01 |
Of the three thresholds tested, 0.8 gives the highest average accuracy on these eight datasets, though the differences are small. Table B reports the main evaluation runs for GCN, SCT-O, [email protected], SCT-G, and SCT-R.
Table B. Node classification accuracy (%, mean ± standard deviation).
| Dataset | GCN | SCT-O | [email protected] | SCT-G | SCT-R |
|---|---|---|---|---|---|
| Cora | 88.48 ± 0.67 | 87.51 ± 0.58 | 90.27 ± 0.10 | 90.58 ± 0.37 | 91.80 ± 0.28 |
| CiteSeer | 73.35 ± 0.65 | 75.27 ± 0.34 | 76.07 ± 0.48 | 76.69 ± 0.24 | 78.08 ± 0.26 |
| PubMed | 89.70 ± 0.35 | 91.08 ± 0.37 | 90.73 ± 0.66 | 91.66 ± 0.73 | 92.20 ± 0.72 |
| Computers | 89.40 ± 0.45 | 88.86 ± 1.27 | 90.10 ± 0.70 | 90.32 ± 1.09 | 91.10 ± 0.51 |
| Photo | 86.00 ± 0.37 | 85.31 ± 0.66 | 87.16 ± 0.89 | 88.17 ± 0.67 | 88.00 ± 0.36 |
| WikiCS | 84.60 ± 0.62 | 85.15 ± 1.25 | 86.74 ± 1.59 | 87.40 ± 0.94 | 87.53 ± 1.07 |
| ogbn-arxiv | 74.20 ± 0.12 | 75.09 ± 1.03 | 75.22 ± 1.19 | 75.58 ± 1.83 | 76.70 ± 2.99 |
| ogbn-products | 81.50 ± 0.21 | 84.85 ± 1.14 | 85.32 ± 1.06 | 86.13 ± 0.76 | 85.40 ± 0.17 |
| Average excluding two social graphs | 83.40 | 84.14 | 85.20 | 85.82 | 86.35 |
| 66.30 ± 3.05 | 48.93 ± 1.17 | 51.32 ± 0.25 | 58.36 ± 1.15 | 61.20 ± 0.20 | |
| 70.40 ± 1.47 | 64.29 ± 2.00 | 65.95 ± 1.67 | 68.01 ± 2.25 | 63.20 ± 3.90 | |
| Average across all ten graphs | 80.39 | 78.63 | 79.89 | 81.29 | 81.52 |
Table B shows that SCT-G outperforms both SCT-O and [email protected] on all ten datasets, with average accuracy improvements of 2.66 and 1.40 percentage points, respectively, indicating that its performance improvements exhibit good consistency across datasets. SCT-R outperforms the two baselines on nine datasets, with average improvements of 2.89 and 1.63 percentage points, respectively. Even after excluding the two social graphs, SCT-R still improve accuray by 1.15 percentage points over [email protected], indicating that its gains are not solely driven by significant improvements on individual social graphs. At the same time, SCT-R exhibited a performance decline on Reddit, suggesting that further structural context aggregation is dataset-dependent. Overall, SCT-G provides more consistent performance improvements, while SCT-R achieves higher accuracy on most datasets.
LLMs performed poorly on Instagram and Reddit, while GNNs performed poorly on WebKB. In both cases, reasoning LLMs were unable to fully distinguish whether the correct judgment stemmed from LLM or GNN (semantics or structure). We rewrite the explanation of Findings 3. "Our results suggest that when LLMs and GNNs have both comparable accuracy and different error pattern on the same dataset, fusion can achieve consistent improvements. It is very difficult to utilize the limited amount of useful information from a weak model while maintaining the strong model’s original correct judgments."
This comparison measures the overall effect of fixed gating threshold versus exposing graph predictions to the LLM; both the accessible information and the decision rule change. We will clarify that our contribution is to study Which modules help approach the oracle upper bound of LLM agents and GNNs? interaction mode (simple fusion or code generation), augmentation content (GNN prediction or topic summary), and aggregation strategy (k-hop or random walk).
4. Appendix reference to the main text
Appendix E gives the prompts and fields available to the agent; Appendix B describes how summaries are constructed and counted. E.1–E.4 are composable interface blocks, while E.5 is a preprocessing template. K and R use the same E.1 + E.2 + E.3 + E.4 classification interface and differ only in aggregation strategy.
We will add explicit references to E.1–E.5 in the corresponding Method paragraphs on information sources, GCN fields, aggregation interfaces, and the LLM interaction loop. Source selection and counting definitions will also reference Appendix B. Readers can then find the relevant template and its additions directly from each method definition.
Thank you for the clarifications. I'll reconsider my score after I've read the revised paper.
Response to Reviewer Efqb
Thank you for your careful review. We address your concerns below. In addition, we will update the paper within three days.
1. SCT variants, topic information, and ablation settings
We will expand the Figure 2 caption to explain how the four configurations fit into the workflow:
The component ablations examine the contributions of GCN fields and topic statistics. Full is SCT-K or SCT-R: it uses all distinct one-hop neighbors of the target, excludes the target itself, and imposes no cap on the number of source nodes.
The GCN predictions consists the predicted class, confidence, prediction margin, and entropy. Topic summaries contain visible label and topic joint counts. Neighbor aggregation consists k-hop or random walk aggregation for GCN fields or Topic summaries. Thus, w/o Graph removes both direct graph predictions and their aggregated statistics, whereas w/o Topic removes visible label and topic joint counts.
2. Revising the claim about effective structural representations in Graph-as-Code
Table 9 of the original GAC paper [1] shows that shuffling adjacency reduces the performance of GAC, supporting the conclusion that Graph-as-Code uses real graph structure. We recognize this capability.
Our use of the term “effective structural representations” here is in contrast to GNN methods. GAC is training-free and can only access the neighbors of the currently visited node, relying primarily on the generalizability of semantic information for inference.
For greater clarity, we have corrected the original text to read: “Compared to GNN methods, GAC cannot guarantee a comprehensive and detailed understanding of structural information.” We emphasize the importance of “global data biases during training” and “the summary of local structures from neighbor aggregation” to support fusion of LLMs and GNNs.
[1] Actions Speak Louder Than Prompts: A Large Scale StudyY of LLMs for Graph Inference, ICLR 2026, Table 9 and Template 3