acceptodds
Under review as a conference paper at ICLR 2027

Beyond Code Generation over Tabular Graphs: A Structural Context Tool for Graph Inference

Abstract

Large Language Models (LLMs) have demonstrated graph inference capabilities on Text-Attributed Graphs (TAGs), which are widely used in social networks and recommendation systems. Graph-as-Code (GAC) enables LLM-Graph interaction in tabular form through code generation, avoiding the need to explicitly serialize large graphs into prompts. In the node classification task, accessibility of tabular graph data does not guarantee that GAC can effectively capture local neighbor structure or global data bias. We observe that the overlap between misclassified nodes of GAC and GNN-based methods (GCN, GAT, GraphSAGE) is substantially smaller than the overlap among GNN-based methods themselves. The oracle upper bound of GAC and GNN indicates complementary predictive performance. To enhance interaction performance, we introduce a Structural Context Tool (SCT) that augments the original tabular graph data with GNN predictions and topic summaries by aggregating over K-hop neighbors or random walks. The SCT can generate augmentation during graph inference. Across ten datasets, our method achieved an average improvement of 3.91% in node classification accuracy compared to baseline GAC. Further analytical experiments indicate that LLM-Graph interactions in tabular form require both the global data bias from GNN training and the local structural summary from neighbor aggregation. Executor code and preprocessed dataset are available at the https://anonymous.4open.science/r/anonymous-artifact-C7A2.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
LLM-Graph Interaction, Structure Context Tool, Graph Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EfqbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper introduces a structural context tool (SCT) for node classification on text-attributed graphs. The authors observe that GAC and GNNs exhibit different error patterns, suggesting potential complementarity. Motivated by this observation, they augment GAC with GCN predictions and neighborhood summaries constructed using either k-hop neighborhoods or random walk-based neighbor selection. The authors evaluate the contributions of these components through experiments and ablation studies. While the motivation and empirical study are interesting, I have several concerns about the paper.

Critical strengths
  1. The motivation and observations from the experimental results are clear and interesting. Figure 1 shows that the error-set overlap between GAC and GNNs is lower than that of GNN methods and quantifies the potential gains. These analyses provide an empirical motivation. Besides, the multi-dataset experiments and component ablations show that GCN prediction augmentation provides consistent gains over SCT-O, whereas the benefits of neighborhood aggregation and topic augmentation depend on the dataset.

  2. These findings offer practical guidance on which information to expose to LLMs.

Critical weaknesses

Although I find the motivation convincing and the experiments extensive, the writing makes it hard for me to fully understand the method's details. Figure 2 looks complex (e.g. many arrows and connections), but the method section does not clearly explain how the components work together or how SCT-O, SCT-G, SCT-K, and SCT-R differ. Appendix E helps, but it is still unclear what information is kept in the "w.o Topic" and "w.o Aggregation" variants. This makes the method hard to assess, and the ablation results difficult to interpret or reproduce.

Questions for authors
  1. Could the authors provide more details about SCT, especially how topic information is used in SCT-K and SCT-R? The current description makes the component analysis hard to interpret.

  2. Since this work builds on GAC, could the authors explain the basis for the claim that GAC "does not guarantee the formation of effective structural representations" (line 53)? After reading the original GAC paper, I am not sure what supports this claim.

  3. Could the authors clarify the added value of the proposed structural summaries? In particular, could they provide comparisons with simple GCN-LLM baselines?

  4. Could the authors check the appendix references and clarify which appendix sections support which parts of the main text? The current references are hard to follow. For example, I need justify Appendix E support which part of your method.

6 comments

Response to Reviewer Efqb

Thank you for your careful review. We address your concerns below. In addition, we will update the paper within three days.

1. SCT variants, topic information, and ablation settings

We will expand the Figure 2 caption to explain how the four configurations fit into the workflow:

Figure 2. Interaction between LLM agents and structured context tools. Table data augmentation includes visible labels, frozen GCN predictions, and topic summaries. K and R denote all one-hop neighbors or aggregate neighborhood information of the target node via a random walk with restarts, respectively, followed by deduplication and lexicographical sorting. GCN predictions include predicted class, confidence, prediction margins, and entropy. Topic summaries include topic support and label joint statistics. SCT-O uses only the GAC original prompt as shown in Appendix E.1; SCT-G adds the GCN predictions prompt as shown in Appendix E.2; and SCT-K/R further incorporates prompt as shown in Appendix E.3 and E.4. Agents query evidence and predict classes through a read-only “code–execute–observe” loop.

The component ablations examine the contributions of GCN fields and topic statistics. Full is SCT-K or SCT-R: it uses all distinct one-hop neighbors of the target, excludes the target itself, and imposes no cap on the number of source nodes.

VariantsRelation to SCT-OComposition of Prompts
Full= SCT-O + GCN predictions + neighbor aggregation + topic summariesAppendix E.1+E.2+E.3+E.4
w/o Graph= SCT-O + neighbor aggregation + topic summariesAppendix E.1+E.3+E.4
w/o Topic= SCT-O + GCN predictions + neighbor aggregationAppendix E.1+E.2+E.3
w/o Aggregation= SCT-O + GCN predictions + topic summariesAppendix E.1+E.2+E.4
w/o All= SCT-OAppendix E.1

The GCN predictions consists the predicted class, confidence, prediction margin, and entropy. Topic summaries contain visible label and topic joint counts. Neighbor aggregation consists k-hop or random walk aggregation for GCN fields or Topic summaries. Thus, w/o Graph removes both direct graph predictions and their aggregated statistics, whereas w/o Topic removes visible label and topic joint counts.

2. Revising the claim about effective structural representations in Graph-as-Code

Table 9 of the original GAC paper [1] shows that shuffling adjacency reduces the performance of GAC, supporting the conclusion that Graph-as-Code uses real graph structure. We recognize this capability.

Our use of the term “effective structural representations” here is in contrast to GNN methods. GAC is training-free and can only access the neighbors of the currently visited node, relying primarily on the generalizability of semantic information for inference.

For greater clarity, we have corrected the original text to read: “Compared to GNN methods, GAC cannot guarantee a comprehensive and detailed understanding of structural information.” We emphasize the importance of “global data biases during training” and “the summary of local structures from neighbor aggregation” to support fusion of LLMs and GNNs.

[1] Actions Speak Louder Than Prompts: A Large Scale StudyY of LLMs for Graph Inference, ICLR 2026, Table 9 and Template 3

3. The simple GCN–LLM fusion baselines and the value of structural summaries

We add a simple combination experiment that uses the frozen GCN prediction when the confidence for the target node is strictly greater than the threshold, otherwise using SCT-O's prediction. We used only the other eight datasets to select the thresholds.

Table A. Average accuracy across the eight non-social graphs in the threshold sweep.

ThresholdAverage accuracy (%)
0.785.12
0.885.20
0.985.01

Of the three thresholds tested, 0.8 gives the highest average accuracy on these eight datasets, though the differences are small. Table B reports the main evaluation runs for GCN, SCT-O, [email protected], SCT-G, and SCT-R.

Table B. Node classification accuracy (%, mean ± standard deviation).

DatasetGCNSCT-O[email protected]SCT-GSCT-R
Cora88.48 ± 0.6787.51 ± 0.5890.27 ± 0.1090.58 ± 0.3791.80 ± 0.28
CiteSeer73.35 ± 0.6575.27 ± 0.3476.07 ± 0.4876.69 ± 0.2478.08 ± 0.26
PubMed89.70 ± 0.3591.08 ± 0.3790.73 ± 0.6691.66 ± 0.7392.20 ± 0.72
Computers89.40 ± 0.4588.86 ± 1.2790.10 ± 0.7090.32 ± 1.0991.10 ± 0.51
Photo86.00 ± 0.3785.31 ± 0.6687.16 ± 0.8988.17 ± 0.6788.00 ± 0.36
WikiCS84.60 ± 0.6285.15 ± 1.2586.74 ± 1.5987.40 ± 0.9487.53 ± 1.07
ogbn-arxiv74.20 ± 0.1275.09 ± 1.0375.22 ± 1.1975.58 ± 1.8376.70 ± 2.99
ogbn-products81.50 ± 0.2184.85 ± 1.1485.32 ± 1.0686.13 ± 0.7685.40 ± 0.17
Average excluding two social graphs83.4084.1485.2085.8286.35
Instagram66.30 ± 3.0548.93 ± 1.1751.32 ± 0.2558.36 ± 1.1561.20 ± 0.20
Reddit70.40 ± 1.4764.29 ± 2.0065.95 ± 1.6768.01 ± 2.2563.20 ± 3.90
Average across all ten graphs80.3978.6379.8981.2981.52

Table B shows that SCT-G outperforms both SCT-O and [email protected] on all ten datasets, with average accuracy improvements of 2.66 and 1.40 percentage points, respectively, indicating that its performance improvements exhibit good consistency across datasets. SCT-R outperforms the two baselines on nine datasets, with average improvements of 2.89 and 1.63 percentage points, respectively. Even after excluding the two social graphs, SCT-R still improve accuray by 1.15 percentage points over [email protected], indicating that its gains are not solely driven by significant improvements on individual social graphs. At the same time, SCT-R exhibited a performance decline on Reddit, suggesting that further structural context aggregation is dataset-dependent. Overall, SCT-G provides more consistent performance improvements, while SCT-R achieves higher accuracy on most datasets.

LLMs performed poorly on Instagram and Reddit, while GNNs performed poorly on WebKB. In both cases, reasoning LLMs were unable to fully distinguish whether the correct judgment stemmed from LLM or GNN (semantics or structure). We rewrite the explanation of Findings 3. "Our results suggest that when LLMs and GNNs have both comparable accuracy and different error pattern on the same dataset, fusion can achieve consistent improvements. It is very difficult to utilize the limited amount of useful information from a weak model while maintaining the strong model’s original correct judgments."

This comparison measures the overall effect of fixed gating threshold versus exposing graph predictions to the LLM; both the accessible information and the decision rule change. We will clarify that our contribution is to study Which modules help approach the oracle upper bound of LLM agents and GNNs? interaction mode (simple fusion or code generation), augmentation content (GNN prediction or topic summary), and aggregation strategy (k-hop or random walk).

4. Appendix reference to the main text

Appendix E gives the prompts and fields available to the agent; Appendix B describes how summaries are constructed and counted. E.1–E.4 are composable interface blocks, while E.5 is a preprocessing template. K and R use the same E.1 + E.2 + E.3 + E.4 classification interface and differ only in aggregation strategy.

We will add explicit references to E.1–E.5 in the corresponding Method paragraphs on information sources, GCN fields, aggregation interfaces, and the LLM interaction loop. Source selection and counting definitions will also reference Appendix B. Readers can then find the relevant template and its additions directly from each method definition.

Thank you for the clarifications. I'll reconsider my score after I've read the revised paper.

Reviewer_vqzxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces SCT to investigate how information types, augmentation strategies, model configurations, and feature characteristics impact graph-augmented LLM inference. The paper evaluates four SCT variants (original, GNN-augmented, K-hop aggregated, and random-walk aggregated) and performs component-wise ablations on graph, topic, and aggregation modules. Experiments show that SCT improves tabular LLM-graph interactions.

Critical strengths

This work features a clear and logical structure, addressing an important question: how GNNs and LLMs complement each other in graph reasoning. The methodological design and main experiments are tightly centered around this objective.

The presentation of the framework is clear and well-organized, making the components and their interactions easy to follow.

The authors offer insightful discussion regarding when each component yields the greatest benefits in LLM-graph reasoning.

The empirical evaluation is comprehensive, covering 10 diverse text-attributed graph datasets.

Critical weaknesses

The paper does not yet clearly distinguish its core contribution from prior work or explain why SCT holds a distinct advantage over simpler GCN prediction-fusion and neighbor-aggregation baselines. For instance, the performance gains of SCT-G over SCT-O might simply stem from aggregating GNN predictions rather than the specific way this information is formatted and presented to the LLM within the SCT structure.

Since the main novelty of SCT lies in combining these components and running empirical verification, the underlying mechanism remains a bit unclear and requires a deeper, more rigorous analysis.

The conclusion that GNN enhancement is ineffective when the text is semantically rich enough lacks sufficient support. Because SCT-O already achieves high performance in these settings, it is difficult to isolate semantic richness as the decisive factor rather than other confounding contributors.

Questions for authors

The authors are suggested to discuss more of the novelty of SCT with deeper analysis of its design choices to better exhibit the necessity and advantage of SCT.

It is also suggested that the authors clarify how they distinguish the effect of text semantic richness from the high SCT-O accuracy and relatively weak GCN performance on WebKB.

4 comments

Response to Reviewer vqzx

Thank you for your insightful review. We address your concerns below. In addition, we will update the paper within three days.

1. Comparison with simple GCN–LLM prediction fusion

We add a simple combination experiment that uses the frozen GCN prediction when the confidence for the target node is strictly greater than the threshold, otherwise using SCT-O's prediction. We used only the other eight datasets to select the thresholds.

Table A. Average accuracy across the eight non-social graphs in the threshold sweep.

ThresholdAverage accuracy (%)
0.785.12
0.885.20
0.985.01

Of the three thresholds tested, 0.8 gives the highest average accuracy on these eight datasets, though the differences are small. Table B reports the main evaluation runs for GCN, SCT-O, [email protected], SCT-G, and SCT-R.

Table B. Node classification accuracy (%, mean ± standard deviation).

DatasetGCNSCT-O[email protected]SCT-GSCT-R
Cora88.48 ± 0.6787.51 ± 0.5890.27 ± 0.1090.58 ± 0.3791.80 ± 0.28
CiteSeer73.35 ± 0.6575.27 ± 0.3476.07 ± 0.4876.69 ± 0.2478.08 ± 0.26
PubMed89.70 ± 0.3591.08 ± 0.3790.73 ± 0.6691.66 ± 0.7392.20 ± 0.72
Computers89.40 ± 0.4588.86 ± 1.2790.10 ± 0.7090.32 ± 1.0991.10 ± 0.51
Photo86.00 ± 0.3785.31 ± 0.6687.16 ± 0.8988.17 ± 0.6788.00 ± 0.36
WikiCS84.60 ± 0.6285.15 ± 1.2586.74 ± 1.5987.40 ± 0.9487.53 ± 1.07
ogbn-arxiv74.20 ± 0.1275.09 ± 1.0375.22 ± 1.1975.58 ± 1.8376.70 ± 2.99
ogbn-products81.50 ± 0.2184.85 ± 1.1485.32 ± 1.0686.13 ± 0.7685.40 ± 0.17
Average excluding two social graphs83.4084.1485.2085.8286.35
Instagram66.30 ± 3.0548.93 ± 1.1751.32 ± 0.2558.36 ± 1.1561.20 ± 0.20
Reddit70.40 ± 1.4764.29 ± 2.0065.95 ± 1.6768.01 ± 2.2563.20 ± 3.90
Average across all ten graphs80.3978.6379.8981.2981.52

Table B shows that SCT-G outperforms both SCT-O and [email protected] on all ten datasets, with average accuracy improvements of 2.66 and 1.40 percentage points, respectively, indicating that its performance improvements exhibit good consistency across datasets. SCT-R outperforms the two baselines on nine datasets, with average improvements of 2.89 and 1.63 percentage points, respectively. Even after excluding the two social graphs, SCT-R still improve accuray by 1.15 percentage points over [email protected], indicating that its gains are not solely driven by significant improvements on individual social graphs. At the same time, SCT-R exhibited a performance decline on Reddit, suggesting that further structural context aggregation is dataset-dependent. Overall, SCT-G provides more consistent performance improvements, while SCT-R achieves higher accuracy on most datasets.

LLMs performed poorly on Instagram and Reddit, while GNNs performed poorly on WebKB. In both cases, reasoning LLMs were unable to fully distinguish whether the correct judgment stemmed from LLM or GNN (semantics or structure). We rewrite the explanation of Findings 3. "Our results suggest that when LLMs and GNNs have both comparable accuracy and different error pattern on the same dataset, fusion can achieve consistent improvements. It is very difficult to utilize the limited amount of useful information from a weak model while maintaining the strong model’s original correct judgments."

This comparison measures the overall effect of fixed gating threshold versus exposing graph predictions to the LLM; both the accessible information and the decision rule change. We will clarify that our contribution is to study What is effctive for approaching the oracle upper bound of LLM agents and GNNs fusion? the interaction mode (simple fusion or code generation), the augmentation content (GNN predictions or topic summaries), and the aggregation strategy (k-hop or random walk).

2. Deeper analysis of interaction modes, augmentation content, and aggregation strategy

(1) In terms of interaction mode, GAC’s code generation on tabular data is effective for fusion with GNN predictions.

The results of SCT-O, [email protected], and SCT-G in Table A show that fixed confidence gating yields a slight improvement across the eight datasets, while SCT-G provides a further accuracy improvement across all datasets. This suggests that code generation on tabular for graph inference is effective, but there is still insufficient evidence to confirm the criteria for fusion.

(2) Graph predictions provide more consistent gains, while the additional contribution of topic aggregation depends on the dataset.

The component ablations examine the contributions of GCN fields and topic statistics. Full is SCT-K or SCT-R: it uses all distinct one-hop neighbors of the target, excludes the target itself, and imposes no cap on the number of source nodes.

VariantsRelation to SCT-OComposition of Prompts
Full= SCT-O + GCN predictions + neighbor aggregation + topic summariesAppendix E.1+E.2+E.3+E.4
w/o Graph= SCT-O + neighbor aggregation + topic summariesAppendix E.1+E.3+E.4
w/o Topic= SCT-O + GCN predictions + neighbor aggregationAppendix E.1+E.2+E.3
w/o Aggregation= SCT-O + GCN predictions + topic summariesAppendix E.1+E.2+E.4
w/o All= SCT-OAppendix E.1

The GCN prediction fields consists the predicted class, confidence, prediction margin, and entropy. Topic summaries contain joint counts of visible labels and topics. Neighbor aggregation applies k-hop or random walk aggregation for GCN fields or Topic summaries. Thus, w/o Graph removes both direct graph predictions and their aggregated statistics, whereas w/o Topic removes joint counts of visible labels and topics.

K ablation accuracy (%)

K configurationCoraComputersogbn-productsReddit
Full (SCT-K)90.5690.4085.2768.63
w/o Aggregation87.5488.6682.7062.13
w/o Graph88.7089.7081.3164.67
w/o Topic90.5389.7285.5761.30
w/o All (SCT-O)87.5188.8684.8564.29

Neighbor Aggregation provides the most consistent gains in ablation study and determines how local structures are organized and utilized. Graph fields show very stable improvements and provide a basis for predictions across the entire graph. Topic summary is designed to simulate the transmission of textual information, and its benefits are concentrated in specific datasets.

On ogbn-products, w/o Topic achieves 85.57%, compared with 85.27% for Full, a difference of 0.30 percentage points. One possible explanation is that topic-conditioned evidence is sparse. This large graph has 47 classes, while preprocessing and context costs limit extraction to 1–4 topics per source node and display to 20 topics per target summary. Topics are matched exactly after normalization, so some topic–class combinations in a local source set may have only a single supporting node. Such statistics may add little discriminative information and distract the agent from class marginals. This observation suggests that the value of topic summaries depends on local statistical support.

(3) Comparing random-walk and k-hop aggregation.

Random walks aggregation is more effective on most datasets than k-hop aggregation, and has a fixed sampling budget. On ogbn-arxiv, SCT-R achieves 76.70%, compared with 75.10% for SCT-K, showing a benefit on this large graph. Popular nodes in the Reddit dataset are relatively concentrated, so a 1-hop approach works better.

3. Interpreting semantic richness, high SCT-O accuracy, and weak GCN performance on WebKB

LLMs performed poorly on Instagram and Reddit, while GNNs performed poorly on WebKB. In both cases, reasoning LLMs were unable to fully distinguish whether the correct judgment stemmed from LLM or GNN (semantics or structure). Both the limited room for improvement and the weak additional graph predictions affect the outcome, so the current experiments cannot isolate semantic richness as the decisive factor. We agree that the statement “GNN enhancement is ineffective when text is sufficiently semantically rich” was too strong.

We rewrite the explanation of the Findings 3. "To achieve consistent improvements, LLMs and GNNs should satisfy both similar accuracy rates and different error mode on the same dataset. It is very difficult to utilize the limited amount of useful information from a weak model while maintaining the strong model’s original correct judgments."

Thank you for the detailed rebuttal. The added confidence-gated baseline, clarification of the SCT variants and ablation settings, and correction of the average gain address my main concerns. I will maintain my positive score.

We have updated the revised paper. Thank you again for your comments and feedback.

Reviewer_jPVtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies LLM agents that classify nodes in text-attributed graphs by writing pandas code over a table of node text, neighbors and visible labels, following Graph-as-Code. It shows these agents and tuned GNNs make largely different errors, and that an oracle picking the right one per node would score 82-95%. The proposed structural context tool lets the LLM query a fronzen GCN's predictions and per-target neighbor summaries, draw from 1-hop neighbors or random-walk neighbors. With DeepSeek-v4-flash on 10 datasets, the authors report that average gain of 3.91 points over Graph-as-Code, with GCN predictions helping on all 10datasets. Abaltions attribute the gain mainly to the GCN predictions and the neighbor aggregation, and the predictions and the neighbor aggregation, and the method still trails tuned GNNs on Instagram and Reddit.

Critical strengths
  1. The paper has a clean motivation and a careful test protocol. The error-overlap and oracle analysis show the LLM agent and GNNs fail on different nodes. Adding GCN predictions helps on all 10 datasets and even uses fewer tokens and fewer steps.
  2. The analysis can explain where it helps.
Critical weaknesses
  1. No simple combination baseline and the gain is small over GNNs.
  2. The results are not substantial. The +3.91% cannot be reproduced from Table 1. Diffference between the three SCT versions are often within one standard deviations.
Questions for authors

How does SCT compare with simple ways of combining the gCN and the plain LLM agent? Like a switch that trust GCN above a confidence threshold. How is the +3.91% computed (baseline, versions, absolute or relative)?

4 comments

Response to Reviewer jPVt

Thank you for your thoughtful review. We address your concerns below. In addition, we will update the paper within three days.

1. Simple combination of GCN and a plain LLM agent.

We add a simple combination experiment that uses the frozen GCN prediction when the confidence for the target node is strictly greater than the threshold, otherwise using SCT-O's prediction. We used only the other eight datasets to select the thresholds.

Table A. Average accuracy across the eight non-social graphs in the threshold sweep.

ThresholdAverage accuracy (%)
0.785.12
0.885.20
0.985.01

Of the three thresholds tested, 0.8 gives the highest average accuracy on these eight datasets, though the differences are small. Table B reports the main evaluation runs for GCN, SCT-O, [email protected], SCT-G, and SCT-R.

Table B. Node classification accuracy (%, mean ± standard deviation).

DatasetGCNSCT-O[email protected]SCT-GSCT-R
Cora88.48 ± 0.6787.51 ± 0.5890.27 ± 0.1090.58 ± 0.3791.80 ± 0.28
CiteSeer73.35 ± 0.6575.27 ± 0.3476.07 ± 0.4876.69 ± 0.2478.08 ± 0.26
PubMed89.70 ± 0.3591.08 ± 0.3790.73 ± 0.6691.66 ± 0.7392.20 ± 0.72
Computers89.40 ± 0.4588.86 ± 1.2790.10 ± 0.7090.32 ± 1.0991.10 ± 0.51
Photo86.00 ± 0.3785.31 ± 0.6687.16 ± 0.8988.17 ± 0.6788.00 ± 0.36
WikiCS84.60 ± 0.6285.15 ± 1.2586.74 ± 1.5987.40 ± 0.9487.53 ± 1.07
ogbn-arxiv74.20 ± 0.1275.09 ± 1.0375.22 ± 1.1975.58 ± 1.8376.70 ± 2.99
ogbn-products81.50 ± 0.2184.85 ± 1.1485.32 ± 1.0686.13 ± 0.7685.40 ± 0.17
Average excluding two social graphs83.4084.1485.2085.8286.35
Instagram66.30 ± 3.0548.93 ± 1.1751.32 ± 0.2558.36 ± 1.1561.20 ± 0.20
Reddit70.40 ± 1.4764.29 ± 2.0065.95 ± 1.6768.01 ± 2.2563.20 ± 3.90
Average across all ten graphs80.3978.6379.8981.2981.52

Table B shows that SCT-G outperforms both SCT-O and [email protected] on all ten datasets, with average accuracy improvements of 2.66 and 1.40 percentage points, respectively, indicating that its performance improvements exhibit good consistency across datasets. SCT-R outperforms the two baselines on nine datasets, with average improvements of 2.89 and 1.63 percentage points, respectively. Even after excluding the two social graphs, SCT-R still improve accuray by 1.15 percentage points over [email protected], indicating that its gains are not solely driven by significant improvements on individual social graphs. At the same time, SCT-R exhibited a performance decline on Reddit, suggesting that further structural context aggregation is dataset-dependent. Overall, SCT-G provides more consistent performance improvements, while SCT-R achieves higher accuracy on most datasets.

LLMs performed poorly on Instagram and Reddit, while GNNs performed poorly on WebKB. In both cases, reasoning LLMs were unable to fully distinguish whether the correct judgment stemmed from LLM or GNN (semantics or structure). We rewrite the explanation of Findings 3. "Our results suggest that when LLMs and GNNs have both comparable accuracy and different error pattern on the same dataset, fusion can achieve consistent improvements. It is very difficult to utilize the limited amount of useful information from a weak model while maintaining the strong model’s original correct judgments."

2. Correction of the average gain

The originally reported gain of 3.91 pp was calculated using SCT-G on Photo, SCT-K on WikiCS, and SCT-R on the remaining eight datasets. For each dataset, we subtracted the GAC* accuracy reported in the original Graph-as-Code paper and then averaged the differences. This calculation combined different SCT variants and used an external baseline.

We now use SCT-O evaluated under our protocol as the baseline throughout. We compute the absolute accuracy difference for each dataset, then take the unweighted mean across all ten datasets.

Choice of SCT resultAverage accuracy gain over SCT-O
Best variant among SCT-G, SCT-K, and SCT-R+3.51 pp
Fixed SCT-R on all ten datasets+2.89 pp

We plan to report +3.51 pp in the revised abstract. This value summarizes the best results among the variants per dataset.

3. Interpreting the magnitude of improvement and standard deviations

Reasons for large standard deviations. Each run uses up to 1,000 target nodes from the same fixed test split. Cora and CiteSeer have fewer than 1,000 test nodes, so all five runs use the same complete target set, and their small standard deviations primarily reflect repeated inference. On the other datasets, standard deviations reflect both changes in the target set and repeated inference.

Limited improvement among variants on Photo and WikiCS. We agree that differences in average accuracy alone do not establish a stable advantage among SCT-G, SCT-K, and SCT-R. A single method's standard deviation is also not the standard deviation of a paired difference between methods. We therefore discuss gains over the base SCT-O interface separately from differences among the three augmented variants. SCT-G has a higher mean than SCT-O on all ten datasets, while K/R yield further gains on several datasets. The augmented variants range from 87.70% to 88.17% on Photo and 87.40% to 88.02% on WikiCS; these small differences do not establish a clear advatage for any single variant. We will make this limitation clear in our conclusions.

Magnitude of improvement and relative error reduction. When baseline accuracy is already high, looking only at a one- or two-percentage-point gain can understate the reduction in errors. For example, increasing accuracy from 90% to 91% is a one-percentage-point gain, but reduces the error rate from 10% to 9%, corresponding to 10% relative error reduction. We therefore report absolute accuracy gain, relative accuracy gain, and relative error reduction (RER) to show how the gain translates into fewer errors.

The five representative datasets below all compare SCT-O with fixed SCT-R under the same protocol. We use the same SCT variant for each dataset.

Dataset (SCT-O→SCT-R, %)Accuracy gain (pp)Relative accuracy gainRelative error reduction (RER)
Cora (87.51→91.80)+4.29+4.90%34.35%
PubMed (91.08→92.20)+1.12+1.23%12.56%
Computers (88.86→91.10)+2.24+2.52%20.11%
Photo (85.31→88.00)+2.69+3.15%18.31%
WikiCS (85.15→87.53)+2.38+2.80%16.03%

Relative accuracy gain is 100 × (SCT-R accuracy − SCT-O accuracy) / SCT-O accuracy; RER is 100 × (SCT-R accuracy − SCT-O accuracy) / (100 − SCT-O accuracy), with accuracies and results expressed as percentages. In these examples, RER ranges from 12.56% to 34.35%: SCT-R removes approximately one-third of the remaining errors on Cora and one-fifth on Computers. These are substantial error reductions over an already strong SCT-O baseline. The five examples illustrate the size of the gains.

We have updated the revised paper. Thank you again for your comments and feedback.

Dear reviewer,

As the discussion period is about to close, we would like to kindly draw your attention to our response above and the updated manuscript. As requested, we have (i) incorporated a simple combination of LLM–GNN fusion with a fixed threshold, (ii) revised the average gain, and (iii) clarified the relationship between the magnitude of improvement and the standard deviation.

Could you please let us know whether these additional analyses and explanations have addressed your concerns? If so, we would greatly appreciate it if you could consider updating your score. If any concerns remain, we would be glad to clarify them further while the discussion period is still open.

Best regards.

Comments on the paper

1 comment

Dear Area Chairs, Program Chairs, and Reviewers,

We thank all reviewers for their valuable feedback and helpful suggestions. We are pleased that reviewers have acknowledged the motivations of our work and recognized the effective of Structural Context Tool. Furthermore, we summarized the strengths of the paper as follows:

Clear, solid and interesting motivation (Reviewer jPVt, vqzx, Efqb)

Ingeniously designed technical approach and experiments (Reviewer vqzx, Efqb)

Soundness and informativeness (Reviewer Efqb)

Well-organized presentation (Reviewer vqzx)

Substantial validation and comprehensive empirical evaluation (Reviewer jPVt)

Consistent gains and Insightful discussion (Reviewer vqzx, Efqb)

In discussion period, our rebuttal addressed the concerns of Reviewer vqzx and Efqb.

Additional experiments: Following the reviewers' suggestions, we added an additional experiments about simple combination of GNN and LLM fusion with a fixed threshold. Additional experiments further demonstrate that SCT has a positive effect on the fusion criteria for calibrating the LLM Agent.

Clarifications: To improve the readability of the paper, we further revised the description of the main results, the precise wording of the three findings, and the clarification of SCT variants in the ablation study. We appreciate the Reviewer Efqb for the constructive feedback and increasing the presentation score.

More discussions: We have reorganized the paper by the impact of LLM-GNN fusion from four perspectives: interaction modes, augmentation context, augmentation strategy, and configuration profile. We also added a finding regarding inference efficiency.

Finally, we thank all reviewers again for their thoughtful reviews. Please let us know if you have any recommendations for further improving the paper.

Best regards, Authors of Paper 44540.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.