- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies which neighbors an LLM should see when the graph sequence has a fixed size. It keeps the LLaGA interface largely fixed and compares random, structural, semantic, and task-specific neighbor selection. The experiments cover node classification and link prediction on four citation datasets. The paper concludes that no method is always best, random selection is often competitive, and a label-informed Oracle can greatly improve node classification.
Critical strengths
-
The question is reasonable. Keeping the interface fixed is a useful way to study the effect of neighbor selection.
-
The paper considers both node classification and link prediction. I also find the experiment that changes the serializer between training and testing useful.
-
Figure 2 reports CN and CN2 coverage separately for positive and negative pairs. The label-agreement results in Table 8 are also helpful for understanding what information is present in the selected neighborhoods.
Critical weaknesses
-
Section 3.4 and section 4.4 are almost the same. Please justify why they are so similar.
-
The paper does not report repeated training runs, variance, or significance tests. Many differences are very small. On ogbn-arxiv, Random scores 0.7635 and PPR scores 0.7636, which is a difference of about five test nodes. The PubMed node classification results are also very close. In addition, Table 7 appears to choose the best serializers from test results, and these choices are used again in the later analyses on the same test set. The authors should use validation results for this choice and report results over multiple seeds.
-
The budget may not be restrictive enough on Cora, CiteSeer, and PubMed. Their median degrees are only two or three in Table 4, while the first-hop budget is ten. For many nodes, all first-hop neighbors can fit, so the methods may mainly change the order of neighbors. For example, Random, Core, and Degree have exactly the same Cora accuracy in Table 1. The paper should report how often each method actually has to discard neighbors at each hop.
-
The Oracle uses the target node’s own test label to rank neighbors. This means that information about the answer affects the input sequence. On ogbn-arxiv, accuracy falls from 0.9237 to 0.5454 when an Oracle-trained model receives Random test sequences (Table 9). This is also consistent with the model learning a shortcut from the label-dependent sequence. The Oracle is useful as a diagnostic, but its result should not be treated as an achievable improvement for a usable selector. The structure-trained model reaches only 0.8079 even when it receives Oracle sequences at test time.
Questions for authors
Please see the above weaknesses.
We thank the reviewer for the detailed feedback and for acknowledging the value of our systematic analysis and experiments. We address the concerns below.
1- Weakness 1:
Section 3.4 presents the cross-serialization experimental design, while Section 4.4 discusses the findings in relation to RQ4, focusing on the model's sensitivity to changes in serialization between training and testing. We understand the possible confusion and can rewrite these sections to reduce repetition.
2- Weakness 2:
Our goal is not to establish the superiority of individual serializers based on small differences in accuracy, but to investigate how neighborhood serialization affects LLM-based graph learning across datasets and tasks. We can provide standard deviations alongside the reported accuracies to better characterize the variability of our results.
The small differences between several methods are consistent with our main finding that no single serializer is consistently dominant. This observation is particularly important given that our experiments cover four datasets with different sizes, structural characteristics, and levels of homophily. For example, PubMed has high homophily (around 80%), which may partly explain the small differences in accuracy among serializers, as even randomly selected neighbors are likely to share the target node’s class. In contrast, ogbn-arxiv has lower homophily (around 65%) and a highly uneven degree distribution (Table 4). Therefore, structural serializers may favor high-degree hubs that do not necessarily provide class-relevant information, as further illustrated by the label-agreement analysis in Table 8. These observations highlight that effective serialization depends on task-relevant information and dataset characteristics rather than method complexity.
Regarding Table 7, we selected the best-performing serializer within each family for each dataset based on the initial experimental test results to keep subsequent analyses manageable. Our goal was to compare different serialization families rather than individual methods, investigating which types of neighborhood information are most suitable for different tasks and datasets. We understand that selecting representative methods based on the test results may introduce bias, and we will address this limitation in the paper.
3- Weakness 3:
Our experiments cover four datasets with different degree distributions (Table 4), allowing us to examine serialization under varying neighborhood conditions. Although Cora, CiteSeer, and PubMed have relatively low median degrees, our framework uses hierarchical two-hop neighborhoods. As shown in Table 1, second-hop truncation occurs for 81.74% of PubMed, 45.02% of Cora, and 21.16% of CiteSeer neighborhoods. In contrast, ogbn-arxiv has higher neighborhood connectivity, with truncation occurring for 42.90% of first-hop and 90.21% of second-hop neighborhoods. Additionally, selecting first-hop neighbors determines which second-hop nodes become available, affecting the selected serialized context. These results show that our evaluation covers different levels of neighborhood truncation, allowing us to study serialization across datasets with diverse structural characteristics.
Table 1: Neighborhood truncation effect at each hop (K=10).
4- Weakness 4:
As explicitly stated in our paper, Oracle is designed as a diagnostic that uses ground-truth labels to assess the availability of class-relevant information in the neighborhood, rather than a deployable serializer. Its purpose is mainly to investigate the potential value of class-relevant context for node classification. The performance gap between Oracle and practical deployed serializers highlights the challenge of identifying such information without access to ground-truth labels. Furthermore, the substantial performance drop when Oracle-trained models are tested on Random sequences reveals a potential trade-off between class-relevant information and robustness to serialization shifts. While Oracle achieves high accuracy under matched training and testing conditions, its strong dependence on label-conditioned sequences makes it sensitive to changes in the input distribution. This suggests that future label-aware selection methods should consider not only class relevance but also robustness across different serialization conditions. These findings support our conclusion that serialization is both an information-selection mechanism and part of the learned graph-to-LLM input distribution.