- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes DeKoR, a GNN–LLM fusion framework for text-attributed graphs. The method encodes nodes with a frozen pre-trained GNN and extracts semantic embeddings from a per-dataset LoRA-fine-tuned LLM using three neighborhood-scale prompts. It then decomposes the LLM knowledge into a local view and a global view. Each view is distilled into a lightweight expert on top of the GNN representations, using cosine and InfoNCE losses for the local view and a KL loss for the global view. A node-adaptive router fuses the two experts predictions.
Critical strengths
- The problem is practical: removing the LLM from inference greatly reduces inference time and memory.
- The method is simple, modular, and easy to reproduce.
- The experiment results are good, covering two tasks, multiple GNN/LLM backbones, ablations, routing analysis, and efficiency comparisons, with consistent gains.
Critical weaknesses
-
Fig. 2 marks the LLM as frozen, but it is actually LoRA fine-tuned on each dataset. Table 5 excludes this cost, along with the three-prompt embedding extraction for all nodes, and there is no ablation without fine-tuning. This cost likely exceeds the total cost of some baselines and is likely to boost performance. Please report end-to-end training and preprocessing cost.
-
The numbers in Table 1 do not show a substantial mismatch, and different representations do not imply harmful fusion. Fig. 1 does not support the claim of different geometry: on arXiv, GCN and GCNII both show blob-like distributions and Qwen is only more fragmented.
-
Challenge II is an observation, not an analysis of why prior methods fall short.
-
Novelty is limited: LLM→GNN distillation (e.g., LinguGKD), prototype-based soft-assignment distillation, and two-expert gating are frequently used techniques in GraphLLMs. The contribution should be clarified.
-
Several results are inconsistent:
- GCN averages differ between Table 2 and Table 4(a) (79.97 vs. 82.09).
- The ablation is non-monotonic on Cora and PubMed: on Cora, w/o L_glo (87.3) and w/o Rt (87.4) fall below removing all components (89.4), and the same holds on PubMed (90.3 vs. 92.2).
- The K range in Fig. 5 contradicts App. D.6.
- The routing weights in Fig. 4 are nearly constant (0.86–0.92), which looks like a nearly fixed weighting rather than node-level and domain-specific routing.
- Comparisons with GNN and LLM backbones are not complete.
Questions for authors
Seen in Weaknesses.
5 comments
Response to Reviewer wk4x (Part 2 of 3)
To W2: We thank the reviewer for this important suggestion. While CKA and UMAP illustrate the representation differences, we agree that evaluating their impact on fusion performance would provide further evidence. To further investigate this question, we have conducted additional experiments to examine how these differences relate to fusion performance. We evaluate three model pairs (GCN & GCNII, GCN & Qwen-8B, and GCNII & Qwen-8B) by concatenating their representations for classification. For each dataset, we report the change in accuracy relative to the better-performing individual model. The results are shown as below:
| Method | Cora | CiteSeer | PubMed | WikiCS | arXiv | Avg. |
|---|---|---|---|---|---|---|
| Qwen-8B + GCN | 1.28 | -2.26 | -2.11 | 0.03 | -2.07 | -1.03 |
| Qwen-8B + GCNII | 0.30 | -1.63 | -1.65 | -1.00 | -1.93 | -1.18 |
| GCN + GCNII | 0.25 | -0.92 | -0.43 | -1.62 | 1.36 | -0.27 |
| DeKoR (Qwen-8B + GCNII) | 1.43 | 2.30 | -2.08 | 2.16 | 2.34 | 1.23 |
As shown by the results and Table 1, the two GNN-LLM pairs have higher average () values (0.32 and 0.29 vs. 0.13) and larger accuracy drops (1.03 and 1.18 vs. 0.27) than GCN + GCNII. This suggests that greater representation discrepancies can lead to harmful fusion. Direct fusion may introduce redundant or conflicting information, making it harder for the model to extract task-relevant signals. On Cora and WikiCS, however, both GNN-LLM pairs yield better accuracy changes than the GNN-GNN pair. A possible explanation is that textual information about research topics and article subjects provide complementary information beyond what the GNN captures, which may offset the effects of redundancy or conflict. With the same Qwen-8B and GCNII backbones, DeKoR outperforms direct concatenation by 2.41 average. This supports DeKoR’ s design of decomposing and transferring semantic knowledge, and then performing fine-grained fusion at the node level.
To W3: We agree with your opinion. One possible explanation is that the benefit of LLM knowledge may be small relative to its contribution to the prediction. Some existing methods invoke an LLM during inference, which adds substantial computational overhead. Although LLMs provide rich semantic information, its usefulness varies across nodes. Some information offers little benefit for prediction. The additional inference cost can therefore bring small gains in accuracy. Based on this insight, DeKoR distills different types of LLM knowledge into separate lightweight experts before deployment. During inference, we adjust how much each expert contributes to each node’s prediction. This allows DeKoR to use LLM knowledge selectively at a lower inference cost. We will add this analysis to the revised manuscript.
To W4: Thank you for raising this point. We agree that distillation and adaptive routing are existing techniques. DeKoR’s major contribution is a paradigm for coordinating structural and semantic knowledge that separates knowledge transfer from adaptive fusion. GLANCE uses a router to decide which nodes need LLM assistance. DeKoR’s router instead decides how much each of two distilled semantic experts contributes to a node’s prediction. Both experts take GNN representations as input, so DeKoR requires no LLM inference. LinguGKD distills LLM knowledge into GNN by aligning LLM and GNN representations at different layers. DeKoR derives two distinct targets from the LLM representations to capture semantic knowledge at different granularities. This design enables finer-grained knowledge selection and fusion for each node.
To W5: Thank you for pointing out this discrepancy. The GCN and GraphSAGE baselines in Table 2 are trained end to end on the downstream task. Table 4(a) evaluates GCN and GraphSAGE as DeKoR backbones with the other DeKoR components removed.
To W6: Thank you for pointing this out. The variant labeled “w/o , , and ” directly uses GCNII representations for classification. The single-component variants retain DeKoR’s expert-based prediction architecture. Removing keeps the global expert but removes its prototype supervision. Removing combines the two experts’ predictions with equal weights. DeKoR’s distillation objectives and adaptive router are designed to work together. Removing a single component changes how the remaining components contribute to prediction and can weaken their coordination. This may explain why some partially ablated variants perform worse than the GCNII-only baseline on Cora and PubMed.
To W7: Thank you for pointing out this oversight. We have updated Appendix D.6 in the revised manuscript to match the range of shown in Figure 5.
Response to Reviewer wk4x (Part 3 of 3)
To W8: The values 0.86–0.92 are the mean local expert weights for four node groups on WikiCS. As shown by the scatter points and box plots, individual node weights vary within each group. To further explore the contributions of node-wise routing, we conduct additional experiments. We replace node-wise routing with seven constant local/global weights. The results are presented in the following table:
| Weights (local / global) | Cora | CiteSeer | PubMed | WikiCS | arXiv |
|---|---|---|---|---|---|
| 0.50 / 0.50 | 87.4 | 80.8 | 92.5 | 86.5 | 74.0 |
| 0.60 / 0.40 | 89.5 | 80.5 | 92.3 | 86.5 | 74.0 |
| 0.70 / 0.30 | 90.0 | 80.8 | 92.0 | 86.7 | 73.6 |
| 0.80 / 0.20 | 90.0 | 81.0 | 92.3 | 87.7 | 74.6 |
| 0.85 / 0.15 | 89.8 | 81.2 | 92.6 | 87.2 | 74.6 |
| 0.90 / 0.10 | 90.3 | 81.2 | 92.3 | 87.8 | 74.7 |
| 0.95 / 0.05 | 90.0 | 81.5 | 92.3 | 87.7 | 74.6 |
| DeKoR | 90.8 | 81.5 | 93.1 | 88.2 | 75.3 |
We make two observations from the table: 1) The best fixed weight varies across datasets. Among the fixed weights, the optimal local weight is 0.90 on Cora, WikiCS, and arXiv, 0.95 on CiteSeer, and 0.85 on PubMed. All five datasets favor a higher weight for the local expert, while the preferred weights between local and global predictions varies by dataset. 2) Adaptive routing outperforms all fixed-weight settings. Compared with the best fixed weight for each dataset, DeKoR improves accuracy by 0.5 on Cora and PubMed, 0.4 on WikiCS, and 0.6 on arXiv. These results support the benefit of adapting the fusion weights to individual nodes.
To W9: To broaden the backbone comparison in Table 4, we added GAT as a GNN backbone and Mistral-7B as an LLM backbone. The results are shown as below:
| Method | Cora | CiteSeer | WikiCS |
|---|---|---|---|
| Mistral-7B | 89.22 | 75.78 | 87.08 |
| Mistral-7B(DeKoR) | 90.25 | 79.84 | 87.80 |
| Method | Cora | CiteSeer | WikiCS | PubMed | arXiv |
|---|---|---|---|---|---|
| GAT | 88.87 | 75.28 | 88.46 | 83.05 | 66.18 |
| GAT(DeKoR) | 90.55 | 80.93 | 87.59 | 91.73 | 73.74 |
DeKoR with GAT gain an accuracy of 4.54 on average. The largest gains are on PubMed (+8.68) and arXiv (+7.56). Accuracy decreases by 0.87 on WikiCS. DeKoR with Mistral-7B gain an accuracy of 1.94 on Cora, Citeseer and Wikics. The largest gain is on CiteSeer (+4.06). Together with Table 4, these results show that DeKoR consistently improves performance across different GNNs and LLMs. Therefore, our method is compatible with main LLM and GNN architectures.
Response to Reviewer wk4x (Part 1 of 3)
We thank Reviewer wk4x for the constructive comments. Our detailed responses are provided below.
To W1: As described in Appendix D.5, we first fine-tune Qwen-8B with LoRA on each dataset. We then freeze the adapted LLM and use it to extract semantic representations. The “frozen” label in Fig.2 therefore refers to representation extraction after LoRA fine-tuning. We will revise the figure and its caption to make this sequence explicit. Table 5 reports only the cost of downstream inference. We now report the end-to-end time of the full pipeline, and compare it with selected baselines. To ensure a fair comparison, all methods use the same 1,000 sampled nodes per dataset, split into 500/250/250 training/validation/test nodes. For DeKoR, the reported cost includes training(prompt construction, embedding extraction, GNN training, LoRA fine-tuning, downstream training) and inference.
Runtime (s)
The results show that DeKoR completes inference in approximately 1 second per dataset. This inference efficiency comes with an upfront cost for LoRA fine-tuning and embedding extraction using three prompts. Even after accounting for these costs, DeKoR has a shorter end-to-end runtime than GLANCE on all five datasets. Compared with lighter baselines such as OFA and GraphGPT, DeKoR incurs a higher overall cost but achieves higher accuracy.
We also evaluate a variant that extracts semantic representations from the Qwen-8B without LoRA fine-tuning on DeKoR and GLANCE. The results are shown as below:
DeKoR without LoRA outperforms GLANCE without LoRA by 2.28 on average. Full DeKoR achieves an average gain of 0.64 over the “w/o LoRA” variant. These results demonstrate that LoRA provides limited benefits, and that DeKoR’s advantage does not depend on LoRA fine-tuning.