acceptodds
Under review as a conference paper at ICLR 2027

DeKoR: Decomposed Knowledge Routing for Efficient GNN–LLM Fusion

Abstract

Large Language Models (LLMs) have shown strong potential for text-attributed graph learning by leveraging rich pre-trained knowledge and language understanding capabilities. Combining LLMs with Graph Neural Networks (GNNs) offers a promising paradigm for expanding the modeling capabilities of graph learning. However, LLMs and GNNs follow distinct learning paradigms, leading to a pronounced mismatch between their knowledge spaces. Such discrepancies make effective knowledge fusion nontrivial. Moreover, existing methods often incur substantial additional computation for only limited performance gains. To address these challenges, we propose DeKoR, an efficient framework for GNN and LLM knowledge fusion. To enable efficient knowledge transfer, DeKoR decomposes LLM knowledge into complementary local and global views and uses the LLM as an offline teacher to distill them into lightweight experts. To enable selective knowledge fusion, an adaptive router learns how much each node should rely on each expert. Extensive experiments on multiple benchmarks demonstrate that DeKoR outperforms existing baselines on both node and subgraph classification, achieving superior effectiveness and efficiency.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Text-attributed Graphs, Large Language Models, Graph Neural Networks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wk4xOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DeKoR, a GNN–LLM fusion framework for text-attributed graphs. The method encodes nodes with a frozen pre-trained GNN and extracts semantic embeddings from a per-dataset LoRA-fine-tuned LLM using three neighborhood-scale prompts. It then decomposes the LLM knowledge into a local view and a global view. Each view is distilled into a lightweight expert on top of the GNN representations, using cosine and InfoNCE losses for the local view and a KL loss for the global view. A node-adaptive router fuses the two experts predictions.

Critical strengths
  1. The problem is practical: removing the LLM from inference greatly reduces inference time and memory.
  2. The method is simple, modular, and easy to reproduce.
  3. The experiment results are good, covering two tasks, multiple GNN/LLM backbones, ablations, routing analysis, and efficiency comparisons, with consistent gains.
Critical weaknesses
  1. Fig. 2 marks the LLM as frozen, but it is actually LoRA fine-tuned on each dataset. Table 5 excludes this cost, along with the three-prompt embedding extraction for all nodes, and there is no ablation without fine-tuning. This cost likely exceeds the total cost of some baselines and is likely to boost performance. Please report end-to-end training and preprocessing cost.

  2. The numbers in Table 1 do not show a substantial mismatch, and different representations do not imply harmful fusion. Fig. 1 does not support the claim of different geometry: on arXiv, GCN and GCNII both show blob-like distributions and Qwen is only more fragmented.

  3. Challenge II is an observation, not an analysis of why prior methods fall short.

  4. Novelty is limited: LLM→GNN distillation (e.g., LinguGKD), prototype-based soft-assignment distillation, and two-expert gating are frequently used techniques in GraphLLMs. The contribution should be clarified.

  5. Several results are inconsistent:

    • GCN averages differ between Table 2 and Table 4(a) (79.97 vs. 82.09).
    • The ablation is non-monotonic on Cora and PubMed: on Cora, w/o L_glo (87.3) and w/o Rt (87.4) fall below removing all components (89.4), and the same holds on PubMed (90.3 vs. 92.2).
    • The K range in Fig. 5 contradicts App. D.6.
    • The routing weights in Fig. 4 are nearly constant (0.86–0.92), which looks like a nearly fixed weighting rather than node-level and domain-specific routing.
    • Comparisons with GNN and LLM backbones are not complete.
Questions for authors

Seen in Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer wk4x (Part 1 of 3)

We thank Reviewer wk4x for the constructive comments. Our detailed responses are provided below.

To W1: As described in Appendix D.5, we first fine-tune Qwen-8B with LoRA on each dataset. We then freeze the adapted LLM and use it to extract semantic representations. The “frozen” label in Fig.2 therefore refers to representation extraction after LoRA fine-tuning. We will revise the figure and its caption to make this sequence explicit. Table 5 reports only the cost of downstream inference. We now report the end-to-end time of the full pipeline, and compare it with selected baselines. To ensure a fair comparison, all methods use the same 1,000 sampled nodes per dataset, split into 500/250/250 training/validation/test nodes. For DeKoR, the reported cost includes training(prompt construction, embedding extraction, GNN training, LoRA fine-tuning, downstream training) and inference.

Runtime (s)

MethodCoraCiteSeerPubMedWikiCSarXiv
TrainingInferenceTotalTrainingInferenceTotalTrainingInferenceTotalTrainingInferenceTotalTrainingInferenceTotal
UniGraph44303110045403443037424504544303509448124430310444534744303477849081
OFA1932121410818126214222361842721111425139
GraphGPT828196102410221241146674131805988267125510963051401
LinguGKD799407994851608516658206582674506745607506075
GLANCE452664532474074747742677433784797856950299511
DeKoR437514376449114492698316984759017591909119092

The results show that DeKoR completes inference in approximately 1 second per dataset. This inference efficiency comes with an upfront cost for LoRA fine-tuning and embedding extraction using three prompts. Even after accounting for these costs, DeKoR has a shorter end-to-end runtime than GLANCE on all five datasets. Compared with lighter baselines such as OFA and GraphGPT, DeKoR incurs a higher overall cost but achieves higher accuracy.

We also evaluate a variant that extracts semantic representations from the Qwen-8B without LoRA fine-tuning on DeKoR and GLANCE. The results are shown as below:

MethodCoraCiteSeerPubMedWikiCSarXiv
GLANCE (w/o LoRA)89.2276.7892.2384.8571.26
DeKoR (w/o LoRA)90.2580.0992.8587.7874.75
DeKoR90.7981.4793.1388.2275.30

DeKoR without LoRA outperforms GLANCE without LoRA by 2.28 on average. Full DeKoR achieves an average gain of 0.64 over the “w/o LoRA” variant. These results demonstrate that LoRA provides limited benefits, and that DeKoR’s advantage does not depend on LoRA fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer wk4x (Part 2 of 3)

To W2: We thank the reviewer for this important suggestion. While CKA and UMAP illustrate the representation differences, we agree that evaluating their impact on fusion performance would provide further evidence. To further investigate this question, we have conducted additional experiments to examine how these differences relate to fusion performance. We evaluate three model pairs (GCN & GCNII, GCN & Qwen-8B, and GCNII & Qwen-8B) by concatenating their representations for classification. For each dataset, we report the change in accuracy relative to the better-performing individual model. The results are shown as below:

MethodCoraCiteSeerPubMedWikiCSarXivAvg.
Qwen-8B + GCN1.28-2.26-2.110.03-2.07-1.03
Qwen-8B + GCNII0.30-1.63-1.65-1.00-1.93-1.18
GCN + GCNII0.25-0.92-0.43-1.621.36-0.27
DeKoR (Qwen-8B + GCNII)1.432.30-2.082.162.341.23

As shown by the results and Table 1, the two GNN-LLM pairs have higher average (1−CKA1-\mathrm{CKA}) values (0.32 and 0.29 vs. 0.13) and larger accuracy drops (1.03 and 1.18 vs. 0.27) than GCN + GCNII. This suggests that greater representation discrepancies can lead to harmful fusion. Direct fusion may introduce redundant or conflicting information, making it harder for the model to extract task-relevant signals. On Cora and WikiCS, however, both GNN-LLM pairs yield better accuracy changes than the GNN-GNN pair. A possible explanation is that textual information about research topics and article subjects provide complementary information beyond what the GNN captures, which may offset the effects of redundancy or conflict. With the same Qwen-8B and GCNII backbones, DeKoR outperforms direct concatenation by 2.41 average. This supports DeKoR’ s design of decomposing and transferring semantic knowledge, and then performing fine-grained fusion at the node level.

To W3: We agree with your opinion. One possible explanation is that the benefit of LLM knowledge may be small relative to its contribution to the prediction. Some existing methods invoke an LLM during inference, which adds substantial computational overhead. Although LLMs provide rich semantic information, its usefulness varies across nodes. Some information offers little benefit for prediction. The additional inference cost can therefore bring small gains in accuracy. Based on this insight, DeKoR distills different types of LLM knowledge into separate lightweight experts before deployment. During inference, we adjust how much each expert contributes to each node’s prediction. This allows DeKoR to use LLM knowledge selectively at a lower inference cost. We will add this analysis to the revised manuscript.

To W4: Thank you for raising this point. We agree that distillation and adaptive routing are existing techniques. DeKoR’s major contribution is a paradigm for coordinating structural and semantic knowledge that separates knowledge transfer from adaptive fusion. GLANCE uses a router to decide which nodes need LLM assistance. DeKoR’s router instead decides how much each of two distilled semantic experts contributes to a node’s prediction. Both experts take GNN representations as input, so DeKoR requires no LLM inference. LinguGKD distills LLM knowledge into GNN by aligning LLM and GNN representations at different layers. DeKoR derives two distinct targets from the LLM representations to capture semantic knowledge at different granularities. This design enables finer-grained knowledge selection and fusion for each node.

To W5: Thank you for pointing out this discrepancy. The GCN and GraphSAGE baselines in Table 2 are trained end to end on the downstream task. Table 4(a) evaluates GCN and GraphSAGE as DeKoR backbones with the other DeKoR components removed.

To W6: Thank you for pointing this out. The variant labeled “w/o Lloc\mathcal{L}_{loc}, Lglo\mathcal{L}_{glo}, and RtR_{t}” directly uses GCNII representations for classification. The single-component variants retain DeKoR’s expert-based prediction architecture. Removing Lglo\mathcal{L}_{glo} keeps the global expert but removes its prototype supervision. Removing RtR_{t} combines the two experts’ predictions with equal weights. DeKoR’s distillation objectives and adaptive router are designed to work together. Removing a single component changes how the remaining components contribute to prediction and can weaken their coordination. This may explain why some partially ablated variants perform worse than the GCNII-only baseline on Cora and PubMed.

To W7: Thank you for pointing out this oversight. We have updated Appendix D.6 in the revised manuscript to match the range of KK shown in Figure 5.

AuthorsOpenReview ↗

Response to Reviewer wk4x (Part 3 of 3)

To W8: The values 0.86–0.92 are the mean local expert weights for four node groups on WikiCS. As shown by the scatter points and box plots, individual node weights vary within each group. To further explore the contributions of node-wise routing, we conduct additional experiments. We replace node-wise routing with seven constant local/global weights. The results are presented in the following table:

Weights (local / global)CoraCiteSeerPubMedWikiCSarXiv
0.50 / 0.5087.480.892.586.574.0
0.60 / 0.4089.580.592.386.574.0
0.70 / 0.3090.080.892.086.773.6
0.80 / 0.2090.081.092.387.774.6
0.85 / 0.1589.881.292.687.274.6
0.90 / 0.1090.381.292.387.874.7
0.95 / 0.0590.081.592.387.774.6
DeKoR90.881.593.188.275.3

We make two observations from the table: 1) The best fixed weight varies across datasets. Among the fixed weights, the optimal local weight is 0.90 on Cora, WikiCS, and arXiv, 0.95 on CiteSeer, and 0.85 on PubMed. All five datasets favor a higher weight for the local expert, while the preferred weights between local and global predictions varies by dataset. 2) Adaptive routing outperforms all fixed-weight settings. Compared with the best fixed weight for each dataset, DeKoR improves accuracy by 0.5 on Cora and PubMed, 0.4 on WikiCS, and 0.6 on arXiv. These results support the benefit of adapting the fusion weights to individual nodes.

To W9: To broaden the backbone comparison in Table 4, we added GAT as a GNN backbone and Mistral-7B as an LLM backbone. The results are shown as below:

MethodCoraCiteSeerWikiCS
Mistral-7B89.2275.7887.08
Mistral-7B(DeKoR)90.2579.8487.80
MethodCoraCiteSeerWikiCSPubMedarXiv
GAT88.8775.2888.4683.0566.18
GAT(DeKoR)90.5580.9387.5991.7373.74

DeKoR with GAT gain an accuracy of 4.54 on average. The largest gains are on PubMed (+8.68) and arXiv (+7.56). Accuracy decreases by 0.87 on WikiCS. DeKoR with Mistral-7B gain an accuracy of 1.94 on Cora, Citeseer and Wikics. The largest gain is on CiteSeer (+4.06). Together with Table 4, these results show that DeKoR consistently improves performance across different GNNs and LLMs. Therefore, our method is compatible with main LLM and GNN architectures.

Reviewer_d2cgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to leverage semantic knowledge from LLMs to improve text-attributed graph classification while reducing computational overhead during deployment. The proposed method employs a trained GNN and an LLM to construct structural and semantic node representations, respectively. The LLM representations and their prototype-associated distributions are adopted as two distillation targets to train two lightweight experts. A node-level router assigns fusion weights to the two experts based on structural representations and expert predictions. Node classification, subgraph classification, ablation, and efficiency experiments are conducted across five datasets. The average node classification accuracy reaches 85.78%.

Critical strengths

1.LLM inference is eliminated at test time, substantially lowering deployment costs compared with on-the-fly LLM invocation. 2.It jointly learns node representations and prototype distributions, providing multi-granularity training objectives for the student model. 3.It achieves the best average node classification accuracy on four out of five datasets, demonstrating consistent gains rather than dataset-specific improvements. 4.It outperforms GLANCE by 3.51 percentage points on the reported arXiv benchmark, indicating promising performance gains for further validation. 5.Ablations over different backbones and components, together with routing weight analyses, are provided to inspect the configuration dependence of the method.

Critical weaknesses

1.The reported random seeds, data splits, and label budgets used for pre-training and final evaluation are inconsistent. It remains unclear whether identical supervision and test boundaries are maintained across all stages.

2.The averaged results of GCN and GraphSAGE across five datasets in Table 2 conflict with those in Table 4(a), hindering reliable assessment of the improvement over backbone baselines.

3.Baselines including vanilla distillation with matched parameter counts and fixed-weight fusion are missing, making it hard to attribute gains to the decomposition design and node-wise routing.

4.The reported CKA discrepancies and two theoretical propositions do not constitute rigorous proof that fusion conflicts are mitigated, nor that the two experts exhibit task complementarity.

5.The efficiency experiments lack clear descriptions of caching strategies, GNN computation scope and timing boundaries, which limits cross-method comparability of latency and GPU memory usage.

Questions for authors

1.Please clarify whether the three random seeds alter data splits, and whether the final test set remains strictly isolated from supervised labels used in all training stages.

2.Please report the total number of training labels used by DeKoR and all baselines, and clarify whether extra labels are consumed during GNN pre-training.

3.Please explain the discrepancy between the GCN/GraphSAGE averages in Table 2 and Table 4(a), and confirm whether the two tables adopt different experimental settings.

4.Please specify the implementation of w/o Rt, and verify whether node routing retains advantages over fixed fusion weights tuned on the validation set.

5.Please clarify whether the timing measurements in Table 5 cover GNN encoding, graph sampling and data transfer, and whether identical caching and batching configurations are enforced across all methods.

4 comments
AuthorsOpenReview ↗

Response to Reviewer d2cg (Part 1 of 2)

We thank Reviewer d2cg for the constructive comments. Our detailed responses are provided below.

To Q1 and W1: Thank you for asking for clarification. We use three random seeds 123/456/789 with the fixed data splits. Across all training stages(GNN pre-training, LLM fine-tuning, knowledge distillation and router training), all supervision comes from the same training split.

To Q2 and W1: DeKoR and all baselines use the same number of labeled nodes for training: 1,354 for Cora, 1,593 for CiteSeer, 9,858 for PubMed, 5,850 for WikiCS, and 50,000 for arXiv. GNN training uses only the labels within this budget.

To Q3 and W2: Thank you for pointing out this discrepancy. The GCN and GraphSAGE baselines in Table 2 are trained end to end on the downstream task. Table 4(a) evaluates GCN and GraphSAGE as DeKoR backbones with the other DeKoR components removed.

To W3 and Q4: In the w/o RtR_t variant, we disable the learnable node-wise router and replace its outputs with fixed, equal fusion weights. To further explore the contributions of node-wise routing and decomposition, we evaluate two additional baselines. Vanilla Distillation. We replace DeKoR’s two experts and routing module with an MLP and a classification head. The baseline is trained with a classification loss and an InfoNCE loss for alignment with LLM representations, while its parameter count is matched to that of DeKoR by adjusting the MLP width. The results are presented in the following table:

MetricMethodCoraCiteSeerPubMedWikiCSarXiv
Accuracy (%)Vanilla Distillation89.2178.4192.2585.0973.62
Accuracy (%)DeKoR90.7981.4793.1388.2275.30
Parameters (M)Vanilla Distillation12.87112.83412.82212.90813.277
Parameters (M)DeKoR12.86812.83912.81712.90713.275

DeKoR outperforms the parameter-matched vanilla distillation baseline, with accuracy gains of 0.88-3.13. The two methods differ by at most 0.005M parameters (about 0.04%) on any dataset. These results indicate that DeKoR’s performance gains stem from its decomposition mechanism rather than increased model capacity.
Fixed-Weight Fusion. We replace node-wise routing with seven constant local/global weights. The results are presented in the following table:

Weights (local / global)CoraCiteSeerPubMedWikiCSarXiv
0.50 / 0.5087.480.892.586.574.0
0.60 / 0.4089.580.592.386.574.0
0.70 / 0.3090.080.892.086.773.6
0.80 / 0.2090.081.092.387.774.6
0.85 / 0.1589.881.292.687.274.6
0.90 / 0.1090.381.292.387.874.7
0.95 / 0.0590.081.592.387.774.6
DeKoR90.881.593.188.275.3

We make two observations from the table: 1)The best fixed weight varies across datasets. Among the fixed weights, the optimal local weight is 0.90 on Cora, WikiCS, and arXiv, 0.95 on CiteSeer, and 0.85 on PubMed. All five datasets favor a higher weight for the local expert, while the preferred weights between local and global predictions varies by dataset. 2)Adaptive routing outperforms all fixed-weight settings. Compared with the best fixed weight for each dataset, DeKoR improves accuracy by 0.5 on Cora and PubMed, 0.4 on WikiCS, and 0.6 on arXiv. These results support the benefit of adapting the fusion weights to individual nodes.

AuthorsOpenReview ↗

Response to Reviewer d2cg (Part 2 of 2)

To W4: CKA analysis reveals representation differences, while our theoretical propositions provide insights into DeKoR’s design. To further investigate this question, we have conducted additional experiments to examine how these differences relate to fusion performance. We evaluate three model pairs (GCN & GCNII, GCN & Qwen-8B, and GCNII & Qwen-8B) by concatenating their representations for classification. For each dataset, we report the change in accuracy relative to the better-performing individual model. The results are shown as below:

MethodCoraCiteSeerPubMedWikiCSarXivAvg.
Qwen-8B + GCN1.28-2.26-2.110.03-2.07-1.03
Qwen-8B + GCNII0.30-1.63-1.65-1.00-1.93-1.18
GCN + GCNII0.25-0.92-0.43-1.621.36-0.27
DeKoR (Qwen-8B + GCNII)1.432.30-2.082.162.341.23

As shown by the results and Table 1, the two GNN-LLM pairs have higher average (1−CKA1-\mathrm{CKA}) values (0.32 and 0.29 vs. 0.13) and larger accuracy drops (1.03 and 1.18 vs. 0.27) than GCN + GCNII. This suggests that greater representation discrepancies can lead to harmful fusion. Direct fusion may introduce redundant or conflicting information, making it harder for the model to extract task-relevant signals. On Cora and WikiCS, however, both GNN-LLM pairs yield better accuracy changes than the GNN-GNN pair. A possible explanation is that textual information about research topics and article subjects provide complementary information beyond what the GNN captures, which may offset the effects of redundancy or conflict. With the same Qwen-8B and GCNII backbones, DeKoR outperforms direct concatenation by 2.41 average. This supports DeKoR’ s design of decomposing and transferring semantic knowledge, and then performing fine-grained fusion at the node level.

To illustrate how the local and global experts complement each other, we present three nodes from Cora, WikiCS, and arXiv. The experts disagree and only one expert predicts the correct label. DeKoR correctly classifies all three nodes.

Dataset (node ID)True labelLocal expertGlobal expert
Cora (1270)Neural NetworksNeural Networks (0.9801)Reinforcement Learning (0.0199)
WikiCS (7536)Operating SystemsComputer Architecture (0.0005)Operating Systems (0.9995)
arXiv (8623)cs.SYcs.SY (0.6249)cs.IR (0.3751)

Values in parentheses are routing weights, and bold indicates the correct expert prediction. On Cora (node 1270) and arXiv (node 8623), the local expert is correct and receives high routing weights of 0.9801 and 0.6249, respectively. On WikiCS (node 7536), the global expert correctly predicts Operating Systems and receives a routing weight of 0.9995. These cases show that some nodes that are difficult for one expert can be correctly classified by the other. DeKoR uses this complementarity by assigning more weight to the correct expert.

To Q5 and W5: We used the same timing protocol for all methods in Table 5. We timed each method’s encoding step, including GNN and LM encoding. We excluded node sampling because all methods used the same pre-sampled nodes. We also excluded CPU-GPU data transfers because their overhead was nearly identical across methods. Each method ran inference once with a batch size of 8 and no intermediate caching.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Responses

Dear Reviewer d2cg,

Thank you again for your time and effort in reviewing our work!

We understand that this might be a busy period for you. We have provided detailed responses to your questions and concerns. We have also uploaded a revised manuscript with the changes highlighted in blue.

Whenever you have a chance, we would be grateful for your feedback on whether our responses have addressed your concerns.

Thank you!

Best regards,
Authors of Paper 15072

AuthorsOpenReview ↗

A Gentle Follow-up on Our Responses

Dear Reviewer d2cg,

Thank you again for your time and effort in reviewing our work.

We understand that this may be a busy period for you. We have provided detailed responses to your questions and concerns, along with additional experiments, and uploaded a revised manuscript with the changes highlighted in blue\textcolor{blue}{\text{blue}}.

We would be grateful to know whether these responses and revisions have addressed your concerns. If so, would you kindly consider reassessing your rating in light of the clarifications and additional results? If you have any other questions, we would be happy to provide further clarification.

Thank you for your time and consideration.

Best regards,

Authors of Paper 15072

Reviewer_Lmi9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on the problem of combing GNNs and LLMs for TAG learning, with an emphasis on reducing inference cost in comparison to previous methods. The authors argue that one understudied issue is the difference in the LLM and GNNs representation spaces, making direct fusion a challenging learning problem. As a result, the authors consider separating the knowledge transfer from knowledge fusion via distillation. Specifically, their method DeKoR constructs global and local views of the LLM’s semantic understanding of the nodes, which are distilled into lightweight modules that can focus on the fusion strategy. Then, a final router decides how to weight these two signals. While each part is relatively simple in isolation, I think the connection of all these pieces together make for a compelling architecture. While its performance improvements relative to previous methods (e.g. GLANCE) is not massive the inference speed benefit makes for a very practical system.

Critical strengths
  1. I think the design of the method is promising and very practical. While there are quite a few pieces to train, once done, it should be fairly easy to use. This is a nice property for the integration of LLM models for graph learning.
  2. While the performance results are somewhat marginal over the baselines, the benefit in terms of inference speed seems to be most promising. I think this also speaks to the design of the model and focusing on extraction vs fusion, allowing the LLM to be removed from the inference process.
  3. The experimental results in terms of ablations feel fairly comprehensive, providing a solid understanding of partial contributions of the objectives and backbones.
Critical weaknesses
  1. While I think DeKoR’s reuse of previous ideas is fine, I do think clarifying the novelty is important. As far as I can tell, the two core ideas of distillation and adaptive usage already exist in previous methods, such as LinguGKD and GLANCE, respectively. While the ablations show that the individual pieces contribute, can the authors help clarify the novelty over just piecing these two designs together?
  2. While the biggest benefit seems to come in the subgraph classification, I am curious how careful the authors were in making sure the aggregation is fair across baselines? Were all the baselines amenable to this design of aggregation? Is there anything that could be unfair in how the baselines would need to be changed to process this task given not all of them expose embeddings in the same way that DeKoR does?
  3. I appreciate the visualization about representation mismatch, but at the same time, it is unclear how real this problem is. Specifically, just because there is low similarity or differing UMAP visualizations doesn’t mean the fusion will be harmful. I am wondering if there is a way the authors can explicitly connect some notion of representation mismatch to performance to show there is a connection here. Otherwise, it is possible DeKoR is doing something else to improve performance and not the actual thing being used as motivation.
Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Lmi9

We thank Reviewer Lmi9 for their support and valuable feedback. Our detailed responses are provided below.

To W1: Thank you for raising this point. We agree that distillation and adaptive routing are existing techniques. DeKoR’s major contribution is a paradigm for combining structural and semantic knowledge that separates knowledge transfer from adaptive fusion. GLANCE uses a router to decide which nodes need LLM assistance. DeKoR’s router instead decides how much each of two distilled semantic experts contributes to a node’s prediction. Both experts take GNN representations as input, so DeKoR requires no LLM inference. LinguGKD distills LLM knowledge into GNN by aligning LLM and GNN representations at different layers. DeKoR derives two distinct targets from the LLM representations to capture semantic knowledge at different granularities. This design enables finer-grained knowledge selection and fusion for each node.

To W2: Thank you for raising this important concern. We ensured a fair comparison when adapting the baselines to subgraph classification. Our experiments use a unified subgraph construction method and settings. For embedding-based baselines(GNNs, GTs, GLANCE, LinguGKD), a graph-level classifier is needed to perform subgraph classification. We therefore apply mean pooling to obtain subgraph representations, followed by a graph-level classifier. For baselines(OFA, GraphGPT, UniGraph) that have their own readout designs, we follow their original architectures. We didn’t force all baselines into DeKoR’s aggregation design. These adaptations ensure a fair comparison while preserving the original designs of each baseline.

To W3: We thank the reviewer for this important suggestion. While CKA and UMAP illustrate the representation differences, we agree that evaluating their impact on fusion performance would provide further evidence. To further investigate this question, we have conducted additional experiments to examine how these differences relate to fusion performance. We evaluate three model pairs (GCN & GCNII, GCN & Qwen-8B, and GCNII & Qwen-8B) by concatenating their representations for classification. For each dataset, we report the change in accuracy relative to the better-performing individual model. The results are shown as below:

MethodCoraCiteSeerPubMedWikiCSarXivAvg.
Qwen-8B + GCN1.28-2.26-2.110.03-2.07-1.03
Qwen-8B + GCNII0.30-1.63-1.65-1.00-1.93-1.18
GCN + GCNII0.25-0.92-0.43-1.621.36-0.27
DeKoR (Qwen-8B + GCNII)1.432.30-2.082.162.341.23

As shown by the results and Table 1, the two GNN-LLM pairs have higher average (1−CKA1-\mathrm{CKA}) values (0.32 and 0.29 vs. 0.13) and larger accuracy drops (1.03 and 1.18 vs. 0.27) than GCN + GCNII. This suggests that greater representation discrepancies can lead to harmful fusion. Direct fusion may introduce redundant or conflicting information, making it harder for the model to extract task-relevant signals. On Cora and WikiCS, however, both GNN-LLM pairs yield better accuracy changes than the GNN-GNN pair. A possible explanation is that textual information about research topics and article subjects provides complementary information beyond what the GNN captures, which may offset the effects of redundancy or conflict. With the same Qwen-8B and GCNII backbones, DeKoR outperforms direct concatenation by 2.41 average. This supports DeKoR’ s design of decomposing and transferring semantic knowledge, and then performing fine-grained fusion at the node level.

Reviewer_Lmi9OpenReview ↗

Thank you

Thank you for the clarifications and follow-up experiments! I think the new results look reasonable to me, and I appreciate the additional context on experimental details. As I already leaned positive on the paper, I will maintain my score.

Dear Reviewer Lmi9\textcolor{blue}{\text{Lmi9}},

Thank you for taking the time to review and for your recognition of our efforts. We are glad that the new results were helpful. We appreciate your thoughtful feedback, which helped us clarify the paper.

If you have any further questions, please feel free to reach out. We would be more than happy to discuss them with you at your convenience.

Thanks again for your thoughtful review and your support!

Best regards,

Authors of 15072

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank our reviewers for their time and effort in providing thorough and constructive feedback on our paper. In response to their comments, we have added the following experiments and analyses, and clarified some details about methods and evaluation. The revised parts are highlighted in blue.

Revisions to the main text

  1. We clarify the relationship between representation differences and fusion performance. (Introduction)
  2. We make an analysis of Challenge II. (Introduction)
  3. We clarify DeKoR’s contribution by distinguishing it from LinguGKD and GLANCE. (Related Work)
  4. We revise Figure 2 to make clear that the LLM is first fine-tuned with LoRA. (Figure 2)
  5. We explain why the reported average accuracy differs in Table2 and Table4(a). (Experiments)
  6. We include GAT as a GNN backbone and Mistral-7B as an LLM backbone. (Table 4)
  7. We discuss why the ablation results are non-monotonic on Cora and PubMed. (Experiments)
  8. We clarify the same inference protocol and what was timed in Table 5. (Experiments)

Revisions to the appendix

  1. We directly fuse the representations of three backbone pairs to examine how representation differences relate to fusion performance. (Appendix E.1)
  2. We add a Vanilla Distillation experiment to assess whether the gains can be explained by model capacity. (Appendix E.2)
  3. We compare node-wise routing with seven fixed weights. (Appendix E.3)
  4. We present some cases to show how the two experts complement each other. (Appendix E.4)
  5. We report DeKoR and GLANCE without LoRA fine-tuning to assess the contribution of LoRA. (Appendix E.5)
  6. We report the runtime(pre-processing, training, inference) for the full pipeline. (Appendix E.6)
  7. We clarify all methods use the same subgraph construction procedure, and that baselines retain their original designs. (Appendix D.3)
  8. We clarify the data splits are fixed across different seeds and training stages, we also report the label budgets of five datasets. (Appendix D.6)
  9. We correct the range of KK to match Figure 5. (Appendix D.6)

Dear Reviewers,

Thank you for your thoughtful reviews and contributions. The authors have now submitted their rebuttals. Please take some time to review their responses and assess whether your concerns have been adequately addressed. If appropriate, please update your comments and ratings accordingly.

Thank you for your time and continued efforts.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.