acceptodds
Under review as a conference paper at ICLR 2027

SkillGNN: Large Language Model-Guided Component Evolution for Knowledge Graph Completion

Abstract

Graph neural networks (GNNs) for knowledge graph completion still rely heavily on manually designed components. Large language models (LLMs) can generate alternative designs, but feedback on individual candidates does not by itself ensure that design experience accumulates across iterations. We present SkillGNN, a framework that couples component-wise GNN evolution with the refinement of a reusable textual design Skill. The Skill guides an LLM agent to propose local computations within a fixed component interface while preserving the surrounding GNN architecture. A task-grounded evaluator combines validity and structural checks with subgraph training, providing diagnostic feedback that SkillOpt uses to refine the Skill for subsequent proposals. Experiments on WN18RR and FB15K-237 show that the evolved NBFNet scorers improve full-graph link prediction over NBFNet and one-shot LLM component design when trained from scratch. Additional experiments examine the influence of the design LLM and show improvements from evolving NBFNet's node-update component and CompGCN's scoring component.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Neural Networks, Automated Machine Learning, Component-wise Evolution, Skill Evolution, LLM Agents, Knowledge Graph Completion

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PNRGOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SkillGNN uses an LLM agent to iteratively develop a trainable component within a fixed KGC GNN under a strict input-output contract. Candidates are compiled into typed expression trees and then undergo filtering based on type, gradient, semantic, and structural criteria. Eligible candidates are then trained on a fixed development subgraph with reflect and gate validation splits. A textual “Skill” holding design guidance is revised by SkillOpt (Yang et al., 2026), and a revision is accepted only if the component generated with it enhances gate MRR. The best-performing gate component is retrained from scratch on the entire graph. The authors report MRR gains over the published NBFNet of +0.0100 on WN18RR (0.5610) and +0.0026 on FB15K-237 (0.4176). Gains over one-shot LLM design are +0.0104 and +0.0039. Additionally, they report a +0.0044 gain on a reproduced CompGCN, a smaller gain from node-update evolution, results with three design LLMs, triple-classification gains of +0.0006 and +0.0009, and WN18RR ablations.

Critical strengths
  • Candidates are evaluated on a fixed, documented subgraph with reflect/gate splits. In the end, the authors also exclude the test set from search. A typed DSL with validity and structural checks limits the number of invalid proposals. The interface contract and zero-initialized gating isolate the scorer’s effect from propagation and the objective. So I would say, this is a reusable engineering contribution, regardless of the final numbers.
  • The reported direction of improvement looks consistent (even though the magnitudes are small).
Critical weaknesses
  • In my opinion, the central claim is not supported by the evidence. When I look at Table 3, the component generated from the initial Skill with no evolution already reaches 0.5583 MRR. That is +0.0073 over NBFNet, about 73% of the total gain, versus +0.0027 attributable to the evolution loop, from a single run. Figure 3 and Section 4.3 show no steady improvement: gate MRR for GPT-5.6 Sol peaks and then "return[s] to a lower range," and cleary DeepSeek runs fluctuate. This is simply best-of-N selection under a noisy objective looks like. There is no equal-budget baseline (best-of-N with a fixed initial Skill, random or evolutionary search over the same DSL, an EoH/FunSearch-style loop), and the one-shot baseline gets one sample versus hundreds of LLM calls. The initial Skill is authored by the researchers and encodes design knowledge, and I couldn’t see that text. But the paper cannot separate the effect of the prompt, selection from many samples, and Skill evolution.
  • The bigger problem is empirical setting. Clearly, the empirical improvements are too small and too weakly controlled to be reliable. All results are single runs, with no seeds, confidence intervals, or tests, for gains of +0.0100 (WN18RR) and +0.0026 (FB15K-237) MRR, and +0.0006 and +0.0009 accuracy in triple classification.
Questions for authors
  • Does Skill evolution help beyond sampling? In Table 3, the component generated from the initial Skill S0 (no evolution) reaches 0.5583 MRR, which is basically +0.0073 over NBFNet and +0.0077 over one-shot design. That is most of the +0.0100 total gain. How would SkillGNN compare, at equal numbers of LLM calls and trained candidates, with (a) best-of-N using the fixed S0 and (b) random or mutation-based search over the same typed DSL?
  • Does the evolved Skill transfer? The paper claims design experience accumulates in the Skill. If you generate fresh components (e.g., 20 per condition) from the final Skill S_T and from S0 under the same protocol, how do their gate MRR distributions compare? Could you also show S0 and S_T? Figure 3 shows gate MRR returning to a lower range after its peak, so I could not tell whether the Skill improves generation or the search is picking the best of many noisy samples.
  • How large is the variance relative to the gains? All results appear to be single runs. Please report mean ± std and a paired significance test over at least 5 full-graph training seeds for NBFNet-PyG (trained under your settings, not the published number), one-shot design, the S0 component, and SkillGNN. How many full-graph runs were done for each reported number, and what rule picked the reported run? The Figure 2 caption says node-update results use "the run with highest test MRR." Was that rule also used for the scorer and CompGCN results?
2 comments
AuthorsOpenReview ↗

Response to Reviewer PNRG

W1. Thank you for examining how much of the gain comes from the initial Skill. The initial Skill makes a substantial contribution, and the additional gain from evolution is worth examining on a mature benchmark under a limited search budget. For context, RotatE improved MRR by 0.0060 over TransE when both used the same negative-sampling setup (Sun et al., 2019).

Figure 3 tracks the scores of individual candidates proposed during search. The search retains the best component found so far even when later proposals score lower. The fluctuations in gate MRR therefore do not by themselves establish that evolution is ineffective.

We added two controls to test what Skill evolution contributes beyond repeated sampling. For the first, we matched the budget at 100 generation attempts per method and used the same DSL, subgraph, and training and evaluation protocol. Duplicate and invalid proposals counted toward that budget. The best gate validation MRRs were 0.5507 for best-of-N with fixed S_0, 0.5488 for random search, and 0.5536 for SkillGNN. SkillGNN exceeded the two baselines by 0.0029 and 0.0048, respectively. In the second comparison, we independently generated 20 fresh candidates from the initial Skill S_0 and 20 from the evolved Skill S_T at step 100, keeping both Skills fixed during generation. The initial and evolved Skills yielded gate MRRs of 0.5401 ± 0.0098 and 0.5430 ± 0.0112, respectively (mean ± standard deviation). SkillGNN found a higher-scoring candidate at the matched generation budget, and the fresh proposals showed a trend toward higher gate MRR under the evolved Skill.

The initial Skill is already in the open-source code repository. One example of the design guidance it contains is “Prefer topology, geometry, or sharing changes over weight micro-tuning.”

The initial prompt and design prior give the search its starting point. Selection among repeated samples determines which candidate the search retains, and Skill evolution uses evaluation feedback to revise the guidance for later proposals. We will make these roles explicit in the revision.

Sun et al. (2019). RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. ICLR.

W2. We appreciate the focus on empirical reliability. The size of a gain should be interpreted alongside the difficulty of the task and the experimental controls. We evaluated the final best component with three random seeds. Its test MRR, H@1, H@3, and H@10 were 0.5598 ± 0.0011, 0.5051 ± 0.0006, 0.5820 ± 0.0019, and 0.6699 ± 0.0018, respectively (mean ± standard deviation), with modest variability across runs. Our NBFNet reproduction also yielded MRRs of 0.5517 on WN18RR and 0.4155 on FB15K-237, close to the published results. For each backbone comparison, SkillGNN used the same training pipeline and configuration, with only the corresponding component replaced.

The triple classification accuracy gains are small (0.0006 and 0.0009), with positive changes on both datasets. We regard them as supplementary evidence of improvement on this task. Statistical significance has yet to be established.

AuthorsOpenReview ↗

Response to Reviewer PNRG

Q1. To assess whether evolution helps beyond sampling, we matched the generation budget at 100 attempts, counting duplicate and invalid proposals. The best validation MRR on the subgraph gate split was 0.5507 for (a) best-of-N with fixed S_0 and 0.5488 for (b) random search over the same typed DSL. The first 100 generation attempts in the existing SkillGNN evolution trace reached 0.5536, exceeding (a) by 0.0029 and (b) by 0.0048. These comparisons provide evidence that Skill evolution can help beyond repeated sampling under the matched generation budget.

Q2. Thank you for suggesting the comparison on fresh proposals. Using the same generation model, subgraph, and training and evaluation protocol, we generated 20 fresh proposals from the initial Skill S_0 and 20 from the final Skill S_T at step 100. Both Skills stayed fixed throughout generation. The gate MRRs (mean ± sample standard deviation) were 0.5401 ± 0.0098 for the initial Skill and 0.5430 ± 0.0112 for the evolved Skill. These results suggest that the evolved Skill can generate proposals with higher gate MRR on average.

Q3. The main results in the original submission were reported as single-seed point estimates. Reporting point estimates follows prior work by Sun et al. (2019) and Zhu et al. (2021). To quantify variability for the final best component, we added full-graph runs with three random seeds. The test MRR, H@1, H@3, and H@10 on WN18RR were 0.5598 ± 0.0011, 0.5051 ± 0.0006, 0.5820 ± 0.0019, and 0.6699 ± 0.0018, respectively (mean ± standard deviation). The final model showed modest variability across these runs. Our NBFNet reproduction yielded MRRs of 0.5517 on WN18RR and 0.4155 on FB15K-237, close to the published results.

We selected the runs for the node-update, scorer, and CompGCN results by highest validation MRR. The reference to test MRR in the Figure 2 caption was a wording error. The node-update runs had validation MRRs of 0.5389 and 0.5374, so we selected the first. Its test MRR, H@1, H@3, and H@10 were 0.5541, 0.4978, 0.5795, and 0.6677, respectively. For reference, the run with lower validation MRR yielded 0.5573, 0.5041, 0.5764, and 0.6688 on the same test metrics.

Sun et al. (2019). RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. ICLR.

Zhu et al. (2021). Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. NeurIPS.

Reviewer_KQW8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses an LLM to evolve a local component of a GNN for knowledge graph completion, guided by a reusable textual Skill that is refined from evaluation feedback. The setting is well posed, the interface contract is clean, and the two datasets are standard. The problem is what the ablation says about the central claim. In Table 3 the first component already reaches 0.5583, and the full evolution adds 0.0027 MRR on a 252-query gate split, which is a fraction of a standard error. The NBFNet comparison also uses published numbers, and the one reproduced backbone runs below its published score. My decision is weak reject.

Critical strengths

First, the design object is well chosen. Restricting evolution to a local component under a fixed input-output contract keeps the rest of the GNN fixed and makes each proposal comparable, which is the right way to isolate a component's effect.

Second, the evaluator is more than a score. Program checks, structural distinctness, and subgraph training give the optimizer diagnostic feedback, and the CompGCN and node-update extensions show the framework is not tied to one component. The ablation that separates the first component from the evolved one is exactly the experiment a reader needs, even though it undercuts the main claim.

Critical weaknesses

First, the evidence does not support the claim that iterative Skill refinement accumulates design experience. Table 3 gives one-shot 0.5506, first component 0.5583, and best component 0.5610. So 0.0077 of the 0.0104 gain over one-shot comes from the initial Skill and only 0.0027 from evolution. The gate split has 252 queries, which puts the sampling standard error near 0.022 MRR, and no seed variance is reported. The evolution effect is therefore a fraction of a standard error, and the best component is a maximum over roughly one hundred noisy estimates.

Second, the comparison against NBFNet mixes training pipelines. The NBFNet numbers, 0.5510 and 0.4150, are taken from the literature, while SkillGNN is trained in the authors' own pipeline. The only backbone they reproduce, CompGCN, lands at 0.4664 against a published 0.479, a shortfall of 0.0126 that is five times the 0.0026 gain claimed on FB15K-237. Without a reproduced NBFNet number, the headline gain cannot be separated from the pipeline.

Questions for authors
  1. How many seeds does each configuration use? With 252 gate queries and no variance reported, can you show that 0.0027 is outside noise?
  2. Is NBFNet re-run under your pipeline? If the CompGCN reproduction is 0.0126 below its published score, what is the reproduced NBFNet score?
  3. What is the token and wall-clock cost of the full evolution, and why did FB15K-237 run forty steps when the budget is stated as one hundred?
  4. The node-update result is described as the run with highest test MRR. Was that component selected on the test set, and how does that affect the reported 0.5541?
1 comment
AuthorsOpenReview ↗

Response to Reviewer KQW8

W1. We appreciate the reviewer's careful breakdown of the gains. Their magnitude should be read in the context of two well-studied knowledge graph completion benchmarks, WN18RR and FB15K-237, where further improvements are difficult. On FB15K-237, Sun et al. (2019) report a 0.0130 MRR gain for RotatE over ConvE and a 0.0060 gain over TransE in the comparison with matched negative sampling. The finite budget limits how far SkillGNN can explore. The current results describe its performance at that budget and leave its performance ceiling open.

The 252 queries come from the subgraph gate split used during search. Table 3 reports full-graph test performance, so the gate split's sampling standard error cannot directly quantify uncertainty in the full-graph difference. Selecting the best component by validation MRR and reporting point estimates in the main tables follow precedents in Sun et al. (2019) and Zhu et al. (2021). To assess variability, we also report full-graph results across three random seeds using validation MRR for selection. Test MRR, H@1, H@3, and H@10 were 0.5598 ± 0.0011, 0.5051 ± 0.0006, 0.5820 ± 0.0019, and 0.6699 ± 0.0018, respectively (mean ± standard deviation).

Sun et al. (2019). RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. ICLR.

Zhu et al. (2021). Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. NeurIPS.

W2. We appreciate the concern about pipeline differences. We reproduced NBFNet using the training pipeline from the original paper. In the order MRR, H@1, H@3, and H@10, the reproduced scores were 0.5517, 0.4962, 0.5731, and 0.6643 on WN18RR and 0.4155, 0.3222, 0.4557, and 0.5972 on FB15K-237. The reproduced MRRs are close to the published values of 0.5510 and 0.4150. SkillGNN used the same pipeline and configuration, with only the corresponding component replaced. The gains thus hold with the training settings fixed.

Using the public CompGCN code and configuration, we reproduced an MRR of 0.4664, below the published 0.4790. Holding the pipeline and configuration fixed, we replaced only the corresponding component. SkillGNN then reached 0.4708, a gain of 0.0044 over the reproduced CompGCN.

Q1. Each of the three results in the original submission was reported from a single seed. To assess variability, we evaluated the final best component with three random seeds. The test results (mean ± standard deviation) were 0.5598 ± 0.0011 for MRR, 0.5051 ± 0.0006 for H@1, 0.5820 ± 0.0019 for H@3, and 0.6699 ± 0.0018 for H@10. These runs showed modest variability for the final model. The 252 queries belong to the subgraph gate split used during evolution. The 0.0027 MRR gain was measured on the full-graph test set, so the gate split's sampling standard error cannot directly determine whether that gain exceeds noise.

Q2. Yes. We reran NBFNet using the training pipeline and configuration from the original paper. In the order MRR, H@1, H@3, and H@10, we obtained 0.5517, 0.4962, 0.5731, and 0.6643 on WN18RR and 0.4155, 0.3222, 0.4557, and 0.5972 on FB15K-237. These MRR values are close to the published 0.5510 and 0.4150. Using the public code and configuration, we reproduced a CompGCN MRR of 0.4664, below the published 0.4790. With only the corresponding component replaced, SkillGNN reached 0.4708, an increase of 0.0044. Within each comparison, the models shared the same training pipeline and configuration.

Q3. Thank you for asking about the search cost and for flagging the discrepancy in the stated budget. The full 100-step GPT-5.6 Sol run on WN18RR took about 81 hours of wall-clock time. The usage logs give a lower bound of approximately 18,960,000 tokens.

For FB15K-237, we planned 100 evolution steps. Given the experiment schedule and available compute, we completed four full update cycles (40 steps) by the reporting cutoff, and the paper reports the results from those cycles. We plan to complete the remaining 60 evolution steps on FB15K-237.

Q4. We appreciate the chance to clarify the selection criterion. We used validation MRR alone to select the node-update run. Figure 2 mistakenly labels the criterion as test MRR. The selected run had a validation MRR of 0.5389, compared with 0.5374 for the other run. Its test MRR, H@1, H@3, and H@10 were 0.5541, 0.4978, 0.5795, and 0.6677, respectively, so the reported test MRR remains 0.5541. For reference, the run with lower validation MRR yielded 0.5573, 0.5041, 0.5764, and 0.6688 on the same test metrics.

Reviewer_kn8NOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents SkillGNN, a framework that uses LLMs to design local components for knowledge graph GNNs. The framework maintains a textual design Skill. An LLM generates a component under a fixed interface, and an evaluator checks the program and trains it on a development subgraph. The main experiments evolve the scoring function of NBFNet on WN18RR and FB15K-237. The paper also studies NBFNet node updates, CompGCN scoring functions, and different design LLMs.

Critical strengths
  1. The paper studies an interesting problem: using LLM feedback to automate the design of trainable GNN components.
  2. The evaluator includes program, tensor, gradient, semantic, and structural checks before model training.
  3. The paper studies several component types and design LLMs and reports the substantial search cost.
Critical weaknesses
  1. The LLM-driven baselines are insufficient and empirical gains are small. In Table 2, the only direct LLM-driven baseline is one-shot LLM design. SkillGNN improves over it by only 0.0104 MRR on WN18RR and 0.0039 on FB15K-237s.

  2. One result uses the test set for model selection. Figure 2 states that the node-update result uses the run with the highest test MRR. Runs should be selected using validation performance before the test set is evaluated.

  3. The search is expensive and not currently reproducible. A single evolution run can consume tens of millions of LLM tokens and many GPU hours. Some usage records are missing and therefore remain lower bounds.

Questions for authors
  1. Under the same number of generated candidates, component-training runs, and LLM tokens, does SkillGNN outperform a fixed-Skill best-of-N search?

  2. Can the node-update runs be selected only by validation MRR rather than test MRR?

  3. Can a Skill evolved on WN18RR improve search on FB15K-237 or another unseen graph dataset?

1 comment
AuthorsOpenReview ↗

Response to Reviewer kn8N

W1. Thank you for raising the question of how to interpret the gains. WN18RR and FB15K-237 are well-studied knowledge graph completion benchmarks where further improvements are difficult. On FB15K-237, Sun et al. (2019) report a 0.0130 MRR gain for RotatE over ConvE. They also report a 0.0060 gain over TransE when both methods use the same negative-sampling setup. The finite evolution budget limits how far SkillGNN can explore, so the results describe what SkillGNN achieved within that budget and leave its performance ceiling open.

Sun et al. (2019). RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. ICLR.

W2. Thank you for catching the caption error. We selected the run with higher validation MRR (0.5389 versus 0.5374) before evaluating it on the test set. The selected run's test MRR, H@1, H@3, and H@10 were 0.5541, 0.4978, 0.5795, and 0.6677, respectively. For reference, the other run yielded 0.5573, 0.5041, 0.5764, and 0.6688 on the same test metrics. Those additional test scores played no role in model selection.

W3. We appreciate the concern about cost and reproducibility. The Skill, components, and search trajectory can be saved and reused, allowing later runs to resume evolution from a saved state. The 100-step GPT-5.6 Sol run on WN18RR logged approximately 18,960,000 tokens. Subgraph training and validation took approximately 19.7500 GPU hours in total, averaging about 11 minutes per evaluation. LLM calls and candidate training are separate stages. During evolution, subgraph evaluation provides performance feedback. Full-graph training follows the search. A fresh search may follow a different trajectory because LLM sampling is stochastic. The source code accompanying the paper supports reproduction of the evolution workflow.

Q1. The budget comparison matters for interpreting the gains. We could not match all three budgets simultaneously in our setup, so we ran two comparisons against fixed-Skill best-of-N on WN18RR. The baseline used GPT-5.6 Sol and held its Skill fixed throughout generation, so it accumulated no experience across proposals. For SkillGNN, we used the existing evolution trace. Both methods used the same subgraph, random seed, and evaluation protocol, with 20 training epochs per evaluated candidate. We counted duplicate and invalid proposals toward the generation budget and compared the highest validation MRR on the subgraph gate split.

At 100 generation attempts per method, the best gate validation MRR was 0.5507 for best-of-N and 0.5536 for SkillGNN, a difference of 0.0029 in SkillGNN's favor.

With roughly equal token usage, best-of-N used 194,624 tokens to generate 152 candidates and reached a best gate validation MRR of 0.5527. SkillGNN used 199,227 tokens to generate 4 candidates and reached 0.5426. The fixed-Skill baseline performed better at this lower budget, with token usage differing by 4,603. In this run, SkillGNN found better candidates only after further exploration. Iterative feedback and Skill updates add search overhead, and reducing that cost is an area for improvement.

Q2. Yes. We selected the node-update run using validation MRR alone. Thank you for drawing attention to the wording in Figure 2, which mistakenly names test MRR as the criterion. The two runs had validation MRRs of 0.5389 and 0.5374, so we selected the first. Its test MRR, H@1, H@3, and H@10 were 0.5541, 0.4978, 0.5795, and 0.6677, respectively. For reference, the run with lower validation MRR yielded 0.5573, 0.5041, 0.5764, and 0.6688 on the same test metrics.

Q3. Thank you for suggesting this transfer test. We used the Skill evolved on WN18RR to generate 10 candidate components on FB15K-237. The best candidate reached a subgraph gate validation MRR of 0.3475. This was 0.0024 below the best value of 0.3499 from the 40-step evolution of FB15K-237's own Skill. The result provides evidence of transfer in this setting, with fewer candidate generations yielding performance close to that of the target dataset's own evolution run.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.