- Rating
- 1
- Confidence
- 2(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SkillGNN uses an LLM agent to iteratively develop a trainable component within a fixed KGC GNN under a strict input-output contract. Candidates are compiled into typed expression trees and then undergo filtering based on type, gradient, semantic, and structural criteria. Eligible candidates are then trained on a fixed development subgraph with reflect and gate validation splits. A textual “Skill” holding design guidance is revised by SkillOpt (Yang et al., 2026), and a revision is accepted only if the component generated with it enhances gate MRR. The best-performing gate component is retrained from scratch on the entire graph. The authors report MRR gains over the published NBFNet of +0.0100 on WN18RR (0.5610) and +0.0026 on FB15K-237 (0.4176). Gains over one-shot LLM design are +0.0104 and +0.0039. Additionally, they report a +0.0044 gain on a reproduced CompGCN, a smaller gain from node-update evolution, results with three design LLMs, triple-classification gains of +0.0006 and +0.0009, and WN18RR ablations.
Critical strengths
- Candidates are evaluated on a fixed, documented subgraph with reflect/gate splits. In the end, the authors also exclude the test set from search. A typed DSL with validity and structural checks limits the number of invalid proposals. The interface contract and zero-initialized gating isolate the scorer’s effect from propagation and the objective. So I would say, this is a reusable engineering contribution, regardless of the final numbers.
- The reported direction of improvement looks consistent (even though the magnitudes are small).
Critical weaknesses
- In my opinion, the central claim is not supported by the evidence. When I look at Table 3, the component generated from the initial Skill with no evolution already reaches 0.5583 MRR. That is +0.0073 over NBFNet, about 73% of the total gain, versus +0.0027 attributable to the evolution loop, from a single run. Figure 3 and Section 4.3 show no steady improvement: gate MRR for GPT-5.6 Sol peaks and then "return[s] to a lower range," and cleary DeepSeek runs fluctuate. This is simply best-of-N selection under a noisy objective looks like. There is no equal-budget baseline (best-of-N with a fixed initial Skill, random or evolutionary search over the same DSL, an EoH/FunSearch-style loop), and the one-shot baseline gets one sample versus hundreds of LLM calls. The initial Skill is authored by the researchers and encodes design knowledge, and I couldn’t see that text. But the paper cannot separate the effect of the prompt, selection from many samples, and Skill evolution.
- The bigger problem is empirical setting. Clearly, the empirical improvements are too small and too weakly controlled to be reliable. All results are single runs, with no seeds, confidence intervals, or tests, for gains of +0.0100 (WN18RR) and +0.0026 (FB15K-237) MRR, and +0.0006 and +0.0009 accuracy in triple classification.
Questions for authors
- Does Skill evolution help beyond sampling? In Table 3, the component generated from the initial Skill S0 (no evolution) reaches 0.5583 MRR, which is basically +0.0073 over NBFNet and +0.0077 over one-shot design. That is most of the +0.0100 total gain. How would SkillGNN compare, at equal numbers of LLM calls and trained candidates, with (a) best-of-N using the fixed S0 and (b) random or mutation-based search over the same typed DSL?
- Does the evolved Skill transfer? The paper claims design experience accumulates in the Skill. If you generate fresh components (e.g., 20 per condition) from the final Skill S_T and from S0 under the same protocol, how do their gate MRR distributions compare? Could you also show S0 and S_T? Figure 3 shows gate MRR returning to a lower range after its peak, so I could not tell whether the Skill improves generation or the search is picking the best of many noisy samples.
- How large is the variance relative to the gains? All results appear to be single runs. Please report mean ± std and a paired significance test over at least 5 full-graph training seeds for NBFNet-PyG (trained under your settings, not the published number), one-shot design, the S0 component, and SkillGNN. How many full-graph runs were done for each reported number, and what rule picked the reported run? The Figure 2 caption says node-update results use "the run with highest test MRR." Was that rule also used for the scorer and CompGCN results?
2 comments
Response to Reviewer PNRG
Q1. To assess whether evolution helps beyond sampling, we matched the generation budget at 100 attempts, counting duplicate and invalid proposals. The best validation MRR on the subgraph gate split was 0.5507 for (a) best-of-N with fixed S_0 and 0.5488 for (b) random search over the same typed DSL. The first 100 generation attempts in the existing SkillGNN evolution trace reached 0.5536, exceeding (a) by 0.0029 and (b) by 0.0048. These comparisons provide evidence that Skill evolution can help beyond repeated sampling under the matched generation budget.
Q2. Thank you for suggesting the comparison on fresh proposals. Using the same generation model, subgraph, and training and evaluation protocol, we generated 20 fresh proposals from the initial Skill S_0 and 20 from the final Skill S_T at step 100. Both Skills stayed fixed throughout generation. The gate MRRs (mean ± sample standard deviation) were 0.5401 ± 0.0098 for the initial Skill and 0.5430 ± 0.0112 for the evolved Skill. These results suggest that the evolved Skill can generate proposals with higher gate MRR on average.
Q3. The main results in the original submission were reported as single-seed point estimates. Reporting point estimates follows prior work by Sun et al. (2019) and Zhu et al. (2021). To quantify variability for the final best component, we added full-graph runs with three random seeds. The test MRR, H@1, H@3, and H@10 on WN18RR were 0.5598 ± 0.0011, 0.5051 ± 0.0006, 0.5820 ± 0.0019, and 0.6699 ± 0.0018, respectively (mean ± standard deviation). The final model showed modest variability across these runs. Our NBFNet reproduction yielded MRRs of 0.5517 on WN18RR and 0.4155 on FB15K-237, close to the published results.
We selected the runs for the node-update, scorer, and CompGCN results by highest validation MRR. The reference to test MRR in the Figure 2 caption was a wording error. The node-update runs had validation MRRs of 0.5389 and 0.5374, so we selected the first. Its test MRR, H@1, H@3, and H@10 were 0.5541, 0.4978, 0.5795, and 0.6677, respectively. For reference, the run with lower validation MRR yielded 0.5573, 0.5041, 0.5764, and 0.6688 on the same test metrics.
Sun et al. (2019). RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. ICLR.
Zhu et al. (2021). Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. NeurIPS.
Response to Reviewer PNRG
W1. Thank you for examining how much of the gain comes from the initial Skill. The initial Skill makes a substantial contribution, and the additional gain from evolution is worth examining on a mature benchmark under a limited search budget. For context, RotatE improved MRR by 0.0060 over TransE when both used the same negative-sampling setup (Sun et al., 2019).
Figure 3 tracks the scores of individual candidates proposed during search. The search retains the best component found so far even when later proposals score lower. The fluctuations in gate MRR therefore do not by themselves establish that evolution is ineffective.
We added two controls to test what Skill evolution contributes beyond repeated sampling. For the first, we matched the budget at 100 generation attempts per method and used the same DSL, subgraph, and training and evaluation protocol. Duplicate and invalid proposals counted toward that budget. The best gate validation MRRs were 0.5507 for best-of-N with fixed S_0, 0.5488 for random search, and 0.5536 for SkillGNN. SkillGNN exceeded the two baselines by 0.0029 and 0.0048, respectively. In the second comparison, we independently generated 20 fresh candidates from the initial Skill S_0 and 20 from the evolved Skill S_T at step 100, keeping both Skills fixed during generation. The initial and evolved Skills yielded gate MRRs of 0.5401 ± 0.0098 and 0.5430 ± 0.0112, respectively (mean ± standard deviation). SkillGNN found a higher-scoring candidate at the matched generation budget, and the fresh proposals showed a trend toward higher gate MRR under the evolved Skill.
The initial Skill is already in the open-source code repository. One example of the design guidance it contains is “Prefer topology, geometry, or sharing changes over weight micro-tuning.”
The initial prompt and design prior give the search its starting point. Selection among repeated samples determines which candidate the search retains, and Skill evolution uses evaluation feedback to revise the guidance for later proposals. We will make these roles explicit in the revision.
Sun et al. (2019). RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. ICLR.
W2. We appreciate the focus on empirical reliability. The size of a gain should be interpreted alongside the difficulty of the task and the experimental controls. We evaluated the final best component with three random seeds. Its test MRR, H@1, H@3, and H@10 were 0.5598 ± 0.0011, 0.5051 ± 0.0006, 0.5820 ± 0.0019, and 0.6699 ± 0.0018, respectively (mean ± standard deviation), with modest variability across runs. Our NBFNet reproduction also yielded MRRs of 0.5517 on WN18RR and 0.4155 on FB15K-237, close to the published results. For each backbone comparison, SkillGNN used the same training pipeline and configuration, with only the corresponding component replaced.
The triple classification accuracy gains are small (0.0006 and 0.0009), with positive changes on both datasets. We regard them as supplementary evidence of improvement on this task. Statistical significance has yet to be established.