- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether a designed synthetic pretraining corpus can replace real knowledge graph pretraining for zero-shot knowledge graph completion while keeping the ULTRA reasoning model unchanged. CAMP-KG independently samples rule templates and graph statistics, constructs provenance-aware supervision that preserves rule premises and excludes evaluation targets, and selects checkpoints on held-out synthetic graphs. On 54 zero-shot graphs, its overall MRR is close to that of real-graph pretraining, improves on graphs not directly derived from the pretraining data sources, and regresses on source-proximal graphs. The authors analyze these phenomena through a corpus source × supervision factorial experiment, generator ablations, corpus and compute controls, representation probes, and a reduced-scale TRIX evaluation.
Critical strengths
-
The research question is clear and has some novelty. This paper studies a worthwhile question: for zero-shot knowledge graph reasoning, can the design of the pretraining data distribution, rather than ever more complex model architectures, improve cross-knowledge-graph generalization?
-
The paper provides relatively sufficient controlled evidence around the pretraining pipeline. The corpus source × supervision factorial experiment separates their effects under the specific setting; the rule × statistical diversity ablation, edge- and compute-matched controls, mixed real/synthetic training, and second-backbone evaluation check various alternative explanations. Recomputing means and win/tie/loss counts from the values shown in Table A1 is consistent with the main-text results within rounding. This evidence makes the contribution more convincing than a single synthetic-vs-real comparison.
Critical weaknesses
-
The overall performance advantage is limited, and the core conclusions depend on dataset design. Table 1 provides a four-arm comparison, but the “provenance-aware” supervision under the two corpora is not fully equivalent: the synthetic graphs use rule derivations provided by the generator, while the real graphs use rules mined by AnyBURL (Appendix I). Their coverage, reliability, and premise visibility may differ, and checkpoint selection also uses different validation domains. In addition, Real-Subgraphs uses standard supervision, whereas CAMP-KG uses provenance-aware supervision (Appendix H). Therefore, these results support the value of the full pipeline, but it remains difficult to determine how much of the out-of-provenance gain comes from corpus design rather than from the supervision and selection protocol. Moreover, the 54 test graphs actually belong to 11 provenance clusters, and some same-source graphs are strongly dependent. The authors are advised to: (1) report macro-MRR aggregated by provenance family and per-family performance changes, not only graph-level averages; (2) add a leave-one-family-out sensitivity analysis to verify whether the source-disjoint gain depends on a small number of data families.
-
The role of “rule–statistics decoupling” lacks direct causal evidence. Table 2 compares the presence/absence of rules and statistical diversity and supports that both contribute, but it does not compare independent sampling with correlated sampling while holding the respective distributions of rule and statistical parameters fixed. Thus, it has not isolated the additional effect of “decoupling” beyond rule coverage and distributional diversity. The distance and representation probes in §5 provide correlational evidence, but after excluding source-proximal families the interval for the distance effect crosses zero, and lower statistical decodability does not equal lower predictive reliance. The existing evidence is consistent with the decoupling explanation, but insufficient to establish it as the main mechanism for improved transfer.
Questions for authors
-
Clarification of supervision construction: How are premise/conclusion role conflicts handled for symmetry rules in synthetic graphs and for mined rules in real graphs, respectively? Please report the number of conflicts, the retained derivation coverage, and the target-exclusion check after adding inverse edges. A distinction should be made between visible logical premises and artificial inverse edges of evaluation targets. This answer could significantly strengthen or weaken my assessment.
-
Clarification of mechanism evidence: Can the authors provide the fitted curves for each method corresponding to Eq. (5), along with statistical uncertainty accounting for provenance clusters? The right panel of Figure 3 and Figure A2 currently show correlations of MRR differences, rather than the fits described in the captions. At the same time, please limit interpretations of controls to genuinely matched comparisons: 3-Cycled in Table 3 has only 213K edges, and Appendix H states that Real-Subgraphs uses standard supervision.
-
Experimental design details: The official ULTRA repository states that its released 3-graph checkpoint used 800K training steps, whereas this paper uniformly uses 200K. Although the paper’s ULTRA reproduction results are close to the published results, it is still worth explaining why the official checkpoint was not used as an additional practical reference.
-
More baselines need to be compared in the experiment, such as [1, 2, 3, 4, 5]
[1] How Expressive are Knowledge Graph Foundation Models? ICML 2025
[2] TRIX: A More Expressive Model for Zero-Shot Domain Transfer in Knowledge Graphs. LoG 2024
[3] A Prompt-Based Knowledge Graph Foundation Model for Universal In-Context Reasoning. NeurIPS 2024
[4] SEMMA: A Semantic Aware Knowledge Graph Foundation Model. EMNLP 2025
[5] Knowledge Reasoning Language Model: Unifying Knowledge and Language for Inductive Knowledge Graph Reasoning. ICLR 2026
6 comments
Q1: How are premise and conclusion conflicts handled?
Thank you for highlighting this important correctness issue. We resolve premise and conclusion conflicts by retaining premises, excluding conflicting conclusions from prediction targets, and verifying target exclusion after inverse augmentation. We clarify how premise–conclusion conflicts are resolved and how target masking is implemented under inverse-edge augmentation.
Table R2. Premise conflict resolution
Source Candidate Conflicts Retained Retention Synthetic symmetry 2,399,063 41,737 2,357,326 98.26% Synthetic other rules 31,600,000 0 31,600,000 100.00% Real mined rules 133,424 9,336 124,088 93.00% All counts refer to distinct canonical triples. Premise conflicts are resolved before assigning conclusion targets: when a triple serves as both a premise and a candidate conclusion, it is retained as an observable premise and excluded from the prediction targets. Non-symmetry synthetic rules have zero conflicts by construction because their head relations never serve as premises. The synthetic counts precede incompleteness deletion. For the real graphs, 124,088 candidate conclusions are retained after conflict resolution, corresponding to approximately 22.79% of the 544,534 real training triples.
For each prediction query, we remove both the target triple and its artificially introduced inverse edge from the observation graph. This operation is performed after resolving premise–conclusion conflicts, ensuring that retained logical premises remain observable while withheld conclusions are excluded. In particular, for symmetry rules, we distinguish the inverse edge of an observable premise from the inverse edge of a withheld conclusion. The former remains visible as valid evidence, whereas the latter is excluded together with the prediction target.
Symmetry requires distinguishing logical conclusions from artificial inverse edges.
Role Canonical triple Artificial inverse Observable premise Withheld conclusion The inverse of an observable premise remains legitimate evidence. The withheld conclusion and its artificial inverse must both be absent from the evaluation observation graph. If a conclusion already serves as a premise, premise precedence excludes it from the prediction targets.
Q2: Are the structural-distance figures and corpus controls correctly described?
We agree that the visualization is inconsistent. We correct the figures to present method-specific fitted relationships with provenance-cluster-bootstrap uncertainty and clarify that 3-Cycled and Real-Subgraphs are not fully matched controls, limiting the conclusions drawn from these comparisons.
We correct the right panel of Figure 3 and Figure A2 to show the fitted relationship between MRR and standardized structural distance for each method, rather than correlations of MRR differences. The fitted slopes are −0.021 for ULTRA-Real and −0.010 for CAMP-KG, yielding an interaction of +0.011 with a 95% provenance-cluster-bootstrap confidence interval of +0.004 to +0.019. After excluding source-proximal families, the interaction decreases to +0.007 with a confidence interval of −0.001 to +0.016. We therefore interpret this analysis as supporting an association rather than establishing a causal mechanism.
We agree that the controls should not be interpreted as isolating individual factors when their experimental settings are not matched. Specifically, 3-Cycled contains only 213K edges and is not edge-matched to CAMP-KG; thus, its comparison cannot isolate the effect of cycle structure. Similarly, Real-Subgraphs uses standard supervision rather than provenance-aware supervision, as correctly specified in Appendix H, and therefore cannot isolate the corpus effect under matched supervision. We restrict both comparisons to descriptive evidence and withdraw stronger causal interpretations.
Q3: Why use 200K steps rather than the released 800K-step ULTRA checkpoint?
You are correct that the released three-graph ULTRA checkpoint was trained for 800k steps. Our 200k-step experiment is a budget-matched reproduction, not a reproduction of the official training duration.
We selected 200k steps to compare the real and synthetic pipelines under identical training budgets, retaining the same architecture, optimizer, loss, and inference procedure. Our reproduced baseline closely matches the published results across the three evaluation settings, but this agreement does not eliminate the training-duration difference. The released 800k-step checkpoint is an important practical reference and should be reported separately from the controlled 200k-step comparison. Retraining both pipelines for 800k steps would test a different budget and require substantially more computation. We have not evaluated the official 800K-step checkpoint on our complete 54-graph benchmark, and therefore do not claim superiority over that checkpoint.
Q4: How does CAMP-KG compare with additional foundation models?
We agree that additional architectural baselines would improve the broader empirical context. Our primary claim concerns pretraining-data design under a fixed reasoner, rather than superiority over models using different architectures or additional semantic inputs.
We agree that broader architectural comparisons would improve the empirical context. Table 1 isolates the effect of the pretraining pipeline within ULTRA. Appendix L additionally evaluates TRIX using real and synthetic pretraining, with the same backbone within each comparison. Across 15 graphs, TRIX improves from 0.392 to 0.398 MRR with the synthetic pipeline. Its source-disjoint gain is +0.019, closely matching the corresponding ULTRA gain of +0.021 on this subset.
MOTIF, KG-ICL, SEMMA, and the KRLM are relevant architectural references. They differ in model architecture, pretraining resources, access to textual semantics, or evaluation protocol. A fair direct comparison requires a common graph suite and consistent inference conditions. We distinguish the role of these methods as broader performance baselines from the controlled comparisons needed to identify the effect of the pretraining distribution.
Your rigorous assessment of provenance dependence, supervision correctness, and baseline comparability has substantially strengthened the robustness of our evaluation and the credibility of our empirical conclusions. We hope these clarifications and additional results address your main concerns. We would be happy to provide further clarification if needed.
Thank you for your detailed comments and for identifying important reporting issues. The corrections strengthen the interpretation of the main result by separating the demonstrated transfer behavior from claims requiring additional experimental verification.
We address your concerns as follows: Additional analyses establish robustness across provenance families and the transfer benefit of independent sampling. We clarify target masking, correct the structural distance figures, and restrict interpretations of unmatched baselines.
W1: Does the source-disjoint gain depend on a small number of provenance families?
Following your helpful suggestion, we reanalyzed the results by provenance family. The family-macro MRR improves from 0.325 to 0.337 across 11 clusters, and the source-disjoint gain remains positive after excluding any single family. These results show that the improvement is not driven by a single favorable family.
W2: Does independent rule–statistics sampling improve transfer beyond rules and statistical diversity?
You are correct that the existing ablation establishes the separate contributions of rules and statistical diversity, rather than the causal benefit of sampling them independently. we therefore conducted a matched independent-versus-correlated sampling experiment and found that independent sampling improves MRR by +0.0124 overall and +0.0197 on source-disjoint graphs, providing direct evidence that decoupling contributes to transfer beyond marginal rule coverage and statistical diversity.