- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates negative sampling for knowledge graph foundation models (KGFMs) during cross-graph pretraining. The authors argue that negatives obtained by uniformly replacing entities are often too easy to provide sufficiently informative supervision, and propose KMAS to address this limitation.
KMAS uses the foundation model’s current relation encoder to compute similarities between relations and constructs candidate-entity sampling distributions from the head and tail entity sets associated with similar relations. During training, it combines uniformly sampled negatives with these relation-aware negatives and follows a warmup–increase–decrease schedule for the proportion of hard negatives. The sampling distributions evolve as the relation encoder is updated, whereas the mixture schedule is predetermined by training progress.
The main contribution is a training-time negative sampling module applicable to multiple existing KGFMs, rather than a new foundation model architecture or a theoretical advance in expressiveness.
The authors pretrain on three knowledge graphs and evaluate five foundation models on 44 datasets. Average MRR improves for all five models, with absolute gains ranging from 0.004 to 0.008. Ablation studies support the effectiveness of hybrid sampling, continuously updated relation representations, and the proposed mixture schedule. The paper also compares against existing negative sampling methods and reports cross-dataset significance tests for selected models, along with computational overhead. Overall, the experiments support average performance improvements, but do not fully establish the source of these gains or their robustness across random seeds and dataset families.
Critical strengths
- Compatibility with Existing Models without Changing Inference
KMAS reuses the existing relation encoder, requires no separate negative-generation network, and leaves the inference procedure unchanged. For models with inexpensive relation encoders, it can potentially improve performance with limited additional training overhead.
If the gains are robustly established, a training module that requires neither architectural redesign nor additional inference cost would have practical value. Simplicity is not itself a limitation: a simple method that consistently improves strong baselines can constitute a meaningful contribution.
However, the low-overhead claim should remain architecture-specific, as demonstrated by the substantially higher training cost for FLOCK.
- Evaluation across Multiple Foundation Models
Average performance improves for all five evaluated foundation models. The authors also provide ablations examining hybrid sampling, frozen relation representations, and alternative mixture schedules.
Since the central claim is that KMAS enhances existing KGFMs, effectiveness across architectures is more relevant than achieving the best result on an individual dataset. The reported results provide initial evidence of broad applicability and reduce the likelihood that the method’s success depends entirely on tuning for a single architecture.
The strength of this evidence is nevertheless limited by the absence of reported multi-seed results and the potential dependence among datasets.
Critical weaknesses
- The Added Value of the Core Mechanism over Simpler Alternatives Is Not Sufficiently Established
KMAS constructs candidate distributions from the head and tail entity sets of similar relations. This mechanism may primarily impose entity-type or relational-role constraints. The current evaluation does not adequately disentangle the benefit of these constraints from the additional benefit attributed to continuously updated, transferable relation representations.
The most relevant controlled comparisons would include:
- Sampling from the head and tail entity sets of the target relation itself;
- Sampling based on static relation–entity co-occurrence information;
- Using the same mixture schedule without learned relation similarities.
The main technical contribution rests on using transferable relation representations to guide negative sampling. If simpler relational-role constraints combined with the same schedule achieve comparable performance, the current experiments would not establish the necessity of the proposed mechanism, and the contribution would appear to be a relatively incremental combination of existing ideas.
Comparisons with existing negative sampling methods do not fully substitute for controlled experiments that isolate the paper’s central mechanism.
- The Modest Average Gains Are Not Yet Supported by Sufficient Evidence of Robustness and Fair Comparison
The average MRR gains range from 0.004 to 0.008, but the paper does not clearly report means, variances, or confidence intervals across multiple random seeds. The per-dataset results also include noticeable regressions, while some of the larger gains are concentrated on individual datasets.
For example, SEMMA improves by 0.108 MRR on Metafam. Under an equally weighted average over 44 graphs, this single result contributes approximately 0.00245, or roughly 30% of the average gain of 0.008 reported in Table 1. This does not invalidate the improvement on Metafam, but it motivates examining how sensitive the aggregate conclusion is to individual datasets.
In addition, the random-sampling ablation KMASsub achieves an average MRR of 0.364 with ULTRA, whereas the original ULTRA baseline in Table 1 achieves 0.360. This discrepancy needs to be explained to isolate the contribution of the proposed sampling mechanism.
Questions for authors
A. Clarifications Regarding the Existing Experiments and Implementation
Question 1: Were the original baselines and their KMAS-enhanced counterparts evaluated using identical training, validation, and model-selection procedures?
Please explain the difference between the original ULTRA random-negative baseline and KMASsub. In particular, were the random seeds, training steps, loss weighting, checkpoint-selection procedures, and hyperparameter search budgets matched?
Question 2: How was overlap between the pretraining and evaluation datasets checked, and which triples were used for negative masking?
Please specify how entity, relation, and triple overlap was assessed. Please also clarify whether training-time negative masking used only facts permitted by the training protocol.
B. Additional Evidence Needed to Support the Current Claims
Question 3: With the mixture schedule and training configuration held fixed, does KMAS significantly outperform simpler relational-role-constrained sampling methods?
The most informative comparisons would be sampling from the target relation’s own head and tail entity sets and sampling based on static relation–entity co-occurrence information.
Question 4: Are the main gains reproducible across random seeds and robust at the dataset-family level?
Please provide multi-seed results for at least representative models. It would also be important to examine whether the conclusions hold when individual high-gain datasets are excluded and when results are aggregated by dataset family.
3 comments
Questions
Q1: Each original baseline and its KMAS-enhanced counterpart use matching random seeds, training steps, loss weighting, checkpoint-selection procedures, and hyperparameter search budgets. Hyperparameters and checkpoints are selected only on the validation sets of FB15k-237, WN18RR, and CoDEx-Medium; the evaluation datasets are not used for model selection. Please see our response to W2 for an explanation of the difference between the original ULTRA sampler and .
Q2: We follow the dataset construction, splits, and pre-training and zero-shot evaluation protocol of previous KGFMs: pre-training on FB15k-237, WN18RR, and CoDEx-Medium and evaluating on the other benchmark datasets without training or fine-tuning on them. For training-time negative masking, strict_negative_mask excludes tails for which is observed in the current pre-training graph when corrupting a query ; it analogously excludes observed heads for . Thus, “the entire set of triples” in Section 3.3 refers to the observed triples in the current pre-training graph. Validation and test facts used for filtered ranking belong to the evaluation procedure and are not supplied to the training negative sampler.
Q3: Thank you for specifying the controlled comparison. Table 4 in our response to W1 compares KMAS with and , using the same warmup, mixture schedule, and training configuration. The former assigns equal probability to distinct entities in the corresponding set; the latter weights them by their observed occurrence counts, without learned relation similarities or updates during training. Known positives are filtered in both variants; an empty candidate set falls back to the target relation's combined head and tail entity set under the same rule. The random-negative proportion, total negative sample budget, and model-selection procedure are unchanged. This comparison examines the additional contribution of learned relation similarities, while the frozen variant separately examines their continuous updating.
Q4: Table 2 in our response to Reviewer xpBs reports five-seed results for two representative KGFMs, ULTRA and SEMMA, and their KMAS-enhanced counterparts to assess variability across training runs. For ULTRA and SEMMA, Table 5 below reports MRR gains on all 44 datasets and by dataset family.
Table 5. MRR gains from adding KMAS to ULTRA and SEMMA: mean ± standard deviation over five seeds. Datasets within each family are equally weighted.
| Evaluation collection or dataset family | Number of datasets | ULTRA MRR gain | SEMMA MRR gain |
|---|---|---|---|
| All evaluation datasets | 44 | +0.0068 ± 0.0029 | +0.0091 ± 0.0030 |
| CoDExSmall / CoDExLarge | 2 | +0.0058 ± 0.0056 | +0.0016 ± 0.0055 |
| NELL995 / NELL23k | 2 | +0.0130 ± 0.0158 | +0.0062 ± 0.0116 |
| WDsinger | 1 | +0.0197 ± 0.0130 | +0.0061 ± 0.0070 |
| FB15k237(10) / FB15k237(20) / FB15k237(50) | 3 | +0.0061 ± 0.0053 | +0.0051 ± 0.0026 |
| Hetionet | 1 | +0.0140 ± 0.0133 | +0.0061 ± 0.0290 |
| WN18RRInductive | 4 | +0.0114 ± 0.0309 | +0.0013 ± 0.0070 |
| FB15k237Inductive | 4 | +0.0044 ± 0.0037 | +0.0034 ± 0.0056 |
| NELLInductive | 3 | +0.0050 ± 0.0101 | +0.0158 ± 0.0105 |
| ILPC2022 | 2 | +0.0074 ± 0.0044 | +0.0035 ± 0.0035 |
| HM | 3 | -0.0014 ± 0.0036 | +0.0009 ± 0.0038 |
| FBIngram | 4 | +0.0060 ± 0.0026 | +0.0024 ± 0.0014 |
| WKIngram | 4 | +0.0022 ± 0.0079 | +0.0028 ± 0.0035 |
| NLIngram | 3 | -0.0070 ± 0.0091 | +0.0092 ± 0.0131 |
| WikiTopics | 6 | +0.0053 ± 0.0090 | +0.0039 ± 0.0039 |
| Metafam | 1 | +0.0713 ± 0.1191 | +0.1950 ± 0.1511 |
| FBNELL | 1 | +0.0062 ± 0.0053 | +0.0164 ± 0.0145 |
Thank you for the clarifications and additional experiments. They substantially address my main concern, and I now view the contribution of this article more positively. My remaining concerns mainly concern the methodological novelty. I view the novelty primarily as a useful mechanism design tailored to KGFM pretraining, rather than a fundamentally new negative sampling or curriculum learning principle. The contribution is valuable, but relatively incremental. Overall, I am updating my score from 2 (Weak rejection) to 3 (Weak acceptance), while retaining reservations about the strength of the novelty and the broader mechanistic claims.
We thank you for your careful assessment and constructive suggestions.
Weaknesses
W1: Thank you for these suggestions. Table 4 compares KMAS with three alternatives: KMASrelation−type, which samples uniformly from the target relation's head or tail entity set; KMASco−occurrence, which samples according to static relation–entity co-occurrence counts in the training graph; and KMASfrozen, which retains the relation embeddings obtained after warmup for negative sample construction while the KGFM itself continues to train. The first two alternatives use no learned relation similarities, while the frozen variant examines the benefit of updating the relation embeddings. All variants use the same mixture schedule, negative sample budget, masking, training configuration, and validation procedure. KMAS outperforms all three alternatives in both MRR and Hits@10 on ULTRA and SEMMA, demonstrating the benefits of learned relation similarities and continuously updated relation embeddings.
Table 4. Negative sample construction with the same adjustment strategy, averaged over 44 datasets.
W2: We appreciate this concern. Table 2 in our response to Reviewer xpBs reports five-seed results for two representative KGFMs, ULTRA and SEMMA, and their KMAS-enhanced counterparts. Regarding the MRR difference between the original ULTRA and its KMASsub variant in the submitted paper, the original sampler generates all random negatives in one random sampling, whereas KMASsub uses two independent random sampling, replacing the relation-aware part with a second random group. Consequently, matching seeds do not guarantee identical sampled negatives or training trajectories. This sampling difference may lead to either higher or lower performance, as illustrated by SEMMA's Hits@10 decreasing from 0.529 to 0.527 with KMASsub in the submitted paper.