- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
This paper presents a knowledge graph foundation model (KGFM) called Muze for link and relation prediction. Generally, KGFMs aim to overcome the limitations of knowledge graph embeddings (KGE) by being able to support new entities and relations at inference time, including zero-shot prediction on a completely new KG.
I consider the positioning of this paper misleading, and recommend to reject the paper for this reason. At its heart, the approach trains a KGE model on the inference graph and uses it for prediction as in KGE models. The paper does not write it like this, however, but rephrases the gradient-based training process of KGE models with GNN-terminology (e.g., random initialization becomes random node embeddings, gradient computation becomes message computation/aggregation, optimizer step becomes embedding update) to make it look more like a KGFM. The relationship to KGE models and their training process is not clearly exposed.
Critical strengths
S1. The approach may add something valuable to KGE training (such as requiring less epochs or preventing overfitting).
S2. The approach suggests that KGE-style models are competitive for inductive LP, if one is willing to retrain on the inductive split.
Critical weaknesses
W1. Does not clearly state that it essentially fits a KGE model (or an ensemble of KGE models, if multiple runs are used). KGE model training with gradient descent works as follows:
- Initialize embeddings of all entities and relations randomly.
- Compute the gradient of each entity and relation embedding using negative sampling.
- Update each entity and relation embedding using an optimizer.
- Repeat for a certain number of epochs.
- Predict via ranking with scoring function.
The proposed method works as follows:
- Directly.
- Directly, as a_{x,t} matches exactly the KGE gradient.
- Using a custom "optimizer", which has momentum/adaptive LR terms (4), an LR schedule (eta, gamma, alpha), and normalization. The main difference to KGE training is that the optimizer uses a small trained model to determine the update.
- Directly, each layer corresponds to an epoch.
- Directly.
KGE models are typically trained using mini-batches of triplets; the proposed paper does the same (App B.2). Moreover, "MUZE encodes each KG once and reuses the embeddings across queries" also holds for KGE models. And like KGE models, MUZE needs to rerun everything as soon as anything in the KG changes (such as new entities, relations, triplets, ...).
This relationship is not clearly exposed in the paper, which is why I consider the paper misleading.
Questions for authors
None.
6 comments
We noticed that the review was updated. Below, we respond to the newly added comments.
The authors argued in the discussion that the use of a small, trained MLP in step 3 is the main novelty of this paper. This is at odds with the entire positioning of the paper;
We did not argue that a trained MLP is the main novelty of our paper. The discussion was not about novelty; we were simply explaining why the reviewer's claim that MUZE does "exactly the same thing" as KGE training does not hold. We chose the gated residual refinement in Eq. (6), which consists of multiple MLPs, as the most direct counterexample where the difference is immediately visible. Providing a simple counterexample to directly refute an equivalence claim does not mean we are claiming it as our main novelty.
The main novelty of MUZE lies in an architectural shift for zero-shot KG reasoning: from per-query encoding to query-agnostic encoding. Existing KGFMs support inference with unseen entities and relations but require per-query encoding, whereas classical KGE models utilize query-agnostic embeddings for scalable inference but require separate training on each KG. MUZE, the first query-agnostic KGFM, combines both strengths by using the gradients of a shared, learned KGE scoring function as messages in the encoder and the same function as the decoder.
This architectural shift provides concrete empirical benefits. Inference becomes substantially more scalable than in existing KGFMs (Lines 61-66; Tables 2-3), and a single pre-trained model supports both entity and relation prediction and transfers to entity alignment without additional training, whereas existing KGFMs are trained separately for each task (Lines 76-78 and 253-256). MUZE thus paves the way for scalable, general-purpose KGFMs.
it constitutes a form of learning to optimize for KGE models.
MUZE is not trained to optimize a given KGE objective but is formulated for end-to-end link prediction. Its scoring and update functions are learned jointly on triplets held out from message passing (Lines 238-242 and Appendix B.2), using a prediction loss distinct from the KGE loss that defines the encoder's messages (Eq. (1)).
We respectfully ask that the paper be assessed on its formulation and contributions, not on a claim we did not make.
I agree with the authors that their approach is not /exactly/ the same as learning KGE models, and I toned down the respective comments in the review. The approach also may add something valuable to KGE training (such as requiring less epochs or preventing overfitting).
I disagree with the positioning of the paper, however, as the relationship to KGE training is not clearly exposed. The authors were unwilling to clearly acknowledge this relationship in the discussion, too. If one accepts the proposed method as a KGFM for inductive link prediction, one would have to accept other transductive link predictions methods as well.
PS: App B.2, which the authors mention in the response, is also puzzling: According to B.2, the training process actually uses mini-batch GD instead of GD. The latter corresponds to the GNN interpretation given in the main paper, whereas the former corresponds to KGE training.
Response to Reviewer upr1
We respectfully disagree with the review’s claim that we simply rephrase the KGE training process to make it look like a KGFM. The review’s central premise, that MUZE trains a standard KGE model on an inference graph, is factually incorrect.
MPNN Inference Is Not KGE Training
MUZE is an MPNN that performs zero-shot inference on unseen KGs after pre-training. During pre-training, MUZE learns the parameterized scoring function (Lines 238-242) and MLP-based update functions (Eqs. (5)-(6)). At inference, it simply performs a forward pass on the input KG with the learned parameters (Lines 149-152). Computing representations via message passing with fixed, learned parameters is standard MPNN inference, not standard KGE training.
In Table 21 of Appendix D in the revised manuscript, we trained a KGE model with MUZE's scoring function on each of the 13 transductive KGs for K=12 epochs, matching the number of layers of MUZE. MUZE substantially outperforms the resulting KGE model: 0.331 (MUZE) vs. 0.022 (KGE) MRR for entity prediction and 0.886 (MUZE) vs. 0.236 (KGE) MRR for relation prediction. Note that MUZE performs zero-shot inference on these datasets. These results show that MUZE's inference is different from training a KGE model on the inference graph.
MUZE is a GNN-based KGFM
MUZE is a GNN-based KGFM, and thus, the GNN terminology describes its actual computations. MUZE is an MPNN that couples message passing with decoding via a KGE scoring function (Lines 16-22, 54-59, 70-71, 112-114, 142-145, and 184-199). Its message functions are analytically derived from the gradients of a KGE loss defined using a parameterized scoring function, while its MLP-based update functions compute the representations from the aggregated messages (Eqs. (4)-(6)). Together, these components form a neural architecture that serves as a KGFM, not a terminological rephrasing of the standard KGE training process. By using the same scoring function to derive the messages in encoding and to score the resulting representations in decoding, MUZE produces representations aligned with its decoder (Lines 197-199 and 228-229).
MUZE Makes Significant Progress Towards KGFMs
MUZE is the first query-agnostic KGFM. While existing KGFMs re-encode the entire KG for every query (Lines 42-49 and 94-102), MUZE encodes each KG once and reuses the embeddings across queries (Lines 50-52 and 59-61). This architectural shift accelerates inference by 11x over the fastest baseline and 1389x over the most accurate baseline (Table 2), while achieving competitive performance in entity prediction and establishing new state-of-the-art performance in relation prediction (Table 1). On Wikidata-5M, with approximately 5 million entities and 20 million triplets, MUZE achieves an MRR of 0.291 in 9 minutes, whereas ULTRA achieves 0.174 in 120 minutes and TRIX and FLOCK do not finish within 24 hours (Table 3). Also, while existing KGFMs cannot be directly applied to tasks other than KG completion, MUZE can perform zero-shot entity alignment (Table 7 in the submitted version; Table 8 in the revised version), broadening the scope of tasks a KGFM can handle.
We hope this clarification resolves the misunderstanding about our paper.