- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes, MERIT, a knowledge graph foundation models for inductive link prediction, where the main contributions is to replace the relational NBFNets in ULTRA by a variant of transformer. The entity encoder and the overall link prediction framework, including the extension to complex query answering, are borrowed from ULTRA and ULTRAQuery. The authors motivate MERIT through the computational costs of recent KGFMs and the improvement of the overall acrruacy.
Critical strengths
The empirical results are promising, and the evaluations are standard and extensive. The author also motivates over the complex query answering, unlike most of the KGFM literature. Additionally, I also appreciate the effort to report computational costs, including runtime, FLOPs, and memory consumption. Such comparisons are useful for understanding the practical trade-offs among existing KG foundation models.
Critical weaknesses
As described in the paper summary, MERIT largely retains the architecture of ULTRA, including the four relational motif types, query-conditioned relation representations, the entity-level NBFNet encoder, and the overall link prediction procedure. The only difference between ULTRA and MERIT is only the relation encoders. This, by itself, is not an issue, as existing KGFM such as MOTIF are also the same. However, the motivations of MERIT are not convincing. The authors motivate MERIT by arguing that motif-constrained message passing restricts interactions between relation types and that dense attention can overcome these limitations. First, the relation-level MPNN in ULTRA already allows information to propagate between relations through multiple message-passing layers. The absence of a direct motif connection does not necessarily prevent two relations from exchanging information. Second, MERIT continues to rely on exactly the same four handcrafted relational motifs as ULTRA. Although dense attention relaxes the connectivity restrictions, the structural information available to the relation encoder is still derived from these motifs. As a result, the claimed expressivity gain is precisely MPNN v.s. Graph Transformer, which is not limited to KGFM but a general graph learning property that is well-studied and does not only limited to MPNN here. I would appreciate a more precise explanation of what structural or relational patterns MERIT can capture that ULTRA cannot, ideally supported by theoretical analysis or controlled experiments demonstrating the limitations of motif-constrained message passing, i.e., by presenting counter-examples over expressivity limitations similar to MOTIF or TRIX, or even by creating synthetics datasets similar to FLOCK.
Another major concern is the computational efficiency motivation. Replacing sparse relation-level message passing with dense self-attention does not inherently improve computational scalability (it actually makes it worse theoretically!). The complexity of relation-level message passing depends on the number of edges in the relation graph, while dense self-attention introduces quadratic complexity in the number of relation types. The empirical efficiency of MERIT appears to depend substantially on the relatively small relation vocabularies of the evaluated benchmarks and the availability of highly optimized attention implementations, such as FlexAttention. These implementation advantages do not establish an algorithmic improvement in computational complexity over ULTRA. In fact, Figure B.2 reports that MERIT requires approximately 5.5 PFLOPs compared with 5.2 PFLOPs for ULTRA, despite their similar inference times. As a result, the motivation for efficiency is hardly convincing to me.
Questions for authors
Beyond replacing ULTRA's relation-level MPNN with a structurally biased Transformer, what is the main new methodological contribution of MERIT? How does it differ conceptually from existing graph Transformer architectures?
Since MERIT still uses the same four relational motifs as ULTRA, what additional structural information can it capture? Can the authors provide concrete examples or theoretical analysis showing limitations of ULTRA that MERIT overcomes?
Can the authors provide controlled scaling experiments with larger relation vocabularies and varying relation-graph sparsity? How much of the observed efficiency comes from optimized attention kernels rather than reduced computational work?