acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Identification and Semantic Meanings: Difference-based Knowledge Graph Embedding

Abstract

Knowledge graph embedding (KGE) encodes entities and relations as low-dimensional representations and relies on model architectures and scoring functions to capture link patterns, such as symmetry and inversion. However, existing methods directly use the embedding of each entity or relation for identification and as its semantic meaning. We challenge this from a difference-based perspective, which emphasize that semantic meaning is not intrinsic or self-identical, but instead arises from systematic differences among representations. Motivated by this, we propose a simple yet effective representation-level transformation that redefines the semantic representation of an entity or relation as its embedding differences with respect to all other entities or relations. This transformation can be seamlessly applied as a plug-in to a wide range of state-of-the-art KGE models, preserving overall computational efficiency while consistently improving link prediction performance across multiple benchmark datasets.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Knowledge Graph Embedding, Difference-based Meanings

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wuUAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that existing knowledge graph embedding (KGE) models treat each embedding's absolute coordinates as the "semantic meaning" of an entity or relation, and proposes DiffE as an alternative. In DiffE, each entity (relation) embedding is replaced by the mean, over all other entities (relations), of its embedding differences passed through a one-layer feed-forward layer and a tanh. The resulting representations are fed into the unchanged scoring function of an existing KGE model (TransE, DistMult, RotatE, HousE, GoldE). Experiments on WN18RR, FB15k-237 and YAGO3-10 report gains of roughly one MRR point over each base model.

Critical strengths

The MRR improvement from DiffE over the base models is consistent; it holds for every base model and dataset in Table 2.

Critical weaknesses
  1. The contribution is overstated. DiffE averages over all entities (or relations). The paper presents this as a change of paradigm in how semantic meaning is represented, but it is an aggregation layer on top of the same learned embeddings. Every entity's representation is a mean over the same full set of entities; this can be seen as a single step of message passing performed globally: each entity receives the messages from every other entity and aggregates them with a mean, as if on a complete graph, independently of the edges of the knowledge graph. Also note that every representation is still built from a learned per-entity embedding table, so DiffE cannot handle entities that were not seen during training, nor new graphs. "Rethinking identification" does not change what the model can or cannot represent.

  2. Message-passing link predictors are missing from the comparison. Link prediction is the only task evaluated, so the method should be compared with the strongest link predictors, not only with other KGE models, e.g., message passing methods like NBFNet [1] is neither cited nor compared against. Its published results on the same benchmarks (MRR 0.415 on FB15k-237, 0.551 on WN18RR) are higher than every row of Table 2, including GoldE+DiffE (0.380 and 0.537). This is critical: since DiffE is a form of message passing (Weakness 1), it is not surprising that it improves over the base models, but message passing performed along the structure of the graph is known to work better. NBFNet propagates messages along the edges of the knowledge graph, over multiple layers and conditioned on the query, and obtains much larger gains while also adding inductive capabilities.

[1] Zhu, Zhaocheng, et al. "Neural bellman-ford networks: A general graph neural network framework for link prediction." Advances in neural information processing systems 34 (2021): 29476-29490.

Questions for authors

See weaknesses

Reviewer_VSxBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a difference-based representation transformation for knowledge graph embedding. Instead of directly using the original entity and relation embeddings, it constructs pairwise differences between each embedding and all other embeddings of the same type, and aggregates these differences using a feed-forward network, Tanh activation, and mean pooling. The resulting representations replace the original embeddings in existing KGE scoring functions without changing their functional forms.

The paper studies the translation-invariance properties of the proposed representation and provides a theoretical discussion of its potential generalization benefits. Experiments on WN18RR, FB15k-237, and YAGO3-10 demonstrate improvements across several KGE baselines, including TransE, DistMult, RotatE, HousE, and GoldE. The authors also investigate entity- and relation-level contributions, embedding dimensions, and computational efficiency.

Critical strengths
  1. Broad empirical effectiveness across different KGE architectures. DiffE consistently improves link prediction performance across several representative KGE models and three benchmark datasets. The improvements extend to strong baselines such as HousE and GoldE, suggesting that the proposed transformation may be useful across different embedding architectures and geometries.

  2. Simple and flexible integration with existing models. DiffE operates at the representation level and preserves the original scoring functions. The entity-only and relation-only ablations provide useful evidence that both transformations contribute to performance improvements. The reported runtime comparisons also suggest that the additional computational overhead can be modest in the evaluated settings.

Critical weaknesses
  1. A logical issue in the proof of Theorem 4.1. Theorem 4.1 claims that a global translation of all entity embeddings leaves the DiffE representations unchanged. While this property is valid because global translations cancel out in pairwise differences, the provided proof appears to proceed in the reverse direction. Specifically, it assumes equality of the aggregated representations and directly infers equality of the underlying pairwise differences. However, equality after nonlinear transformation and mean aggregation does not necessarily imply equality of the original differences. Therefore, the current proof contains an unjustified implication, even though the stated theorem itself appears correct.

  2. Insufficient theoretical justification for improved generalization. Section 5 argues that translation invariance leads to improved generalization through covering numbers and Rademacher complexity. However, translation invariance alone does not guarantee better generalization. In particular, it remains unclear why the specific hypothesis class induced by DiffE necessarily admits tighter generalization bounds than the original KGE hypothesis class. Since DiffE also introduces additional trainable nonlinear transformations, their effects on hypothesis class complexity and the theoretical comparison need to be clarified.

Overall Assessment: I appreciate the simplicity and broad applicability of the proposed method, as well as its consistent empirical improvements across multiple KGE models and datasets. However, the theoretical concerns discussed above weaken my confidence in the paper's core claims, particularly regarding the relationship between translation invariance and improved generalization. I would be willing to reconsider my rating if the authors can convincingly address these concerns during the rebuttal.

Questions for authors

Q1. Could the authors clarify the proof of Theorem 4.1? In particular, why does equality of the nonlinear aggregated representations imply equality of the underlying pairwise differences? Since the stated translation-invariance property follows directly from the cancellation of global translations in pairwise differences, could the authors provide a proof that directly establishes the claimed implication?

Q2. Could the authors clarify under what specific assumptions the hypothesis class induced by DiffE achieves tighter generalization bounds than the original KGE hypothesis class? In particular, how are the additional trainable nonlinear transformations accounted for in the covering-number or Rademacher-complexity analysis?

Reviewer_cUx7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the limitations of conventional KGE methods by introducing a plug-and-play transformation that constructs new entity and relation representations via trainable nonlinear mappings and mean aggregation applied to pairwise embedding differences. The authors argue for improved generalization, and the experiments on three link-prediction benchmarks demonstrate consistent improvements across several KGE models with limited computational overhead.

Critical strengths

1- The paper proposes a plug-and-play framework that offers flexibility and versatility to be applied to a wide range of KGE scoring methods.

2- The evaluated KGE scoring methods span various families and mechanisms. The empirical results consistently improve performance across these methods, showing that the proposed approach is not limited to any particular method type.

Critical weaknesses

1. The methodological novelty is limited, as the proposed approach essentially applies two trainable one-layer transformations to pairwise embedding differences, followed by mean aggregation. The paper does not establish how this differs meaningfully from simpler alternatives, such as mean-centering or parameter-matched MLPs, and the ablation studies do not isolate the contribution of the difference-based mechanism. The limited discussion of related work further weakens the novelty claim.

2. The theoretical contributions are relatively straightforward, as the translation-invariance property follows directly from subtraction, while the Rademacher-complexity lemma largely restates an established bound. More importantly, the analysis does not provide a convincing comparative generalization guarantee for DiffE over its base models, particularly considering the additional trainable parameters.

3. The experimental evaluation is limited to three old benchmarks and a single transductive link-prediction task, without considering inductive settings. Moreover, the comparisons focus primarily on improvements over classical KGE baselines, without evaluating against more recent GNN- and path-based or transferable KG reasoning approaches such as NBFNet [1], A*Net [2], and ULTRA [3]. This makes it difficult to assess the practical significance of the reported improvements relative to contemporary methods.

4. The paper does not report the final selected hyperparameter configurations or sufficient implementation details for the additional transformation layers. More importantly, the code is not publicly available and is promised only upon acceptance, making it difficult to verify the results or reproduce the implementation. These omissions are particularly concerning.

5. The computational complexity analysis relies on the assumption that the number of negative samples scales with the number of entities, which does not generally hold for KGE training. The paper also does not explain how the all-pairs transformations are implemented efficiently or how their memory requirements are handled. Furthermore, the runtime evaluation covers only two model-dataset configurations and omits peak memory usage on the largest dataset.

[1] Zhu et al., Neural Bellman-Ford Networks, NeurIPS 2021 [2] Zhu et al., ANet: A Scalable Path-based Reasoning Approach for Knowledge Graphs*, NeurIPS 2023 [3] Galkin et al., Towards Foundation Models for Knowledge Graph Reasoning (ULTRA), ICLR 2024

Questions for authors

1. How does DiffE perform compared with simpler alternatives, such as mean-centered embeddings, affine transformations, and parameter-matched MLPs applied to original or centered embeddings? Can the authors provide ablations that isolate the contributions of the difference operation, nonlinear transformation, and aggregation strategy?

2. Can the authors provide a formal comparison of the generalization bounds or hypothesis-class complexity of DiffE and its base models, accounting for the additional trainable parameters?

3. Can the authors provide controlled comparisons with recent neural or path-based KG link-prediction methods to better establish the practical relevance of DiffE? Additionally, will the authors release the code, final hyperparameter configurations, and implementation details needed to verify the reported efficiency, including the strategy used to avoid explicitly encoding all pairwise differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.