acceptodds
Under review as a conference paper at ICLR 2027

ReRoute: Beyond Static Representations in Multimodal Knowledge Graph Completion

Abstract

Multimodal knowledge graph completion (MMKGC) integrates structural, visual, and textual information to predict missing facts. Static candidate representations use fixed modality combinations, although evidence requirements vary across relations and head/tail roles. Missing modalities further limit the available evidence. We propose ReRoute to adapt entity representations to these requirements. Its relation-role-aware multimodal routing (RRMR) adjusts visual and textual enhancement components on both the query and candidate sides. These adjustments depend on the relation and head/tail role. Structure-aware missing-modality compensation (SMMC) aggregates available modality-specific enhancement components from one-hop neighbors in the training graph. RRMR then modulates these supplementary signals. Decomposable scoring evaluates all entities without explicitly constructing conditioned candidate representations for each query. Across four evaluation metrics on DB15K, MKG-W, and MKG-Y, ReRoute ranks first in 10 of the 12 comparisons and second in the remaining 2. On MKG-W and MKG-Y, ReRoute achieves relative MRR improvements of 5.02% and 6.93%, respectively, over the strongest baseline on each dataset.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Multimodal Knowledge Graph Completion, Link Prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_szKBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReRoute for multi-modal knowledge graph completion (MMKGC), built on a token-based encoder with a TuckER decoder. It adds two things. The first is a small router that picks how much visual vs. textual signal to mix in, depending on the relation and whether the entity is the head or the tail. This is applied on both the query and candidate sides, and the scoring stays decomposable, so all entities can be scored efficiently. The second is a training-free trick: if an entity is missing a modality, it borrows that modality's signal from its one-hop neighbors. The numbers are strong on MKG-W and MKG-Y and roughly on par on DB15K.

Critical strengths
  • The motivation is clear and practical: the useful modality depends on the relation and role, and reweighting can not recover a modality that is absent.
  • The empirical gains on MKG-W and MKG-Y are sizable, and the ablations cover the main design choices.
  • The robustness experiment with extra masked text and the SMMC coverage statistics are useful.
Critical weaknesses
  • The design is overly complicated for what it delivers. The overall gains don't seem worth the added complexity, and on DB15K there is no improvement on the key metrics (Hit@1/MRR).
  • DB15K has 279 relations and relatively few training triples, so the router for each (relation, role) pair may be under-trained. The authors should analyze this, otherwise readers will suspect the method's applicability is narrow.
  • The experiments focus on transductive MMKGC settings, which can not present the generalization capability of this method.
  • Case study is missing in this paper.
  • The paper says the ablations "confirm the contribution of each module," yet SMMC is never tested on DB15K. The "robustness" claim rests on a single setting: masking text at test time on MKG-Y.
Questions for authors
  • How does SMMC compare with simple neighbor-mean imputation of the modality features?
  • Does the gain hold on larger or different benchmarks (e.g. with different missing rates), or when the visual modality is masked at test time?
Reviewer_1cQKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReRoute, which combines relation-role-aware multimodal routing (RRMR) with structure-aware missing-modality compensation (SMMC). RRMR dynamically adjusts modality contributions based on relations and head/tail roles, while SMMC leverages neighboring entities to compensate for missing modalities. Experiments demonstrate promising performance over existing methods.

Critical strengths

The paper addresses the limitations of static multimodal representations and missing modalities through a unified framework, achieving competitive results.

Critical weaknesses
  1. The methodology is difficult to follow, with several implementation details insufficiently explained. For example, in Eq. (1), how exactly is masked mean pooling performed? Similarly, although the authors mention that a Transformer produces the multimodal base representation zez_e, it remains unclear how the zez_e is initialized and calculated. A more detailed description would improve reproducibility.

  2. In Eq. (2), neighbors are derived from the adjacency graph constructed using training triples. Does that form the same neighbor sets even with different modalities? Moreover, the neighbor aggregation in Eq. (3) does not consider relation types, but only the entity info. The authors should explain how SMMC provides meaningful modality-specific compensation beyond conventional graph-based neighborhood aggregation. That is, since graph-based methods such as GNNs already aggregate neighborhood information, the distinction and technical novelty of SMMC over these approaches should be better justified. Additionally, Eq. (4)/(5) generate routing weights based only on the relation and head/tail role, meaning that all candidate entities under the same relation–role combination share identical modality weights. This raises questions about whether the routing mechanism can sufficiently capture entity-specific evidence requirements.

  3. Around Lines 231, the authors state that TuckER produces the query vector qξq_\xi, but the exact computation is not provided. While TuckER typically models entity and relation interactions through Tucker decomposition, how is it integrated in this paper to obtain the query vector? Furthermore, Eq. (8) involves both query vectors and entity representations. Are the query encoder and entity representations jointly optimized during training?

  4. Additional comparisons with conventional GNN aggregation, simple neighbor averaging, and relation-aware neighborhood aggregation are preferred. Furthermore, the missing-modality robustness experiments should be also considered for visual information in MKG-W.

Questions for authors

see Weakness.

13 comments
AuthorsOpenReview ↗

Response to Reviewer 1cQK: Method Clarifications and Additional Experiments

Thank you for your valuable feedback. Our responses are provided below, with revisions highlighted in blue in the PDF.

Response to Reviewer 1cQK:

W1. Pooling and entity encoding. (a) Each entity uses up to Lv=16L_v=16 BEiT visual and Lt=24L_t=24 BERT textual tokens, padded to fixed lengths. For token representations after projection, layer normalization, dropout and addition of modality position embeddings, masked mean pooling computes x‾e(m)=∑k=1Lmbe,k(m)xe,k(m)/max⁡(1,∑k=1Lmbe,k(m))\overline{\mathbf{x}}_e^{(m)}=\sum_{k=1}^{L_m}b_{e,k}^{(m)}\mathbf{x}_{e,k}^{(m)}/\max(1,\sum_{k=1}^{L_m}b_{e,k}^{(m)}), where be,k(m)=1b_{e,k}^{(m)}=1 for non-padding tokens and 0 otherwise. A missing modality has a zero summary; ae(m)=0a_e^{(m)}=0 gives δe(m)=0\boldsymbol{\delta}_e^{(m)}=\mathbf{0} in Eq.(1).

(b) The input is [entity token; structural embedding; visual tokens; textual tokens]. Frozen BEiT/BERT embeddings are projected to d=256d=256, and unavailable tokens are masked out. A one-layer, four-head Transformer outputs ze\mathbf{z}_e at the entity-token position. Structural embeddings, the entity token, position embeddings and projection weights use Xavier uniform initialization; Transformer sublayers and layer norms use PyTorch defaults (Appendix D.4).

W2. Modality-specific compensation and entity-specific evidence. (a) The shared graph contains undirected one-hop edges from training triples. Eq.(2) selects neighbors with the required modality: Nm(e)=j∈N(e):aj(m)=1\mathcal{N}_m(e)=\\{j\in\mathcal{N}(e):a_j^{(m)}=1\\}. On MKG-W, Nv(e)≠Nt(e)\mathcal{N}_v(e)\neq\mathcal{N}_t(e) for 72% of entities, with 2.43 vs. 4.01 available neighbors averaged over all entities. Overlapping sets still supply different components: δj(v)\boldsymbol{\delta}_j^{(v)} vs. δj(t)\boldsymbol{\delta}_j^{(t)}.

(b) SMMC supplies missing modality-specific components for RRMR to reweight. It aggregates available neighbors' pre-compensation components using base-representation similarity, only for missing modalities when ηm≥τmiss\eta_m\ge\tau_{\mathrm{miss}}. This one-hop operation preserves observed components and adds no trainable parameters (Eqs.(2)-(3)). SMMC weights neighbors by similarity rather than connecting edge types; RRMR then modulates the resulting evidence by query relation and role. This separates reusable neighborhood compensation from query-dependent routing. The comparisons in W4 show that SMMC outperforms mean, learnable GNN and relation-aware aggregation on both datasets.

(c) The gates set modality weights for each relation-role pair, while ze\mathbf{z}_e and δ~e(m)\widetilde{\boldsymbol{\delta}}_e^{(m)} carry each candidate's evidence (Sec.4.3). Shared gates scale the enhancement components, yielding candidate-dependent score contributions and enabling decomposable scoring (Eq.(8)). For (Sam Wood, directed, ?) on MKG-Y, Ambush (1950 film) ranks 2. On the same ReRoute checkpoint, removing all candidate-side enhancement terms, or only the text term, changes its rank to 88; removing only the visual term leaves it at 2. With the shared text gate gt=0.1993g_t=0.1993, the text contributions are +1.015 for the answer and −1.759-1.759 for the top competitor. The same gate thus increases the answer's score and decreases the competitor's.

W3. TuckER and joint optimization. (a) Let cξ,r^ξ\mathbf{c}_\xi,\widehat{\mathbf{r}}_\xi denote the triple context encoder outputs at the prediction and relation positions. For core tensor W∈Rd×d×d\mathcal{W}\in\mathbb{R}^{d\times d\times d} indexed by relation, input feature and output feature, inference computes:

(Mξ)ij=∑k=1d(r^ξ)kWkij,qξ=BN⁡1(Mξ⊤BN⁡0(cξ)).(\mathbf{M}_\xi)_{ij}=\sum_{k=1}^{d}(\widehat{\mathbf{r}}_\xi)_k\mathcal{W}_{kij},\qquad\mathbf{q}_\xi=\operatorname{BN}_1(\mathbf{M}_\xi^\top\operatorname{BN}_0(\mathbf{c}_\xi)).

BN uses running statistics at inference; training dropout details are in Appendix D.4.

(b) One optimizer jointly trains the entity encoder, triple context encoder, TuckER core tensor and routing parameters end-to-end. Gradients flow through both query and candidate representations in Eq.(8).

W4. Aggregation comparisons and visual robustness. (a) MRR(%):

AggregationMKG-WMKG-Y
Without SMMC41.4541.90
Available-neighbor mean41.9842.32
GNN41.4741.96
Relation-aware GNN41.5242.39
SMMC42.0842.56

All aggregators use the same enhancement components, neighbor sets and missing-trigger conditions. The GNN applies a learnable linear update with ReLU to the neighbor mean; the relation-aware GNN uses basis-decomposed relation-conditioned messages. SMMC achieves the highest MRR on both datasets.

(b) We extended RQ4 to visual masking on MKG-W (Fig.8, Appendix D.8). At a 90% total visual missing rate, ReRoute achieves MRR 40.77% vs. 40.05% (RRMR) and 38.76% (Backbone). Its margin over RRMR widens from 0.63 percentage points at the original approximately 40% missing rate to 0.72 percentage points at 90%.

AuthorsOpenReview ↗

Follow-up to Reviewer 1cQK — W4: Full aggregation comparisons and MKG-W visual-masking results

Thank you for your feedback. Due to the character limit, our initial response to W4 included only the MRR results for aggregation comparisons and selected results for visual masking on MKG-W. We provide the complete results below.

(a) Comparison with learnable GNN aggregation.

We compare SMMC with no compensation, uniform averaging over available neighbors, a learnable GNN, and a relation-aware GNN. All aggregation variants use the same modality-specific enhancement components, available-neighbor sets, missing-trigger conditions, and training settings. The GNN applies a learnable linear transformation followed by ReLU to the neighbor mean; the relation-aware GNN uses basis-decomposed relation-conditioned messages. SMMC instead weights available neighbors by base-representation similarity and introduces no additional trainable parameters.

All metrics below are reported as percentages; higher is better.

MethodMKG-WMKG-Y
MRRHits@1Hits@3Hits@10MRRHits@1Hits@3Hits@10
w/o SMMC41.4534.3344.2554.8641.9037.1643.6650.75
Available neighbor mean41.9835.0444.5254.8942.3237.6243.7551.12
GNN41.4734.7043.8854.6941.9637.3143.5950.16
Relation-aware GNN41.5234.4444.4254.9642.3937.7043.7651.14
SMMC42.0835.1344.8555.2642.5637.8044.2351.29

SMMC achieves the highest values on all four metrics on both datasets. Its MRR improvements over the GNN are 0.61 and 0.60 percentage points on MKG-W and MKG-Y, respectively; the corresponding improvements over the relation-aware GNN are 0.56 and 0.17 points. Relative to uniform neighbor averaging, SMMC improves MRR by 0.10 and 0.24 points. These results support similarity-weighted compensation for missing modality-specific components in our setting, even when compared with learnable and relation-aware aggregation alternatives.

(b) Visual-masking robustness on MKG-W.

We extend the robustness evaluation to visual masking on MKG-W, starting from its original visual missing rate of approximately 40% and increasing the total missing rate to 90%. The rates below refer to total visual missingness, including originally missing visual inputs. RRMR denotes ReRoute without SMMC. The complete MRR and Hits@10 results are:

ModelMissing rateMRRHits@10
0.440.1053.49
0.539.7853.15
0.639.5052.86
Backbone0.739.2452.65
0.839.0252.30
0.938.7652.14
0.441.4554.86
0.541.0154.37
0.640.7354.05
RRMR0.740.4353.71
0.840.2553.48
0.940.0553.23
0.442.0855.26
0.541.8654.86
0.641.5154.50
ReRoute0.741.2854.24
0.840.9953.93
0.940.7753.59

ReRoute achieves the best MRR and Hits@10 at all six missing rates. Its MRR advantage over RRMR is 0.63, 0.85, 0.78, 0.85, 0.74, and 0.72 percentage points, respectively. Thus, SMMC provides a consistent benefit across the evaluated missing rates, including severe visual missingness. At 90% missingness, ReRoute exceeds RRMR and Backbone by 0.72 and 2.01 MRR points, respectively, and by 0.36 and 1.45 Hits@10 points. Together with the textual-masking experiment on MKG-Y, this extends the robustness evaluation across two datasets and two modalities.

Reviewer_1cQKOpenReview ↗

Response to authors

Thank you for your responses and additional experiments (W4). Yet, some of my concerns remain insufficiently addressed.

W1. The motivation behind these design choices remains unclear. In particular, why is masked mean pooling adopted instead of alternative pooling or aggregation strategies? More importantly, why can the difference between a mean-pooled modality representation and the Transformer-generated multimodal base representation can capture modality-specific enhancement information? What is the difference between such enhancement and an existing data augmentation?

W2. Thank you for clarifying the link between SMMC and RRMR. However, the relations connecting neighboring entities in SMMC are different from the query relation considered in RRMR. It remains unclear why similarity-based weights in eq. 2 (that ignores neighborhood relation types) is more suitable for missing-modality compensation. I might overlook some details but please point out if any. Similarly, the provided case study demonstrates that shared routing weights can yield different score contributions. However, it does not establish whether shared weights are sufficiently expressive. Maybe an ablation study can be considered to include entity representation into eq. (4), for instance, and then check both predictive performance and computational efficiency.

W3. While the authors have clarified the entire framework is trained end-to-end, the motivation for incorporating TuckER and jointly optimizing these components remains unclear. Importantly, the complexity analysis in Appendix C.3 doesnot account for TuckER-based query generation. What exactly TuckER core tensor is? Given that the computational overhead of TuckER is non-negligible, a more comprehensive analysis is needed.

W4. From the MKG-Y, why GNN/Relation-aware GNN performance is so high? Also SMMC - (Relation-aware GNN) is only 0.17 in MRR. Then link back to W2, the improvement over graph-based neighborhood aggregation is marginal. Please again check how SMMC provides meaningful modality-specific compensation beyond conventional graph-based neighborhood aggregation. At last, given this method is based on missing modality, missing-modality compensation methods should be included in the experiment (see Sec 5.5).

Again, I might overlook some details so please point out if any. The key point is, while the authors have provided some more results, my main concern is the lack of sufficient justification, either the detail is missing or inconsistent. For instance, several designs appear somewhat ad hoc, such as setting the missing-modality threshold to 0.1 (line 189) and adopting cosine similarity (eq. 2); that is link to W2. Although these choices may work empirically, the paper does not sufficiently explain why they are appropriate for such a problem.

Reviewer_NeoaOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript presents two key challenges in the field of multimodal knowledge graph completion: inconsistent demands for modal evidence from different relationships and entity roles, and difficulty in obtaining useful evidence information when modal information is missing. To address these challenges, the author proposes ReRoute, which mainly consists of two components: Relationship Role Aware Multimodal Routing (RRMR) and Structure Aware Missing Modal Compensation (SMMC). Specifically, ReRoute first constructs a multimodal base representation and modal specific enhancement components. Subsequently, SMMC compensates for missing modal information by aggregating enhancement components of neighboring entities, while RRMR dynamically adjusts the contributions of visual and textual enhancement components based on relationships and entity head/tail roles, thereby achieving effective and efficient knowledge graph completion. The experimental results showed that ReRoute achieved a relative MRR improvement of 5.02% and 6.93% compared to the strongest baseline on the MKG-W and MKG-Y datasets, respectively.

Critical strengths
  1. The manuscript is written clearly, well-organized, and easy to understand.
  2. The manuscript focuses on two important issues in the field of multimodal knowledge graph completion, including the varying needs of different relationships and entities for evidence. How to effectively complete missing evidence, especially the latter, is an interesting question. The manuscript proposes effective solutions to these two issues, and the experimental results strongly prove this point. In addition, the experiments in the manuscript are relatively complete, including the ablation study of bilateral routing, robustness evaluation under missing text information, and numerical equivalence verification of decomposable scoring mechanism.
Critical weaknesses

How scalable is the method proposed in the manuscript for large-scale multimodal knowledge graphs? According to Algorithm 1, the model will recalculate the representations of all entities and perform SMMC in each training batch. Although decomposable scoring reduces the computation and memory overhead of candidate scores for query conditions, it does not eliminate the need for repetitive full entity encoding during training, which may be inefficient on large-scale knowledge graphs?

Questions for authors
  1. According to Algorithm 1, the model will recalculate the representations of all entities and perform SMMC in each training batch. Will this lead to a significant increase in training time compared to other baseline methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.