- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReRoute for multi-modal knowledge graph completion (MMKGC), built on a token-based encoder with a TuckER decoder. It adds two things. The first is a small router that picks how much visual vs. textual signal to mix in, depending on the relation and whether the entity is the head or the tail. This is applied on both the query and candidate sides, and the scoring stays decomposable, so all entities can be scored efficiently. The second is a training-free trick: if an entity is missing a modality, it borrows that modality's signal from its one-hop neighbors. The numbers are strong on MKG-W and MKG-Y and roughly on par on DB15K.
Critical strengths
- The motivation is clear and practical: the useful modality depends on the relation and role, and reweighting can not recover a modality that is absent.
- The empirical gains on MKG-W and MKG-Y are sizable, and the ablations cover the main design choices.
- The robustness experiment with extra masked text and the SMMC coverage statistics are useful.
Critical weaknesses
- The design is overly complicated for what it delivers. The overall gains don't seem worth the added complexity, and on DB15K there is no improvement on the key metrics (Hit@1/MRR).
- DB15K has 279 relations and relatively few training triples, so the router for each (relation, role) pair may be under-trained. The authors should analyze this, otherwise readers will suspect the method's applicability is narrow.
- The experiments focus on transductive MMKGC settings, which can not present the generalization capability of this method.
- Case study is missing in this paper.
- The paper says the ablations "confirm the contribution of each module," yet SMMC is never tested on DB15K. The "robustness" claim rests on a single setting: masking text at test time on MKG-Y.
Questions for authors
- How does SMMC compare with simple neighbor-mean imputation of the modality features?
- Does the gain hold on larger or different benchmarks (e.g. with different missing rates), or when the visual modality is masked at test time?
Response to Reviewer 1cQK: Method Clarifications and Additional Experiments
Thank you for your valuable feedback. Our responses are provided below, with revisions highlighted in blue in the PDF.
Response to Reviewer 1cQK:
W1. Pooling and entity encoding. (a) Each entity uses up to Lv=16 BEiT visual and Lt=24 BERT textual tokens, padded to fixed lengths. For token representations after projection, layer normalization, dropout and addition of modality position embeddings, masked mean pooling computes xe(m)=∑k=1Lmbe,k(m)xe,k(m)/max(1,∑k=1Lmbe,k(m)), where be,k(m)=1 for non-padding tokens and 0 otherwise. A missing modality has a zero summary; ae(m)=0 gives δe(m)=0 in Eq.(1).
(b) The input is [entity token; structural embedding; visual tokens; textual tokens]. Frozen BEiT/BERT embeddings are projected to d=256, and unavailable tokens are masked out. A one-layer, four-head Transformer outputs ze at the entity-token position. Structural embeddings, the entity token, position embeddings and projection weights use Xavier uniform initialization; Transformer sublayers and layer norms use PyTorch defaults (Appendix D.4).
W2. Modality-specific compensation and entity-specific evidence. (a) The shared graph contains undirected one-hop edges from training triples. Eq.(2) selects neighbors with the required modality: Nm(e)=j∈N(e):aj(m)=1. On MKG-W, Nv(e)=Nt(e) for 72% of entities, with 2.43 vs. 4.01 available neighbors averaged over all entities. Overlapping sets still supply different components: δj(v) vs. δj(t).
(b) SMMC supplies missing modality-specific components for RRMR to reweight. It aggregates available neighbors' pre-compensation components using base-representation similarity, only for missing modalities when ηm≥τmiss. This one-hop operation preserves observed components and adds no trainable parameters (Eqs.(2)-(3)). SMMC weights neighbors by similarity rather than connecting edge types; RRMR then modulates the resulting evidence by query relation and role. This separates reusable neighborhood compensation from query-dependent routing. The comparisons in W4 show that SMMC outperforms mean, learnable GNN and relation-aware aggregation on both datasets.
(c) The gates set modality weights for each relation-role pair, while ze and δe(m) carry each candidate's evidence (Sec.4.3). Shared gates scale the enhancement components, yielding candidate-dependent score contributions and enabling decomposable scoring (Eq.(8)). For (Sam Wood, directed, ?) on MKG-Y, Ambush (1950 film) ranks 2. On the same ReRoute checkpoint, removing all candidate-side enhancement terms, or only the text term, changes its rank to 88; removing only the visual term leaves it at 2. With the shared text gate gt=0.1993, the text contributions are +1.015 for the answer and −1.759 for the top competitor. The same gate thus increases the answer's score and decreases the competitor's.
W3. TuckER and joint optimization. (a) Let cξ,rξ denote the triple context encoder outputs at the prediction and relation positions. For core tensor W∈Rd×d×d indexed by relation, input feature and output feature, inference computes:
(Mξ)ij=∑k=1d(rξ)kWkij,qξ=BN1(Mξ⊤BN0(cξ)).
BN uses running statistics at inference; training dropout details are in Appendix D.4.
(b) One optimizer jointly trains the entity encoder, triple context encoder, TuckER core tensor and routing parameters end-to-end. Gradients flow through both query and candidate representations in Eq.(8).
W4. Aggregation comparisons and visual robustness. (a) MRR(%):
All aggregators use the same enhancement components, neighbor sets and missing-trigger conditions. The GNN applies a learnable linear update with ReLU to the neighbor mean; the relation-aware GNN uses basis-decomposed relation-conditioned messages. SMMC achieves the highest MRR on both datasets.
(b) We extended RQ4 to visual masking on MKG-W (Fig.8, Appendix D.8). At a 90% total visual missing rate, ReRoute achieves MRR 40.77% vs. 40.05% (RRMR) and 38.76% (Backbone). Its margin over RRMR widens from 0.63 percentage points at the original approximately 40% missing rate to 0.72 percentage points at 90%.