- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RHDN addresses multimodal knowledge graph completion through two modules. R-HAlign combines relation- and role-conditioned optimal transport (OT) over visual and textual sub-entities with pairwise interactions and hierarchical role prototypes. FDNS reweights negatives using model-based difficulty and multimodal role-consistency evidence. A ConvE-style decoder scores triples. Experiments on DB15K, MKG-W, and MKG-Y report the highest MRR among the listed baselines, with component ablations and pseudo-false-negative diagnostics.
Critical strengths
S1. The non-separable pairwise aggregation in Eq. (11) makes the representation sensitive to the transport coupling. The Independent Coupling ablation provides a useful control by retaining the same marginals. S2. RHDN improves MRR over the strongest listed baseline by 0.59, 1.17, and 0.88 percentage points on the three benchmarks. Removing R-HAlign or FDNS consistently reduces performance. S3. The appendix documents feature extraction, missing-modality handling, relation clustering, and evaluation. The frozen-pool and score-matched diagnostics directly examine whether risk reweighting contributes beyond model difficulty.
Critical weaknesses
W1. FDNS does not condition its risk estimate on the other entity in the query (Eqs. 17-19). It therefore measures candidate role compatibility without distinguishing which specific triples are true. The score-matched diagnostic does not control for semantic type, leaving the claimed false-negative discrimination insufficiently established. W2. The diagnostic statistics need consistent normalization. In Fig. 5, the actor column sums to approximately 0.17, while its plotted marginal is 0.20. In Fig. 6, the two largest normalized weights sum to more than one. Table 5 does not specify how average weights are computed, and Appendix D.2 omits candidate-pool selection and score-matching counts. W3. Tables 1, 2, and 8 omit variability across seeds, despite reporting three-run averages. This matters for gains such as the 0.59-point DB15K MRR improvement. Baseline result sources, feature inputs, and tuning budgets are unspecified, and no matched RGAT/ConvE structural baseline is provided. W4. The contribution needs sharper positioning against MyGO and NativE, and against the omitted OTKGE (NeurIPS 2022) and ComCon (Information Fusion 2026). Existing ablations do not isolate the benefit of cluster-level sharing or compare OT with learned cross-attention. These comparisons are needed to identify what the proposed alignment and weighting add to prior MMKGC methods. W5. The claim of controlled computational overhead is unsupported by runtime or memory measurements. Appendix D.4 identifies the substantial pairwise-encoding cost but reports only asymptotic complexity. Full-entity ranking, representation caching, and head prediction under Eq. (15) also lack implementation details. W6. Appendix D.3.3 defines Relation Selectivity, Role Selectivity, and Prototype Purity@5 without reporting their values or label sources. The mapping from the implemented 4 by 4 patch grid and BERT tokens to the semantic regions and phrases in Fig. 5 and Tables 9-10 is also unspecified.
Questions for authors
Q1. (W1) Does FDNS distinguish pseudo-FNs from negatives matched by both score and semantic type? How does its risk estimate distinguish queries with different opposite-side entities? Q2. (W2) How are the weights in Table 5 and Fig. 6 normalized? What explains the marginal discrepancy in Fig. 5? Please report pool-selection rules and matched-pair counts. Q3. (W3) Which baselines were reproduced, using which features and tuning budgets? Please report per-seed results and a matched RGAT/ConvE structural baseline. Q4. (W4) What distinguishes RHDN from OTKGE and ComCon? How does it perform with cross-attention replacing OT and with relation-level prototypes alone? Q5. (W5) What are the training time, peak memory, and full-ranking time relative to strong baselines? How are representations cached and head queries evaluated? Q6. (W6) What are the selectivity and purity results, and where do their labels come from? How are tokens and patches mapped to the displayed phrases and regions?
4 comments
Response to Reviewer DbdL
Response to Reviewer DbdL (continued: Part 2)
Q3/W3. Baseline controls and seed variability
We added a reproducibility-oriented comparison using validation-MRR model selection, a stated budget of 12 configurations per method and dataset, and seeds 42, 3407, and 2027 for the controlled reruns. The matched structure-only model uses RHDN's two-layer RGAT (four heads, ) and the same ConvE-style decoder, taking only RGAT structural embeddings and using the same self-adversarial weighting as RHDN w/o FDNS. Data splits, negative-candidate budgets, validation selection, and filtered head/tail evaluation are held consistent.
| Dataset | RGAT/ConvE | RHDN w/o FDNS | RHDN | Reproduced comparator |
|---|---|---|---|---|
| DB15K | 34.53 ± 0.35 | 38.56 ± 0.24 | 40.24 ± 0.26 | C²RS: 39.68 ± 0.34 |
| MKG-W | 34.89 ± 0.27 | 38.45 ± 0.28 | 40.92 ± 0.27 | C²RS: 39.73 ± 0.21 |
| MKG-Y | 36.67 ± 0.26 | 38.80 ± 0.28 | 40.38 ± 0.23 | SNAG: 39.49 ± 0.16 |
MRR in %, reported as mean ± sample SD over three seeds.
Relative to the matched structural model, the multimodal pipeline without FDNS improves MRR by 4.03/3.56/2.13 points; FDNS adds 1.68/2.47/1.58 points. Against the controlled competing reruns, full RHDN improves mean MRR by 0.56/1.19/0.89 points. On DB15K, the individual RHDN–C²RS differences are +1.10/−0.08/+0.65 points. These figures make the run-to-run variability visible alongside the average improvement. We also report seed-level MRR for four FDNS variants (e.g., 39.76 ± 0.18 without evidence disagreement versus 40.24 ± 0.26 for full RHDN).
Manuscript additions: Appendix C.2, Baseline Provenance and Seed Variability, with three-seed comparisons in Tables 6–7; the matched structure-only model is also included in Appendix D.4.1 (Table 25).
Q4/W4. Distinction from prior methods and targeted OT/prototype controls
OTKGE aligns modality-level distributions through OT; RHDN instead uses relation–role-conditioned region–token transport whose entries weight non-separable pairwise interactions (Eq. 11). This makes the learned representation sensitive to the correspondence pattern itself. Relative to MyGO's fine-grained entity representation and NativE's relation-adaptive modality fusion, R-HAlign explicitly combines sub-entity correspondence with head/tail prototypes. Compared with ComCon's complementary–contradictory regularization and weighted negative sampling, FDNS separately estimates score-based difficulty and multimodal role-consistency risk before KL-regularized weighting. The Related Work now discusses OTKGE and ComCon.
The targeted controls give the following MRR (%) comparisons:
| Dataset | Row attention | Sinkhorn (both w/o OT loss) | Relation-only prototypes | Full RHDN |
|---|---|---|---|---|
| DB15K | 39.64 | 39.83 | 40.03 | 40.24 |
| MKG-W | 40.13 | 40.29 | 40.53 | 40.92 |
| MKG-Y | 39.69 | 39.80 | 40.19 | 40.38 |
The parameter-matched cross-attention and Sinkhorn variants share the pairwise costs, marginals, aggregation, and parameter count, and both set . Enforcing the second marginal increases MRR by 0.19/0.16/0.11 points. The relation-only control retains relation-specific head/tail prototypes, OT, and FDNS, while removing the cluster-prototype input and relation–cluster consistency loss. Restoring cluster-level sharing adds 0.21/0.39/0.19 points over relation-only prototypes. The original Independent Coupling control also shows 0.67/0.81/0.76-point MRR losses when internal transport correspondence is removed.
Manuscript additions: Sections 2.2–2.3 for prior-work positioning; Section 4.3 (Tables 2–3) and Appendix D.1.1 (Table 8) for the OT, attention, and prototype controls.
Response to Reviewer DbdL
Response to Reviewer DbdL
Q5/W5. Training cost, caching, and full-entity ranking
We measured training and full-test evaluation on a single NVIDIA RTX A6000 (48 GB) against C²RS on DB15K/MKG-W, SNAG on MKG-Y, and the matched RGAT/ConvE structure-only model. The table reports RHDN values alongside the relevant strong baseline:
| Dataset | s/epoch (RHDN / baseline) | Peak train GB (RHDN / baseline) | Full-test s (RHDN / baseline) |
|---|---|---|---|
| DB15K | 322.8 / 228.4 | 34.56 / 19.82 | 151.9 / 32.9 |
| MKG-W | 151.2 / 108.3 | 27.96 / 15.36 | 95.3 / 20.2 |
| MKG-Y | 96.8 / 73.7 | 26.85 / 13.74 | 63.1 / 17.4 |
Full-test time includes cache construction and filtered head/tail ranking. For RHDN, cache construction accounts for 121.8/76.3/49.2 seconds, while subsequent head-plus-tail ranking takes 30.1/19.0/13.9 seconds. Total training times for RHDN are 45.7/21.5/13.0 hours on the three datasets, respectively. This quantifies both the end-to-end cost and the specific cache-building bottleneck.
Caching and scoring. Frozen ViT/BERT region/token features are precomputed for training. At evaluation, representations are cached one relation at a time: stores ConvE query vectors for candidate heads, and stores tail-role embeddings. For , head scores are ; for , tail scores are . Chunking changes memory use, not Eq. (15); the evaluator filters other known true completions while retaining the target. Batched and direct candidate-triple scoring yield identical filtered ranks under the same tie rule.
Manuscript additions: Appendix D.4.1, Runtime, Memory, and Full-Entity Ranking, including the profiling protocol, caching/scoring equations, and Table 25. The isolated FDNS training increment is reported separately in Appendix D.2 (Table 16).
Q6/W6. Selectivity, type labels, and token/region traceability
We report the numerical diagnostics on the DB15K validation set using 1,500 relation pairs, 1,800 role pairs, and 416 eligible prototypes. Relation Selectivity holds entity/input/role fixed while varying relation; Role Selectivity holds entity/input/relation fixed while varying the head/tail role. Both measure mean base-2 Jensen–Shannon divergence of visual and textual attention marginals and require no type labels.
| Variant | Relation Selectivity | Role Selectivity | Purity@5 |
|---|---|---|---|
| w/o Role Prototypes | 0.073 | 0.046 | — |
| Row attention (w/o OT loss) | 0.094 | 0.069 | 79.76% |
| Sinkhorn (w/o OT loss) | 0.098 | 0.073 | 80.48% |
| RHDN Full | 0.103 | 0.078 | 81.35% |
Purity@5 uses entity-type assertions from a fixed source-KG snapshot, matched to DB15K entity IDs and mapped to coarse categories (e.g., person, work, organization, place, event). Ambiguous or missing types are excluded. Each relation–role slot receives its expected type from labeled training triples; the nearest five validation entities are then checked against that type. Full RHDN achieves 1,692 matching neighbors out of 2,080 retrieved occurrences.
The displayed phrases and image-region names are post-hoc labels, not additional model inputs. Phrases map to character spans and retained, non-special BERT subword indices; images use 16 pooled ViT cells in a row-major grid. For Geoffrey Rush, for instance, actor maps to token 5, Academy Award to tokens 17–18, Face to cells , and Trophy to . Displayed region–phrase weights sum the corresponding cells of the full transport plan, retaining all remaining mass in Other. Thus, the Geoffrey Rush actor column sums to 0.171 + 0.029 = 0.200, matching its textual marginal.
Manuscript additions: Appendix D.3.3, Dataset-Level Validation, including Table 21 (diagnostics), Table 22 (token/grid index mapping), and Tables 23–24 (grouped transport plans corresponding to Fig. 5 and the qualitative examples).
We appreciate the reviewer's careful technical suggestions, which have motivated more precise mechanism diagnostics and evaluation reporting.
Dear Reviewer,
Thank you again for your thoughtful review. We have responded to your comments and revised the manuscript accordingly.
As the discussion period is approaching its end, we would be grateful if you could review our response when convenient. If any concerns remain, we would be happy to clarify them.
Best regards,
The Authors
Response to Reviewer DbdL
Response to Reviewer DbdL
We thank the reviewer for the detailed technical assessment. We have expanded the controlled diagnostics, added architecture-matched comparisons, quantified runtime and memory, and made the interpretation of the OT visualizations more explicit. The additions are highlighted in blue in the revised paper.
Q1/W1. Score-, type-, and role-matched false-negative discrimination
The reviewer is correct that the evidence-based FDNS risk in Eqs. (17)–(19) depends on the corrupted entity, relation, and head/tail role, not explicitly on the opposite-side entity. The latter instead affects the complete-triple score si=f(hi,r,ti) and thus the difficulty component of qi∝exp(si/τ−λfnri).
To test discrimination beyond score and coarse type, we extend the DB15K diagnostic from 240 score-matched pairs (∣Δs∣≤0.005) to two nested subsets. Group A matches type and modality availability within the same pool and corruption direction (136 pairs); Group B additionally matches mean role support (∣Δaˉ∣≤0.01, 64 pairs). Full FDNS improves paired risk concordance over the mean-support-only estimator from 61.8% to 67.6% in Group A and 50.0% to 62.5% in Group B. For Group B, the mean paired risk gap is 0.0116, and the geometric-mean weight ratio is 0.9965. This supports additional risk ordering among closely matched candidates without equating role compatibility with triple-level truth.
Manuscript additions: Appendix D.2, Type- and role-matched FDNS diagnostic and Dependence on the opposite-side entity (Tables 13–14).
Q2/W2. Weight normalization, marginal mass, and pool statistics
Weight calculation. For every frozen candidate pool Cb, self-adversarial and FDNS weights are normalized over all valid candidates in that pool, not separately by category:
q_{bi}=\frac{e^{s_{bi}/\tau-\lambda_{\mathrm{fn}}r_{bi}}} {\sum_{j\in C_b}e^{s_{bj}/\tau-\lambda_{\mathrm{fn}}r_{bj}}}.$$ The weight reported for a category is the **mean of its candidate-occurrence weights after full-pool normalization**. Pseudo-FNs are identified by hidden-set membership; among remaining candidates, the highest and lowest 20% by score in each pool are designated hard and easy, with the middle group containing the remainder. The corresponding DB15K category means are: | Method | Pseudo-FN | Hard | Hard/FN | |---|---:|---:|---:| | Self-adversarial | 0.03790 | 0.03506 | 0.925 | | Score-only risk | 0.03582 | 0.03325 | 0.928 | | **FDNS** | **0.03341** | **0.03477** | **1.041** | **Pool selection and matching.** The 5% hiding protocol retains **75,261 pools** from 79,222 training triples, with **4,808,912 valid candidate occurrences**, **1,248 pseudo-FN occurrences**, **1,176 positive-hit pools (1.56%)**, and **240 one-to-one score-matched pairs (19.23% coverage)**. Pools are generated by standard corruption, without enrichment based on hidden-set membership. **Fig. 5.** The marginal refers to the **full grouped Sinkhorn plan**, including the aggregated *Other* region. In the Geoffrey Rush example, the named visual regions contribute **0.171** to the *actor* column and the remaining region contributes **0.029**, giving the textual marginal **0.200**. For a reduced visualization, grouping omitted entries preserves total mass. **Fig. 6.** Training weights likewise use the **full-pool denominator**. In the reported 64-candidate pool, the two highlighted highest-weight candidates have normalized weights **0.118** and **0.096**; the remaining 62 candidates account for **0.786**, so the complete distribution sums to one. Selecting candidates for display does not change their training normalization. **Manuscript additions:** **Appendix D.2 (Tables 9–10 and 12–14)** defines the normalization, category averages, and diagnostic coverage; **Appendix D.3.3 (Tables 22–24)** documents the grouped transport masses associated with Fig. 5. The full-pool weight interpretation is clarified here for Fig. 6.