acceptodds
Under review as a conference paper at ICLR 2027

Beyond Readability: Rethinking LLM-based MMKGC as Parallel Semantic Ranking

Abstract

Multimodal knowledge graph completion (MMKGC) remains challenging due to structural sparsity, despite its importance for semantic search and reasoning. LLM-based approaches typically address this by formulating MMKGC as a text generation problem. However, existing methods primarily rely on textual inputs and underexploit visual information in multimodal knowledge graphs. Morover, this formulation introduces two key limitations: autoregressive (AR) decoding incurs high latency, and optimizing for readability leads to hallucinations in the entity space. To address the aforementioned issues, we argue that MMKGC is fundamentally a semantic ranking problem within the entity space, rather than a natural language generation task. Driven by this core insight, we propose RIN, a Non-Autoregressive (NAR) parallel inference framework that reframes the LLM-based completion task into a process of semantic ranking over candidate entities. By decoupling the model’s inference trajectory from contextual coherence constraints, RIN not only enables efficient non-autoregressive semantic reasoning but also mechanically circumvents hallucinations typically induced by the pursuit of sequential coherence. Specifically, RIN first employs the Instruction Fine-tuning Module to achieve task reformulation and distribution alignment. Subsequently, the One-shot Generation Module distills core semantic features and filters redundant noise, thereby enabling efficient one-shot parallel inference. Finally, the Hallucination Suppression Module utilizes token-level joint probabilities to perform global scoring of candidate entities. Experiments on three benchmark datasets show that RIN consistently outperforms 22 state-of-the-art methods, achieving both superior accuracy and significantly improved inference efficiency.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
knowledge graph completion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mVqVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RIN formulates multimodal knowledge graph completion as ranking over candidate entities. It first encodes the multimodal query through a single LLM forward pass to obtain a semantic anchor. A lightweight recurrent predictor then combines this anchor with the previous state and predicted token to construct representations for successive token positions. Gating and causal attention refine these representations, and a shared prediction head produces token distributions. Each candidate entity is scored by summing the log-probabilities of its constituent tokens, enabling ranking over the full entity set without further LLM backbone calls. The method reports MRRs of 40.85%, 42.84%, and 45.69% on DB15K, MKG-W, and MKG-Y, respectively, and a 5.9× inference-throughput improvement over a token-by-token baseline.

Critical strengths
  1. The framework offers a coherent way to connect LLM representations with full-entity ranking. Restricting predictions to the candidate entity set avoids outputs that cannot be mapped to valid graph entities.

  2. Evaluation across three datasets supports the method’s ranking effectiveness. The expanded three-million-entity candidate-pool experiment provides additional evidence of robustness to distractor entities.

Critical weaknesses
  1. The methodological distinction from K-ON is insufficiently established. K-ON already combines one-shot multi-token prediction, conditional attention for token dependencies, and entity-level contrastive learning. The additional contribution of RIN’s recurrent predictor and gating remains unclear relative to this existing framework.

  2. Recent relevant methods, including DSoM, M-Hyper, and MDBGF, are missing from the comparisons. These methods should be included in the experimental evaluation.

  3. Differences in LLM backbones limit the fairness of the comparisons, particularly with K-ON. A controlled comparison with K-ON using the same backbone is needed to assess RIN’s contribution.

  4. The token-by-token baseline in Figure 2 is not clearly identified. The efficiency evaluation should specify this baseline and include quantitative comparisons with methods such as K-ON.

  5. Computational resource reporting is incomplete. Although Figure 2(a) reports training time, the paper does not specify fine-tuning token counts, the number of training epochs or steps, or peak GPU memory usage. The computational overhead of the recurrent predictor and attention-based refinement is also not quantified separately, limiting assessment of the method’s resource requirements.

  6. Figure 1 is overly dense and difficult to follow as the main architectural overview. Numerous arrows, repeated token-probability tables, and tightly packed module details make the overall data flow unclear. Simplifying the main pipeline and moving detailed computations to separate figures would improve clarity.

Questions for authors
  1. Which methodological contributions distinguish RIN from K-ON’s one-shot prediction, conditional attention, and entity-level training?

  2. Can you include recent baselines such as DSoM, M-Hyper, and MDBGF in the comparisons?

  3. Can you compare RIN with K-ON using the same LLM backbone?

  4. Which token-by-token baseline is used in Figure 2, and how does RIN compare quantitatively with K-ON in terms of efficiency?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mVqV

mVqV-1: What do recurrence and gating add beyond K-ON?

Covers: CW1; Q1.

Our response:

We thank the reviewer for the question. RIN shares K-ON's single-pass setting, and we will state this common ground more clearly in Section 3. The two methods differ in how the token positions are predicted and how the predictor is kept stable.

The key difference lies in how the LLM's ability is used. An LLM learns its understanding during pre-training by predicting each token from the tokens before it, so its ability is best used when the way we use it matches the way it was trained. K-ON feeds the final hidden state of the LLM directly into multiple MLPs and predicts all positions in parallel from the same vector. A position does not see the tokens before it, and dependency is added only afterwards by the conditional attention, which does not fully exploit the potential of the LLM. RIN keeps the single backbone call but restores this token-by-token pattern in a lightweight predictor. Each step takes the previous state and the token just predicted (Equation 5), and reuses the backbone's own embeddings, feed-forward network and prediction head (Equations 6–9). The evidence for an entity thus builds up token by token, in the same way the LLM was trained, and the attention refinement works on states that already carry this context.

Feeding predicted tokens back into the state makes the recurrence prone to error accumulation, and the content condenser is designed for this problem. Its state-dependent gate (Equation 8) filters each step's representation and keeps noise from building up across steps. Without it, training does not converge on DB15K and MKG-W (Table 2), which shows that the gate is critical for stable training of the recurrent predictor.

These designs lead to clear gains over K-ON. Without visual input, RIN with the text-only Qwen2.5-7B backbone reaches MRR of 40.73 / 42.65 / 45.38 on DB15K / MKG-W / MKG-Y, compared with 38.10 / 36.64 / 33.59 for K-ON with its original Llama-2-7B backbone (Table 1). When K-ON is re-run on the same Qwen2.5-VL-7B backbone as RIN, the gap becomes even larger (40.73 / 42.46 / 44.97 vs. 34.35 / 31.39 / 31.79; see our response to Q3). We will add a side-by-side comparison with K-ON in Section 3 to make these differences explicit.

=================================================

mVqV-2: Can DSoM, M-Hyper, and MDBGF be included?

Covers: CW2; Q2.

Our response:

We sincerely thank the reviewer for pointing out these recent methods. DSoM, M-Hyper and MDBGF are indeed closely related to our work, and we will include them in the experimental comparison (Table 1) of the revised version.

================================================================

mVqV-3: Can RIN and K-ON be compared using the same LLM backbone?

Covers: CW3; Q3.

Our response:

We thank the reviewer for this suggestion. Following it, we re-ran K-ON on the same Qwen2.5-VL-7B-Instruct backbone as RIN, using K-ON's official code with its original LoRA and training configuration and changing only the backbone. Since K-ON is a text-only method, it receives only textual input. For a matched comparison, we also list RIN without visual features (RIN w/o e_v in Table 2), which uses the same backbone and likewise takes no visual input.

DatasetModelMRRHits@1Hits@3Hits@10
DB15KK-ON (Qwen2.5-VL-7B)34.3525.3038.9651.85
DB15KRIN w/o e_v40.7333.1044.4754.38
DB15KRIN40.8533.1545.2755.24
MKG-WK-ON (Qwen2.5-VL-7B)31.3924.6534.1343.89
MKG-WRIN w/o e_v42.4637.5145.0051.73
MKG-WRIN42.8437.8345.2752.22
MKG-YK-ON (Qwen2.5-VL-7B)31.7926.2334.5841.21
MKG-YRIN w/o e_v44.9739.8247.2954.56
MKG-YRIN45.6940.3148.4855.20

With the backbone and input modality matched, RIN still outperforms K-ON by 6.38, 11.07 and 13.18 MRR points on DB15K, MKG-W and MKG-Y. The gap is even larger than in Table 1, since K-ON performs worse on Qwen2.5-VL-7B than on its original Llama-2-7B backbone. This shows that the gains of RIN come from its step-wise prediction design rather than from the choice of backbone. We will add this same-backbone comparison to Table 1 of the revised version.

AuthorsOpenReview ↗

Response to Reviewer mVqV

mVqV-4: Which baseline is used in Figure 2, and how efficient is RIN relative to K-ON?

Covers: CW4; Q4.

Our response:

We thank the reviewer for the question, and we will name the baseline clearly in Figure 2 and Section 4.4. The token-by-token baseline is the same Qwen2.5-VL-7B-Instruct backbone fine-tuned with the same instruction inputs and the same LoRA configuration, and trained on the same 8× NVIDIA H20 GPUs. It generates the entity name autoregressively, one token at a time. RIN calls the backbone once per query and scores all candidates in parallel.

For the comparison with K-ON, we report the training time on DB15K together with accuracy. K-ON uses the same Qwen2.5-VL-7B backbone as in our response to Q3, and all training times are measured on the same 8× NVIDIA H20 GPUs, with each method using its own training configuration.

MethodBackbone calls per queryTraining time (DB15K)MRR (DB15K)
Token-by-token baselineOne per output token20 h–
K-ONOne1.2 h34.35
RINOne3.5 h40.85

Like K-ON, RIN calls the backbone only once per query, so both methods avoid the cost of token-by-token generation and train far faster than the baseline. On the same backbone, the full RIN also improves MRR over K-ON by 6.50 points on DB15K, and by 11.45 and 13.90 points on MKG-W and MKG-Y (6.38, 11.07 and 13.18 points without visual input; see our response to Q3). We will add K-ON to the efficiency comparison in Figure 2 of the revised version.

===================================================================

mVqV-5: Computational resources and module overhead are incompletely reported

Covers: CW5.

Our response:

We thank the reviewer for pointing out these missing details, and we will add a full resource table to the appendix.

For training, RIN is fine-tuned with LoRA on Qwen2.5-VL-7B-Instruct for 5 epochs on 8× NVIDIA H20 GPUs, with batch size 8, learning rate 1e-4 and 128 negative entities per query. Every query outputs a fixed length of 8 tokens. Each training triple yields one head query and one tail query, so one epoch contains 158,444 queries on DB15K, 68,392 on MKG-W and 42,620 on MKG-Y. This gives 1.27M, 0.55M and 0.34M target tokens per epoch, and 6.3M, 2.7M and 1.7M over the full training. The peak GPU memory during training is 45.81 GiB per GPU on MKG-W with a per-device batch size of 4.

For the two modules, the recurrent predictor has about 105M parameters and the attention-based refinement has about 51M. Together with the LoRA on the shared prediction head, they add about 0.16B parameters, which is 1.2%, 0.6% and 1.9% of the 8.45B parameters used in the forward pass, including the frozen backbone. Their computation is also small. The backbone processes the whole instruction with its text and image tokens, while both modules run only on the 8 output positions after the single backbone pass.

The token-by-token baseline has no such modules and uses only the 8.29B backbone, so it has fewer parameters than RIN. Yet it takes 20 hours to train against 3.5 hours for RIN, and handles 12 queries per second at inference against 71 (Figure 2). The baseline runs the full backbone once for every output token, while RIN runs it once per query. The cost is therefore set by the number of backbone calls, not by the parameter count.

==========================================================

mVqV-6: Figure 1 is too dense to communicate the main pipeline

Covers: CW6.

Our response:

We thank the reviewer for this helpful suggestion. In the revised version, we will simplify Figure 1 to show only the main pipeline and move the detailed computations to separate figures, which will make the overall data flow easier to follow.

Thank you for taking the time to address my questions. While your response has clarified some points, some of my concerns still remain. I will consider the points you have raised during the further discussion.

Reviewer_2kZKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RIN, a method for multimodal knowledge graph completion based on large language models. Its core process involves obtaining query representations through a single forward pass of the backbone model, then constructing representations for multiple token positions using a lightweight recurrent module and an attention refinement module, ultimately scoring predefined candidate entities. Training combines entity-level contrastive loss with token-level supervision loss. The paper reports superior results to listed baselines on DB15K, MKG-W, and MKG-Y, claiming significant reductions in training cost and improvements in inference throughput. Additionally, the authors explore the scalability of their method by expanding the candidate set to three million entities.

Critical strengths
  1. For knowledge graph completion tasks with a well-defined output space, directly ranking candidate entities is reasonable. This design ensures that the final output corresponds to an entity within the candidate set, thereby avoiding the issue of entity resolution after free-text generation.

  2. The reported results are strong. On the three datasets, RIN achieves the strongest performance in the table.

  3. Certain components are supported by empirical evidence.

Critical weaknesses
  1. The approach overlaps significantly with the method proposed in K-ON. Reframing the KGC task from token-by-token generation to a candidate entity scoring paradigm cannot be considered a primary contribution of this paper. The potential contributions of this work likely lie in the incorporation of multimodal information.

  2. The entity-level scoring in Equation 14 exhibits structural bias. This formula employs the same set of positional distributions for all candidates and sums the log-probabilities of the individual tokens constituting an entity:

    $ s(e)=\sum_{i=1}^{m}\log P(t_i\mid L_i). $

    If the token sequences of the two candidate entities are respectively

    $ e_a=(t_1,\ldots,t_m),\qquad e_b=(t_1,\ldots,t_m,u_1,\ldots,u_r),\quad r\ge 1, $

    Since the logarithmic probability of each token is no greater than zero, it is necessary that

    $ s(e_b)=s(e_a)+\sum_{j=1}^{r}\log P(u_j\mid L_{m+j})\le s(e_a). $

    Therefore, when the token sequence of one candidate entity is a strict prefix of another, the score of the longer candidate can never exceed that of the shorter one, even if the longer candidate is the actual correct answer. This limitation is inherent to the scoring formulation itself and cannot be eliminated through training alone.

  3. The ablation results do not sufficiently support the claim that “multimodal instruction design provides a necessary foundation” (Line 451-452). When visual information is removed in the ablation experiments, the MRR on the three datasets drops by only 0.12, 0.38, and 0.72 percentage points, respectively; similarly, replacing the backbone model with a text-only model results in only a minor decline. This indicates that the text-based pathway accounts for the majority of the performance, which fails to substantiate the strong assertion that "the multimodal design provides an essential foundation."

  4. In the efficiency experiments, the authors claim to compare their approach against a token-by-token LLM baseline, yet they fail to specify the exact models or implementation configurations used. It is difficult to discern whether the observed speedup stems from the architecture, the number of the training parameters, or differences in the baseline implementation.

Questions for authors
  1. Could you clarify the technical contributions of this paper relative to K-ON?

  2. How does Equation 14 avoid the inherent ordering constraints among prefix candidates?

  3. Section 3.5 states that auxiliary supervision is applied to the intermediate predictions, whereas Equation 17 uses logits obtained from the refined representations. Which predictions are actually used to compute the auxiliary loss?

  4. What is the complete configuration for the efficiency baseline?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2kZK

2kZK-1: What are RIN's technical contributions relative to K-ON?

Covers: CW1; Q1.

Our response:

We thank the reviewer for the careful comparison with K-ON. We agree that the candidate entity scoring formulation originates from K-ON, and RIN builds on it. Beyond this formulation, RIN differs from K-ON mainly in the following three aspects.

From parallel projection to step-wise composition. K-ON feeds the final hidden state of the LLM directly into multiple MLPs and obtains all positional representations in parallel from the same vector. This does not fully exploit the potential of the LLM, whose semantic competence comes from composing meaning token by token. RIN instead introduces a lightweight recurrent predictor outside the backbone, in which each position is built on the previous step and the previously predicted token while reusing the backbone's own embedding and feed-forward layers. Step-wise composition is thus recovered without additional backbone forward passes, allowing RIN to distinguish semantically related candidates whose names differ in only a few tokens.

Content gating for stable recurrence. Feeding predicted tokens back into the state makes the recurrence prone to error accumulation. The content condenser, a state-dependent gate (Equation 8), filters each step's output to keep only informative features. It is critical for stable training: without it, training fails to converge on DB15K and MKG-W (Table 2).

Neighbor-aware multimodal instruction. Because the backbone is invoked only once, its input must carry all the evidence the predictor needs. We design a dedicated instruction template (Appendix 6.2) that jointly encodes the entity description, the image, and the local neighbor facts of the query entity, with the target triple excluded to prevent information leakage. This design reflects the multimodal contribution noted by the reviewer, and further supplies structural context from the graph beyond text and images.

These designs translate into consistent gains. RIN improves MRR over K-ON by 7.2%, 16.9%, and 36.0% on DB15K, MKG-W, and MKG-Y, respectively (Table 1). Moreover, the improvement does not come from multimodal information alone. The table below compares K-ON with RIN variants that take no visual input (MRR). K-ON is reported both with its original Llama-2-7B backbone (Table 1) and re-run with its official code on the same Qwen2.5-VL-7B backbone as RIN.

MethodBackboneDB15KMKG-WMKG-Y
K-ON (Table 1)Llama-2-7B38.1036.6433.59
RIN with text-only backboneQwen2.5-7B40.7342.6545.38
K-ON (re-run)Qwen2.5-VL-7B34.3531.3931.79
RIN w/o e_vQwen2.5-VL-7B40.7342.4644.97

Whether K-ON uses its original backbone or the same backbone as RIN, RIN without visual input outperforms it on all three datasets, by 2.63, 6.01 and 11.79 MRR points in the first case and by 6.38, 11.07 and 13.18 points in the second. This suggests that a substantial part of the gain comes from the prediction design rather than from visual information.

In the revision, we will add a dedicated comparison with K-ON to the related work and method sections, clearly separating the entity-scoring formulation inherited from K-ON from the three contributions of RIN described above.

AuthorsOpenReview ↗

Response to Reviewer 2kZK

2kZK-2: How can Equation 14 avoid fixed ordering between prefix candidates?

Covers: CW2; Q2.

Our response:

We thank the reviewer for the careful derivation. Equation 14 omits the length normalization used in our implementation, which is an oversight in the manuscript; all reported results use the normalized score:

Score(e)=1m∑i=1mαi Li[ti],\mathrm{Score}(e) = \frac{1}{m} \sum_{i=1}^{m} \alpha_i \, \mathbf{L}_i[t_i],

where Li[ti]\mathbf{L}_i[t_i] is the logit of token tit_i at position ii and αi\alpha_i is a fixed position weight. For the reviewer's prefix pair,

Score(eb)=m⋅Score(ea)+∑j=1rαm+j Lm+j[uj]m+r,\mathrm{Score}(e_b) = \frac{m \cdot \mathrm{Score}(e_a) + \sum_{j=1}^{r} \alpha_{m+j} \, \mathbf{L}_{m+j}[u_j]}{m + r},

so the longer candidate ranks higher whenever its additional tokens score above the prefix average. Since the scores are raw logits rather than non-positive log-probabilities, the ordering is not fixed by the formula, and the entity-level contrastive objective (Equation 15) can learn it from data. We will correct Equation 14 accordingly in the revision.

========================================================================

2kZK-3: Do the small visual gains support calling multimodal design a necessary foundation?

Covers: CW3.

Critical Weaknesses — item 3 (verbatim):

  1. The ablation results do not sufficiently support the claim that “multimodal instruction design provides a necessary foundation” (Line 451-452). When visual information is removed in the ablation experiments, the MRR on the three datasets drops by only 0.12, 0.38, and 0.72 percentage points, respectively; similarly, replacing the backbone model with a text-only model results in only a minor decline. This indicates that the text-based pathway accounts for the majority of the performance, which fails to substantiate the strong assertion that "the multimodal design provides an essential foundation."

Our response:

We thank the reviewer for this careful reading. We agree that on the current benchmarks, textual information carries most of the discriminative evidence, as the ablation results below show (MRR).

VariantDB15KMKG-WMKG-Y
RIN40.8542.8445.69
RIN w/o e_v (no visual input)40.7342.4644.97
RIN with text-only backbone (Qwen2.5-7B)40.7342.6545.38

Removing the visual input lowers MRR by 0.12, 0.38 and 0.72 points, and replacing the backbone with its text-only counterpart lowers it by 0.12, 0.19 and 0.31 points. This pattern is common in MMKGC and reflects the nature of these benchmarks. Each entity comes with a textual description that often states its type, attributes and key relations directly, while the images are mostly collected from web search and, for many entities such as abstract concepts or events, carry little identifying information. Prior work on the same benchmarks reports similar drops: removing the visual modality reduces MRR by 0.28 for NativE on DB15K and by 0.88 for M2GDT on MKG-W, and RSME finds that visual context is not always helpful for knowledge graphs. Even without visual input, RIN still outperforms K-ON on the same Qwen2.5-VL-7B backbone (MRR 40.73 / 42.46 / 44.97 vs. 34.35 / 31.39 / 31.79), so its main advantage does not depend on the visual modality.

On top of this text-dominated pathway, visual information provides consistent complementary gains. Adding images improves all four metrics on all three datasets, and the gain grows on MKG-W and MKG-Y, reaching 1.19 points in Hits@3 on MKG-Y. Images supply information that descriptions seldom state, such as appearance, shape and scene, which helps separate candidates whose descriptions are similar. We will therefore revise Lines 451–452 to state that multimodal instruction design provides complementary gains on top of a text-dominated prediction pathway, rather than a necessary foundation.

We still regard visual input as an important part of RIN, because its value is likely to grow. Image sources for knowledge graphs are becoming richer, moving from single retrieved pictures to curated, multi-view and higher-quality images, and multimodal LLMs are becoming much stronger at understanding them. As both improve, images may provide completion evidence that is as sufficient as text, or more direct, for example for long-tail entities whose descriptions are short or missing. RIN is built to benefit from this trend. Visual features enter through the same instruction and the same single backbone pass as text, so better images or a stronger vision-language backbone can be used directly without changing the architecture.

AuthorsOpenReview ↗

Response to Reviewer 2kZK

2kZK-4: Is the auxiliary loss computed from intermediate or refined predictions?

Covers: Q3.

Our response:

We thank the reviewer for spotting this inconsistency. Equation 17 is correct: the auxiliary loss is computed from the refined logits L in Equation 13, i.e., the same token distributions used for entity scoring. The step predictions p_k are only used to select the input token for the next step via argmax and do not receive any loss. The phrase "to provide auxiliary supervision" in Section 3.5 is a writing error, and we will remove it in the revision.

This design keeps a single set of outputs for both objectives: the auxiliary loss stabilizes the same token distributions that the contrastive loss ranks, and its gradients reach the step-wise predictor through the refinement module, so no separate supervision of intermediate predictions is needed.

========================================================================================

2kZK-5: Does the efficiency gain come from the architecture, parameter count, or implementation differences?

Covers: CW4; Q4.

Our response:

We thank the reviewer for the question and will add the full efficiency configuration to the appendix. The token-by-token baseline uses the same setup as RIN: the Qwen2.5-VL-7B-Instruct backbone, the same instruction inputs, the same LoRA configuration (rank 32, α = 64, on the same backbone modules), 8× NVIDIA H20 GPUs with mixed precision, and batch size 8. The two differ only in how the output is produced. The baseline generates the entity name autoregressively, one token at a time, while RIN calls the backbone once and scores all candidates in parallel.

With these settings matched, the speedup does not come from the parameter budget or the implementation. RIN actually trains more parameters than the baseline, as its step-wise predictor and refinement module add parameters on top of the shared LoRA. The gain comes from the architecture. The baseline runs the full backbone once for every generated token, whereas RIN runs it once per query and handles the step-wise recurrence in small modules outside the backbone. This is what raises inference throughput from 12 to 71 queries per second.

Thank you for the additional experiments and clarifications. After carefully reviewing your responses, I find that some of my concerns have been addressed. However, I still consider the overall contribution of the paper to be somewhat limited, and I believe the manuscript requires substantial further refinement. My initial rating remains appropriate at this stage.

Reviewer_RBTQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multimodal knowledge graph completion and proposes RIN, a framework that reformulates entity name generation as candidate entity ranking. The paper argues that autoregressive generation introduces unnecessary decoding costs and may produce outputs outside the valid entity set. RIN uses an instruction tuned multimodal language model to encode entity descriptions, images, and graph neighborhoods in a single backbone forward pass. A lightweight recurrent predictor then produces position wise token distributions, which are refined through causal attention and used to score all candidate entities. The model is trained with both an entity level contrastive objective and a supervised fine tuning objective. Experiments on DB15K, MKG W, and MKG Y report strong ranking performance, reduced training time, higher inference throughput, and scalability to three million candidate entities.

Critical strengths

The paper identifies a practically important mismatch between autoregressive generation and multimodal knowledge graph completion. Since the target is selected from a fixed entity set, generating an entity name token by token introduces unnecessary decoding cost and may produce outputs outside the valid candidate space. Reformulating the task as constrained entity ranking provides a clear motivation for improving both prediction validity and inference efficiency.

The empirical evaluation is extensive and closely aligned with the main claims of the paper. RIN is evaluated on three MMKGC benchmarks against 24 baselines and achieves the best results across all reported ranking metrics. The efficiency analysis shows substantial reductions in training time and a 5.9 times improvement in inference throughput. The additional evaluation with three million candidate entities provides useful evidence of scalability, while the component ablations help explain the sources of the reported performance gains.

Critical weaknesses

The entity scoring formulation does not appear to compute the joint probability of each candidate token sequence as claimed. The position wise logits are produced from a single shared trajectory driven by the model’s argmax tokens, and all candidate entities are scored using the same set of distributions. A candidate’s own preceding tokens therefore do not condition the probabilities assigned to its later tokens. The attention refinement captures dependencies along the shared prediction trajectory. It remains unclear how these shared dependencies represent token consistency within each candidate entity. The method may still learn useful discriminative scores through contrastive training, although the probabilistic interpretation of Equation 14 currently lacks sufficient justification.

The methodological contribution appears moderate relative to the paper’s framing. Candidate ranking is already a standard formulation in knowledge graph completion, and RIN builds on several familiar components, including an instruction tuned multimodal language model, a lightweight recurrent predictor, attention refinement, and contrastive ranking. I see the main contribution as an efficient way to combine these components for MMKGC. This is useful, although the current evidence does not fully support the broader claim of fundamentally rethinking the task. Since efficiency is central to the value of the method, I would also like to see a clearer description of the Token by Token baseline and the timing protocol used to obtain the reported speedup.

Questions for authors

Are the logits used in Equation 14 conditioned on each candidate entity’s own preceding tokens? If all candidates share a trajectory generated from the model’s argmax tokens, how does the resulting score capture candidate specific token dependencies, and in what sense can it be interpreted as a joint sequence probability?

Could the authors provide the complete configuration of the Token by Token baseline and clarify which components are included in the reported training time and inference throughput? In particular, are the backbone, multimodal inputs, hardware, numerical precision, batch size, maximum entity length, and end to end candidate scoring matched between the two systems?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RBTQ

RBTQ-1: Does the shared trajectory represent a candidate-conditioned joint probability?

Covers: CW paragraph 1; Q1.

Our response:

We thank the reviewer for this careful and accurate analysis. The logits used in Equation 14 are conditioned on the query and on the shared prediction trajectory, not on each candidate's own preceding tokens. Candidate specificity instead comes from position-aligned lookup: at each position, the shared prediction head outputs a score vector over the full token vocabulary, and each candidate, whose entity name is tokenized in advance, takes the entry indexed by its own token ID at each position and aggregates these values over its tokens. Different candidates therefore collect different combinations of values from the same outputs. Attention refinement models dependencies among these shared positional representations, and the entity-level contrastive objective, discussed below, trains the correct entity's token combination to score higher than those of negative entities. Equation 14 is therefore an entity-level ranking score rather than an exact candidate-conditioned joint sequence probability. Our original description did not make this distinction clear, and we will correct it throughout the main text and appendix.

We adopt a shared trajectory primarily to make scoring over the full entity set efficient. Candidate-specific conditioning would require running the positional predictor separately for each candidate prefix, so the cost would grow with both the number of candidates and their token lengths. In RIN, each query requires one backbone invocation and one lightweight recurrence. Because all candidates index the same positional outputs, scoring the full entity set reduces to parallel lookup and aggregation. This keeps full-entity ranking tractable, including in the expanded three-million-entity candidate pool.

During training, the positive entity provides auxiliary token supervision, while the entity-level contrastive objective directly trains the aggregated candidate scores to distinguish positive from negative entities. Negative candidates can therefore contribute to training without being separately generated. The ablation below shows the benefit of this objective.

DatasetModelMRRHits@1Hits@3Hits@10
DB15KRIN40.8533.1545.2755.24
DB15KRIN w/o L_KGC32.3825.7035.9445.56
MKG-WRIN42.8437.8345.2752.22
MKG-WRIN w/o L_KGC39.7834.5942.4948.46
MKG-YRIN45.6940.3148.4855.20
MKG-YRIN w/o L_KGC40.8235.9743.3450.71

As shown in the table, removing the entity-level contrastive training decreases MRR by 8.47, 3.06, and 4.87 percentage points on DB15K, MKG-W, and MKG-Y, respectively. These results confirm the reviewer's observation that RIN learns useful discriminative scores through contrastive training: entity-level contrastive supervision substantially improves ranking beyond positive-token supervision alone.

In the revision, we will replace “joint probability” with “entity-ranking score” in the description of Equation 14 and in the appendix, state explicitly that attention refinement models dependencies among shared positional representations, and present the shared trajectory as an efficiency-motivated design choice.

AuthorsOpenReview ↗

Response to Reviewer RBTQ

RBTQ-2: Is the contribution framed too broadly?

Covers: The contribution-framing concern in CW paragraph 2.

Our response:

We thank the reviewer for recognizing the usefulness of RIN as an efficient framework for MMKGC. RIN is more than a combination of existing components. Its innovations lie mainly in the following three aspects.

Step-wise semantic composition with a single backbone call. Our design follows a simple principle: to make full use of a pretrained LLM, the way it is used should resemble the way it was trained, namely composing meaning token by token. Methods such as K-ON feed the final hidden state of the LLM directly into multiple MLPs, which does not fully exploit the potential of the LLM, while full token-by-token decoding requires one backbone invocation per token. RIN resolves this trade-off with a lightweight recurrent predictor that recovers step-wise composition outside the backbone, so the backbone is invoked only once and semantically related candidates whose names differ in only a few tokens can still be distinguished. As shown in Table 1, RIN improves MRR over K-ON by 7.2%, 16.9%, and 36.0% on DB15K, MKG-W, and MKG-Y, respectively, and removing the attention refinement over the recurrent outputs alone reduces MRR by 2.14, 2.09, and 2.80 percentage points (Table 2).

Stable recurrence through content gating. Noise can accumulate through recurrence. The content condenser, a state-dependent gate, filters each step's output to keep only informative features, and is critical for stable training: without it, training fails to converge on DB15K and MKG-W (Table 2).

Neighbor-aware multimodal instruction. Because the backbone is invoked only once, its input must carry all the evidence the predictor needs. We design a dedicated instruction template (Appendix 6.2) that injects the local neighbor facts of the query entity alongside its textual and visual information, so the single forward pass also encodes structural context from the graph, with the target triple excluded to prevent information leakage.

At the same time, we agree that candidate ranking is a standard formulation in KGC and that the current evidence does not fully support the broader claim of fundamentally rethinking the task. We will adjust the framing throughout the paper to position RIN as an efficient LLM-based ranking framework for MMKGC, highlight the step-wise recurrent predictor, content gating, and neighbor-aware multimodal instruction as its three main technical contributions, and explicitly acknowledge prior one-shot approaches such as K-ON. The Token-by-Token baseline and timing protocol are addressed in the following response.

=========================================================================

RBTQ-3: What is the Token-by-Token baseline, and how is efficiency measured?

Covers: The efficiency concern in CW paragraph 2; Q2.

Our response:

We thank the reviewer for emphasizing the importance of the efficiency evaluation. The Token-by-Token baseline uses the same Qwen2.5-VL-7B-Instruct backbone as RIN and predicts the tail entity in the standard generative way, producing its name one token at a time, with one full backbone forward pass per generated token. Both systems are run under identical conditions: the same backbone, eight NVIDIA H20 GPUs, bf16 mixed precision, and the same training configuration. The timing protocol is likewise matched. Training time covers only the training process itself, excluding evaluation runs. Inference throughput covers the model forward pass together with scoring and ranking the full entity set, so both systems are measured up to the point at which a complete ranking over all entities is produced, while data loading and preprocessing, which are shared by both systems, are excluded.

Under this protocol, the speedup follows directly from the architectural difference. The Token-by-Token baseline requires one backbone invocation per generated token, whereas RIN requires a single backbone invocation per query followed by a lightweight recurrence, and scores all candidate entities in parallel by indexing the shared positional outputs. As shown in Figure 2, this reduces the training time on DB15K from approximately 20 hours to 3.5 hours under the same configuration and increases inference throughput from 12 to 71 queries per second, a 5.9× speedup.

In the revision, we will add a dedicated paragraph and an appendix table describing the Token-by-Token baseline and the full timing protocol, including the backbone, hardware, numerical precision, batch sizes, maximum entity length, and the components included in each measurement.

Thank you for the detailed response. The clarification of Equation 14 and the additional details on the efficiency comparison address some of my concerns. I still view the overall methodological contribution as moderate, especially since candidate ranking is already standard and the shared trajectory does not model candidate-specific token dependencies. I will keep my rating at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.