- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RIN formulates multimodal knowledge graph completion as ranking over candidate entities. It first encodes the multimodal query through a single LLM forward pass to obtain a semantic anchor. A lightweight recurrent predictor then combines this anchor with the previous state and predicted token to construct representations for successive token positions. Gating and causal attention refine these representations, and a shared prediction head produces token distributions. Each candidate entity is scored by summing the log-probabilities of its constituent tokens, enabling ranking over the full entity set without further LLM backbone calls. The method reports MRRs of 40.85%, 42.84%, and 45.69% on DB15K, MKG-W, and MKG-Y, respectively, and a 5.9× inference-throughput improvement over a token-by-token baseline.
Critical strengths
-
The framework offers a coherent way to connect LLM representations with full-entity ranking. Restricting predictions to the candidate entity set avoids outputs that cannot be mapped to valid graph entities.
-
Evaluation across three datasets supports the method’s ranking effectiveness. The expanded three-million-entity candidate-pool experiment provides additional evidence of robustness to distractor entities.
Critical weaknesses
-
The methodological distinction from K-ON is insufficiently established. K-ON already combines one-shot multi-token prediction, conditional attention for token dependencies, and entity-level contrastive learning. The additional contribution of RIN’s recurrent predictor and gating remains unclear relative to this existing framework.
-
Recent relevant methods, including DSoM, M-Hyper, and MDBGF, are missing from the comparisons. These methods should be included in the experimental evaluation.
-
Differences in LLM backbones limit the fairness of the comparisons, particularly with K-ON. A controlled comparison with K-ON using the same backbone is needed to assess RIN’s contribution.
-
The token-by-token baseline in Figure 2 is not clearly identified. The efficiency evaluation should specify this baseline and include quantitative comparisons with methods such as K-ON.
-
Computational resource reporting is incomplete. Although Figure 2(a) reports training time, the paper does not specify fine-tuning token counts, the number of training epochs or steps, or peak GPU memory usage. The computational overhead of the recurrent predictor and attention-based refinement is also not quantified separately, limiting assessment of the method’s resource requirements.
-
Figure 1 is overly dense and difficult to follow as the main architectural overview. Numerous arrows, repeated token-probability tables, and tightly packed module details make the overall data flow unclear. Simplifying the main pipeline and moving detailed computations to separate figures would improve clarity.
Questions for authors
-
Which methodological contributions distinguish RIN from K-ON’s one-shot prediction, conditional attention, and entity-level training?
-
Can you include recent baselines such as DSoM, M-Hyper, and MDBGF in the comparisons?
-
Can you compare RIN with K-ON using the same LLM backbone?
-
Which token-by-token baseline is used in Figure 2, and how does RIN compare quantitatively with K-ON in terms of efficiency?
3 comments
Response to Reviewer mVqV
mVqV-4: Which baseline is used in Figure 2, and how efficient is RIN relative to K-ON?
Covers: CW4; Q4.
Our response:
We thank the reviewer for the question, and we will name the baseline clearly in Figure 2 and Section 4.4. The token-by-token baseline is the same Qwen2.5-VL-7B-Instruct backbone fine-tuned with the same instruction inputs and the same LoRA configuration, and trained on the same 8× NVIDIA H20 GPUs. It generates the entity name autoregressively, one token at a time. RIN calls the backbone once per query and scores all candidates in parallel.
For the comparison with K-ON, we report the training time on DB15K together with accuracy. K-ON uses the same Qwen2.5-VL-7B backbone as in our response to Q3, and all training times are measured on the same 8× NVIDIA H20 GPUs, with each method using its own training configuration.
| Method | Backbone calls per query | Training time (DB15K) | MRR (DB15K) |
|---|---|---|---|
| Token-by-token baseline | One per output token | 20 h | – |
| K-ON | One | 1.2 h | 34.35 |
| RIN | One | 3.5 h | 40.85 |
Like K-ON, RIN calls the backbone only once per query, so both methods avoid the cost of token-by-token generation and train far faster than the baseline. On the same backbone, the full RIN also improves MRR over K-ON by 6.50 points on DB15K, and by 11.45 and 13.90 points on MKG-W and MKG-Y (6.38, 11.07 and 13.18 points without visual input; see our response to Q3). We will add K-ON to the efficiency comparison in Figure 2 of the revised version.
===================================================================
mVqV-5: Computational resources and module overhead are incompletely reported
Covers: CW5.
Our response:
We thank the reviewer for pointing out these missing details, and we will add a full resource table to the appendix.
For training, RIN is fine-tuned with LoRA on Qwen2.5-VL-7B-Instruct for 5 epochs on 8× NVIDIA H20 GPUs, with batch size 8, learning rate 1e-4 and 128 negative entities per query. Every query outputs a fixed length of 8 tokens. Each training triple yields one head query and one tail query, so one epoch contains 158,444 queries on DB15K, 68,392 on MKG-W and 42,620 on MKG-Y. This gives 1.27M, 0.55M and 0.34M target tokens per epoch, and 6.3M, 2.7M and 1.7M over the full training. The peak GPU memory during training is 45.81 GiB per GPU on MKG-W with a per-device batch size of 4.
For the two modules, the recurrent predictor has about 105M parameters and the attention-based refinement has about 51M. Together with the LoRA on the shared prediction head, they add about 0.16B parameters, which is 1.2%, 0.6% and 1.9% of the 8.45B parameters used in the forward pass, including the frozen backbone. Their computation is also small. The backbone processes the whole instruction with its text and image tokens, while both modules run only on the 8 output positions after the single backbone pass.
The token-by-token baseline has no such modules and uses only the 8.29B backbone, so it has fewer parameters than RIN. Yet it takes 20 hours to train against 3.5 hours for RIN, and handles 12 queries per second at inference against 71 (Figure 2). The baseline runs the full backbone once for every output token, while RIN runs it once per query. The cost is therefore set by the number of backbone calls, not by the parameter count.
==========================================================
mVqV-6: Figure 1 is too dense to communicate the main pipeline
Covers: CW6.
Our response:
We thank the reviewer for this helpful suggestion. In the revised version, we will simplify Figure 1 to show only the main pipeline and move the detailed computations to separate figures, which will make the overall data flow easier to follow.
Thank you for taking the time to address my questions. While your response has clarified some points, some of my concerns still remain. I will consider the points you have raised during the further discussion.
Response to Reviewer mVqV
mVqV-1: What do recurrence and gating add beyond K-ON?
Covers: CW1; Q1.
Our response:
We thank the reviewer for the question. RIN shares K-ON's single-pass setting, and we will state this common ground more clearly in Section 3. The two methods differ in how the token positions are predicted and how the predictor is kept stable.
The key difference lies in how the LLM's ability is used. An LLM learns its understanding during pre-training by predicting each token from the tokens before it, so its ability is best used when the way we use it matches the way it was trained. K-ON feeds the final hidden state of the LLM directly into multiple MLPs and predicts all positions in parallel from the same vector. A position does not see the tokens before it, and dependency is added only afterwards by the conditional attention, which does not fully exploit the potential of the LLM. RIN keeps the single backbone call but restores this token-by-token pattern in a lightweight predictor. Each step takes the previous state and the token just predicted (Equation 5), and reuses the backbone's own embeddings, feed-forward network and prediction head (Equations 6–9). The evidence for an entity thus builds up token by token, in the same way the LLM was trained, and the attention refinement works on states that already carry this context.
Feeding predicted tokens back into the state makes the recurrence prone to error accumulation, and the content condenser is designed for this problem. Its state-dependent gate (Equation 8) filters each step's representation and keeps noise from building up across steps. Without it, training does not converge on DB15K and MKG-W (Table 2), which shows that the gate is critical for stable training of the recurrent predictor.
These designs lead to clear gains over K-ON. Without visual input, RIN with the text-only Qwen2.5-7B backbone reaches MRR of 40.73 / 42.65 / 45.38 on DB15K / MKG-W / MKG-Y, compared with 38.10 / 36.64 / 33.59 for K-ON with its original Llama-2-7B backbone (Table 1). When K-ON is re-run on the same Qwen2.5-VL-7B backbone as RIN, the gap becomes even larger (40.73 / 42.46 / 44.97 vs. 34.35 / 31.39 / 31.79; see our response to Q3). We will add a side-by-side comparison with K-ON in Section 3 to make these differences explicit.
=================================================
mVqV-2: Can DSoM, M-Hyper, and MDBGF be included?
Covers: CW2; Q2.
Our response:
We sincerely thank the reviewer for pointing out these recent methods. DSoM, M-Hyper and MDBGF are indeed closely related to our work, and we will include them in the experimental comparison (Table 1) of the revised version.
================================================================
mVqV-3: Can RIN and K-ON be compared using the same LLM backbone?
Covers: CW3; Q3.
Our response:
We thank the reviewer for this suggestion. Following it, we re-ran K-ON on the same Qwen2.5-VL-7B-Instruct backbone as RIN, using K-ON's official code with its original LoRA and training configuration and changing only the backbone. Since K-ON is a text-only method, it receives only textual input. For a matched comparison, we also list RIN without visual features (RIN w/o e_v in Table 2), which uses the same backbone and likewise takes no visual input.
With the backbone and input modality matched, RIN still outperforms K-ON by 6.38, 11.07 and 13.18 MRR points on DB15K, MKG-W and MKG-Y. The gap is even larger than in Table 1, since K-ON performs worse on Qwen2.5-VL-7B than on its original Llama-2-7B backbone. This shows that the gains of RIN come from its step-wise prediction design rather than from the choice of backbone. We will add this same-backbone comparison to Table 1 of the revised version.