acceptodds
Under review as a conference paper at ICLR 2027

Think Before You Link: On-Policy Distillation Enhanced Graph Reasoning for Multimodal Knowledge Graph Completion

Abstract

Multimodal knowledge graph completion (MKGC) predicts missing links from graph structure, entity images, and textual descriptions. Existing methods first construct static multimodal entity representations and then predict links by matching these representations. This representation–prediction procedure does not create a query-specific reasoning process before prediction. We propose MKGC-CoT, which reformulates MKGC as reasoning followed by entity prediction. MKGC-CoT uses Multimodal Fusion to prepare the query context, MKG Chain-of-Thought (CoT) Reasoning to assemble relation-specific evidence, and Reasoning-Based Entity Prediction to rank all entities from the hidden state after the CoT. We construct Answer-Free MKG CoT Data, in which the student receives neither the gold edge nor the gold answer and an answer-aware teacher provides structured traces only during training. Two-Stage Training first uses SFT to learn the reasoning format, then uses Answer-Aware On-Policy Distillation (OPD) to strengthen reasoning on the student's own sampled CoT prefixes. At inference, the student receives only the answer-free query. On the DB15K and MKG-Y datasets, MKGC-CoT achieves the best results: it reaches 50.44 MRR on DB15K, a 26.8% relative MRR improvement over the strongest prior multimodal method, and leads on MRR, Hits@1, and Hits@10 on MKG-Y.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Knowledge Graph, Chain of Thought;On Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yg7kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MKGC-CoT, a reasoning-then-ranking framework for multimodal knowledge graph completion. It integrates multimodal fusion, Chain-of-Thought (CoT) reasoning, and hidden-state-based entity ranking. The authors construct answer-free MKG CoT data and adopt a two-stage training strategy combining supervised fine-tuning (SFT) with answer-aware on-policy distillation (OPD). Experiments on DB15K and MKG-Y report MRR scores of 50.44% and 40.21%, respectively, outperforming the strongest listed baselines.

Critical strengths
  • The paper addresses a meaningful limitation of representation-based MKGC methods and provides an interesting motivation for introducing query-specific reasoning into link prediction.

  • The reasoning-then-ranking framework effectively integrates CoT generation with full-entity prediction, avoiding the limitations of candidate-restricted retrieval.

  • The two-stage SFT and OPD training strategy is reasonably motivated and demonstrates measurable improvements, particularly on MKG-Y.

  • The experiments include extensive baseline comparisons, component ablations, and parameter sensitivity analyses. The reported performance on DB15K is particularly strong.

Critical weaknesses
  • The contribution of CoT reasoning is insufficiently established. The prediction head uses the hidden state after the generated Answer field, making it unclear whether improvements arise from intermediate reasoning or answer generation. The no-CoT ablation also changes the scoring pathway and removes language-model supervision, preventing a controlled assessment of reasoning itself.

  • The methodological novelty is relatively limited. Multimodal fusion, CoT generation, hidden-state ranking, and on-policy distillation are established techniques. Their integration is reasonable, but the paper does not sufficiently demonstrate a new relation-specific evidence selection mechanism.

  • The faithfulness of generated reasoning remains unverified. Answer-aware teacher supervision may produce plausible post-hoc explanations rather than evidence-grounded reasoning. The qualitative examples in Figure 5 are illustrative schematics rather than actual model-generated reasoning trajectories.

  • The experimental comparisons are not fully controlled. Most baseline results are taken from previous publications with potentially different data splits, features, and evaluation settings. A matched LLM-based ranking baseline without CoT is also missing.

  • Experimental validation of reliability and efficiency is incomplete. The paper lacks multiple-seed results, statistical significance tests, and computational cost comparisons. The MRR improvement on MKG-Y is only 0.18 percentage points over the strongest baseline, while the proposed approach introduces substantial autoregressive reasoning overhead.

Questions for authors
  1. Can the authors compare the full model with answer-only generation and ranking from the hidden state before the Answer field to isolate the contribution of intermediate CoT reasoning?

  2. How is the factual correctness and evidence grounding of teacher-generated CoT verified? Can the authors provide actual student-generated traces and corresponding predictions?

  3. Can the authors distinguish the contribution of answer-aware OPD from additional training by comparing it with answer-unaware distillation and continued SFT under matched budgets?

  4. Can the authors provide controlled comparisons with an LLM-based ranker without CoT, along with multiple-seed results and training/inference efficiency measurements?

Reviewer_ixSNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MKGC-CoT, which generates structured reasoning from multimodal graph context and ranks candidate entities using post-CoT hidden states. Training first uses supervised fine-tuning to learn the structured reasoning format, then applies answer-aware on-policy distillation on student-generated reasoning prefixes to reduce the mismatch between teacher-forced training and inference. The method reports MRR scores of 50.44% on DB15K and 40.21% on MKG-Y, compared with 42.68% and 40.03% for the strongest listed baselines, respectively.

Critical strengths
  1. The method provides a coherent way to connect query-conditioned language-model reasoning with full-entity ranking, addressing the mismatch between free-form generation and knowledge graph completion.

  2. The method reports MRR improvements on both DB15K and MKG-Y. The substantial improvement on DB15K provides positive evidence for the effectiveness of the overall framework.

Critical weaknesses
  1. The motivating claim in lines 39–45 is not directly evaluated. The experiments do not establish whether the reasoning stage improves relation-specific evidence selection or rejection of conflicting evidence. The qualitative cases are illustrative schematics rather than actual generated traces and predictions.

  2. Table 2 and Figure 4 show greater sensitivity to the ranking-loss weight during SFT than during OPD, but the reason remains unclear. The OPD ablation also removes additional ranking training, leaving the distillation contribution unisolated. OPD variant that removes only the KL term under matched training conditions, together with component ablations on DB15K, would help clarify the sources of the largest reported gain.

  3. The experiment without CoT variant also removes (\mathcal{L}{LM}) and changes the scoring state. Although a separate (\mathcal{L}{LM}) ablation is provided, the contribution of intermediate reasoning itself remains unisolated.

  4. Figure 1 could illustrate the specific limitation of the previous approach more clearly. Pseudocode accompanying Figure 3 could also help clarify the data-construction procedure.

Questions for authors
  1. What empirical evidence shows that the method addresses the claimed evidence-selection failures? Please provide actual generated traces and corresponding candidate rankings for representative queries.

  2. Can you report the Table 2 component ablations on DB15K, including the CoT and loss-function variants?

  3. Does CoT outperform answer-only supervision when the language-model objective and ranking interface are kept comparable?

Reviewer_4C3COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MKGC-CoT, which formulates MKGC as CoT reasoning followed by entity prediction. It integrates structural, visual, and textual information and uses the hidden state after CoT generation to predict missing entities. A two-stage training strategy of supervised fine-tuning (SFT) and answer-aware on-policy distillation (OPD) is employed, achieving promising results on two benchmarks.

Critical strengths

The proposed method combines SFT with answer-aware OPD, achieving competitive performance on two benchmarks demonstrating its potential effectiveness.

Critical weaknesses
  1. The motivation for using a large teacher–student framework is insufficiently justified. It is unclear why such a framework is necessary for MKGC rather than directly fine-tuning an LLM.

  2. Generating CoT supervision with a large teacher introduces substantial computational costs. The paper lacks evidence that the gains come from meaningful CoT reasoning rather than additional supervision, as well as an analysis of training and inference complexity.

  3. The method is evaluated on only two datasets. Moreover, comparisons with non-LLM methods such as MyGO may not be fair without considering differences in model capacity and scales.

  4. The teacher has access to the gold answer during distillation, whereas the student does not. Additional ablations should distinguish the effects of answer-aware supervision, on-policy student trajectories, and extra training iterations. Furthermore, what happens if the student fails to generate meaningful CoT during training? The paper should clarify how incorrect or low-quality reasoning trajectories are handled during on-policy distillation and whether the training process remains stable.

  5. Some implementation details are missing. For example, grg_r and τr\tau_r in Eq. (3) are not clearly defined. In addition, CoT generation settings, such as the decoding strategy and maximum generation length, are not specified.

Questions for authors

see Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.