- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MKGC-CoT, a reasoning-then-ranking framework for multimodal knowledge graph completion. It integrates multimodal fusion, Chain-of-Thought (CoT) reasoning, and hidden-state-based entity ranking. The authors construct answer-free MKG CoT data and adopt a two-stage training strategy combining supervised fine-tuning (SFT) with answer-aware on-policy distillation (OPD). Experiments on DB15K and MKG-Y report MRR scores of 50.44% and 40.21%, respectively, outperforming the strongest listed baselines.
Critical strengths
-
The paper addresses a meaningful limitation of representation-based MKGC methods and provides an interesting motivation for introducing query-specific reasoning into link prediction.
-
The reasoning-then-ranking framework effectively integrates CoT generation with full-entity prediction, avoiding the limitations of candidate-restricted retrieval.
-
The two-stage SFT and OPD training strategy is reasonably motivated and demonstrates measurable improvements, particularly on MKG-Y.
-
The experiments include extensive baseline comparisons, component ablations, and parameter sensitivity analyses. The reported performance on DB15K is particularly strong.
Critical weaknesses
-
The contribution of CoT reasoning is insufficiently established. The prediction head uses the hidden state after the generated Answer field, making it unclear whether improvements arise from intermediate reasoning or answer generation. The no-CoT ablation also changes the scoring pathway and removes language-model supervision, preventing a controlled assessment of reasoning itself.
-
The methodological novelty is relatively limited. Multimodal fusion, CoT generation, hidden-state ranking, and on-policy distillation are established techniques. Their integration is reasonable, but the paper does not sufficiently demonstrate a new relation-specific evidence selection mechanism.
-
The faithfulness of generated reasoning remains unverified. Answer-aware teacher supervision may produce plausible post-hoc explanations rather than evidence-grounded reasoning. The qualitative examples in Figure 5 are illustrative schematics rather than actual model-generated reasoning trajectories.
-
The experimental comparisons are not fully controlled. Most baseline results are taken from previous publications with potentially different data splits, features, and evaluation settings. A matched LLM-based ranking baseline without CoT is also missing.
-
Experimental validation of reliability and efficiency is incomplete. The paper lacks multiple-seed results, statistical significance tests, and computational cost comparisons. The MRR improvement on MKG-Y is only 0.18 percentage points over the strongest baseline, while the proposed approach introduces substantial autoregressive reasoning overhead.
Questions for authors
-
Can the authors compare the full model with answer-only generation and ranking from the hidden state before the Answer field to isolate the contribution of intermediate CoT reasoning?
-
How is the factual correctness and evidence grounding of teacher-generated CoT verified? Can the authors provide actual student-generated traces and corresponding predictions?
-
Can the authors distinguish the contribution of answer-aware OPD from additional training by comparing it with answer-unaware distillation and continued SFT under matched budgets?
-
Can the authors provide controlled comparisons with an LLM-based ranker without CoT, along with multiple-seed results and training/inference efficiency measurements?