acceptodds
Under review as a conference paper at ICLR 2027

Visual Dependence and Utility in Multimodal Knowledge Graph Completion: A Controlled Pilot Study

Abstract

Multimodal knowledge graph completion experiments often use changes in aggregate ranking metrics to assess the value of visual information. Interpreting these changes requires specifying which inputs, model parameters, and evaluation labels remain fixed. We report a controlled pilot study that examines these choices through three diagnostics: alpha-channel image preprocessing, positive-label filtering with fixed scores, and visual-package reassignment in separately trained MyGO models. Correcting transparent-image handling changes hundreds of source images, while absolute mean new-target MRR changes remain below in our fixed linear-transfer setup under its original validation filter. Adding withheld training positives to the evaluation filter changes MRR without changing model scores and reverses one descriptive fusion comparison. In a separate transductive experiment, five new visual mappings reduce the original-image model's MRR by 0.0544–0.0587. The permanently permuted model changes by less than 0.0006 in either direction, despite more than 58% of query ranks changing under every mapping. A zero-visual control remains exactly invariant. These observations distinguish input dependence, net ranking utility, and evaluation sensitivity within the tested configurations. The experiments are exploratory: the MyGO comparison uses one training seed and 100 epochs, and the evaluation data have informed development. The study motivates reporting query-level responses alongside aggregate metrics and identifying the intervention that supports each claim.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
multimodal knowledge graph completion, graph representation learning, visual modality evaluation, controlled experiments, filtered ranking, modality perturbation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_inU4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This controlled pilot examines three distinct issues in multimodal knowledge graph completion: transparent-image preprocessing, positive-label filtering with model scores held fixed, and visual-package reassignment in saved MyGO models. Its central observation is that a small mean MRR change can coexist with changes to many individual query ranks.

Critical strengths
  1. The paper separates changes to input preprocessing, evaluation labels, and fixed-model visual assignments. This makes its diagnostic measurements easier to interpret.

  2. The query-level absolute changes and zero-visual control reveal behavior that aggregate MRR alone conceals. The limitations are stated candidly.

Critical weaknesses
  1. The main MyGO comparison uses one training seed and 100 epochs, with no evidence of convergence; the referenced implementation uses a much longer training budget. Five new input mappings do not substitute for independent training runs. Consequently, the central difference between training conditions remains preliminary.

  2. The three experiments examine image preprocessing, evaluation rules, and visual inputs, but use different experimental settings and do not yet support a unified, convincing conclusion. It remains unclear whether these findings generalize to other models and datasets.

Questions for authors

Do the original-, permuted-, and zero-visual patterns persist across multiple training seeds, matched execution conditions, a sufficiently long training budget, and an independently selected evaluation split?

Reviewer_HonBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the role of visual information in multimodal knowledge graph completion. Instead of relying solely on aggregate metrics such as MRR to assess the usefulness of visual information, the paper examines how different factors influence model predictions. The core idea is that a small change in average MRR does not necessarily indicate that the model is independent of visual information. To investigate this, the authors conduct three controlled experiments involving image preprocessing, positive-label filtering, and visual input reassignment. The experiments are conducted using several knowledge graph datasets and the MyGO model. The results show that visual perturbations can affect many individual predictions even when the overall MRR remains nearly unchanged. These findings suggest that aggregate metrics alone may not fully capture a model's dependence on visual information.

Critical strengths
  1. This paper investigates an interesting issue in multimodal knowledge graph completion. I think it is important to examine whether aggregate metrics such as MRR can fully capture the contribution of visual information. The experiments show that visual perturbations can affect a large number of query rankings even when the average MRR remains nearly unchanged. This provides a useful perspective on evaluating multimodal models beyond aggregate performance.

  2. The paper conducts several controlled experiments to examine different factors that may influence evaluation results. Specifically, the authors investigate image preprocessing, positive-label filtering, and visual input reassignment. These experiments help distinguish changes caused by model inputs from those introduced by evaluation protocols. In addition, the zero-visual control provides a useful sanity check for the visual reassignment experiments.

Critical weaknesses
  1. The main MyGO experiment is relatively limited. Each model is trained with only one random seed for 100 epochs, whereas the original MyGO implementation uses a training budget of 1,500 epochs. The paper does not provide evidence that the models have converged. Although five different visual mappings are evaluated, they are all applied to the same trained checkpoints and therefore cannot establish robustness across independent training runs. As a result, it remains unclear whether the observed differences reflect the contribution of visual information or are influenced by incomplete training.

  2. The experiments do not clearly establish whether visual information is actually useful for knowledge graph completion. In the frozen-feature experiments, all visual fusion configurations perform worse than their text-only counterparts on newcomer-target prediction, suggesting that the current fusion approach may not effectively utilize visual information. Furthermore, randomly reassigning visual inputs introduces distribution shifts and does not directly establish whether the model relies on meaningful visual content. Consequently, the current experiments primarily demonstrate sensitivity to visual inputs rather than providing a clear explanation of their actual contribution to model performance.

Questions for authors
  1. Why are the MyGO models trained for only 100 epochs when the original implementation uses 1,500 epochs? Would the results remain consistent with longer training and multiple random seeds?

  2. How can the experiments distinguish the contribution of meaningful visual information from arbitrary visual features or distribution shifts caused by visual reassignment?

  3. Why do all visual fusion configurations perform worse than the text-only models on newcomer-target prediction? Would using a stronger fusion method change the conclusions?

Reviewer_dppvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper examines input dependency and metric fluctuations in multimodal knowledge graph completion through three approaches: correcting transparent-image preprocessing, adjustments to the filtered set of positive examples, and the redistribution of visual information. Key observations include the following: positive and negative shifts at the query level may cancel each other out, resulting in a near-zero net change in mean reciprocal rank (MRR); furthermore, modifications to the filtered evaluation set can alter model rankings even when the underlying scores remain constant. While these diagnostic insights offer practical value, the paper fails to sufficiently demonstrate how the three sets of experiments collectively constitute a scientific contribution that advances the current state of knowledge.

Critical strengths

This submission appears to be an AI-driven research draft that has not undergone a full human review. I can not explicitly provide its strengths before carefully verifying experimental results by authors. (Please see the weakness below.)

Critical weaknesses
  1. How the three diagnostic studies collectively address important scientific questions has not yet been sufficiently demonstrated.

  2. The primary findings largely reiterate existing diagnostic principles, with insufficient new scientific contributions.

  3. The implementation and audit details are quite extensive, which detracts from readability and obscures the core contributions.

This submission appears to be an AI-driven research draft that has not undergone a full human review. The AI Use Statement explicitly says, “This working draft has not yet completed a full human-author review” ( Line 420–421). The authors also acknowledge that establishing a distinct scientific contribution “requires further evidence” ( Line 103–104), and that the new experiments use previously examined validation resources “without independent data confirmation” (Line 237–238). These statements raise serious concerns about whether the work is ready for submission.

Questions for authors
  1. What is the most significant scientific contribution of this paper, and how does it distinguish itself from existing work?

  2. What scientific insight emerges from considering the three diagnostics together?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.