acceptodds
Under review as a conference paper at ICLR 2027

On Evaluating the Adversarial Robustness of Foundation Models for Multimodal Entity Linking

Abstract

Multimodal entity linking (MEL) aims to identify a target entity from a set of semantically related candidates using visual and textual evidence. Unlike conventional image classification, MEL requires fine-grained candidate disambiguation, and small changes in cross-modal similarity may reverse the ranking between the correct entity and hard negatives. However, the adversarial robustness of MEL remains underexplored. We conduct a systematic evaluation of visual adversarial robustness under two candidate-constrained settings: Image-to-Text Entity Linking (I2T-EL) and Image+Text-to-Text Entity Linking (IT2T-EL). Our evaluation covers general-purpose vision-language encoders, multimodal large language models, three adversarial attacks, and five MEL benchmarks. The results show that visual perturbations can substantially reduce linking accuracy, although the degradation varies across models, datasets, and attack strengths. We further observe that informative textual context can partially preserve entity-level semantics under visual attacks. Motivated by this observation, we propose LLM-RetLink, a training-free framework that converts visual evidence into multiple discrete entity hypotheses, retrieves candidate-grounded knowledge, filters the retrieved evidence, and performs evidence-aware entity linking. Experiments show that LLM-RetLink improves both clean and adversarial linking accuracy by 0.4%-35.7% across the evaluated settings. Our study provides an adversarial MEL benchmark and highlights a new direction for improving entity-level robustness through discrete semantic grounding.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Entity Linking, Robustness, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oM9sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual adversarial robustness in multimodal entity linking, considering image only and image+text inputs. It evaluates the traditional MEL methods and multimodal LLMs using PGD, APGD, and CW attacks at two strengths. Results on Wikidata-MEL, WikiDiverse, and M³EL reveal substantial vulnerability, with textual context providing partial protection. The paper further proposes LLM-RetLink, a framework requiring no parameter updates that combines multiple entity hypotheses, candidate grounded retrieval, evidence filtering, and final entity selection. Comparative experiments report broad improvements in clean accuracy and accuracy under the evaluated attacks, complemented by component ablations.

Critical strengths
  1. Improved MEL accuracy on both clean and adversarial inputs. LLM-RetLink provides substantial gains in both settings. For example, on M³EL I2T-EL, it improves MiniGPT-4’s clean accuracy from 47.5% to 79.3% and its accuracy under CW-S attacks from 36.9% to 52.7% (Table 3). This dual benefit strengthens the method’s practical value, as the same framework can improve everyday entity linking performance while also mitigating the effects of visual adversarial attacks

  2. A promising approach to improvement without retraining. LLM-RetLink combines discrete entity hypotheses with retrieved semantic evidence to support disambiguation. For MiniGPT-4 on M³EL I2T-EL, it improves clean accuracy from 47.5% to 79.3% and CW-S accuracy from 36.9% to 52.7% (Table 3). Gains across many configurations support semantic grounding as a practically useful direction for improving entity linking under visual perturbations.

Critical weaknesses
  1. Inconsistency between the ablation results and their interpretation. According to Table 4, removing Evidence Reliability Filtering causes the largest decrease in accuracy. However, §4.2.2 states that “the Entity Hypothesis Generation module has the largest impact on performance.” This contradiction creates substantial confusion about the contributions of individual components. The authors should resolve this inconsistency or provide further clarification.

  2. Insufficient description of the ablation configurations. The ablation study removes E.G., C.R., and E.F. individually to assess the contribution of each component. However, the paper does not explain how downstream stages operate when a component on which they depend is removed. For example, when E.G. is removed, how are the queries in Equation (6), which depend on hjh_j, constructed? When C.R. is removed, is E.F. still executed? A complete description of these configurations is essential for reproducing the ablation study.

  3. Unclear adaptive attack evaluation of the complete system. The paper does not clearly specify whether attacks are adapted to the complete system incorporating LLM-RetLink, or whether the evaluation reuses adversarial images generated against the original models or other models. If the paper does not evaluate the adaptive attacks, the current results are insufficient to establish robustness against an attacker who knows the defense and adapts the attack accordingly. The authors should clarify the attack targets and generation procedure, and state whether adaptive attacks against the complete enhanced system were evaluated.

  4. Insufficient analysis of errors introduced by the added stages. The proposed method involves several stages: entity hypothesis generation, candidate retrieval, evidence filtering, and final entity selection. Could these stages introduce additional errors, and how would the method detect them? For identical test inputs, what proportion of samples correctly classified by the original model become incorrect after adding LLM-RetLink? One potential error pathway deserves investigation: a visual misinterpretation produces incorrect hypotheses, which bias retrieval and evidence scoring; misleading evidence then receives a high score, passes filtering, and reinforces the final incorrect prediction.

  5. Missing implementation details for evidence reliability scoring. §3.4.3 provides only semantic descriptions of sctxs_{\mathrm{ctx}}, sents_{\mathrm{ent}}, ssrcs_{\mathrm{src}}, and sconflicts_{\mathrm{conflict}} in Equation (7), without specifying reproducible procedures for computing them. The paper also omits the values and selection criteria for the weights λ1\lambda_1, λ2\lambda_2, λ3\lambda_3, and λ4\lambda_4, and the filtering threshold τ\tau. These settings directly determine which evidence is retained, so their omission limits both understanding and reproducibility.

  6. Insufficient separation of general accuracy gains from reduced sensitivity to perturbations. The experiments show that LLM-RetLink generally improves MEL accuracy on both clean and adversarial inputs. These gains are valuable, but the current evaluation does not sufficiently distinguish improvements in general entity linking ability from reductions in sensitivity to adversarial perturbations. Additional controlled comparisons should quantify how much the proposed hypothesis generation, evidence filtering, and other components contribute beyond simply providing the model with additional reference information, and whether these additional gains specifically improve resistance to attacks.

  7. Numerous writing and presentation errors. The manuscript contains many errors that make it difficult to read. Examples include:

    • 7.1. Citation formatting: Author names are repeatedly duplicated. For example, “Moon et al. (2018) first” appears as “Moon Moon et al. (2018)first,” and “Adjali et al. (2020) introduced” appears as “Adjali Adjali et al. (2020)introduced.”

    • 7.2. Unresolved figure reference: Page 4, line 180 contains “Figure ??”.

    • 7.3. Inconsistent notation: Equation (9) on page 7 does not contain MiM_i, yet the accompanying explanation states, “where MiM_i is the candidate, EE is the correct entity.”

    • 7.4. Inconsistent component names: The four stages in Figure 4 use different names from those in the main text. In particular, the third stage is labeled “Attribute Alignment” in the figure but called “Evidence Reliability Filtering” in the text. The terminology should be made consistent across Figure 4 and the corresponding subsections.

    • 7.5. Other typographical errors: Examples include “Duo to the task” in Figure 4.

Questions for authors
  1. Which model or system are the adversarial examples generated against, and are adaptive attacks evaluated against the complete enhanced system? For each model in Tables 2–3, are adversarial images generated specifically against that model or transferred from another model? After adding LLM-RetLink, does attack generation account for the hypothesis generation, retrieval, and evidence filtering stages? Please clarify the attack targets and generation procedure.

  2. How much additional benefit does the complete method provide beyond directly supplying candidate knowledge? Could the authors compare the complete method with a simple baseline that directly provides candidate knowledge to the MLLM for entity selection, using the same MLLM, the same knowledge source, and comparable information budgets? How much additional improvement do hypothesis generation, evidence filtering, and the other components provide on clean and adversarial inputs, respectively? Do these gains reflect reduced sensitivity to adversarial perturbations, beyond improvements in general entity linking accuracy?

  3. How are the evidence reliability scores computed, and how are the scoring weights and filtering threshold determined? How are s_ctx, s_ent, s_src, and s_conflict in §3.4.3 computed? Please provide reproducible computational procedures and specify the values and selection criteria for the weights λ₁, λ₂, λ₃, and λ₄ and the filtering threshold τ in Equation (7).

Reviewer_XvhQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates visual attacks on multimodal entity linking and proposes LLM-RetLink, a retrieval-enhanced, training-free defense. The problem is relevant, but the evaluation scope is inconsistent and the defense gains are confounded by extra retrieval and inference. I do not recommend acceptance in the current form.

Critical strengths
  • The paper studies a meaningful failure mode: small visual changes can swap the correct entity with a hard negative.
  • It covers multiple model families, attacks, datasets, and both image-only and image-plus-text settings.
  • The module ablation for LLM-RetLink is useful.
Critical weaknesses
  • Inconsistent benchmark scope. The paper claims five datasets, but Tables 2–3 report only Wikidata-MEL, WikiDiverse, and M3EL; Richpedia-MEL and WIKIPerson are missing.

  • Unclear MLLM threat model. Attacks are generated from cosine-similarity label logits, while MLLMs answer through discrete candidate-index prompts. The paper must state whether attacks are white-box, surrogate-transfer, or reused across models.

  • Strong attacks are not shown to be imperceptible. Strong PGD/APGD uses ε=0.2, approximately 51/255, and CW changes its constraint from 20 to 100. No PSNR, SSIM, human study, or perturbation success rate is reported.

  • Robustness remains weak. Qwen2.5-VL drops from 63.5% to 27.7% under CW-S on Wikidata-MEL, and MiniGPT-4 drops from 91.1% to 52.2% in IT2T-EL. Confidence intervals and per-sample attack success are needed.

Questions for authors

See Weakness.

Reviewer_A91gOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the robustness of multimodal entity linking (MEL) under visual adversarial perturbations. It considers two tasks, Image-to-Text Entity Linking (I2T-EL) and Image+Text-to-Text Entity Linking (IT2T-EL), and uses PGD, APGD, and CW attacks to evaluate general-purpose vision-language encoders, dedicated MEL models, and multimodal large language models. The authors further propose LLM-RetLink, a training-free framework that attempts to mitigate visual perturbations.

Critical strengths
  • Adversarial robustness has been extensively studied in image classification and VQA. This paper brings this topic to multimodal entity linking.

  • The design of LLM-RetLink is intuitive.

Critical weaknesses
  • The motivations of LLM-RetLink are not sufficiently explained. The Introduction only briefly describes LLM-RetLink in the contribution list, without explaining the motivation behind its four-stage design.

  • Many terms lack clear definitions or explanations, which makes the paper quite difficult to follow. For example, Sec. 3.1 provides the objectives for PGD and CW, but does not clearly define the two MEL settings (i.e., I2T-EL and IT2T-EL) or explain how the attacks are instantiated for them. The module names in Figure 4 (Entity Perception, Dynamic Retrieval, and Attribute Alignment) are inconsistent with those used in the text: Entity Hypothesis Generation, Candidate-Grounded Retrieval, and Evidence Reliability Filtering. Attribute Alignment is also labeled with “cross-attention” in the figure, but this is not explained anywhere in the main text. Similarly, Figure 2 introduces “Cross-Entity Loss” without defining it.

  • The number of evaluated datasets are inconsistent across the paper. The abstract (Line 21) and several other passages (Lines 81, 90, 94, 177, 210, and 222) state that the evaluation covers five datasets, whereas Sec. 4.1 (Lines 313–314) lists only three public datasets. Results for Richpedia-MEL and WIKIPerson are entirely missing.

  • The ablation analysis contradicts the results in Table 4. Sec. 4.2.2 states that Entity Hypothesis Generation has the largest impact, with an average accuracy drop of 13.6% when removed, followed by Candidate-Grounded Retrieval (9.1%), while Evidence Reliability Filtering has the smallest impact (approximately 6%). However, after carefully checking Table 4, I could not reproduce these numbers. In fact, the table appears to support the opposite conclusion regarding the relative impact of these modules. Could the authors explain this discrepancy? How to get this number?

minor:

  • The notation is inconsistent. For example, cc is defined as a “balancing hyperparameter” in Sec. 3.1, but is referred to as a “constraint” in Sec. 3.2 and Table 1. “Figure ?? ” lacks reference in Line 180.

Overall, given the numerous errors and lack of rigor, I do not think this paper meets the standards of ICLR.

Questions for authors

Please address the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.