- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies visual adversarial robustness in multimodal entity linking, considering image only and image+text inputs. It evaluates the traditional MEL methods and multimodal LLMs using PGD, APGD, and CW attacks at two strengths. Results on Wikidata-MEL, WikiDiverse, and M³EL reveal substantial vulnerability, with textual context providing partial protection. The paper further proposes LLM-RetLink, a framework requiring no parameter updates that combines multiple entity hypotheses, candidate grounded retrieval, evidence filtering, and final entity selection. Comparative experiments report broad improvements in clean accuracy and accuracy under the evaluated attacks, complemented by component ablations.
Critical strengths
-
Improved MEL accuracy on both clean and adversarial inputs. LLM-RetLink provides substantial gains in both settings. For example, on M³EL I2T-EL, it improves MiniGPT-4’s clean accuracy from 47.5% to 79.3% and its accuracy under CW-S attacks from 36.9% to 52.7% (Table 3). This dual benefit strengthens the method’s practical value, as the same framework can improve everyday entity linking performance while also mitigating the effects of visual adversarial attacks
-
A promising approach to improvement without retraining. LLM-RetLink combines discrete entity hypotheses with retrieved semantic evidence to support disambiguation. For MiniGPT-4 on M³EL I2T-EL, it improves clean accuracy from 47.5% to 79.3% and CW-S accuracy from 36.9% to 52.7% (Table 3). Gains across many configurations support semantic grounding as a practically useful direction for improving entity linking under visual perturbations.
Critical weaknesses
-
Inconsistency between the ablation results and their interpretation. According to Table 4, removing Evidence Reliability Filtering causes the largest decrease in accuracy. However, §4.2.2 states that “the Entity Hypothesis Generation module has the largest impact on performance.” This contradiction creates substantial confusion about the contributions of individual components. The authors should resolve this inconsistency or provide further clarification.
-
Insufficient description of the ablation configurations. The ablation study removes E.G., C.R., and E.F. individually to assess the contribution of each component. However, the paper does not explain how downstream stages operate when a component on which they depend is removed. For example, when E.G. is removed, how are the queries in Equation (6), which depend on , constructed? When C.R. is removed, is E.F. still executed? A complete description of these configurations is essential for reproducing the ablation study.
-
Unclear adaptive attack evaluation of the complete system. The paper does not clearly specify whether attacks are adapted to the complete system incorporating LLM-RetLink, or whether the evaluation reuses adversarial images generated against the original models or other models. If the paper does not evaluate the adaptive attacks, the current results are insufficient to establish robustness against an attacker who knows the defense and adapts the attack accordingly. The authors should clarify the attack targets and generation procedure, and state whether adaptive attacks against the complete enhanced system were evaluated.
-
Insufficient analysis of errors introduced by the added stages. The proposed method involves several stages: entity hypothesis generation, candidate retrieval, evidence filtering, and final entity selection. Could these stages introduce additional errors, and how would the method detect them? For identical test inputs, what proportion of samples correctly classified by the original model become incorrect after adding LLM-RetLink? One potential error pathway deserves investigation: a visual misinterpretation produces incorrect hypotheses, which bias retrieval and evidence scoring; misleading evidence then receives a high score, passes filtering, and reinforces the final incorrect prediction.
-
Missing implementation details for evidence reliability scoring. §3.4.3 provides only semantic descriptions of , , , and in Equation (7), without specifying reproducible procedures for computing them. The paper also omits the values and selection criteria for the weights , , , and , and the filtering threshold . These settings directly determine which evidence is retained, so their omission limits both understanding and reproducibility.
-
Insufficient separation of general accuracy gains from reduced sensitivity to perturbations. The experiments show that LLM-RetLink generally improves MEL accuracy on both clean and adversarial inputs. These gains are valuable, but the current evaluation does not sufficiently distinguish improvements in general entity linking ability from reductions in sensitivity to adversarial perturbations. Additional controlled comparisons should quantify how much the proposed hypothesis generation, evidence filtering, and other components contribute beyond simply providing the model with additional reference information, and whether these additional gains specifically improve resistance to attacks.
-
Numerous writing and presentation errors. The manuscript contains many errors that make it difficult to read. Examples include:
-
7.1. Citation formatting: Author names are repeatedly duplicated. For example, “Moon et al. (2018) first” appears as “Moon Moon et al. (2018)first,” and “Adjali et al. (2020) introduced” appears as “Adjali Adjali et al. (2020)introduced.”
-
7.2. Unresolved figure reference: Page 4, line 180 contains “Figure ??”.
-
7.3. Inconsistent notation: Equation (9) on page 7 does not contain , yet the accompanying explanation states, “where is the candidate, is the correct entity.”
-
7.4. Inconsistent component names: The four stages in Figure 4 use different names from those in the main text. In particular, the third stage is labeled “Attribute Alignment” in the figure but called “Evidence Reliability Filtering” in the text. The terminology should be made consistent across Figure 4 and the corresponding subsections.
-
7.5. Other typographical errors: Examples include “Duo to the task” in Figure 4.
-
Questions for authors
-
Which model or system are the adversarial examples generated against, and are adaptive attacks evaluated against the complete enhanced system? For each model in Tables 2–3, are adversarial images generated specifically against that model or transferred from another model? After adding LLM-RetLink, does attack generation account for the hypothesis generation, retrieval, and evidence filtering stages? Please clarify the attack targets and generation procedure.
-
How much additional benefit does the complete method provide beyond directly supplying candidate knowledge? Could the authors compare the complete method with a simple baseline that directly provides candidate knowledge to the MLLM for entity selection, using the same MLLM, the same knowledge source, and comparable information budgets? How much additional improvement do hypothesis generation, evidence filtering, and the other components provide on clean and adversarial inputs, respectively? Do these gains reflect reduced sensitivity to adversarial perturbations, beyond improvements in general entity linking accuracy?
-
How are the evidence reliability scores computed, and how are the scoring weights and filtering threshold determined? How are s_ctx, s_ent, s_src, and s_conflict in §3.4.3 computed? Please provide reproducible computational procedures and specify the values and selection criteria for the weights λ₁, λ₂, λ₃, and λ₄ and the filtering threshold τ in Equation (7).