acceptodds
Withdrawn from ICLR 2027

JOLT: Joint Text–Image Search for Grounding Disruption

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Grounded multimodal reasoning systems select image regions as evidence for answering, exposing an attack interface when those selections are observable. We study a reference-assisted, query-bounded attack that jointly modifies questions and applies bounded image perturbations, using only returned boxes as target feedback. The challenge is that localization depends on the complete image–question pair: textual and visual changes must be evaluated together within a limited query budget. We introduce JOLT, which represents question variants and accumulated image perturbations as persistent paired states. Geometric feedback guides their joint search, allowing candidates to retain earlier image changes while exploring new input combinations without model internals or downstream answer feedback. We assess both localization and answer outcomes. Preliminary reported results across six datasets show that JOLT reduces mean answer accuracy from approximately 70.2% before attack to 29.0% after attack. These findings demonstrate downstream degradation in the evaluated systems and identify a robustness concern for multimodal pipelines that expose intermediate visual grounding and use selected regions as evidence for answering.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vision-language models, adversarial attacks, visual grounding, multimodal reasoning, black-box optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fhkjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JOLT, a joint text–image search framework designed to disrupt the grounding process in multimodal models with observable intermediate localization outputs. JOLT exploits the localization feedback exposed by the model to iteratively modify both the input question and the image, aiming to shift the predicted grounding regions and ultimately reduce answer accuracy. Experiments on six Visual-CoT dataset subsets show that JOLT reduces average answer accuracy from 70.2% to 29.0%, outperforming several baseline search strategies. The paper addresses an interesting problem at the intersection of multimodal robustness and agentic interaction with intermediate model outputs. However, the current evaluation does not sufficiently isolate the contributions of question modification, image perturbation, and their interaction, nor does it establish that the observed degradation is primarily caused by disrupting the grounding process rather than changing the effective task. These limitations weaken the evidence for the paper’s central insight and restrict the generalizability of its conclusions.

Critical strengths
  1. An interesting and potentially important research question. The paper investigates whether observable intermediate localization outputs can become an attack surface in multimodal systems. Rather than focusing solely on perturbing the final input–output mapping, it considers how feedback from intermediate states can be exploited to guide an iterative attack. This is a meaningful direction for understanding the robustness of multimodal systems that expose intermediate predictions.
  2. A unified framework for jointly optimizing text and image inputs. JOLT formulates the attack as a joint search problem, using localization feedback to guide modifications to both the question and the image. The framework offers a systematic way to investigate interactions between textual and visual inputs rather than treating them as independent attack channels.
  3. Empirical evidence that feedback-guided search can degrade task performance. The experiments show substantial reductions in answer accuracy compared with clean inputs and several baseline search strategies. The paper also includes cross-model experiments and analyses relating localization displacement to answer accuracy, providing initial evidence that the proposed approach can affect downstream model behavior.
Critical weaknesses
  1. Insufficient evidence that the attack preserves the original task semantics. This is the most important concern. JOLT jointly modifies the question and the image, but the paper does not adequately disentangle the effects of these two components. In particular, it remains unclear whether the accuracy degradation is caused by disruption of the grounding process or by changes to the original task introduced through question rewriting and image perturbation. The perplexity and embedding-similarity analyses are insufficient to establish semantic preservation: linguistic fluency and embedding similarity do not guarantee that the relevant entities, attributes, relations, or answer conditions remain unchanged. The paper should include controlled ablations that separately evaluate question rewriting, image perturbation, and their joint effect, ideally supplemented by human verification of task preservation.
  2. The central mechanism and its causal contribution are not sufficiently established. JOLT optimizes geometric differences between predicted and reference bounding boxes, whereas its ultimate objective is to induce incorrect answers. These objectives are not equivalent. The paper's own results indicate that substantial localization disruption does not necessarily lead to incorrect answers, and that incorrect answers can occur even when localization remains relatively accurate. Consequently, the experiments demonstrate that grounding feedback can guide a search procedure that degrades answer accuracy, but they do not fully establish that disruption of the intermediate grounding state is the primary cause of the degradation. Further analysis is needed to connect changes in localization to downstream answer failures and to determine whether the proposed intermediate-state attack mechanism provides benefits beyond those of ordinary input perturbations.
  3. Limited evaluation scope weakens the generality of the conclusions. The main experiments focus on one Visual-CoT model and six related dataset subsets. The additional experiments cover a small number of other models and use different datasets and sample sizes. This provides preliminary evidence of transferability but is insufficient to establish that the vulnerability generalizes broadly across contemporary multimodal models and reasoning settings. Evaluation on more recent models and more diverse benchmarks would strengthen the paper. It would also be important to distinguish models that expose explicit localization outputs from those that do not, since the proposed attack depends on access to such intermediate feedback.
Questions for authors

See the weaknesses.

Reviewer_7PCqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies attacks on multimodal question answering with Visual-CoT, which uses visual grounding to identify question-relevant regions and answers questions using both the selected crop and the full image. Assuming that the attacker can observe grounding outputs, access reference-region annotations, and modify both the question and the image, the paper proposes JOLT, a joint text–image attack that uses no model internals or downstream answer feedback. JOLT searches over pre-generated question variants and bounded image perturbations, guided by the IoU and center displacement between the returned and reference regions. It generates candidates through component recombination and local mutation, and uses NSGA-II for population selection. On 3,000 samples from six source subsets of Visual-CoT, JOLT reduces mean judge-based answer accuracy from 70.2% to 29.0%.

Critical strengths

S1. The method is well suited to the defined attack setting.

Assuming that the attacker can observe grounding outputs, access reference-region annotations, and modify both the question and the image, JOLT uses the IoU and center displacement between the returned and reference regions to guide joint search over question variants and bounded image perturbations. This provides a concrete search signal without requiring access to target-model internals or downstream answer feedback, making the approach relevant to systems that expose intermediate localization outputs.

S2. The experiments include a broad range of search-strategy controls.

Beyond the single-modality Prompt Search and Image Search baselines, the paper includes joint random search, joint greedy search, and two sequential search strategies. These controls provide useful comparisons of different ways to organize joint search, strengthening the empirical evaluation beyond comparisons with single-modality attacks alone. Independent searches on LLaVA-1.6-Vicuna-7B and Qwen2.5-VL-7B-Instruct provide additional evidence of the method’s applicability beyond the primary VisCoT target.

Critical weaknesses

W1. The strong attacker assumptions limit the method’s practical applicability.

The paper assumes that, in addition to modifying the question and image, the attacker can observe the target model’s returned grounding regions and access reference-region annotations. These conditions may not be jointly satisfied in practical applications, limiting the method’s scope. In particular, for systems that expose only final answers, the attacker cannot directly access intermediate grounding outputs.

In my view, a more practically informative evaluation would use an open-source surrogate grounding model, distinct from the target model, to provide search feedback, explicitly specify how reference regions are obtained or approximated, and then test whether the resulting adversarial inputs remain effective against the target model. The current experiments do not establish effectiveness under these conditions. The dependence on target-model grounding feedback and reference annotations therefore remains an important reservation in my assessment of the method’s practical value.

W2. The experimental validation has the following limitations.

W2.1. Task preservation of the question variants is insufficiently validated.

Although the paper constrains question rewriting through generation instructions and evaluates the variants using perplexity and embedding similarity, these measures do not establish that the rewritten questions preserve the original task. Human verification of the final selected questions is needed to assess whether the target entity, queried attribute or relation, and correct answer remain unchanged. I would like the authors to report task-preservation rates and re-evaluate JOLT and the main baselines on comparable samples after excluding rewrites that alter the original task semantics.

W2.2. The cost comparison is incomplete.

Both the localization-query cap and the proposal cap are set to 200, while duplicate proposals consume only the latter (Section 3.5 and Appendix A.3). Under the stated update rules, JGS has an approximately 20% probability of repeating its current input by skipping the image update and retaining the same question index. In contrast, JOLT may produce a new input pair through recombination even without mutation. Equal nominal caps therefore do not guarantee equal numbers of actual queries or valid scored candidates.

I would like the authors to report the actual localization-query counts, duplicate-proposal rates, and numbers of valid candidates for each method. Reporting runtime under consistent hardware and parallelization settings, with explicit accounting for additional overhead such as question generation, would also enable a more complete assessment of cost.

W2.3. The limited effectiveness on closed-source models requires further explanation.

The attacks reduce accuracy on GPT-4o and Claude Sonnet 3.5 by 7 and 8 percentage points, respectively. I would like the authors to provide a deeper analysis of these results.

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_vcf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JOLT, a black-box adversarial attack that jointly optimizes question variants and image perturbations using visual grounding feedback. The method employs population-based search guided by geometric objectives. Experiments across six datasets demonstrate substantial reductions in answer accuracy compared with several baseline attacks and search strategies.

Critical strengths
  1. Meaningful attack setting. The paper demonstrates that observable grounding outputs can be exploited to disrupt multimodal reasoning without accessing model parameters or downstream answer feedback. This highlights a relevant vulnerability in grounding-based reasoning systems.

  2. Strong empirical results. JOLT is evaluated on six datasets with several joint-search baselines. It reduces average answer accuracy from 70.2% to 29.0%, outperforming joint greedy search by 8.3 percentage points under the same nominal localization-query budget.

Critical weaknesses
  1. Uncertain validity of the attack evaluation. The attack modifies both questions and images, but semantic preservation of the rewritten questions is not adequately verified. Since answers are evaluated against the original questions, the reported accuracy degradation may partly reflect changes in question meaning rather than genuine adversarial failures. This affects the validity of the main experimental conclusion.

  2. Insufficient evidence for the methodological contribution. JOLT combines existing techniques, including NSGA-II, Square-style perturbations, and question rewriting. Without component-level ablations, it is unclear which mechanisms account for the improvements over simpler joint-search strategies. This weakens the evidence for the proposed method's technical contribution.

Questions for authors
  1. Can the authors provide a human evaluation of semantic preservation for the modified questions? How much of the accuracy degradation can be attributed specifically to grounding disruption rather than question modification?

  2. Can the authors provide ablations for population selection, cross-parent recombination, and persistent image perturbations to clarify their individual contributions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.