acceptodds
Withdrawn from ICLR 2027

Learning Transferable Semantic Image Edits for VLM Resource-Exhaustion Attacks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The inference cost of a vision-language model (VLM) service can be amplified by an adversary-supplied image that induces unusually long outputs under a fixed benign prompt. blackHowever, the effectiveness of pixel-space attacks does not consistently generalize across VLM architectures. To improve transfer, we introduce Visual Editing for Token Amplification (VETA), an image-only resource-exhaustion attack that uses reinforcement learning (RL) to train an image-editing planner from the returned responses of one surrogate VLM. The planner specifies changes to visible objects, text, and scene relations; a frozen diffusion editor executes these instructions while preservation rewards encourage retention of the original scene. Two-stage RL first learns reliable output-lengthening edits and then rewards exceptionally long and limit-reaching outputs. All compared methods generate four candidates per source image and use the same evaluation rules. Across six target VLMs, VETA achieves macro amplification when averaging the four candidates and when selecting using only surrogate feedback, without target-side selection. Separately, the maximum over four target-model responses reaches , measuring the longest observed output rather than average request cost. Same-editor controls and ablations assess the contribution of learning, while controlled serving measurements of target-selected candidates show increased latency and energy use and reduced throughput. Learned semantic image editing thus enables transferable resource amplification.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-language models, Multimodal security, Resource-exhaustion attacks, Adversarial transferability, Black-box attacks, Semantic image editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C7M8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VETA, a resource exhaustion attack on vision language models based on semantic image editing, aiming to improve cross model transferability of the attack. Using only black box feedback from a surrogate model, VETA learns semantic editing policies via reinforcement learning that induce target models to generate abnormally long responses.

Critical strengths
  1. Practical attack formulation. VETA learns transferable image-editing policies through reinforcement learning, using only black-box response feedback without access to target model gradients.

  2. Fairly broad evaluation. The attack is tested across diverse VLM architectures, with ablations on training stages and reward components, and measurements of real inference costs.

Critical weaknesses
  1. The attack mechanism lacks sufficient validation. The paper fails to effectively distinguish between anomalies induced by OCR and the effects of other semantic edits. Consequently, it is difficult to determine whether the attack relies primarily on OCR-related recognition errors.

  2. The contribution of reinforcement learning remains unclear. Since standard image editing alone can already significantly amplify the attack’s output, the existing baselines do not adequately control for the content and magnitude of the edits.

Questions for authors
  1. Could the authors provide an ablation that excludes text-based edits and report the attack efficacy of each of the four edit types under the final policy?
  2. Could the authors include a simple semantic editing baseline under the same edit constraints to verify the actual contribution of reinforcement learning?
  3. Minor: Figures 1 and 2 are somewhat stylized. Could the authors consider a simpler, more conventional style for Figures 1 and 2 to make the method easier to follow?
Reviewer_zdvSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Visual Editing for Token Amplification (VETA), an image-only resource-exhaustion attack against vision-language models (VLMs). Unlike existing approaches that optimize pixel-level perturbations, VETA learns a semantic image-editing policy that modifies visible scene content to induce longer model responses under a fixed benign prompt.

The framework consists of a trainable planner, a frozen diffusion-based image editor, and a black-box surrogate VLM. The planner generates structured editing instructions, which are executed by the image editor. The resulting images are evaluated by the surrogate, and response-based rewards are used to optimize the planner through a two-stage reinforcement learning procedure. The first stage encourages consistent output-length amplification, while the second emphasizes long-tail responses, reaching output-token limits, and repetition.

Experiments evaluate transfer from a single surrogate to six open-weight target VLMs and three proprietary models. VETA achieves 7.58× average amplification across six targets when averaging four candidates, 7.24× with surrogate-only candidate selection, and 22.34× when selecting the maximum response on each target. Additional experiments examine serving costs, visual preservation, and the contributions of the training stages and reward components.

Critical strengths

S1. Strong cross-model transfer under a black-box training setting.

The paper addresses an important limitation of prior image-based resource-exhaustion attacks: their effectiveness often deteriorates substantially when transferred across model architectures. VETA trains using response-only feedback from a single surrogate and achieves meaningful amplification across multiple model families without target-side adaptation. The distinction between surrogate-selected and target-selected results is particularly valuable, as the former provides stronger evidence of practical transferability. The additional evaluation on proprietary VLMs further supports the relevance of the proposed attack beyond the training architecture.

S2. A reusable semantic attack policy with meaningful system-level evaluation.

Rather than optimizing perturbations independently for every image, VETA trains a reusable planner that produces scene-conditioned editing instructions. This provides a potentially useful direction for transferable adversarial optimization. The two-stage training procedure also addresses a practical difficulty caused by censored output-length feedback at token limits. Furthermore, the paper goes beyond reporting token counts by measuring latency, energy consumption, and throughput, helping establish the system-level implications of resource-amplifying inputs.

Critical weaknesses

W1. The comparison with pixel-space baselines does not sufficiently isolate the benefit of semantic editing.

The central claim is that semantic image editing enables stronger cross-model transfer than existing pixel-space attacks. However, the compared methods operate under substantially different perturbation constraints. Verbose Images and LingoLoop are restricted to an L-infinity perturbation budget of 8/255, whereas VETA can introduce new objects, alter object states, and modify scene relationships. Consequently, VETA has access to a significantly larger and more expressive modification space.

Although Table 4 acknowledges this distinction, the experimental results do not establish whether the transfer improvement is primarily attributable to the proposed semantic optimization strategy or simply to the greater freedom to modify image content. This is especially important because increased visual complexity may naturally elicit longer descriptions from VLMs. The same-editor controls partially address this issue, but they do not establish an equal-modification-budget comparison with alternative attack strategies.

This limitation weakens the interpretation of the main comparisons and makes it difficult to determine how much of the reported improvement should be attributed to the proposed method rather than the attack space itself.

W2. The contribution of reinforcement learning and the generality of the learned policy require stronger evidence.

The paper attributes much of its effectiveness to learning transferable semantic editing strategies through reinforcement learning. However, the Neutral editing baseline already achieves 4.68× candidate-average amplification, compared with 7.58× for VETA (Table 5). This suggests that ordinary image editing can account for a substantial portion of the observed amplification.

While the VETA-Init comparison and training-stage ablations demonstrate that learning improves effectiveness within the proposed pipeline, they do not fully explain what transferable strategies the policy acquires or whether the improvements generalize beyond the limited evaluation setting. The main experiments use only 32 held-out images, and the primary evaluation fixes the downstream prompt to a generic image-description question. Although the paper evaluates additional paraphrases, these remain within the same task family.

As a result, the evidence supports transfer across model architectures more strongly than transfer across tasks, image distributions, or substantially different prompting conditions. This limits the generality of the learned-policy claim.

Questions for authors

Q1. How much of the observed improvement comes from the larger semantic modification space rather than the learned editing strategy?

Could the authors provide additional controlled comparisons that match image modification magnitude, introduced visual content, or other measures of editing complexity? For example, comparing VETA with non-RL semantic editing methods under similar modification budgets would help isolate the contribution of the learned planner. This clarification would materially affect my assessment of the paper's central claim.

Q2. What specific transferable behaviors does the RL policy learn beyond ordinary semantic editing?

Table 5 shows that Neutral editing already achieves substantial amplification. Could the authors provide a more detailed analysis of the differences between Neutral, VETA-Init, and the trained VETA policy, such as changes in editing patterns, object additions, ambiguity, or response characteristics? This would help establish whether the policy learns generalizable attack strategies rather than simply producing more complex images.

Q3. To what extent does the amplification reflect unnecessary computation rather than legitimate increases in descriptive content?

Since VETA modifies scene content, longer responses may partially result from additional objects or relationships that legitimately require more description. Could the authors further distinguish informative output growth from redundant continuation, for example by analyzing non-repetitive output length, semantic information density, or task-relevant content? The existing repetition analysis and VQA agreement audit are useful, but do not fully resolve this distinction.

Q4. How stable are the transfer results across larger evaluation sets and different task families?

The main experiments use 32 held-out images and primarily focus on image-description prompts. Would the reported amplification persist on substantially larger test sets or on tasks such as visual question answering, OCR, or grounded reasoning, where the expected response format is more constrained? Additional evidence would help clarify the scope of the transferability claim.

Reviewer_hkFyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates resource-exhaustion attacks against Vision-Language Models (VLMs). The objective is to induce the model to generate excessively long responses—thereby increasing inference costs—by modifying the input image while keeping the text prompt unchanged.

The authors propose VETA (Visual Editing for Token Amplification), which employs reinforcement learning to train an image-editing planner. This planner generates semantic editing instructions that are then executed by a frozen diffusion-based image-editing model. The edits encompass objects, text, object states, and spatial relationships, with a "fidelity reward" used to constrain the extent to which the original scene is altered.

Critical strengths
  1. Technical novelty and innovation The paper integrates reinforcement learning-based planning, semantic image editing, and resource-exhaustion attacks, exploring how semantic-level image modifications can enhance the transferability of attacks across different models—an approach that demonstrates a notable degree of innovation. Unlike methods that rely solely on pixel-level perturbations, semantic editing alters the scene content as perceived by the model, offering a fresh perspective for investigating resource-based attacks across diverse visual architectures.

  2. Experimental rigor and validation The paper distinguishes between attack capabilities across various metrics by reporting the candidate average amplification factor, the amplification factor for surrogate models, and the maximum amplification factor for target models. Furthermore, the authors conduct a comprehensive set of experiments, including control experiments using the same image editor, ablation studies on reward components, and system-level evaluations of latency, energy consumption, and throughput.

  3. Significance of contributions By addressing the risks of resource consumption in multimodal inference services, this research sheds light on the relationship between natural semantic variations and anomalous output lengths, providing valuable insights for the security of Vision-Language Model (VLM) services.

Critical weaknesses
  1. Technical limitations or concerns The core issue is that VETA’s threat model does not fully align with that of pixel-level attack methods. The pixel-attack baselines in the paper are constrained by an ℓ∞\ell_\infty perturbation bound of 8/2558/255, whereas VETA allows for direct modifications to objects, text, and the relationships between elements within a scene. Because the permissible scope of image alteration differs fundamentally between these approaches, the attack conditions cannot be considered strictly equitable, even when the number of candidates is held constant. Furthermore, an increase in output length may simply stem from the fact that the modified image contains more information requiring description. For instance, if text, objects, or complex relationships are added to an original image, a longer model-generated description may be a reasonable response to the task, rather than an indicator of anomalous reasoning behavior. Additionally, the second-stage reward explicitly encourages excessively long outputs and lexical repetition; however, the paper does not sufficiently explain the specific mechanisms by which various semantic edits influence the model’s termination behavior or its tendency to generate repetitive content.

  2. Experimental gaps or methodological issues The primary transferability evaluation relies on only 32 unseen images from the GQA dataset. This sample size is too small to robustly support conclusions regarding the generalizability of attacks across different models. The baselines lack comparisons with reinforcement learning (or non-reinforcement learning) methods operating under an equivalent "semantic edit budget." Moreover, the study fails to adequately account for control conditions such as random semantic edits, manually designed semantic edits, or searches conducted via diffusion models.

Furthermore, the experiments on system overhead utilize candidate images selected by the target model itself—a methodology that contradicts the core premise of transfer-based attacks, which should ideally function without relying on feedback from the target model. Consequently, the attack capabilities demonstrated in these experiments may not be representative of true transfer-based scenarios.

The paper also lacks validation across various prompt types—specifically those involving short-form answers, structured Q&A, OCR, counting, and visual reasoning tasks.

  1. Clarity or presentation issues

A clear distinction must be drawn between standard, lengthy textual descriptions and anomalous generations that lead to resource exhaustion.

While the "maximum amplification factor" across four candidates is defined separately in the paper, it could easily be misinterpreted as the average, realized effect of a single transfer-based attack.

It is recommended that the case studies include the original image, the edited image, the full model response, and an analysis of the change in image complexity.

  1. Missing related work or comparisons

The paper should incorporate comparisons with existing research on semantic adversarial editing, diffusion-model-based image attacks, visual prompt injection, and the enhancement of scene complexity. There is a particular need for general image-editing methods that do not rely on external resources to target an object, yet are capable of enhancing scene details or semantic complexity.

Questions for authors

VETA allows for significant semantic editing, whereas pixel-based attack baselines are constrained by an L∞L_\infty norm of 8/2558/255; how is a fair comparison ensured?

To what extent does the increase in output length stem from the introduction of new objects, text, scene complexity, or visual ambiguity?

Could the study incorporate random semantic edits, human-performed semantic edits, and a non-reinforcement-learning search baseline using the same diffusion-based editor?

Does VETA remain effective for tasks such as concise response generation, OCR, counting, and structured question answering?

Within the ultra-long outputs, what proportion consists of valid descriptive information, and what proportion is attributable to repetitive generation or premature/abnormal termination?

To what degree do factors such as editor stochasticity, failure to follow instructions, and image fidelity thresholds influence the experimental results?

Under conditions of real-world concurrent load and server-side output control, do the increases in latency and energy consumption remain significant?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.