- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents DarkLLM, a language-driven framework for generating visual adversarial perturbations. An LLM interprets attack instructions and produces latent representations that condition dedicated noise generators. Through instruction tuning, surrogate-model optimization, and a decoupled training strategy, the framework supports targeted, untargeted, segmentation, and multi-model attacks. The evaluation covers multiple vision and multimodal tasks, including transfer to commercial MLLMs.
Critical strengths
-
Unifying heterogeneous attack settings within a common framework is an important contribution. Although the individual components are largely established, their integration is well motivated and practically relevant.
-
The experiments cover classification, retrieval, captioning, and segmentation. The segmentation results demonstrate strong transferability, while the targeted attack results show improvements over AnyAttack on several evaluated MLLMs.
-
The appendix strengthens the empirical evaluation through analyses of unseen instruction templates, alternative target descriptions, performance on 1,000 images, threshold sensitivity, repeated API evaluations, robustness to defenses, and perceptual quality.
Critical weaknesses
-
The perturbation generator itself appears to offer limited architectural novelty.
-
The paper presents DarkLLM as an efficient and scalable alternative to repeated optimization, but its extensive evaluation of attack effectiveness is not accompanied by a systematic analysis of computational cost.
-
The ablation with a frozen text encoder provides useful evidence for the effectiveness of LLM conditioning. However, since the LLM is fine-tuned while the baseline encoder remains frozen, the comparison does not fully distinguish the benefits of the LLM from those of task-specific adaptation.
-
Table 20 appropriately evaluates two semantic-similarity thresholds. Although DarkLLM outperforms AnyAttack at both thresholds, its average ASR decreases from 55.5% at 0.3 to 11.1% at 0.5. The discussion should therefore distinguish between inducing broadly target-related content and fulfilling the complete target description.
Questions for authors
-
What are the total training GPU-hours and latency for generating a perturbation from a new instruction? How do these compare with representative baselines, and how does reuse across images affect the amortized cost?
-
Can a smaller or trainable text encoder provide a better effectiveness-efficiency trade-off under matched training conditions?
6 comments
Response to Reviewer PdWK (Part 2/2)
Q4: Can a smaller or trainable text encoder provide a better effectiveness-efficiency trade-off under matched training conditions?
A4: We sincerely appreciate this insightful suggestion. To investigate this, we conducted the requested experiment by unfreezing the CLIP ViT-L/14 text encoder and training it under matched conditions. The results are summarized below:
| Conditioning Model | GPT-4o ASR (%) | GPT-5-mini ASR (%) | Gemini-2.5-Flash ASR (%) | Average |
|---|---|---|---|---|
| Frozen Text Encoder | 62.0 | 50.0 | 59.0 | 57.0 |
| Trainable Text Encoder | 63.6 | 55.6 | 64.6 | 61.3 |
| DarkLLM | 67.7 | 58.6 | 63.6 | 63.3 |
As the results indicate, fine-tuning the text encoder yields improvements over the frozen baseline, raising the average ASR from 57.0% to 61.3%. However, DarkLLM maintains a clear advantage in overall attack effectiveness. Although the trainable encoder performs competitively on Gemini-2.5-Flash, DarkLLM consistently outperforms this baseline on both GPT-4o and GPT-5-mini by clear margins, achieving a higher overall average ASR.
A trainable text encoder indeed offers a more computationally efficient alternative. However, relying solely on conventional text encoders fundamentally limits the framework's capacity for deep semantic understanding and flexible interaction, which is our core motivation for exploring an LLM-driven adversarial framework.
Indeed, our framework is motivated by the broader success of foundational LLMs. Today, LLMs have become the cornerstone of numerous successful applications, enabling breakthroughs in areas such as complex visual reasoning (e.g., LLaVA, NeurIPS ’23), image generation (e.g., BAGEL, arXiv ’25; OmniGen2, CVPR ’26), and robotic manipulation (e.g., OpenVLA, CoRL ’24; RT-2, CoRL ’23).
In the adversarial domain, previous works have explored using CLIP (e.g., AnyAttack, CVPR ’25; CGNC, ECCV ’24) or diffusion models (e.g., DiffAttack, TPAMI ’24) for adversarial generation. DarkLLM takes a fundamentally different path. To the best of our knowledge, DarkLLM is the first work in the field to leverage an LLM for generating visual adversarial perturbations.
Crucially, empowering an LLM to perform diverse adversarial attacks is non-trivial. To make this possible, we conducted careful architectural designs and systematic optimizations for each specific attack objective to maximize performance. Furthermore, this architecture provides many possibilities for future exploration. Promising directions include integrating reinforcement learning, incorporating Diffusion Transformers (DiT), and enabling joint outputs of visual perturbations and adversarial text. We believe our framework and empirical results will provide the community with valuable insights.
Reviewer-Author Discussion Period Ends in ONE Day
Dear Reviewer PdWK,
Thank you again for your valuable review. We have provided a detailed response to address the concerns you raised. As the discussion period deadline is approaching, we would be very grateful if you could revisit our clarifications. Please let us know if any questions or ambiguities remain. We are eager to engage in further discussion and would be more than happy to address any additional questions.
We sincerely appreciate your time and effort.
Warm regards,
Authors of Submission 9607
Response to Reviewer PdWK (Part 1/2)
We sincerely thank the Reviewer PdWK for recognizing the practical relevance of our unified framework and extensive evaluations. We address your concerns below.
Q1: The perturbation generator itself appears to offer limited architectural novelty.
A1: We acknowledge the reviewer's question regarding the generator architecture. We intentionally adopted established, lightweight components for the perturbation generator to prioritize training stability and computational efficiency.
We believe that our primary novelty lies at the framework level rather than in low-level neural architectures. By introducing the Embedding as Attack mechanism, DarkLLM successfully leverages an instruction-following LLM as a central semantic controller for adversarial generation, effectively bridging natural language instructions with pixel-level adversarial noise.
In our extensive experiments, we demonstrate that even with this simple generator, DarkLLM achieves state-of-the-art or highly competitive adversarial performance across heterogeneous tasks. We consider exploring more advanced generator architectures, such as Diffusion Transformers (DiT), as a promising direction to further enhance its generative capacity.
Q2: The discussion should distinguish between inducing broadly target-related content and fulfilling the complete target description.
A2: We sincerely appreciate the reviewer's careful reading and the constructive suggestion. We agree that distinguishing between these semantic levels is crucial. The scoring criteria in Appendix Table 11 provide a concrete basis for this distinction:
The decrease in average ASR from 55.5% to 11.1% explicitly demonstrates that enforcing strict subject-level alignment is substantially more challenging than inducing broad semantic redirection. Nevertheless, DarkLLM consistently outperforms AnyAttack across both difficulty levels.
Following your advice, we will incorporate this criteria-based discussion into Section C.2 to explicitly clarify the differences between broad semantic redirection and strict subject-level alignment.
Q3: What are the total training GPU-hours and latency for generating a perturbation from a new instruction? How do these compare with representative baselines, and how does reuse across images affect the amortized cost?
A3: We appreciate the opportunity to clarify the computational requirements. Developing the unified DarkLLM framework requires an upfront, one-time training of approximately 1,040 GPU-hours (130 hours on 8 H200 GPUs). Once trained, generating a perturbation for a new instruction requires only a single forward pass, taking just 0.25 seconds on a single H200 GPU.
To contextualize this, it is essential to compare the time complexities between different adversarial paradigms. Sample-wise targeted attacks (e.g., M-Attack) scale with O(X×Y) complexity, as they require iterative backpropagation for every combination of X images and Y targets. For instance, optimizing a perturbation for a single image-target pair takes roughly 40 seconds using M-Attack on a single H200 GPU.
In contrast, the computational advantage of DarkLLM emerges from perturbation reuse. Because DarkLLM generates an image-agnostic Universal Adversarial Perturbation (UAP), its generation cost scales with O(Y) during inference. The 0.25-second generation cost is incurred only once per target instruction, and applying this perturbation across X images requires no further optimization.
To illustrate the practical efficiency, consider a standard robustness evaluation setup using X=1,000 images and just Y=100 target instructions. A sample-wise approach would require optimizing 100,000 independent pairs, resulting in approximately 1,111 GPU-hours of computation. For DarkLLM, the total computational cost is 1,040 GPU-hours for training plus merely 25 seconds for inference. Therefore, X and Y do not need to be significantly large. DarkLLM becomes computationally advantageous even at moderate evaluation scales.