SGA: Stamp Guided Black-box Adversarial Attack against Vision-Language Models

Vision-language models (VLMs) have demonstrated remarkable multimodal capabilities, yet remain vulnerable to jailbreak attacks that circumvent safety alignment and elicit harmful answers. Existing visual attacks typically either optimize pixel-level adversarial perturbations with limited explicit semantics and white-box model access, or construct semantically meaningful multimodal inputs through specific content transformation and external model assistance. In this work, we investigate whether a simple visual cue with widely recognized semantics can itself serve as an effective attack medium. To this end, we propose Stamp Guided Attack (SGA), a straightforward black-box adversarial attack framework that overlays an authorization-themed visual stamp onto the original image and augments the textual query with a fixed, scenario-agnostic priming suffix to induce a harmful response. SGA introduces the reusable checkmark as an explicit semantic hint and represents it with a compact, interpretable parameterization over its shape, size, position, color, and opacity. This formulation reduces attack generation to a low-dimensional black-box optimization problem, which is efficiently solved using a mixed-integer Covariance Matrix Adaptation Evolution Strategy (CMA-ES) through only input-output queries to the victim model. Extensive experiments on seven representative VLMs demonstrate the effectiveness and generalizability of SGA, achieving average Attack Success Rates (ASRs) of 78.78% across five open-source models and 54.25% across two closed-source models. The results reveal that VLMs are susceptible to high-level visual semantics introduced beyond the original image content, highlighting the importance of accounting for such cues in multimodal safety evaluation and alignment. Code is available at https://anonymous.4open.science/r/Stamp_Guided_Attack-7EFA/.