SimAttack: A Simple and Effective Transferable Untargeted Adversarial Attack on Large Vision-Language Models

Large Vision-Language Models (LVLMs) have demonstrated strong multimodal understanding and reasoning capabilities, yet remain vulnerable to transferable adversarial attacks, underscoring the need for rigorous robustness evaluation under black-box settings. Existing transfer-based untargeted attacks often rely on ground-truth textual annotations to obtain transferable optimization signals. However, requiring such annotations for every attacked input imposes a strong practical assumption, since they are generally unavailable in real-world scenarios. To address this limitation, we propose SimAttack, a simple and effective transfer-based black-box untargeted attack that derives transferable optimization signals solely from surrogate vision encoder representations without ground-truth textual supervision. By leveraging visual representations from different local regions and encoder layers, SimAttack reduces reliance on a limited subset of surrogate encoder representations and improves adversarial transferability. Extensive experiments demonstrate that our method consistently outperforms existing state-of-the-art attack methods across heterogeneous LVLMs and diverse downstream tasks.