Thermal Plume Attack: Physics-Inspired Attacks on Infrared Vision-Language Models

Infrared vision-language models (IR-VLMs) support classification, image captioning, and visual question answering in low-visibility environments, yet their robustness to non-contact thermal variations remains underexplored. We propose Thermal Plume Attack, a physics-inspired black-box framework that models a rising thermal plume and optimizes its configuration with CMA-ES. A shared temperature field couples refractive displacement with radiance variation, yielding a structured thermal perturbation. Experiments on person/vehicle crops from the FLIR and LLVIP datasets show effective top-1 flips for LanguageBind Thermal and IRGPT. Evaluations across four CLIP backbones adapted to infrared data and matched controls further reveal that this sensitivity is not confined to a single scorer or architecture. These results establish Thermal Plume Attack as a controlled testbed for studying IR-VLM behavior under structured thermal variations.