acceptodds
Under review as a conference paper at ICLR 2027

Thermal Plume Attack: Physics-Inspired Attacks on Infrared Vision-Language Models

Abstract

Infrared vision-language models (IR-VLMs) support classification, image captioning, and visual question answering in low-visibility environments, yet their robustness to non-contact thermal variations remains underexplored. We propose Thermal Plume Attack, a physics-inspired black-box framework that models a rising thermal plume and optimizes its configuration with CMA-ES. A shared temperature field couples refractive displacement with radiance variation, yielding a structured thermal perturbation. Experiments on person/vehicle crops from the FLIR and LLVIP datasets show effective top-1 flips for LanguageBind Thermal and IRGPT. Evaluations across four CLIP backbones adapted to infrared data and matched controls further reveal that this sensitivity is not confined to a single scorer or architecture. These results establish Thermal Plume Attack as a controlled testbed for studying IR-VLM behavior under structured thermal variations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
infrared vision-language models, physics-inspired adversarial attacks, black-box adversarial attacks, adversarial robustness, thermal imaging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z7iFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Thermal Plume Attack, a physics-inspired black-box attack against infrared vision-language models (IR-VLMs). The method simulates thermal plumes using a shared temperature field to generate coupled refractive and radiance perturbations, and employs CMA-ES to optimize five plume parameters without accessing model gradients. Experiments on FLIR and LLVIP demonstrate its effectiveness against LanguageBind Thermal, IRGPT, and several infrared-adapted CLIP models, revealing vulnerabilities of IR-VLMs to structured thermal disturbances.

Critical strengths
  1. This paper introduces a low-dimensional thermal plume attack framework that couples refraction and radiance through a shared temperature field, offering a physically motivated and potentially novel perturbation mechanism.
  2. The evaluation covers multiple infrared-specialized and infrared-adapted VLMs under direct black-box attacks, with a clear distinction between target-specific flips and open-set errors.
Critical weaknesses
  1. This paper lacks essential ablation studies on refraction-only, radiance-only, and uncoupled perturbations, making it difficult to isolate the contribution of the proposed shared temperature field.
  2. The plume renderer relies on simplified physical models and heuristic radiance approximations, without real-world validation using thermal sources and infrared cameras, leaving its physical realism and feasibility uncertain.
  3. The attack assumes access to ground-truth bounding boxes and continuous model scores, raising concerns about practical applicability and generalization.
Questions for authors
  1. Can the authors provide ablations isolating refraction, radiance, and their coupling, along with baseline comparisons under matched LPIPS and perturbation footprints?
  2. Can the authors validate the physical plausibility of the generated perturbations through real infrared imaging experiments or calibrated physical simulations?
  3. How effective is the proposed method when ground-truth bounding boxes or continuous model scores are unavailable?
3 comments

Thank you for recognizing the thermal plume framework and for your suggestions on evidence and presentation. For presentation revisions, see our reply to Reviewer 4v4L (Readability comment).

A1

To isolate both components' roles, we added refraction and radiance ablations. All arms used 1,016 clean-correct LanguageBind Thermal crops (499 person-to-vehicle, 517 reverse), CMA-ES, ~1,000 candidates/crop and caps of LPIPS ≤0.1 and changed-pixel fraction ≤0.5. Refraction changes geometry; radiance approximates IR intensity.

ArmSeed 1, successes/1,016 (ASR)Seed 2, successes/1,016 (ASR)
Refraction only330 (32.48%)312 (30.71%)
Radiance only220 (21.65%)234 (23.03%)
Coupled387 (38.09%)382 (37.60%)

Coupling improved ASR over each component in both seeds.

To compare mechanisms, we added separate baseline evaluations with common caps of LPIPS ≤0.1 and changed-pixel fraction ≤0.5. On the respective subsets, plume ASR was 41.7%; stripe noise and hot/cold blocks achieved 4.17% and 8.33%. Local-pixel and local smooth-random-field CMA-ES each achieved 54.17%. Mean LPIPS and mean changed-pixel fraction varied across methods.

Our plume parameters and IR image formation impose physically meaningful constraints. Real thermal plumes provide the physical basis, with disturbances not readily noticeable in real-world settings. Comparisons should consider ASR, physical plausibility and visibility.

A2

To evaluate physical plausibility, we used a mean temperature field following the Morton-Taylor-Turner (MTT) integral plume model of real plumes' mean buoyant rise and spreading, and added IR recordings from a stove combustion source. Visual comparison showed high similarity between real and simulated disturbances.

To further evaluate model effects, we added tests of all five models on the same recorded IR frames.

The HIKMICRO FALCON FH25 camera was ~1 m from the source between camera and person. All four CLIP encoders are IR-adapted.

All evaluated frames contained a person. Physical-scene ASR is the proportion classified as vehicle in person-versus-vehicle classification.

MetricLanguageBind ThermalOpenAI CLIP L/14EVA-G/14MetaCLIP L/14OpenCLIP B/16
ASR83.33%53.13%53.13%41.67%32.29%

A3

The attack remains feasible. Without ground-truth boxes, automatic detection/segmentation can locate target regions; without continuous scores, a label-only variant can generate perturbations.

To test effectiveness without continuous scores, we added label-only tests on four IR-adapted CLIP models via local inference interfaces with original target crops, achieving 54.03%–74.32% ASR.

The six-parameter variant adds dimensionless radiance gain gg to five plume parameters to control IR intensity-change amplitude. It uniformly samples all six, using only predicted labels as model feedback, and stops at the first candidate satisfying perturbation constraints and receiving the predetermined target label (at most 1,000 evaluations/sample).

Local label-only results

ModelnnLabel-only ASRMean candidate evaluations
OpenAI CLIP L/1476774.32%336.92
EVA-G/141,03269.82%385.22
MetaCLIP L/1484272.80%340.90
OpenCLIP B/161,00554.03%525.39

Both runs use the same nn clean-correct samples per model. Values are two-run means; mean evaluations include all samples, counting failures as 1,000. Overall, our label-only variant remains effective.

Target regions and continuous scores can also be readily acquired. Without ground-truth boxes, YOLO, Grounding DINO or boxes enclosing segmentation masks can locate target regions for cropping and plume-parameter optimization. If the attacked model itself is locally deployable or returns scores, its class scores can guide CMA-ES. Score-interface examples are Hugging Face's InferenceClient.zero_shot_image_classification (labels and scores) and CLIP-as-service's rank() (image-text matching scores). For a label-only attacked interface, use the variant above.

Thank you for your response. It addressed my questions, and I will maintain my score.

We are pleased to hear that our response has addressed your concerns. In the revised manuscript, we will incorporate the additional ablations, baseline comparisons, real infrared imaging results, and label-only evaluations, clarify the alternatives to ground-truth bounding boxes, and further refine the methodological descriptions, experimental details, and overall presentation.

Reviewer_KPdGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes the Thermal Plume Attack, a physics-inspired black-box attack on infrared vision-language models. Inspired by how heat changes in the air cause refractive distortion and thermal radiance in IR imagery, it renders a plume field from a compact five-parameter model onto a clean IR crop. An adaptive search (CMA-ES, 20 candidates x 50 rounds, about 1,000 queries per image) uses the target model's score feedback to find the plume configuration that flips the model's answer to an intended wrong label. The attack is evaluated on infrared-specialized models and infrared-adapted encoders, covering optimization vs. transfer, target-specific vs. open-set flips, and closed-set classification vs. open-ended VQA.

Critical strengths
  1. Interpretable black-box attack space. Replacing unconstrained pixel optimization with a five-parameter plume model keeps the search space small and the optimization manageable, and makes it possible to map back which physical parameters were changed.
  2. Careful and broad experimental protocol. The paper reports query budgets, gating, denominators and limitations more carefully than many attack papers, includes baseline comparisons and generalization checks, and reports insights and failure cases. It also opens a new application scenario for IR VLMs.
Critical weaknesses
  1. The experiments do not support the claimed framework contribution. The primary contribution is presented as a new attack framework, but the contribution of the physics-inspired structure is not isolated: comparisons do not match perturbation magnitude or footprint, and there are no ablations separating refraction, radiance and optimization effects. The evaluation shows per-image synthetic susceptibility rather than a practical attack. As a benchmark the paper would be stronger; as a framework paper the evidence is insufficient.
  2. Physical validity is not established, and the paper is hard to follow. It is not justified that the simulated perturbation is physically valid, i.e., that a real thermal plume would produce a similar effect in infrared images, and there are no real-world or hardware experiments. The introduction does not give a formal definition or an example of what a thermal plume attack is, which makes it hard for readers outside this community to judge its novelty and importance.
Questions for authors
  1. Can you show that real thermal plumes produce perturbations comparable to the simulated ones (e.g., a physical or hardware experiment)?
  2. Can you provide ablations that isolate the contribution of the physics-inspired parameterization from perturbation magnitude and optimization?
  3. Please give a formal definition and a concrete example of the thermal plume attack early in the paper.
3 comments

Thank you for recognizing the low-dimensional, interpretable black-box space and rigorous evaluation, and for your presentation feedback.

A1

Our mean temperature field follows the Morton-Taylor-Turner (MTT) integral plume model of buoyant rise and spreading. We recorded stove-combustion IR imagery; visual comparison showed high similarity to simulated disturbances.

We added real-imaging tests to further assess model responses.

The source lay between person and camera, ~1 m from the HIKMICRO FALCON FH25. All five models classified the same person-containing frames as person/vehicle; all four CLIP encoders are IR-adapted. Physical-scene ASR is the fraction classified as vehicle.

MetricLanguageBind ThermalOpenAI CLIP L/14EVA-G/14MetaCLIP L/14OpenCLIP B/16
ASR83.33%53.13%53.13%41.67%32.29%

These observations provide real-imaging evidence of similarities between simulated and physical thermal-plume disturbances, while the model evaluations reveal recognition vulnerabilities in real thermal-plume scenes.

A2

We added component, optimizer and cap controls on 1,016 clean-correct LanguageBind Thermal crops (499 person→vehicle, 517 reverse; ~1,000 candidates/crop). Means are achieved values.

Components. To isolate components, we added single-component and coupled arms with the main five-parameter model, CMA-ES and caps (LPIPS ≤0.1, changed-pixel fraction ≤0.5). Refraction alters geometry; radiance approximates IR intensity.

ArmSeed 1, successes/1,016 (ASR)Seed 2, successes/1,016 (ASR)
Refraction only330 (32.48%)312 (30.71%)
Radiance only220 (21.65%)234 (23.03%)
Coupled387 (38.09%)382 (37.60%)

Coupling raised ASR by 5.61/6.89 percentage points over refraction and 16.44/14.57 over radiance (seeds 1/2), supporting joint use.

Optimization. To isolate optimizer effects, we added uniform sampling. The six-parameter supplement adds dimensionless radiance gain gg scaling IR intensity changes. Both methods share parameter ranges, budget and caps (LPIPS ≤0.1, changed-pixel fraction ≤0.25). CMA-ES uses 20×50 evaluations, selecting a feasible candidate by target margin with displacement penalty.

MethodSuccesses/1,016 (ASR)Mean LPIPSMean changed-pixel fraction
CMA-ES416 (40.94%)0.05260.1657
Uniform sampling285 (28.05%)0.06380.1639

At similar mean changed-pixel fractions, CMA-ES gave 131 more successes (+12.89 percentage points ASR) and about 17.6% lower mean LPIPS here.

Constraints. To test cap effects, we added comparisons with the same six-parameter model, CMA-ES and budget.

LPIPS capChanged-pixel fraction capSuccesses/1,016 (ASR)Mean LPIPSMean changed-pixel fraction
0.10.5394 (38.78%)0.05950.3264
0.050.5258 (25.39%)0.03850.3084
0.10.25416 (40.94%)0.05260.1657

Tighter LPIPS reduced distortion, retaining 25.39% ASR; the tighter pixel cap cut the mean fraction by ~49%, with 40.94% ASR.

Coupling raised ASR; CMA-ES raised ASR and lowered LPIPS here; attacks remained effective under stricter caps.

A3

We will add this definition and example to Sec. 3.1. For presentation, see our reply to Reviewer 4v4L (Readability comment).

A thermal plume attack is a physics-inspired black-box attack on IR VLMs. For a clean-correct IR crop xx and ground-truth box, a shared temperature field couples local refraction and approximate IR intensity changes via the renderer, xadv=RIR(x;θ)x_{\mathrm{adv}}=R_{\mathrm{IR}}(x;\theta). Five parameters θ=(κ,ρ,δp,ϕ,ζ)\theta=(\kappa,\rho,\delta_p,\phi,\zeta) are condition index, scale, target-relative anchor offset, lean angle and target-region 95th-percentile displacement. CMA-ES optimizes them from continuous class scores under perturbation caps and a candidate budget, without gradients. For predictor ff and preset target ytgt≠ytrue=f(x)y_{\mathrm{tgt}}\ne y_{\mathrm{true}}=f(x), success requires a feasible candidate with f(xadv)=ytgtf(x_{\mathrm{adv}})=y_{\mathrm{tgt}}.

For an IR image correctly classified as person, we target vehicle. Class scores guide iterative parameter updates and rerendering of a stove plume between camera and person; a feasible selected candidate predicted as vehicle succeeds.

Thanks for the rebuttal. It addresses my concerns, and I've updated my rating.

We are pleased to hear that our rebuttal has addressed your concerns, and we appreciate your careful reassessment and constructive feedback. We will incorporate the additional results and clarifications into the revised manuscript and further refine the methodological descriptions, experimental details, and overall presentation.

Reviewer_4v4LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a novel adversarial attack method for Infrared Vision-Language Models.
In a nutshell, the proposed method assumes black-box access to the victim model and attempts to build an adversarial input by adding a rising thermal plume to the image and optimizing its parameters.
This approach does not try to add artificial noise to the original imagery, but rather bases the attack on a physically modeled thermal plume parameterized by a handful of parameters, then searches across these parameters to make the attack successful.
The proposed method showed promising results on a number of infrared benchmarks, outperforming other approaches.

Critical strengths
  • The paper tackles an important and timely problem of VLM adversarial attacks, which has critical use cases in real-world applications.
  • The method is logical, and the authors properly address the motivation behind introducing a new type of attack for infrared vision-language models.
  • The empirical validation features extensive evaluation, demonstrating an improvement over existing approaches.
Critical weaknesses
  • The paper is not necessarily easy to follow. Some models and approaches used in the paper are not necessarily properly explained, which might reduce its reach to an audience less familiar with the domain.
  • In Fig. 3, the provided visual examples suggest that the generated attack is fairly visible in the image, which undermines the overall effectiveness of the adversarial attack.
Questions for authors
  • Can a similar approach be extended to non-infrared images? Is it possible to design similar adversarial attacks on visible-light images?
  • Is there a broader class of perturbations, like plumes, that could be applied in a similar manner?
  • Is it possible to reduce the visibility of the final attacks so they are not immediately apparent in infrared images? How would this affect attack performance?
1 comment

Thank you very much for your valuable feedback on the scope, attack visibility, and presentation of our work. We have addressed your comments individually. In addition, we will continue to refine the organization and presentation of the manuscript to further improve its overall clarity and readability. These refinements will be reflected in the final version.

A1

We focus on infrared images, constructing structured adversarial perturbations against infrared VLMs through physics-inspired plume modeling and parameter optimization. Specifically, a shared synthetic temperature field couples local refractive distortions with approximate thermal-radiance changes (Sec. 3.4). To assess visible-light applicability, we investigated and simulated heated airflow, finding extremely slight disturbances and weak VLM attack effects.

To evaluate NIR applicability, we added tests on NIRPed's miniNIRPed validation subset using IRGPT, trained with NIR data. We simulated NIR perturbations and optimized their parameters, achieving 64.22% untargeted ASR on initially correctly recognized samples.

These results demonstrate adversarial effectiveness in the evaluated NIR recognition task.

Your cross-band suggestion is an interesting research direction. We will explore band-calibrated image-formation models for more bands and target models, and clarify this scope and direction in the revised Limitations and Conclusion.

A2

Thermal turbulence and heated vehicle wakes are broader candidates. In our renderer, a shared temperature field drives both refractive distortion and approximate IR intensity changes. Smoke and gas diffusion are future directions requiring suitable optical models.

Rendering is separate from score-based CMA-ES optimization. The Morton-Taylor-Turner (MTT) model provides the mean field; extensions can reuse this workflow with adapted field models, parameters and constraints.

We choose plumes as an interpretable non-contact mechanism. Sources include vehicle exhaust, space heaters and heat guns (see App. A). We will clarify the scope and outlook in Sec. 3.1 and Secs. 5–6.

A3

Attacks remain effective at lower perceptual distortion.

Visibility in infrared images. Fig. 3 (Sec. 4.2) enlarges the model's IR target crops for display; in full IR frames, the disturbance occupies a small local region. Separately, visible-light camera footage showed it to be almost imperceptible.

Effect of perceptual distortion constraints. On MetaCLIP, increasing the LPIPS penalty weight α\alpha from 0 to 0.5 reduced LPIPS by 82.6% (0.138 to 0.024), while ASR decreased from 80% to 56%. Stronger penalties thus substantially reduce perceptual distortion while retaining effective attacks. See App. F, Tab. 12 for details.

Supplementary evaluation under stricter constraints. To further investigate this trade-off, we conducted additional experiments on LanguageBind Thermal using a six-parameter variant with adjustable radiance gain. CMA-ES optimized the same 1,016 initially correctly classified target crops under two LPIPS caps, with approximately 1,000 candidate evaluations per crop and a changed-pixel fraction cap of 0.5.

LPIPS capMean achieved LPIPSSuccesses/1,016 (ASR)
0.10.0595394 (38.78%)
0.050.0385258 (25.39%)

Tightening the cap gave a paired ASR change of −13.39 percentage points (95% CI −15.93 to −10.87); 25.39% ASR shows effectiveness under stricter constraints.

Readability comment

We have revised and polished the manuscript's motivation, methods and experimental presentation, and adjusted paragraph organization and transitions. Revisions include:

(1) Motivation and organization. We clarified why we focus on IR and how plume imaging physics supports the attack design, improving the links between the research problem, physical mechanism and method.

(2) Methods and key concepts. We will add the attack definition and a person-to-vehicle example in Sec. 3.1 to introduce the attack before technical details. We further explained how a shared temperature field drives both refraction and approximate IR intensity changes, and how rendering connects to parameter optimization. We clarified how the Morton-Taylor-Turner (MTT) model describes the mean temperature field, how turbulence and meandering introduce local variation, and how LanguageBind Thermal/CLIP image-text similarity scores and IRGPT generation log-probabilities guide plume-parameter optimization.

(3) Experiments and interpretation. We clarified each model's role, candidate budgets, perturbation constraints and metrics, linking each experiment's purpose, setup and results to the question it addresses and the conclusion it supports.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.