acceptodds
Under review as a conference paper at ICLR 2027

One Man's Cure Is Another's Poison: Turning JPEG Compression Into Backdoor Triggers for Vision-Language (Action) Models

Abstract

JPEG compression is ubiquitous in practical vision-language model (VLM) pipelines for storage, transmission, and input preprocessing, and has also been evaluated as a lightweight defense against adversarial perturbations. Existing studies largely assume that compression suppresses malicious behavior or represents a distortion that adversarial examples should survive. In this work, we find that such a defense may instead be exploited by the adversary as a trigger for attacks, i.e., the input appears benign but induces attacker-specified behavior after JPEG compression in vision-language(-action) tasks. We propose JPEG-triggered Adversarial attaCK (JACK), a two-stage method that achieves pre-compression stealthiness and post-compression attack effectiveness without modifying the victim model or the JPEG codec. JACK first constructs a candidate image whose JPEG-compressed version induces the target behavior, and then restores benign behavior for the uncompressed counterpart within the same JPEG quantization cell, thereby preserving the attack-inducing compressed representation. Extensive experiments on targeted image classification, visual question answering, multimodal jailbreaking, and multi-view Vision-Language-Action (VLA) policy hijacking validate the effectiveness of the proposed method.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
VLM, adversarial attack, jailbreak attack, JPEG compression, stealthiness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Wv8tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes JACK, a two-stage attack that exploits JPEG quantization to make images elicit benign behavior before compression and attacker-specified behavior afterward in vision-language and robotic models.

Critical strengths

A clever two-stage method that turns JPEG compression into an attack trigger. Broad experiments across vision-language tasks and robotic manipulation.

Critical weaknesses
  1. JPEG compression is not a convincing trigger: (i) unlike a pixel pattern that can be placed on a real road sign, it lacks a clear real-world counterpart explaining how an attacker would cause the victim to receive the compressed input. (ii) JPEG compression lacks trigger secrecy, since anyone can apply it without attacker-specific knowledge, which weakens the claimed practicality of the attack.
 unintentional activation by benign users can expose a backdoor and introduces the False Trigger Rate to measure this issue.
  1. The attack assumes white-box access and a known JPEG quality factor, limiting its practicality, with robustness to different JPEG codecs, resizing, and other preprocessing operations untested.
Questions for authors

Please see the weaknesses above. My main concern is Weakness 1. If the authors provide a convincing justification addressing this concern, I would consider raising my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Wv8t (Part 1/2)

We thank the reviewer for the comments, and for recognizing the strengths of our work:

"A clever two-stage method that turns JPEG compression into an attack trigger. Broad experiments across vision-language tasks and robotic manipulation."

Please find the point-by-point response below.

W1: (i) Trigger Deployment; (ii) Trigger Secrecy; (iii) Unintended Activation (FTR).

Response: We thank the reviewer for this insightful comment. JACK is an inference-time, transformation-triggered adversarial attack against frozen VLMs, rather than a training-time model backdoor. Neither the victim model nor the JPEG codec is modified.

(i) Trigger Deployment. JPEG compression is a widely deployed operation, not an artificial trigger requiring attacker installation. JPEG is a widely used image compression standard [1], and image-sharing platforms routinely compress or re-encode uploaded images [2, 3]. Moreover, JPEG compression has been employed or evaluated as an adversarial defense in classical vision systems [4–6] and VLM security research [7, 8], while recent work highlights compressed-image understanding in deployed VLMs [9].

JACK exploits this existing processing step rather than introducing it. For example, an attacker may submit a crafted image to a service that automatically re-encodes it into JPEG before VLM inference. The attacker controls only the image, while the receiving pipeline performs compression. Related feasibility is supported by Duan et al. (AAAI 2024) [10] on JPEG-conditioned backdoors and Tan et al. (TMLR 2026) [11] on transformation-triggered adversarial examples. Unlike training-time backdoors, JACK requires no modification of the victim model.

(ii) Trigger Secrecy and Stealthiness. JACK does not rely on JPEG compression being secret; its effective trigger is the combination of an attacker-specific optimized image and JPEG processing. Compressing arbitrary benign images does not systematically induce attacker-specified behavior.

In the reviewer's road-sign analogy, a conventional adversarial patch may trigger malicious behavior immediately upon observation, whereas JACK additionally requires JPEG compression for activation. The crafted image elicits benign or task-consistent behavior before compression and attacker-specified behavior afterward. Thus, JPEG-conditioned activation provides behavioral stealthiness against pre-compression inspection, consistent with the motivation for conditionally activated physical attacks [12].

(iii) Unintended Activation (FTR). Unlike persistent model backdoors that may activate on arbitrary compressed inputs, JACK requires an individually optimized adversarial image. We assess unintended target-specific activation on unmodified inputs using clean-input SASR, which measures benign-to-target behavioral switching caused by JPEG compression alone. It is 0.0% on ImageNet (Table 2), 0.0% on MS-COCO VQA (Table 5 in the original submission; Table 11 in the revised manuscript), and 1.25%/1.22% on MM-SafetyBench (Table 3) for LLaVA/Qwen2.5-VL, respectively. These results indicate that unintended target-specific activation is rare on the evaluated clean inputs.

Overall, JACK reveals a distinct security risk: existing preprocessing or defensive operations can activate adversarial behavior concealed during direct, pre-compression inference.

[1] JPEG Committee, “100th JPEG Meeting Celebration Announcement,” JPEG.org, 2023.

[2] F. Bertini et al., “Are Social Networks Watermarking Us or Are We (Unawarely) Watermarking Ourself?,” Journal of Imaging, 2022.

[3] C. Pasquini et al., “Media Forensics on Social Media Platforms: A Survey,” EURASIP Journal on Information Security, 2021.

[4] C. Guo et al., “Countering Adversarial Images Using Input Transformations,” ICLR, 2018.

[5] N. Das et al., “SHIELD: Fast, Practical Defense and Vaccination for Deep Learning Using JPEG Compression,” KDD, 2018.

[6] Z. Liu et al., “Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examples,” CVPR, 2019.

[7] L. Bailey et al., “Image Hijacks: Adversarial Images Can Control Generative Models at Runtime,” ICML, 2024.

[8] Y. Lan et al., “Phi: Preference Hijacking in Multi-modal Large Language Models at Inference Time,” EMNLP, 2025.

[9] Z. Zhang et al., “Benchmarking and Enhancing VLM for Compressed Image Understanding,” ICML, 2026.

[10] Q. Duan et al., “Conditional Backdoor Attack via JPEG Compression,” AAAI, 2024.

[11] Y. Tan et al., “Transform-Triggered Adversarial Examples,” TMLR, 2026.

[12] W. Zhu et al., “TPatch: A Triggered Physical Adversarial Patch,” USENIX Security, 2023.

AuthorsOpenReview ↗

Response to Reviewer Wv8t (Part 2/2)

W2: Robustness to different JPEG codecs and resizing.

Response: We have conducted additional experiments covering zero-shot transferability across JPEG quality-factor mismatches, chroma-subsampling mismatches, and image resizing (without re-optimization), alongside multi-quality joint optimization. Using ImageNet and LLaVA-v1.5-7B with 100 adversarial images per source quality factor (Q∈{10,30,50,70,90}Q\in\{10,30,50,70,90\}), we compare JACK against JPEG-aware PGD and Tan-Joint, reporting the mean SASR (%) across all five source qualities (and across tested offsets or resolutions where applicable):

Transfer SettingJACKJPEG-aware PGDTan-Joint
Quality ±156.448.338.3
Quality ±515.014.84.5
JPEG 4:2:226.825.88.2
JPEG 4:4:49.46.02.2
Resize (384 / 448 / 512)24.721.515.9

(i) JPEG Quality-Factor Mismatch. At ±1\pm 1, JACK achieves 56.4% SASR, exceeding JPEG-aware PGD and Tan-Joint by 8.1 and 18.1 percentage points, respectively. At ±5\pm 5, JACK retains the highest mean SASR despite substantial degradation across all methods.

(ii) JPEG Configuration Mismatch. When images optimized under 4:2:0 subsampling are tested under 4:2:2 and 4:4:4, JACK consistently outperforms both baselines.

(iii) Resizing and Preprocessing Order. When JPEG compression precedes resizing from 336×336336\times336 to 384, 448, and 512, JACK achieves 24.7% mean SASR, outperforming JPEG-aware PGD and Tan-Joint by 3.2 and 8.8 percentage points, respectively.

These results demonstrate partial cross-setting transferability but also reveal performance degradation under larger mismatches. This is expected because the original JACK is designed for a specified JPEG setting without explicitly optimizing cross-setting transferability.

(iv) Preliminary Exploration of Multi-Quality Joint Optimization. To investigate whether this limitation can be mitigated, we further explore extending JACK to jointly optimize a single perturbation for multiple JPEG qualities, using Q30 and Q50 as an illustrative setting.

Unlike Tan-Joint, which jointly optimizes the pre-JPEG benign loss and both post-JPEG targeted losses in one stage, JACK retains its two-stage framework. Stage I minimizes the mean JPEG-targeted loss across Q30 and Q50 to obtain a shared candidate x~adv\tilde{x}_{\mathrm{adv}}, aiming to trigger the target response under both qualities. Stage II constructs the intersection of their quantization cells, Cjoint=C30(x~adv)∩C50(x~adv)\mathcal{C}_{\mathrm{joint}}=\mathcal{C}_{30}(\tilde{x}_{\mathrm{adv}})\cap\mathcal{C}_{50}(\tilde{x}_{\mathrm{adv}}). For feasible samples requiring refinement, JACK optimizes the pre-JPEG benign objective subject to the joint cell constraint and the ℓ∞\ell_\infty perturbation budget using Dykstra-style alternating projection, aiming to preserve adversarial behavior under both qualities.

MethodOptimized QualitiesTest Q30Test Q50Joint SASR (Q30 & Q50)
Tan-Joint (Joint)Q30 + Q5051.0%63.0%51.0%
JACK (Joint)Q30 + Q5087.0%89.0%87.0%

JACK achieves 87.0% joint SASR, outperforming Tan-Joint by 36.0 percentage points. These preliminary results demonstrate the feasibility of extending JACK through multi-quality joint optimization and quantization-cell intersection, providing a promising direction for mitigating its dependence on a single JPEG quality setting.

These experiments and discussions are included in Appendix D of the revised manuscript.

AuthorsOpenReview ↗

Gentle Follow-Up on Our Rebuttal

Dear Reviewer Wv8t,

Thank you for your time and valuable feedback. As the discussion deadline approaches, we would greatly appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns. If so, we would be grateful if you could reconsider your rating accordingly.

If any concerns remain, we would be happy to provide further clarification. Thank you again for your consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer Wv8t,

With the discussion deadline approaching, please let us know if we have resolved all of your concerns or if you require any further clarification. We look forward to your feedback.

Best regards, The Authors

Reviewer_Q789OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies adversarial attacks triggered by JPEG compression and proposes JACK, which preserves benign behavior before compression while inducing attacker-specified outputs afterward. The method first optimizes the post-compression attack objective, then restores benign behavior on the uncompressed input within the same JPEG quantization cell to preserve the post-compression attack. Experiments on image classification, visual question answering, multimodal jailbreaking, and robotic grasping show that JACK achieves higher Stealthy Attack Success Rates than the evaluated baselines.

Critical strengths
  • The paper investigates the risk of preprocessing-triggered behavioral changes, explicitly requiring both benign pre-compression behavior and successful post-compression attacks. This is a clear and relevant security objective.
  • The method exploits the many-to-one nature of JPEG quantization to separate attack activation from benign behavior restoration, closely connecting its design to the problem.
  • Experiments cover two VLMs and one VLA model and report Pre-JPEG Safe Rate, Post-JPEG Attack Success Rate, and their joint success rate, providing a reasonably comprehensive picture of conditional attack performance.
Critical weaknesses
  1. The central claim of quantization-cell preservation is not sufficiently validated. Stage II is claimed to keep the compressed representation strictly unchanged, but Appendix C specifies only three alternating-projection iterations per refinement step, and the final pixel-domain projection may violate the DCT-domain constraints again. The paper does not report constraint residuals, the fraction of changed quantization symbols, or whether the actual JPEG-compressed images remain identical. It therefore remains unclear whether the finite-step implementation satisfies this key design requirement.
  2. Applicability to practical JPEG pipelines remains unclear. The attack assumes knowledge of the quality factor, but the actual quantization results also depend on the codec implementation and processing configuration. The results across different quality factors do not establish how a fixed adversarial input performs when the compression parameters differ from those assumed during optimization. Claims about deployment settings therefore need to be more carefully qualified.
  3. The comparisons do not sufficiently isolate the benefit of the quantization-cell constraint. JPEG-aware PGD does not require benign pre-compression behavior, while Tan-Joint uses a joint objective with fixed equal weights. The paper lacks both a sweep over the joint-objective weights and a two-stage optimization baseline without the quantization-cell constraint. It is therefore unclear how much of the improvement comes from quantization-cell preservation rather than the optimization order or differences in how the objectives are balanced.
  4. The jailbreak evaluation criteria do not adequately support the reported safety conclusions. The main text describes the attack target as a harmful response beginning with “Sure, here is,” but does not sufficiently explain how evaluation distinguishes target-prefix matching from actual harmful content generation, or how safe responses are identified. Since the prefix alone does not establish a successful jailbreak, interpreting the reported Post-ASR and SASR requires clearer evidence of semantic evaluation.
Questions for authors
  1. Does benign pre-compression behavior generalize across queries? Stage II optimizes for a given query and benign target response. Does the same adversarial image retain benign behavior under other reasonable queries, or is its stealthiness largely limited to the query used during optimization?
  2. How is SASR computed for the VLA experiments? Appendix C describes five tasks with ten unseen layouts per task, but the percentage increments in Table 4 do not clearly correspond to direct aggregation over these fifty layouts. Please clarify the evaluation unit, the number of repeated trials, and whether pre- and post-compression rollouts are paired using identical initial layouts.
  3. What is the computational cost of constructing the attacks? In addition to model gradient updates, Stage II requires projections, and the VLA setting introduces constraints across multiple views and trajectories. Please report wall-clock time and computational budgets for each method to clarify the overhead of the two-stage procedure.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Q789 (Part 1/3)

We thank the reviewer for recognizing the relevance of our security objective, principled JPEG-quantization-based attack design, and comprehensive VLM/VLA evaluation with complementary conditional metrics. Our point-by-point responses follow.

W1: Empirical Validation of JPEG-Cell Preservation.

Response: JACK's key innovation is the JPEG quantization-cell constraint, which aims to preserve the Stage-I post-compression adversarial representation, with Dykstra-style projection as its numerical implementation.

To quantify JPEG symbol preservation under actual compression, we evaluate 100 ImageNet images using LLaVA-v1.5-7B, comparing single-pass projection with three Dykstra-style cycles across four JPEG quality factors.

QChanged Coeff. ↓ (1/3 cycles)SASR ↑ (1/3 cycles)
300.05394%/0.04326%92%/93%
500.13104%/0.11227%90%/92%
700.35215%/0.33599%89%/92%
903.42970%/2.17273%77%/83%

Changed Coeff. denotes the percentage of quantized JPEG coefficients differing between Stage-I and Stage-II compressed representations.

Three cycles consistently reduce coefficient changes and improve SASR, providing direct evidence of effective JPEG symbol preservation. At Q=30, 99.96% of coefficients remain unchanged; at Q=90, coefficient changes decrease by 36.6%, with a 6-percentage-point SASR improvement.

Importantly, JACK does not require perfect symbol equality: the constraint effectively preserves the Stage-I representation and post-compression attack effectiveness. Our contribution lies in incorporating this constraint into adversarial optimization, rather than claiming exact convergence within fixed projection cycles. We have clarified the role of Dykstra-style projection in Sections 3.3–3.4 and added these results to Section 4.6 of the revised manuscript.

W2: Transferability across JPEG settings.

Response: We evaluate JPEG quality-factor and chroma-subsampling mismatches without re-optimization on ImageNet using LLaVA-v1.5-7B, with 100 images per source quality (Q∈{10,30,50,70,90}Q\in\{10,30,50,70,90\}). We compare JACK, JPEG-aware PGD, and Tan-Joint, reporting mean SASR (%) across five source qualities (and both offsets for quality mismatches).

Transfer SettingJACKJPEG-aware PGDTan-Joint
Quality ±156.448.338.3
Quality ±515.014.84.5
JPEG 4:2:226.825.88.2
JPEG 4:4:49.46.02.2

(i) JPEG Quality-Factor Mismatch. At ±1\pm 1, JACK achieves 56.4% SASR, exceeding JPEG-aware PGD and Tan-Joint by 8.1 and 18.1 percentage points, respectively. At ±5\pm 5, JACK retains the highest mean SASR despite substantial degradation across all methods.

(ii) JPEG Configuration Mismatch. When images optimized under 4:2:0 subsampling are tested under 4:2:2 and 4:4:4, JACK consistently outperforms both baselines.

These results demonstrate partial cross-setting transferability but also reveal performance degradation under larger mismatches. This is expected because the original JACK is designed for a specified JPEG setting without explicitly optimizing cross-setting transferability.

(ii) Preliminary Exploration of Multi-Quality Joint Optimization. To investigate whether this limitation can be mitigated, we further explore extending JACK to jointly optimize a single perturbation for multiple JPEG qualities, using Q30 and Q50 as an illustrative setting.

Unlike Tan-Joint, which jointly optimizes the pre-JPEG benign loss and both post-JPEG targeted losses in one stage, JACK retains its two-stage framework. Stage I minimizes the mean JPEG-targeted loss across Q30 and Q50 to obtain a shared candidate x~adv\tilde{x}_{\mathrm{adv}}, aiming to trigger the target response under both qualities. Stage II constructs the intersection of their quantization cells, Cjoint=C30(x~adv)∩C50(x~adv)\mathcal{C}_{\mathrm{joint}}=\mathcal{C}_{30}(\tilde{x}_{\mathrm{adv}})\cap\mathcal{C}_{50}(\tilde{x}_{\mathrm{adv}}). For feasible samples requiring refinement, JACK optimizes the pre-JPEG benign objective subject to the joint cell constraint and the ℓ∞\ell_\infty perturbation budget using Dykstra-style alternating projection, aiming to preserve adversarial behavior under both qualities.

Method (Q30 + Q50)Test Q30Test Q50Joint SASR
Tan-Joint51.0%63.0%51.0%
JACK87.0%89.0%87.0%

JACK achieves 87.0% joint SASR, outperforming Tan-Joint by 36.0 percentage points. These preliminary results demonstrate the feasibility of extending JACK through multi-quality joint optimization and quantization-cell intersection, providing a promising direction for mitigating its dependence on a single JPEG quality setting.

We have included these experiments and discussions in Appendix D of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Q789 (Part 2/3)

W3: Unisolated contribution of cell preservation and insufficient analysis of joint-objective weighting.

Response: To isolate the JPEG quantization-cell constraint's contribution, we conduct an additional ablation on the same 500 ImageNet images from Section 4, using LLaVA-v1.5-7B with Q=30Q=30 and ϵ=8/255\epsilon=8/255. We use a matched two-stage baseline without the cell constraint, retaining JACK's Stage-I initialization, objectives, and 300/200-step Stage-I/II budget, with other settings unchanged.

MethodPre-SR ↑Post-ASR ↑SASR ↑
Stage I only81.6%95.6%77.8%
Two-stage w/o cell constraint93.2%82.0%82.0%
JACK (Ours)94.8%94.2%90.2%

Without the cell constraint, Stage II improves Pre-SR from 81.6% to 93.2% but reduces Post-ASR from 95.6% to 82.0%, revealing a benign-restoration/post-JPEG-effectiveness trade-off. In contrast, JACK achieves 94.8% Pre-SR while retaining 94.2% Post-ASR (only 1.4 percentage points below Stage I), yielding an 8.2-percentage-point SASR improvement over the matched baseline. With matched objectives, stage order, and update budget, this supports the contribution of cell preservation beyond two-stage optimization. This ablation appears in revised Section 4.6.

Our main comparisons use Tan-Joint's original equal weighting (1:1) [1]. We also evaluate five pre-/post-JPEG loss-weight ratios {10:1,2:1,1:1,1:2,1:10}\{10:1,2:1,1:1,1:2,1:10\} on 100 ImageNet images using LLaVA-v1.5-7B at Q=90Q=90.

MethodPre-SR ↑Post-ASR ↑SASR ↑
Tan-Joint (10:1)87%56%50%
Tan-Joint (2:1)80%86%69%
Tan-Joint (1:1)79%88%70%
Tan-Joint (1:2)75%95%72%
Tan-Joint (1:10)73%97%70%
JACK (Ours)91%88%83%

Loss weighting shifts the pre-/post-JPEG trade-off but does not consistently improve Tan-Joint's SASR. All tested configurations achieve 50%–72% SASR, below JACK's 83%, demonstrating that JACK's advantage cannot be attributed to joint-objective weighting. Together with the matched two-stage ablation, these results support the contribution of the JPEG quantization-cell constraint beyond optimization order and loss balancing.

[1] Y. Tan et al., “Transform-Triggered Adversarial Examples,” TMLR, 2026.

W4: Clarification and Semantic Validation of Jailbreak Evaluation

Response:

1. Original evaluation protocol. We originally employed a keyword-based heuristic, commonly used in prior jailbreak studies [2, 3]. Pre-SR was determined by refusal-related keywords (e.g., "I'm sorry," "I cannot," "can't assist," and "as an AI"), while Post-ASR required no refusal keywords, a nontrivial response (at least 20 characters, excluding low-information responses), and a benchmark-specific keyword match when available. Importantly, "Sure, here is" is an optimization target, not the jailbreak success criterion.

2. Additional semantic evaluation. We evaluate pre- and post-JPEG responses using the official HarmBench multimodal classifier [4], following recent multimodal jailbreak studies [5, 6]. Unlike keyword matching, HarmBench assesses substantive fulfillment of the specified harmful behavior. Accordingly, Pre-SR measures pre-JPEG non-fulfillment, Post-ASR measures post-JPEG harmful-behavior fulfillment, and SASR requires both conditions simultaneously.

ModelMethodPre-SR ↑Post-ASR ↑SASR ↑
LLaVA-v1.5-7BJPEG-aware PGD41.4467.2913.77
Tan-Joint68.5259.2634.09
JACK69.6867.2941.29
Qwen2.5-VLJPEG-aware PGD80.3645.1927.23
Tan-Joint94.7145.4542.52
JACK98.4845.1944.45

All values are percentages, evaluated using the HarmBench multimodal classifier.

JACK achieves the highest SASR on both models, supporting the effectiveness of quantization-cell-preserving refinement in restoring pre-JPEG non-fulfillment while preserving post-JPEG attack effectiveness. We have clarified the heuristic protocol in Section 4.2 and Appendix C and added the HarmBench results to Section 4.4.

[2] X. Liu et al., “AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models,” ICLR, 2024.

[3] T. Ma et al., “Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models,” ICCV, 2025.

[4] M. Mazeika et al., “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal,” ICML, 2024.

[5] K. Grimes et al., “Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing,” ICLR, 2025.

[6] K. Cui et al., “Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models,” ICLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer Q789 (Part 3/3)

Q1: Transferability across queries.

Response: To assess cross-query transferability, we conducted additional experiments on ImageNet using LLaVA-v1.5-7B, with 100 adversarial images per JPEG quality factor (Q∈{10,30,50,70,90}Q\in\{10,30,50,70,90\}). All adversarial images are optimized using the original query, "Classify the main object in this image using a single concrete noun." We then evaluate the same images without re-optimization under two semantically similar queries:

  • P1: "What is the main object in this picture? Answer with one noun."
  • P2: "Identify the primary object shown. Reply with only its class label."

We compare JACK against JPEG-aware PGD and Tan-Joint, reporting the mean SASR (%) across all five JPEG quality factors.

QueryJACKJPEG-aware PGDTan-Joint
P168.459.649.0
P255.454.439.6

JACK consistently outperforms both baselines under the two alternative queries. These results demonstrate that JACK's attack effectiveness is not strictly tied to the optimization query and exhibits stronger cross-query transferability than the baselines, although its effectiveness remains sensitive to prompt wording. We have incorporated these additional results into Appendix D of the revised manuscript.

Q2: Clarification of the VLA Evaluation Protocol.

Response: SASR is computed at the time-step level, rather than the rollout or layout level. As detailed in revised Appendix C, following TRAP [7], for each of five tasks, we train a task-specific patch on 25 layouts and evaluate it on 10 unseen layouts. For each unseen layout, we collect clean trajectories under benign and target instructions, retaining the first half of the shorter trajectory as evaluation records.

For each record, we obtain benign and target bounding-box references from GraspVLA using the same observation and robot state, and evaluate the same patched observation before and after JPEG compression, with the robot state and benign instruction fixed. Pre-SR and Post-ASR require average IoU > 0.5 across both camera views against their respective references, while SASR requires both conditions on the same record.

This yields 24, 24, 25, 26, and 24 records (123 total) across the five tasks. Metrics are micro-averaged over paired records, with no repeated rollouts. For example, JACK at Q=30 achieves 121/123 (98.37%) Pre-SR, 100/123 (81.30%) Post-ASR, and 98/123 (79.67%) SASR.

[7] Z. Huang et al., “TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches,” ICML, 2026.

Q3: Computational Cost.

Response: We additionally evaluate wall-clock time and peak GPU memory on a single NVIDIA V100 (32 GB) GPU, using 500 gradient updates per method (300/200 for JACK's Stages I/II). We test 100 ImageNet images on LLaVA-v1.5-7B and report per-scene costs for VLA hijacking on GraspVLA.

SettingMethodWall-clock timePeak GPU memory
ImageNetPGD181.14 s/image16.36 GiB
ImageNetTan-Joint364.67 s/image19.37 GiB
ImageNetJACK186.79 s/image16.55 GiB
VLATRAP5.92 min/scene11.82 GB
VLATan-Joint11.63 min/scene14.62 GB
VLAJACK16.58 min/scene11.93 GB

On ImageNet, JACK incurs only 3.1% runtime and 1.2% peak-memory overhead over PGD, while being 48.8% faster than Tan-Joint, which requires two forward passes per update for its joint loss. For VLA, JACK incurs higher runtime due to multi-view and trajectory constraints, but its peak memory remains comparable to TRAP (11.93 vs. 11.82 GB) and 18.4% lower than Tan-Joint. Overall, JACK achieves low computational overhead for VLM attacks and competitive memory efficiency for VLA hijacking, despite increased VLA runtime. These results are included in revised Appendix E.

Reviewer_uxrnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JACK constructs inputs that induce benign responses before JPEG compression and target responses afterward, without changing the victim model or codec. A white-box attacker knowing the JPEG quality first optimizes the compressed branch, then restores benign behavior within the same quantization cell. A shared-patch extension targets multi-view VLA policies. Experiments on classification, VQA, multimodal jailbreaking, and simulated manipulation report improved paired conditional success over the evaluated baselines.

Critical strengths
  1. Relevant security question: The paper examines whether ordinary preprocessing can activate dormant adversarial inputs in frozen generative VLMs and embodied policies.

  2. Structured attack design: JPEG's many-to-one quantization provides a clear motivation for restoring the original response while retaining a compressed target representation.

  3. Useful evaluation breadth: The study considers four tasks, two VLMs, several JPEG qualities, distortion measures, and VLA layouts held out from patch optimization. SASR explicitly requires both desired behaviors on the same input.

Critical weaknesses
  1. Potentially infeasible VLA constraints:

Equation (26) changes only pixels within the patch, but Eqs. (30)–(33) constrain every JPEG coefficient of the full image to an interval with half-width γ around its rounded Stage-I value. With γ = 0.45, an untouched background block can have a fixed normalized coefficient k + 0.47. It rounds to k, yet cannot satisfy the upper bound k + 0.45 for any patch. Therefore, the stated VLA feasible set can be empty even though the original rounding cells contain the Stage-I patch. No restriction to affected coefficients or other feasibility treatment is specified. The authors should provide a feasible formulation consistent with the implementation.

  1. Unverified preservation of the compressed representation:

The central claim is that refinement keeps the JPEG representation strictly unchanged. However, Algorithm 1 uses only three projection cycles per update, ending with pixel clipping, although Section 3.3 acknowledges that one projection can violate the other constraint. No convergence criterion or Stage-I versus final symbol comparison establishes this invariant. Moreover, Eq. (15) is not generally a Euclidean RGB projection because its transform includes nonorthogonal color conversion, so ordinary Dykstra guarantees do not automatically apply. Standard-JPEG ASR supports behavioral success but cannot establish identical compressed representations. The authors should directly validate preservation under the actual JPEG pipeline.

  1. Unspecified ground truth for jailbreak success:

Section 4.2 requires a safe or refusal response before compression and harmful content afterward, while Appendix C specifies an affirmative opening as the optimization target. However, the paper does not describe the harmfulness evaluator, refusal criterion, or validation used to assign these labels. An affirmative opening alone does not establish harmful fulfillment of a request. The qualitative examples show harmful continuations, but the aggregate safety-specific ASR and SASR in Tables 3, 6, and 7 remain unsupported by a documented semantic labeling procedure. The authors should provide independently validated safety and harmfulness judgments for both branches.

  1. Unclear evaluation counts and aggregation:

Appendix C specifies five VLA tasks with ten unseen layouts each and averages performance across tasks. One binary rollout per layout would yield two-percentage-point increments, whereas Table 4 reports 98.37%, 79.67%, and 0.81%. Likewise, 500 VQA image–question pairs would yield 0.2-point increments under one binary evaluation per pair, but Table 5 includes 1.3% and 21.5%. Legitimate repetitions, filtering, or other averaging could explain these values, yet the relevant units and counts are not reported. The authors should report success and trial counts with the aggregation procedure.

  1. Unisolated contribution of cell preservation:

The comparison includes a useful JPEG-aware target-only attack and Tan-Joint, which adapts the simultaneous objective of prior work [1]. However, JACK also changes the schedule to 300 target-generation steps followed by 200 benign-restoration steps. No matched two-stage control starts from the same candidate and restores benign behavior without the cell constraint. Thus, scheduling and cell preservation change together. The reported gains support the complete method but do not isolate the benefit of its central quantization-cell mechanism. The authors should include a matched two-stage control without the cell constraint.

Questions for authors
  1. How is the VLA constraint set kept feasible when an unchanged background coefficient lies outside the tightened interval in Eq. (32)?

  2. How often are the final quantized symbols and decoded JPEG images exactly identical to those of the Stage-I candidate after three projection cycles?

  3. How are safe, refusal, and harmful responses labeled, and how were these judgments independently validated?

  4. What are the actual evaluation units, success counts, repetitions, and aggregation rules behind Tables 4 and 5?

  5. How does JACK compare with the same two-stage schedule and initialization when benign restoration omits the cell constraint?

5 comments
AuthorsOpenReview ↗

Response to Reviewer uxrn (Part 1/3)

We thank the reviewer for recognizing the relevance of our security question, the principled attack design leveraging JPEG quantization, and the comprehensive evaluation across diverse tasks, models, compression settings, and unseen VLA layouts.

Summary of Revisions & New Experiments: All five concerns raised by the reviewer pertain either to notational/reporting clarifications (W1, W4) or to empirical validations that we have now fully conducted (W2, W3, W5). Specifically, every requested experiment—direct symbol-preservation verification (W2), independent semantic evaluation via HarmBench (W3), and a matched two-stage ablation (W5)—consistently confirms our core claims and further strengthens the paper. Please find our point-by-point responses below.

W1 & Q1: Potentially infeasible VLA constraints in Eq. (32).

Response: Our VLA constraint set is feasible; the issue raised by the reviewer stems purely from a notational omission in Eq. (32) in Appendix rather than our actual algorithm or implementation.

In our implementation, tightened JPEG-cell constraints are applied exclusively to the set of DCT blocks affected by the patch, denoted as Bi,v\mathcal{B}_{i,v}, rather than to all blocks across the entire image. For untouched background blocks outside Bi,v\mathcal{B}_{i,v}, the DCT coefficients are independent of the patch and naturally retain their exact Stage-I quantization indices without requiring interval tightening. Consequently, an unchanged background coefficient lying outside the tightened γ\gamma-interval never occurs in the constraint set and never introduces infeasibility.

We have revised Eqs. (32) of the revised Appendix to explicitly restrict the constraints to Bi,v\mathcal{B}_{i,v}, aligning the mathematical formulation with our implementation.

W2 & Q2: JPEG-Cell Preservation and Projection Effectiveness

Response: Direct validation under the actual JPEG pipeline confirms that 3-cycle Dykstra-style projection preserves 97.83% to 99.96% of Stage-I quantized symbols across quality factors while consistently boosting SASR.

We clarify two key methodological and empirical points:

  1. Role of 3-cycle projection under non-orthogonal transforms and clipping: While exact convergence of Dykstra's algorithm relies on exact orthogonal projections, we employ three alternating projection cycles followed by pixel clipping as a computationally efficient numerical approximation to enforce the JPEG quantization-cell constraint alongside the ℓ∞\ell_\infty box constraint. The core contribution of JACK is formulating and incorporating this quantization-cell constraint into two-stage adversarial optimization, rather than claiming exact mathematical convergence within three steps. To avoid overclaiming, we have refined wording such as "strictly unchanged" in Section 3 to accurately reflect high-fidelity approximate preservation.
  2. Direct symbol-preservation verification: To directly answer Q2 under the actual JPEG pipeline (including color conversion, rounding, and clipping), we evaluated 100 ImageNet images on LLaVA-v1.5-7B, comparing single-pass projection against three Dykstra-style cycles across four quality factors QQ:
QDykstraChanged Coeff. ↓Identical Coeff. ↑SASR ↑
30No0.05394%99.94606%92%
303 cycles0.04326%99.95674%93%
50No0.13104%99.86896%90%
503 cycles0.11227%99.88773%92%
70No0.35215%99.64785%89%
703 cycles0.33599%99.66401%92%
90No3.42970%96.57030%77%
903 cycles2.17273%97.82727%83%

Here, Changed Coeff. (and Identical Coeff.) measures the percentage of quantized JPEG coefficients that differ (or remain identical) between the Stage-I candidate and the final Stage-II image.

Despite non-orthogonal color conversion and final pixel clipping, three Dykstra-style cycles consistently reduce coefficient drift across all quality factors—preserving up to 99.96% of quantized symbols at Q=30Q=30, and cutting symbol changes by 36.6% (with a +6% SASR gain) at Q=90Q=90. Moreover, these results show that JACK does not require 100% bitwise symbol identity to preserve post-JPEG adversarial behavior. We have updated Sections 3.3–3.4 and added these validation results to Section 4.6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer uxrn (Part 2/3)

W3 & Q3: Clarification and Semantic Validation of Jailbreak Success Criteria

Response: We have documented our exact heuristic labeling protocol and conducted an independent semantic validation using the official HarmBench classifier [6], where JACK continues to achieve the highest SASR across all evaluated models.

1. Documented original evaluation protocol. Following common practice in prior jailbreak studies [1–5], our original metrics used a rule-based evaluator: Pre-SR requires the presence of standard refusal prefixes (e.g., "I'm sorry," "I cannot," "can't assist," "as an AI"), while Post-ASR requires the absence of refusal keywords, a substantive response length (≥20\ge 20 characters, excluding low-information outputs), and a benchmark-specific keyword match when applicable. Importantly, the affirmative prefix "Sure, here is" serves solely as the Stage-I optimization target, never as the evaluation criterion for jailbreak success.

2. Independent semantic validation via HarmBench. To provide rigorous semantic judgments beyond keyword heuristics, we re-evaluated both pre- and post-compression responses using the official HarmBench multimodal evaluator [6], following recent MLLM red-teaming standards [7, 8]. HarmBench directly judges whether a full response substantively fulfills the harmful request. Here, Pre-SR is the percentage of pre-JPEG responses judged as safe/refusal, Post-ASR is the percentage of post-JPEG responses judged as harmful fulfillment, and SASR requires both conditions simultaneously on the same sample:

ModelMethodPre-SR ↑Post-ASR ↑SASR ↑
LLaVA-v1.5-7BJPEG-aware PGD41.4467.2913.77
Tan-Joint68.5259.2634.09
JACK (Ours)69.6867.2941.29
Qwen2.5-VLJPEG-aware PGD80.3645.1927.23
Tan-Joint94.7145.4542.52
JACK (Ours)98.4845.1944.45

All values are percentages evaluated by the HarmBench multimodal classifier.

Under independent semantic evaluation, JACK consistently outperforms all baselines in SASR (+7.20% on LLaVA-v1.5-7B and +1.93% on Qwen2.5-VL over the strongest baseline), confirming that our quantization-cell constraint genuinely restores pre-compression safety while preserving post-compression harmful fulfillment. We have detailed both protocols in Section 4.2 and Appendix C, and added the HarmBench results to Section 4.4.

[1] X. Liu et al., “AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models,” ICLR, 2024.

[2] T. Ma et al., “Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models,” ICCV, 2025.

[3] B. Yi et al., “Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models,” ICLR, 2025.

[4] A. Basani and X. Zhang, “GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs,” NeurIPS, 2025.

[5] C. Ziakas et al., “Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Experts,” ACL, 2026.

[6] M. Mazeika et al., “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal,” ICML, 2024.

[7] K. Grimes et al., “Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing,” ICLR, 2025.

[8] K. Cui et al., “Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models,” ICLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer uxrn (Part 3/3)

W4 & Q4: Unclear evaluation counts and aggregation.

Response: The reported percentages are derived from 123 VLA time-step records and 400 VQA image–question pairs, respectively, with one paired evaluation per record and no repeated trials. We now explicitly report the evaluation units, exact success counts, and aggregation procedures.

1. Table 4 (VLA hijacking; Table 5 in the revised manuscript): Denominator = 123 time-step records. Following the evaluation protocol of TRAP [9], the evaluation unit is an individual trajectory time-step record rather than a full layout rollout. For each of the 5 tasks, a patch is trained on 25 layouts and tested on 10 unseen layouts. On each unseen layout, we collect clean trajectories under benign and target instructions, take the shorter trajectory, and retain its first 50% of time steps as evaluation records. Because LIBERO does not provide ground-truth bounding boxes, we obtain paired benign and target CoT bounding-box references by querying GraspVLA on the clean observation and robot state.

  • Counts & Aggregation: Across the 5 tasks (10 unseen layouts each), this protocol yields 24, 24, 25, 26, and 24 time-step records, totaling exactly 123 records. Each record is evaluated once before and after JPEG compression (success = mean IoU > 0.5 across the two camera views against the respective reference), and metrics are micro-averaged over all 123 records.
  • Exact Fractions: The percentages cited by the reviewer correspond directly to exact integer counts over 123: for JACK at Q=30Q=30, Pre-SR = 121/123 (98.37%), Post-ASR = 100/123 (81.30%), and SASR = 98/123 (79.67%); for TRAP at Q=70Q=70, SASR = 1/123 (0.81%).

2. Table 5 in the Appendix (MS-COCO VQA; Table 11 in the revised Appendix): Denominator = 400 image–question pairs. Appendix C in the initial submission contained a typo ("500 pairs"); the actual evaluation benchmark contains exactly 400 pairs ( yielding 0.25%-point increments). Each pair is evaluated once without repetition.

  • Exact Fractions: On LLaVA-v1.5-7B, JACK achieves Pre-SR = 324/400 (81.0%), Post-ASR = 108/400 (27.0%), and SASR = 86/400 (21.5%). The 1.3% value cited by the reviewer corresponds to 5/400 (1.25%, rounded to 1.3%), and JPEG-aware PGD's 28.3% corresponds to 113/400 (28.25%, rounded to 28.3%).

We have clarified the evaluation units and aggregation procedures in Appendix C of the revised manuscript.

[9] Z. Huang et al., “TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches,” ICML, 2026.

W5 & Q5: Unisolated contribution of cell preservation.

Response: When compared against a strictly matched two-stage control sharing the exact same initialization and 300+200 step schedule, our quantization-cell constraint independently improves Post-ASR by +12.2 percentage points and overall SASR by +8.2 percentage points.

Following the reviewer's suggestion, we implemented a Matched Two-Stage Baseline (w/o Cell Constraint) on the exact same 500 ImageNet images evaluated in Section 4 (LLaVA-v1.5-7B, Q=30Q=30, ϵ=8/255\epsilon=8/255). This control starts from the identical Stage-I candidate (300 steps) and performs the exact same 200 steps of Stage-II benign-behavior restoration, omitting only the quantization-cell projection:

MethodSchedulePre-SR ↑Post-ASR ↑SASR ↑
Stage I only300 steps81.6%95.6%77.8%
Matched Two-Stage (w/o cell constraint)300 + 200 steps93.2%82.0%82.0%
JACK (Ours, w/ cell constraint)300 + 200 steps94.8%94.2%90.2%

The ablation cleanly isolates the two mechanisms:

  1. Unconstrained Stage-II restoration degrades post-JPEG attack success: While running 200 unconstrained restoration steps improves Pre-SR from 81.6% to 93.2%, it overwrites the JPEG quantization cells, causing Post-ASR to drop sharply by 13.6 percentage points (from 95.6% to 82.0%).
  2. Quantization-cell preservation is essential: Under the identical 300+200 schedule, enforcing the JACK cell constraint preserves Stage-I Post-ASR (94.2% vs. 82.0%, a +12.2% gain) while achieving even higher Pre-SR (94.8%), resulting in a decisive +8.2 percentage-point gain in SASR (90.2% vs. 82.0%). We have incorporated this controlled ablation into Section 4.6 of the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of New Experiments and Revisions

Dear Area Chair and Reviewers,

We thank all reviewers for their constructive feedback and for acknowledging our core strengths: (1) investigating a clear and relevant security risk where ordinary preprocessing activates dormant attacks in frozen VLMs and embodied policies (uxrn, Q789), (2) proposing a clever, principled two-stage design exploiting JPEG's many-to-one quantization (uxrn, Q789, Wv8t), and (3) conducting broad, comprehensive evaluations across vision-language tasks, unseen robotic manipulation layouts, and paired conditional metrics (uxrn, Q789, Wv8t).

We have conducted all requested experiments, clarified every notational/reporting issue, and updated the manuscript. Below summarizes how each concern is resolved.

1. Isolating Quantization-Cell Preservation (uxrn W5, Q789 W3)

  • Matched Two-Stage Ablation: On 500 ImageNet images (LLaVA-v1.5-7B), we compare JACK vs. an identical two-stage baseline (same Stage-I init, objectives, 300+200 schedule; no cell constraint): unconstrained Stage-II drops Post-ASR from 95.6% to 82.0%; enforcing JACK's cell constraint preserves Post-ASR at 94.2% (+12.2%) and boosts SASR from 82.0% to 90.2% (+8.2%).
  • Tan-Joint Weight Sweep: Across five loss ratios (10:1 to 1:10), Tan-Joint achieves 50%–72% SASR, consistently below JACK's 83%, confirming gains stem from cell preservation rather than scheduling or weighting.

2. Direct Verification of Symbol Preservation (uxrn W2, Q789 W1)

  • Actual JPEG Pipeline: Across 100 ImageNet images under the real JPEG pipeline, 3-cycle Dykstra-style projection preserves 97.83%–99.96% of Stage-I quantized symbols (Q∈{30,50,70,90}Q \in \{30,50,70,90\}). At Q=90Q=90, 3 cycles cut coefficient drift by 36.6% and raise SASR by +6% (77% → 83%).
  • Refined Claims: Sections 3.3–3.4 now clearly describe 3-cycle projection as an efficient numerical approximation rather than claiming exact bitwise convergence.

3. Independent Semantic Jailbreak Validation (uxrn W3, Q789 W4)

  • HarmBench Evaluation: Besides documenting our standard keyword protocol ("Sure, here is" is solely an optimization target; evaluation uses refusal keywords like "I'm sorry," "can't assist," "as an AI"), we re-evaluated both branches with the official HarmBench multimodal classifier. JACK achieves the highest SASR on both LLaVA-v1.5-7B (41.29%, +7.20% over Tan-Joint) and Qwen2.5-VL (44.45%, +1.93%), confirming genuine semantic safety before compression and harmful fulfillment afterward.

4. Practicality, Transferability, and Efficiency (Wv8t W1–W2, Q789 W2, Q1, Q3)

  • Threat Model & FTR: Unlike model backdoors, JACK is an inference-time attack jointly triggered by an adversarial image and routine JPEG re-encoding—requiring no trigger secrecy while adding behavioral stealthiness against pre-compression inspection. Clean-input SASR (unintended activation, FTR) is 0.0% on ImageNet/VQA and 1.25%/1.22% on MM-SafetyBench.
  • Zero-Shot Transfer: Under Quality ±1 (56.4%), Quality ±5 (15.0%), subsampling 4:2:2 (26.8%) and 4:4:4 (9.4%), resizing (24.7%), and paraphrased queries (P1: 68.4%, P2: 55.4%), JACK consistently outperforms JPEG-aware PGD and Tan-Joint.
  • Multi-Quality Extension: Joint optimization via cell intersection over Q=30Q=30 and Q=50Q=50 reaches 87.0% joint SASR, outperforming Tan-Joint (51.0%) by +36.0%.
  • Compute Cost: On ImageNet (V100), JACK adds only 3.1% runtime and 1.2% peak memory over PGD while running 48.8% faster than Tan-Joint. On VLA, peak memory is comparable to TRAP and 18.4% lower than Tan-Joint.

5. Formulation & Evaluation Clarifications (uxrn W1, W4, Q789 Q2)

  • Feasible VLA Formulation: Eq. (32) (App. C) now restricts γ\gamma-constraints to patch-affected DCT blocks, matching our implementation and eliminating background infeasibility.
  • Exact Counts: Appendix C now specifies micro-averaging over 123 paired VLA time-step records across 5 tasks (e.g., 121/123 = 98.37%, 98/123 = 79.67%) and 400 MS-COCO VQA pairs (fixing the "500" typo; e.g., 86/400 = 21.5%).

Revised PDF Summary

Integrated into the revised PDF: Secs. 3.3–3.4 (approximate preservation), Secs. 4.2 & 4.4 (jailbreak protocols & HarmBench), Sec. 4.6 (symbol preservation & matched/weight ablations), and Apps. C–E (Eq. (32) fix, exact counts, transferability/multi-quality tests, and runtime/memory benchmarks).

Request for Consideration in the Final Assessment

Despite posting our rebuttal 4 days early with 2 follow-ups, we received no further feedback. We sincerely hope the Area Chair will carefully consider this updated empirical evidence in the final decision rather than relying solely on pre-rebuttal concerns.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.