acceptodds
Under review as a conference paper at ICLR 2027

Behavior-Aligned Visual Optimization for Jailbreaking Vision-Language Models

Abstract

Gradient-based visual jailbreak attacks on vision-language models (VLMs) often optimize image perturbations for a fixed iteration budget using token-level target objectives that only indirectly capture harmful response behavior. We introduce BAVO, a behavior-aligned visual jailbreak framework that learns a lightweight behavior predictor from internal VLM representations to estimate response-level behavior. The predictor provides a differentiable behavioral signal for image optimization and enables instance-adaptive iterate selection without response generation or evaluation at every update. Across 917 samples from multimodal jailbreak benchmarks and four target VLMs, BAVO improves ELITE-based attack success rate (E-ASR) by 16.68–33.26 percentage points and HarmBench ASR by 10.25–24.64 points over affirmative-target PGD. BAVO also reduces the mean optimization steps from 300 to 32.7–161.2, yielding – wall-clock speedups. The performance improvements persist across three defenses, while response-behavior analysis shows shifts from refusal and intermediate responses toward successful responses. These results demonstrate that aligning visual optimization with response-level behavior improves both attack effectiveness and efficiency.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Multimodal Safety, Jailbreak, Behavior-Aligned Optimization, Visual Adversarial Attacks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2VqsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BAVO (Behavior-Aligned Visual Optimization), a white-box visual jailbreak framework for vision-language models (VLMs). The central motivation is that conventional gradient-based attacks optimize the likelihood of a fixed affirmative response prefix, which does not necessarily correspond to substantive compliance with a harmful request. To address this objective–behavior mismatch, BAVO trains a lightweight regression probe to predict ELITE response-level scores from the target model's internal representations. The predicted score provides a differentiable guidance signal for image perturbation optimization and supports instance-adaptive iterate selection, allowing the attack to stop early when a calibrated threshold is reached.

The method is evaluated on 917 samples from the ELITE benchmark across four VLMs: Qwen2.5-VL-7B, LLaVA-1.5-7B, InternVL3-8B, and Llama-3.2-V-11B. Compared with affirmative-target PGD, BAVO improves ELITE-based attack success rate (E-ASR) by 16.68–33.26 percentage points and HarmBench ASR by 10.25–24.64 percentage points. It also reduces the average optimization steps from 300 to 32.7–161.2 and reports wall-clock speedups of 1.8×–10.4×. Additional experiments evaluate the method under three defenses and analyze changes in the generated responses. The results suggest that behavior-guided optimization can improve attack effectiveness while reducing the cost of the optimization stage.

Critical strengths
  1. Well-motivated objective–behavior mismatch.

The paper identifies a meaningful limitation of conventional visual jailbreak attacks: increasing the likelihood of an affirmative prefix does not necessarily produce a substantively harmful response. Predicting an ELITE score from internal representations provides a reasonable way to introduce response-level guidance without generating and evaluating a complete response at every optimization step. This is a useful formulation of the problem and gives the proposed method a clear motivation.

  1. Consistent empirical improvements and useful ablations.

BAVO improves both E-ASR and HarmBench ASR across all four evaluated VLMs. The ablation study further separates the contributions of behavioral guidance and adaptive iterate selection, with both components improving performance over the affirmative-target PGD baseline. The reported efficiency improvements are also promising, and the evaluation under CIDER, JailGuard, and ECSO provides additional evidence that the gains persist in the presence of existing defenses.

Critical weaknesses
  1. The behavior predictor may be unreliable under adversarial distribution shift.

The behavior predictor is trained to regress ELITE scores from representations extracted from clean, unperturbed inputs. However, during attack optimization, the input image is deliberately perturbed to change the target model's internal representation. Consequently, the optimization trajectory may move into regions that are poorly represented in the predictor's training distribution. Since the attack directly maximizes the predictor's estimated behavior score, it may exploit inaccuracies in the surrogate rather than consistently improve the actual generated response.

  1. The comparison with existing attack methods is insufficient to establish the relative contribution and novelty of BAVO.

The primary baseline is affirmative-target PGD, which optimizes a fixed response prefix. Although the paper discusses related representation-based methods, including JailBound and ActMan, and cites the recent visual attack method FORCE, it does not provide direct comparisons with these approaches. As a result, it remains unclear whether the observed improvements primarily arise from the proposed behavior predictor, adaptive iterate selection, or differences between the chosen baseline and stronger existing attacks.

  1. The evaluation does not sufficiently establish generalization beyond the ELITE benchmark distribution.

The paper evaluates 917 held-out samples obtained through a random split of the ELITE benchmark, while training and selecting the behavior predictor using the remaining samples from the same benchmark. Although the individual evaluation examples are disjoint from predictor training, the training and evaluation splits retain the same constituent source datasets and broadly similar data distributions. Thus, the current protocol primarily establishes within-benchmark generalization rather than robustness to previously unseen attack distributions.

  1. The efficiency claims do not fully account for the cost of preparing and deploying BAVO.

The paper reports substantial speedups by measuring the image optimization stage, excluding response generation, judging, and disk I/O. However, BAVO additionally requires model-specific predictor training, which involves generating responses for 3,303 fitting examples, obtaining ELITE supervision, and selecting a representation layer using a validation split. These offline costs are not reflected in the reported attack-time speedups.

Questions for authors
  1. Predictor reliability under adversarial optimization: Can the authors evaluate the correspondence between predicted behavior scores and actual ELITE scores along the adversarial optimization trajectory? In particular, do the predictor's errors increase as perturbations become stronger, and can the authors demonstrate that the selected iterates have consistently higher actual response scores rather than merely higher predicted scores?

  2. Comparison with existing methods: How does BAVO compare with recent representation-based or gradient-based visual jailbreak methods, particularly JailBound, ActMan, and FORCE, under comparable white-box settings? If direct comparisons are not feasible, can the authors clarify the differences in threat models and explain why affirmative-target PGD is the most appropriate baseline for evaluating the proposed contribution?

  3. Generalization and data splitting: Can the authors evaluate BAVO using a source-disjoint split or an independent multimodal jailbreak benchmark? Such experiments would help establish whether the behavior predictor and its calibrated threshold generalize beyond the data distribution represented in ELITE.

  4. End-to-end efficiency: What are the total costs of generating predictor-training responses, computing ELITE supervision, training the probe, and calibrating its threshold? At approximately how many attack instances does the proposed method become computationally advantageous over conventional PGD when these preparation costs are included?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2Vqs [1/5]

Dear Reviewer 2Vqs,

We sincerely thank you for your thoughtful and constructive review. Your comments were extremely helpful in improving the clarity and empirical completeness of the work. 

[W1 & Q1, W3 & Q3] — Predictor Reliability and Source-disjoint Generalization

We thank the reviewer for highlighting concerns regarding predictor reliability under adversarial distribution shift and generalization beyond the predictor's training sources.

Source-disjoint evaluation protocol. To jointly investigate these concerns, we conducted an additional source-disjoint evaluation on 405 MM-SafetyBench inputs. MM-SafetyBench is held out in full for testing, and the behavior predictor is trained and calibrated on the seven remaining ELITE source datasets (SPA-VL, New, FigStep, JailbreakV-28K, VLGuard, MLLMGuard, and SIUO), so no test sample shares a source with the training data. The predictor is retrained, the representation layer ℓ\ell is re-selected by validation Pearson correlation, and τ\tau is recalibrated as the 99th percentile of the predictor outputs on the held-in clean data, so neither the predictor nor its threshold has seen the test distribution.

Predictor–ELITE correspondence along the trajectory. At each fixed iteration index k, we computed the Pearson correlation across the 405 inputs between the predicted behavior score of Eq. (6) and the actual ELITE score.

TargetBaseline (held-out data)k = 10k = 30k = 100k = 200k = 300
Qwen2.5-VL-7B0.5190.3720.3050.1450.2300.188
LLaVA-1.5-7B0.4840.2730.1110.0460.0560.152
InternVL3-8B0.6500.6520.4850.1620.1440.033
Llama-3.2-V-11B0.7300.6030.4440.1930.1340.112

Baseline uses the unperturbed image of the unseen source. Positive predictor–ELITE correlations are observed on these clean inputs across all four VLMs, while the correspondence generally weakens as adversarial optimization proceeds. However, BAVO's instance-adaptive optimization substantially reduces the number of inputs that continue to these later optimization steps, as shown below.

Instance-adaptive optimization limits progression to later iterations. Rather than directly maximizing the predicted behavior score, BAVO employs the saturating hinge loss in Eq. (7), max⁡(0,τ−s^(δ))/τ\max(0,\tau-\hat{s}(\delta))/\tau, which becomes zero once the predicted score reaches τ\tau and provides no further incentive to increase it. Eq. (11) then selects the first threshold-reaching iterate when available. On the source-disjoint evaluation, the median selected iteration ranges from 6 to 29 out of 300. Across the four VLMs, only 0 to 17.0% of inputs remain active at k = 100, and at most 2.0% remain active at k = 300.

The table reports the percentage of the 405 evaluation inputs still being optimized at each iteration.

TargetMedian stopk = 10k = 30k = 100k = 200k = 300
Qwen2.5-VL-7B2975.849.917.05.72.0
LLaVA-1.5-7B628.91.00.00.00.0
InternVL3-8B1051.114.61.20.00.0
Llama-3.2-V-11B844.09.41.00.20.2

To verify that adaptive iterate selection improves actual response quality rather than merely increasing predicted scores, we compare the actual ELITE scores of the selected iterates with those at fixed iteration budgets.

AuthorsOpenReview ↗

Response to Reviewer 2Vqs [2/5]

The retained iterate has the higher actual score. We compare the actual ELITE score at the retained iterate with the scores at fixed iteration budgets, evaluated on the same inputs.

TargetBAVO (Ours)Median kadapt\mathbf{k}_{\mathrm{adapt}}Threshold reached (%)k = 10k = 30k = 100k = 200k = 300
Qwen2.5-VL-7B17.102998.0+1.41*+0.76−0.44−0.41−0.07
LLaVA-1.5-7B16.696100.0+0.83*+2.53*+3.93*+4.77*+5.59*
InternVL3-8B17.6010100.0+2.64*+1.31*+0.16+2.16*+3.02*
Llama-3.2-V-11B16.19899.8+2.20*+1.71*+1.37*+1.64*+2.25*

Each fixed-budget column reports the mean paired difference in actual ELITE scores across 405 inputs. Positive values indicate higher scores for the retained iterate than for the corresponding fixed iteration. Entries marked with * have a 95% confidence interval that excludes zero.

Across all five fixed iteration budgets (k = 10, 30, 100, 200, 300), the retained iterate achieves higher mean actual ELITE scores on LLaVA-1.5-7B, InternVL3-8B, and Llama-3.2-V-11B, with statistically significant improvements in 14 of the 15 comparisons. On Qwen2.5-VL no significant difference is found against the fixed budgets of 100 to 300 iterations, while the median retained iteration is only kadapt=29k_{\mathrm{adapt}} = 29, corresponding to approximately 3× to 10× fewer optimization steps.

We further evaluate the retained iterates using HarmBench, an independent evaluator not used for predictor training or iterate selection. Compared with fixed k=300k=300, the retained iterates improve HarmBench ASR by 8.15–33.58 percentage points across all four VLMs, with paired 95% bootstrap confidence intervals excluding zero. For Qwen2.5-VL-7B, the differences are also positive at k=100k=100 (+1.98 points) and k=200k=200 (+3.21 points).

The above comparisons evaluate the effect of iterate selection within behavior-guided optimization. Notably, the ablation study in Table 3 demonstrates that adding Lbehavior\mathcal{L}_{\mathrm{behavior}} to affirmative-target PGD under the same fixed 300-step budget improves E-ASR by 11.23–26.50 percentage points across all four VLMs. These results highlight the substantial contribution of behavioral guidance itself, alongside the additional benefits of instance-adaptive optimization.

Additionally, the behavior predictor provides a differentiable behavioral signal for image optimization and guides instance-adaptive iterate selection, rather than requiring accurate ELITE-score predictions at every optimization step. Its effectiveness is assessed through improvements in attack success.

Source-disjoint generalization.

MethodQwen2.5-VL-7BLLaVA-1.5-7BInternVL3-8BLlama-3.2-V-11B
Baseline (no attack)62.72 / 60.4978.02 / 52.5941.98 / 38.2727.41 / 21.23
Affirmative-target PGD65.19 / 47.4135.80 / 14.8158.77 / 36.7959.75 / 40.74
BAVO (Ours)77.78 / 79.2682.22 / 55.8081.23 / 76.3081.98 / 63.70
Delta vs PGD+12.59 / +31.85+46.42 / +40.99+22.47 / +39.51+22.22 / +22.96

E-ASR / HarmBench ASR in percent on the 405 held-out source inputs.

BAVO attains the highest E-ASR and the highest HarmBench ASR on all four targets, improving on affirmative-target PGD by 12.59 to 46.42 points of E-ASR and 22.96 to 40.99 points of HarmBench ASR. PGD does not consistently improve attack success over the clean inputs, with the failure most pronounced on LLaVA-1.5, where E-ASR decreases from 78.02 to 35.80. The objective–behavior mismatch therefore persists under a source-disjoint protocol, and behavioral guidance mitigates it while also outperforming affirmative-target PGD under HarmBench. These results indicate that the behavior predictor and its calibrated threshold generalize beyond the data distribution on which they were fitted.

We will add the source-disjoint protocol and the above analyses to the appendix of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 2Vqs [3/5]

[W2 & Q2] — Comparison with existing attacks

We thank the reviewer for the suggestion to compare BAVO with existing methods.

Behavior-aligned visual optimization (BAVO) augments affirmative-target PGD with a learned behavior-guidance loss Ltotal(δ)=Lattack(δ)+λLbehavior(δ)\mathcal{L}_{\mathrm{total}}(\delta) = \mathcal{L}_{\mathrm{attack}}(\delta) + \lambda \mathcal{L}_{\mathrm{behavior}}(\delta) and instance-adaptive optimization. Affirmative-target PGD corresponds to the λ=0\lambda = 0 condition with a full iteration budget, and is therefore the ablation of our own method.

The two conditions share the image-text pair, target model, perturbation budget, maximum iteration budget, affirmative prefix, generation settings, and evaluators. The component-wise ablations in Table 3 of the manuscript further distinguish the contributions of behavioral guidance and instance-adaptive optimization.

The other methods differ from ours in two or more dimensions at once, which makes them unsuitable for a direct performance comparison. We detail these differences and report all four below, each adapted from its authors' released code and run under the protocol of its own paper. JailBound (Song et al., 2025) is excluded, as the authors have not released a code implementation and a faithful reproduction of their results was not feasible.

FORCE (Lin et al., CVPR 2026)

FORCE is a transferability method, which optimizes a perturbation on a source model (LLaVA-1.5-7B) and evaluates it on separate target models. We ran it in two configurations, 1) its own transfer setting with LLaVA-1.5-7B as the source and 2) a white-box setting matching our protocol, where a perturbation is optimized on each target model. We could not complete the white-box evaluation within a practical computational budget because the released optimization loop relies on success-based termination without a fixed iteration limit. Samples converge on LLaVA-1.5 within 109 to 534 steps and the loop ends normally, whereas on the other targets most do not. For example, one Qwen2.5-VL sample ran 3,700 steps without converging, while LLaVA finished the same sample in 406. We therefore report FORCE in its own setting.

ELITE subset, n = 391, E-ASR (%) / HarmBench ASR (%).

TargetNo attackFORCEBAVO (Ours)
Qwen2.5-VL-7B42.71 / 20.4630.18 / 15.0964.19 / 22.25
LLaVA-1.5-7B (white-box, matched threat model)65.47 / 29.9236.83 / 19.1873.91 / 36.57
InternVL3-8B43.48 / 23.7936.06 / 18.9371.87 / 35.04
Llama-3.2-V-11B33.76 / 18.1629.67 / 18.6762.66 / 32.23

On LLaVA-1.5-7B, where both methods optimize the perturbation on the evaluation model, BAVO achieves higher E-ASR and HarmBench ASR than FORCE by 37.1 and 17.4 percentage points, respectively. BAVO also achieves higher E-ASR and HarmBench ASR than FORCE on other three targets, although these comparisons involve different attack settings (white-box versus transfer).

Also, the FORCE comparison is restricted to the 391 samples from SPA-VL (276), VLGuard (105), MLLMGuard (7), and SIUO (3), the four datasets in which the harmful request is carried by the image content rather than by text rendered into the image. The remaining four datasets are excluded for two distinct reasons. FigStep (130), MM-SafetyBench (86), and New (197) use typographic images in which the harmful request is rendered as text, and the perturbation together with the resolution change degrades that text, removing the basis of the attack. In JailbreakV-28k (113), the prompts are long role-play templates that constrain the output format and conflict with FORCE's target prefix "Sure, here is", and no sample we measured converged. Our own results are re-aggregated on the identical indices.

Reviewer_daykOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BAVO, a visual jailbreak attack framework designed to address the misalignment between gradient-based image optimization objectives and the evaluation of final response behaviors in VLMs. By training a lightweight behavior predictor based on the internal hidden states of a VLM, BAVO provides differentiable, response-level behavioral signals for image optimization and enables instance-adaptive iterative selection.

Critical strengths
  1. This paper identifies an objective-behavior mismatch, a discrepancy between the optimization objective and the final evaluation criteria, in existing visual jailbreak attacks, and proposes a theoretically grounded solution.
  2. By training a lightweight behavior predictor using the model’s internal hidden states, the approach avoids the need for repeated generation and evaluation of complete text sequences during the optimization loop. This ensures gradient differentiability, while an instance-adaptive early-stopping mechanism significantly reduces both the number of optimization steps and computational overhead.
  3. Extensive evaluations were conducted across four mainstream VLMs using multimodal samples. The study not only compares the proposed method against various baselines and defense mechanisms but also validates its effectiveness through an analysis of response-behavior distributions.
Critical weaknesses
  1. The predictor is trained solely on clean, unattacked images; however, during optimization, it receives hidden states that have been subjected to adversarial perturbations. The paper fails to provide a dynamic analysis of the correlation between predicted scores and ground-truth scores throughout the optimization process, raising concerns that the predictor’s accuracy may degrade significantly under adversarial distributions.
  2. The paper primarily compares its approach against a simple Affirmative-target PGD baseline, lacking a thorough comparison with more advanced visual jailbreak methods, developed in recent years, that also leverage internal model representations or employ more complex optimization objectives.
  3. Although the number of optimization steps is reduced, training a behavior predictor incurs additional computational costs, as does the generation of responses for clean data. The paper lacks a comprehensive cost-benefit analysis comparing these upfront overheads against the alternative of simply running PGD for a longer duration.
  4. The paper employs a fixed, hard-coded prefix "Sure, here is..." across all samples. This approach may not be adaptable to all types of harmful requests and is easily intercepted by simple keyword-based filtering defenses; furthermore, the paper does not discuss how the choice of target string influences the attack's efficacy. (In fact, this technique was introduced in early 2022-era literature on jailbreaking; empirically, it performs poorly against the models available as of 2026.)
  5. While the study evaluates three defense mechanisms, the analysis is limited to a comparison of success rates. It lacks an in-depth investigation into the underlying mechanisms, explaining why BAVO is able to bypass these specific defenses, or conversely, why it is sometimes successfully intercepted by them.
Questions for authors
  1. During the adversarial optimization process, how does the accuracy of a behavior predictor, trained on clean data, dynamically evolve as the input distribution shifts?
  2. Given that BAVO relies heavily on accessing the internal hidden states of a VLM, how do the authors evaluate the practical threat posed by this method in more realistic transfer-attack scenarios?
  3. Does the consistent use of a fixed, compliant prefix string as an optimization target limit the model’s ability to generalize across harmful requests with different semantic characteristics?
11 comments
AuthorsOpenReview ↗

Response to Reviewer dayk [1/10]

Dear Reviewer dayk,

We sincerely thank you for your thoughtful and constructive review. Your comments were extremely helpful in improving the clarity and empirical completeness of the work. 

[W1 & Q1] — Predictor Reliability under Adversarial Optimization

We thank the reviewer for raising this important concern. Motivated by the reviewer's concern about predictor reliability under adversarial distribution shift, we conducted a trajectory-level analysis under an additional source-disjoint evaluation protocol. This protocol provides a more stringent evaluation by eliminating source overlap between predictor training and testing, thereby assessing reliability under both unseen-source and adversarial distribution shifts. Specifically, we entirely excluded MM-SafetyBench from predictor training, layer selection, and threshold calibration, using only the other seven ELITE source datasets. We then tracked the correspondence between predicted behavior scores and actual ELITE scores from clean inputs through adversarial optimization on 405 inputs from this unseen source.

Predictor–ELITE correspondence along the trajectory. At each fixed iteration index k, we computed the Pearson correlation across the 405 inputs between the predicted behavior score of Eq. (6) and the actual ELITE score.

TargetBaseline (held-out data)k = 10k = 30k = 100k = 200k = 300
Qwen2.5-VL-7B0.5190.3720.3050.1450.2300.188
LLaVA-1.5-7B0.4840.2730.1110.0460.0560.152
InternVL3-8B0.6500.6520.4850.1620.1440.033
Llama-3.2-V-11B0.7300.6030.4440.1930.1340.112

Baseline uses the unperturbed image of the unseen source. Positive predictor–ELITE correlations are observed on these clean inputs across all four VLMs, while the correspondence generally weakens as adversarial optimization proceeds. However, BAVO's instance-adaptive optimization substantially reduces the number of inputs that continue to these later optimization steps, as shown below.

Instance-adaptive optimization limits progression to later iterations. Rather than directly maximizing the predicted behavior score, BAVO employs the saturating hinge loss in Eq. (7), max⁡(0,τ−s^(δ))/τ\max(0,\tau-\hat{s}(\delta))/\tau, which becomes zero once the predicted score reaches τ\tau and provides no further incentive to increase it. Eq. (11) then selects the first threshold-reaching iterate when available. On the source-disjoint evaluation, the median selected iteration ranges from 6 to 29 out of 300. Across the four VLMs, only 0 to 17.0% of inputs remain active at k = 100, and at most 2.0% remain active at k = 300.

The table reports the percentage of the 405 evaluation inputs still being optimized at each iteration.

TargetMedian stopk = 10k = 30k = 100k = 200k = 300
Qwen2.5-VL-7B2975.849.917.05.72.0
LLaVA-1.5-7B628.91.00.00.00.0
InternVL3-8B1051.114.61.20.00.0
Llama-3.2-V-11B844.09.41.00.20.2

To verify that adaptive iterate selection improves actual response quality rather than merely increasing predicted scores, we compare the actual ELITE scores of the selected iterates with those at fixed iteration budgets.

AuthorsOpenReview ↗

Response to Reviewer dayk [2/10]

The retained iterate has the higher actual score. We compare the actual ELITE score at the retained iterate with the scores at fixed iteration budgets, evaluated on the same inputs.

TargetBAVO (Ours)Median kadapt\mathbf{k}_{\mathrm{adapt}}Threshold reached (%)k = 10k = 30k = 100k = 200k = 300
Qwen2.5-VL-7B17.102998.0+1.41*+0.76−0.44−0.41−0.07
LLaVA-1.5-7B16.696100.0+0.83*+2.53*+3.93*+4.77*+5.59*
InternVL3-8B17.6010100.0+2.64*+1.31*+0.16+2.16*+3.02*
Llama-3.2-V-11B16.19899.8+2.20*+1.71*+1.37*+1.64*+2.25*

Each fixed-budget column reports the mean paired difference in actual ELITE scores across 405 inputs. Positive values indicate higher scores for the retained iterate than for the corresponding fixed iteration. Entries marked with * have a 95% confidence interval that excludes zero. Across all five fixed iteration budgets (k = 10, 30, 100, 200, 300), the retained iterate achieves higher mean actual ELITE scores on LLaVA-1.5-7B, InternVL3-8B, and Llama-3.2-V-11B, with statistically significant improvements in 14 of the 15 comparisons. On Qwen2.5-VL no significant difference is found against the fixed budgets of 100 to 300 iterations, while the median retained iteration is only kadapt=29k_{\mathrm{adapt}} = 29, corresponding to approximately 3× to 10× fewer optimization steps.

We further evaluate the retained iterates using HarmBench, an independent evaluator not used for predictor training or iterate selection. Compared with fixed k=300k=300, the retained iterates improve HarmBench ASR by 8.15–33.58 percentage points across all four VLMs, with paired 95% bootstrap confidence intervals excluding zero. For Qwen2.5-VL-7B, the differences are also positive at k=100k=100 (+1.98 points) and k=200k=200 (+3.21 points).

The above comparisons evaluate the effect of iterate selection within behavior-guided optimization. Notably, the ablation study in Table 3 demonstrates that adding Lbehavior\mathcal{L}_{\mathrm{behavior}} to affirmative-target PGD under the same fixed 300-step budget improves E-ASR by 11.23–26.50 percentage points across all four VLMs. These results highlight the substantial contribution of behavioral guidance itself, alongside the additional benefits of instance-adaptive optimization.

Additionally, the behavior predictor provides a differentiable behavioral signal for image optimization and guides instance-adaptive iterate selection, rather than requiring accurate ELITE-score predictions at every optimization step. Its effectiveness is assessed through improvements in attack success.

For reference, the source-disjoint generalization results are summarized below.

Source-disjoint generalization.

MethodQwen2.5-VL-7BLLaVA-1.5-7BInternVL3-8BLlama-3.2-V-11B
Baseline (no attack)62.72 / 60.4978.02 / 52.5941.98 / 38.2727.41 / 21.23
Affirmative-target PGD65.19 / 47.4135.80 / 14.8158.77 / 36.7959.75 / 40.74
BAVO (Ours)77.78 / 79.2682.22 / 55.8081.23 / 76.3081.98 / 63.70
Delta vs PGD+12.59 / +31.85+46.42 / +40.99+22.47 / +39.51+22.22 / +22.96

E-ASR / HarmBench ASR in percent on the 405 held-out source inputs. BAVO attains the highest E-ASR and the highest HarmBench ASR on all four targets, improving on affirmative-target PGD by 12.59 to 46.42 points of E-ASR and 22.96 to 40.99 points of HarmBench ASR. These results indicate that BAVO remains robust under a source-disjoint protocol.

We will add the trajectory-level predictor–ELITE correlation analysis, source-disjoint evaluation protocol, and adaptive stopping statistics to the appendix of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer dayk [3/10]

[W2] — Comparison with existing attacks

We thank the reviewer for the suggestion to compare BAVO with existing methods.

Behavior-aligned visual optimization (BAVO) augments affirmative-target PGD with a learned behavior-guidance loss Ltotal(δ)=Lattack(δ)+λLbehavior(δ)\mathcal{L}_{\mathrm{total}}(\delta) = \mathcal{L}_{\mathrm{attack}}(\delta) + \lambda \mathcal{L}_{\mathrm{behavior}}(\delta) and instance-adaptive optimization. Affirmative-target PGD corresponds to the λ=0\lambda = 0 condition with a full iteration budget, and is therefore the ablation of our own method.

The two conditions share the image-text pair, target model, perturbation budget, maximum iteration budget, affirmative prefix, generation settings, and evaluators. The component-wise ablations in Table 3 of the manuscript further distinguish the contributions of behavioral guidance and instance-adaptive optimization.

The other methods differ from ours in two or more dimensions at once, which makes them unsuitable for a direct performance comparison. We detail these differences and report all four below, each adapted from its authors' released code and run under the protocol of its own paper. JailBound (Song et al., 2025) is excluded, as the authors have not released a code implementation and a faithful reproduction of their results was not feasible.

FORCE (Lin et al., CVPR 2026)

FORCE is a transferability method, which optimizes a perturbation on a source model (LLaVA-1.5-7B) and evaluates it on separate target models. We ran it in two configurations, 1) its own transfer setting with LLaVA-1.5-7B as the source and 2) a white-box setting matching our protocol, where a perturbation is optimized on each target model. We could not complete the white-box evaluation within a practical computational budget because the released optimization loop relies on success-based termination without a fixed iteration limit. Samples converge on LLaVA-1.5 within 109 to 534 steps and the loop ends normally, whereas on the other targets most do not. For example, one Qwen2.5-VL sample ran 3,700 steps without converging, while LLaVA finished the same sample in 406. We therefore report FORCE in its own setting.

ELITE subset, n = 391, E-ASR (%) / HarmBench ASR (%).

TargetNo attackFORCEBAVO (Ours)
Qwen2.5-VL-7B42.71 / 20.4630.18 / 15.0964.19 / 22.25
LLaVA-1.5-7B (white-box, matched threat model)65.47 / 29.9236.83 / 19.1873.91 / 36.57
InternVL3-8B43.48 / 23.7936.06 / 18.9371.87 / 35.04
Llama-3.2-V-11B33.76 / 18.1629.67 / 18.6762.66 / 32.23

On LLaVA-1.5-7B, where both methods optimize the perturbation on the evaluation model, BAVO achieves higher E-ASR and HarmBench ASR than FORCE by 37.1 and 17.4 percentage points, respectively. BAVO also achieves higher E-ASR and HarmBench ASR than FORCE on other three targets, although these comparisons involve different attack settings (white-box versus transfer).

Also, the FORCE comparison is restricted to the 391 samples from SPA-VL (276), VLGuard (105), MLLMGuard (7), and SIUO (3), the four datasets in which the harmful request is carried by the image content rather than by text rendered into the image. The remaining four datasets are excluded for two distinct reasons. FigStep (130), MM-SafetyBench (86), and New (197) use typographic images in which the harmful request is rendered as text, and the perturbation together with the resolution change degrades that text, removing the basis of the attack. In JailbreakV-28k (113), the prompts are long role-play templates that constrain the output format and conflict with FORCE's target prefix "Sure, here is", and no sample we measured converged. Our own results are re-aggregated on the identical indices.

Reviewer_aKtcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of predicting the safety properties of the response from a vision-language model as a result of inputs that attempt to jailbreak the model by using the vision modality as a target. In particular, the paper aims to predict the behaviour of the model without needing to wait for the complete response. Moreover, the behaviour prediction should be in a differentiable form that can be used for attack optimisation. The paper presents a novel formulation of such a behaviour predictor, along with an optimisation framework that uses the predictor as guidance to update sample images to maximise the likelihood of a successful attack. Empirical results across a variety of VLMs demonstrate a clear improvement in attack success rate compared to PGD, in the presence of three baseline defence methods. The proposed method also demonstrates a significant speedup in attack optimisation compared to standard PGD.

Critical strengths

The paper presents a novel approach to the optimisation of jailbreak attacks on vision-language models, based on learning a behaviour prediction model that does not require access to the final response, and is in a differentiable form that can be used for attack optimisation.

The proposed method demonstrates a substantial speed-up in the time required for attack image optimisation in comparison to a traditional PGD method.

Critical weaknesses

The paper only provides a PGD as a non-trivial baseline for comparison.

Questions for authors

What other baselines could be considered in your evaluation, and how does your approach perform compared to other baselines? For example, can the works of Qi et al 2024, Lin et al 2026, Hao et al 2026 or Song et al 2025 be used as baselines?

4 comments
AuthorsOpenReview ↗

Response to Reviewer aKtc [1/2]

Dear Reviewer aKtc,

We sincerely thank you for your thoughtful and constructive review. Your comments were extremely helpful in improving the clarity and empirical completeness of the work. 

[W1 & Q1] — Comparison with existing attacks

We thank the reviewer for the suggestion to compare BAVO with existing methods.

Behavior-aligned visual optimization (BAVO) augments affirmative-target PGD with a learned behavior-guidance loss Ltotal(δ)=Lattack(δ)+λLbehavior(δ)\mathcal{L}_{\mathrm{total}}(\delta) = \mathcal{L}_{\mathrm{attack}}(\delta) + \lambda \mathcal{L}_{\mathrm{behavior}}(\delta) and instance-adaptive optimization. Affirmative-target PGD corresponds to the λ=0\lambda = 0 condition with a full iteration budget, and is therefore the ablation of our own method.

The two conditions share the image-text pair, target model, perturbation budget, maximum iteration budget, affirmative prefix, generation settings, and evaluators. The component-wise ablations in Table 3 of the manuscript further distinguish the contributions of behavioral guidance and instance-adaptive optimization.

The other methods differ from ours in two or more dimensions at once, which makes them unsuitable for a direct performance comparison. We detail these differences and report all four below, each adapted from its authors' released code and run under the protocol of its own paper. JailBound (Song et al., 2025) is excluded, as the authors have not released a code implementation and a faithful reproduction of their results was not feasible.

FORCE (Lin et al., CVPR 2026)

FORCE is a transferability method, which optimizes a perturbation on a source model (LLaVA-1.5-7B) and evaluates it on separate target models. We ran it in two configurations, 1) its own transfer setting with LLaVA-1.5-7B as the source and 2) a white-box setting matching our protocol, where a perturbation is optimized on each target model. We could not complete the white-box evaluation within a practical computational budget because the released optimization loop relies on success-based termination without a fixed iteration limit. Samples converge on LLaVA-1.5 within 109 to 534 steps and the loop ends normally, whereas on the other targets most do not. For example, one Qwen2.5-VL sample ran 3,700 steps without converging, while LLaVA finished the same sample in 406. We therefore report FORCE in its own setting.

ELITE subset, n = 391, E-ASR (%) / HarmBench ASR (%).

TargetNo attackFORCEBAVO (Ours)
Qwen2.5-VL-7B42.71 / 20.4630.18 / 15.0964.19 / 22.25
LLaVA-1.5-7B (white-box, matched threat model)65.47 / 29.9236.83 / 19.1873.91 / 36.57
InternVL3-8B43.48 / 23.7936.06 / 18.9371.87 / 35.04
Llama-3.2-V-11B33.76 / 18.1629.67 / 18.6762.66 / 32.23

On LLaVA-1.5-7B, where both methods optimize the perturbation on the evaluation model, BAVO achieves higher E-ASR and HarmBench ASR than FORCE by 37.1 and 17.4 percentage points, respectively. BAVO also achieves higher E-ASR and HarmBench ASR than FORCE on other three targets, although these comparisons involve different attack settings (white-box versus transfer).

Also, the FORCE comparison is restricted to the 391 samples from SPA-VL (276), VLGuard (105), MLLMGuard (7), and SIUO (3), the four datasets in which the harmful request is carried by the image content rather than by text rendered into the image. The remaining four datasets are excluded for two distinct reasons. FigStep (130), MM-SafetyBench (86), and New (197) use typographic images in which the harmful request is rendered as text, and the perturbation together with the resolution change degrades that text, removing the basis of the attack. In JailbreakV-28k (113), the prompts are long role-play templates that constrain the output format and conflict with FORCE's target prefix "Sure, here is", and no sample we measured converged. Our own results are re-aggregated on the identical indices.

AuthorsOpenReview ↗

Response to Reviewer aKtc [2/2]

ActMan (Hao et al., AAAI 2026)

ActMan replaces the benchmark image with a random benign ImageNet image, so its attack results and the no-attack baseline do not share the same input condition. We therefore compare the two attacked conditions directly without a shared baseline, using its released hyperparameters and default generation configuration.

We evaluate ActMan on three of our four target VLMs. The authors release attackers for four models, none of which is Llama-3.2-V, and the attention term of its objective is not directly applicable to that architecture. This term computes the ratio of attention on harmful text tokens to attention on image tokens, assuming that both are represented in a shared self-attention map. However, Llama-3.2-V incorporates visual features through separate cross-attention layers, so its self-attention map contains an image placeholder rather than the actual visual tokens. Applying the original attention term would therefore require an architecture-specific reformulation rather than a direct reproduction.

The comparison is further restricted to 586 of our 917 evaluation inputs. ActMan derives a harmful-token mask from the text instruction, so it applies only where the prompt states the harmful request rather than referring to the image. Three of our eight datasets satisfy this, SPA-VL (276), New (197) and JailbreakV-28k (113). These datasets were selected based on input format, independently of the observed attack results. Our results are re-aggregated on the identical evaluation indices.

ELITE subset, n = 586, E-ASR (%) / HarmBench ASR (%).

TargetActManBAVO (Ours)
Qwen2.5-VL-7B33.45 / 37.2065.02 / 43.34
LLaVA-1.5-7B55.80 / 48.2974.57 / 55.29
InternVL3-8B23.72 / 28.8470.14 / 52.56

BAVO outperforms ActMan across all three targets under both evaluators, with improvements of 18.8% to 46.4% in E-ASR and 6.1% to 23.7% in HarmBench ASR, despite using half the perturbation budget. These results provide an additional cross-method comparison, although differences in input construction and attack settings preclude a strictly controlled comparison.

VAJM (Qi et al., AAAI 2024) and FigStep (Gong et al., AAAI 2025)

VAJM optimizes a universal adversarial image using an external corpus, whereas FigStep converts harmful instructions into typographic images. Neither preserves the original image-text pairs used by BAVO, so these comparisons are not controlled in the same sense as the affirmative-target PGD ablation. We nonetheless implemented both methods based on their authors' released code and evaluated them on the same 405 harmful requests using common evaluation metrics.

All conditions are evaluated on a source-disjoint split, in which the behavior predictor is trained and calibrated on the seven remaining ELITE source datasets and MM-SafetyBench is held out in full for testing. MM-SafetyBench is a commonly used evaluation benchmark for multimodal jailbreak attacks. For VAJM, we evaluate both the perturbation budget of 16/255 and the unconstrained setting considered in its original paper.

Source-disjoint split (MM-SafetyBench, n = 405), E-ASR (%) / HarmBench ASR (%).

TargetNo attackFigStepVAJM (16/255)VAJM (unconstrained)BAVO (Ours)
Qwen2.5-VL-7B62.72 / 60.4927.41 / 25.6812.35 / 12.1023.70 / 20.9977.78 / 79.26
LLaVA-1.5-7B78.02 / 52.5986.91 / 68.6456.05 / 48.8949.38 / 36.7982.22 / 55.80
InternVL3-8B41.98 / 38.2742.47 / 40.007.65 / 6.422.96 / 2.9681.23 / 76.30
Llama-3.2-V-11B27.41 / 21.2346.91 / 50.371.23 / 0.746.17 / 5.9381.98 / 63.70

The no-attack baseline uses the original MM-SafetyBench image–text pairs. BAVO outperforms VAJM under both perturbation settings across all four targets and achieves higher E-ASR and HarmBench ASR than FigStep on three of the four targets. On LLaVA-1.5-7B, FigStep achieves higher scores under both metrics. The result show that BAVO remains effective on this held-out source, suggesting that the approach generalizes beyond the data distribution used to train the behavior predictor and calibrate its threshold, although differences in input construction and attack settings preclude a strictly controlled comparison with the other methods.

We will include the comparisons with FORCE, ActMan, VAJM, and FigStep and their threat-model differences in the appendix of the revised manuscript.

We sincerely thank the reviewer again for their constructive comments. We will incorporate all clarifications and new experimental results in the revised manuscript.

Please let us know if you have any further questions or concerns.

Best regards,

Authors

Reviewer_aKtcOpenReview ↗

Comment on Authors' Response

Thank you for providing the additional experiments that demonstrate the performance of your approach with respect to the baseline. Overall, based on the contributions, I will retain my positive rating.

AuthorsOpenReview ↗

Follow-up to Reviewer aKtc: Updated Manuscript and Summary

Dear Reviewer aKtc,

Thank you for your encouraging response and positive assessment of our work. We sincerely appreciate your recognition of our contributions.

We have also uploaded a revised manuscript with all revisions highlighted in blue and posted a General Response and Summary of Revisions, summarizing the additional experiments and analyses.

We sincerely hope that these updates further strengthen your confidence in the contributions of our work, and we would be grateful if you could take them into account in your final assessment.

Thank you once again for your time and consideration.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

Dear Reviewers and AC,

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the objective–behavior mismatch motivating our work (dayk, 2Vqs), the novel use of a lightweight behavior predictor that provides differentiable response-level guidance (aKtc, dayk, 2Vqs), and the instance-adaptive optimization that reduces unnecessary updates (dayk, 2Vqs). The reviewers also highlighted the improvements in attack effectiveness and computational efficiency (aKtc, dayk, 2Vqs), supported by extensive evaluations, component-wise ablations, and defense analyses (dayk, 2Vqs).

To address the reviewers' concerns, we conducted additional experiments and analyses and revised the manuscript accordingly. The main additions and revisions are summarized below.

For ease of review, all newly added and revised content is highlighted in blue in the updated manuscript.

Additional Experiments and Revisions

  • Comparison with existing visual jailbreak methods (aKtc, dayk, 2Vqs; Sec. 4.1, App. F): Differences in input construction, optimization objectives, and attack settings preclude strictly controlled comparisons. Nevertheless, we evaluated FORCE, ActMan, VAJM, and FigStep using their authors' released code and report the results as supplementary comparisons.
  • Predictor reliability under adversarial optimization (dayk, 2Vqs; Sec. 3.2, App. G): Analyzed predictor–ELITE correspondence along optimization trajectories in our response and showed that instance-adaptive iterate selection improves ASR while generally achieving higher actual ELITE scores.
  • Preparation cost and end-to-end efficiency (dayk, 2Vqs; Sec. 4.4, App. H): Measured the one-time preparation cost and determined the break-even points, demonstrating that BAVO achieves lower amortized GPU time per successful attack than affirmative-target PGD across all four VLMs.
  • Source-disjoint generalization (2Vqs; Secs. 4.1 and 4.3, App. G): Evaluated BAVO on all 405 MM-SafetyBench inputs held out from predictor training, layer selection, calibration, achieving improvements of 12.59–46.42 E-ASR points and 22.96–40.99 HarmBench ASR points over affirmative-target PGD.
  • Sensitivity to the affirmative target string (dayk; Sec. 3.1, App. E): Evaluated three alternative affirmative target strings and found no statistically significant E-ASR differences across the four VLMs. Additional keyword-filtering and hazard-category analyses are provided in our response.
  • Defense mechanism analysis (dayk; Sec. 4.6, App. I): Conducted sample-level analyses of CIDER, JailGuard, and ECSO, examining their detection statistics and intervention outcomes to explain why BAVO remains successful or is intercepted under each defense.
  • White-box setting and transferability (dayk; Sec. 5): Clarified BAVO's intended white-box threat model and its focus on addressing the objective–behavior mismatch, while also examining cross-model transferability for BAVO and affirmative-target PGD under matched conditions, as discussed in our response.

We sincerely appreciate the reviewers' time and careful consideration throughout the review and discussion process. We hope that our detailed responses, additional experimental evidence, and revised manuscript have addressed the reviewers' concerns and further clarified the contributions of BAVO.

We welcome any further feedback or questions and would be happy to provide additional clarification.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.