- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BAVO (Behavior-Aligned Visual Optimization), a white-box visual jailbreak framework for vision-language models (VLMs). The central motivation is that conventional gradient-based attacks optimize the likelihood of a fixed affirmative response prefix, which does not necessarily correspond to substantive compliance with a harmful request. To address this objective–behavior mismatch, BAVO trains a lightweight regression probe to predict ELITE response-level scores from the target model's internal representations. The predicted score provides a differentiable guidance signal for image perturbation optimization and supports instance-adaptive iterate selection, allowing the attack to stop early when a calibrated threshold is reached.
The method is evaluated on 917 samples from the ELITE benchmark across four VLMs: Qwen2.5-VL-7B, LLaVA-1.5-7B, InternVL3-8B, and Llama-3.2-V-11B. Compared with affirmative-target PGD, BAVO improves ELITE-based attack success rate (E-ASR) by 16.68–33.26 percentage points and HarmBench ASR by 10.25–24.64 percentage points. It also reduces the average optimization steps from 300 to 32.7–161.2 and reports wall-clock speedups of 1.8×–10.4×. Additional experiments evaluate the method under three defenses and analyze changes in the generated responses. The results suggest that behavior-guided optimization can improve attack effectiveness while reducing the cost of the optimization stage.
Critical strengths
- Well-motivated objective–behavior mismatch.
The paper identifies a meaningful limitation of conventional visual jailbreak attacks: increasing the likelihood of an affirmative prefix does not necessarily produce a substantively harmful response. Predicting an ELITE score from internal representations provides a reasonable way to introduce response-level guidance without generating and evaluating a complete response at every optimization step. This is a useful formulation of the problem and gives the proposed method a clear motivation.
- Consistent empirical improvements and useful ablations.
BAVO improves both E-ASR and HarmBench ASR across all four evaluated VLMs. The ablation study further separates the contributions of behavioral guidance and adaptive iterate selection, with both components improving performance over the affirmative-target PGD baseline. The reported efficiency improvements are also promising, and the evaluation under CIDER, JailGuard, and ECSO provides additional evidence that the gains persist in the presence of existing defenses.
Critical weaknesses
- The behavior predictor may be unreliable under adversarial distribution shift.
The behavior predictor is trained to regress ELITE scores from representations extracted from clean, unperturbed inputs. However, during attack optimization, the input image is deliberately perturbed to change the target model's internal representation. Consequently, the optimization trajectory may move into regions that are poorly represented in the predictor's training distribution. Since the attack directly maximizes the predictor's estimated behavior score, it may exploit inaccuracies in the surrogate rather than consistently improve the actual generated response.
- The comparison with existing attack methods is insufficient to establish the relative contribution and novelty of BAVO.
The primary baseline is affirmative-target PGD, which optimizes a fixed response prefix. Although the paper discusses related representation-based methods, including JailBound and ActMan, and cites the recent visual attack method FORCE, it does not provide direct comparisons with these approaches. As a result, it remains unclear whether the observed improvements primarily arise from the proposed behavior predictor, adaptive iterate selection, or differences between the chosen baseline and stronger existing attacks.
- The evaluation does not sufficiently establish generalization beyond the ELITE benchmark distribution.
The paper evaluates 917 held-out samples obtained through a random split of the ELITE benchmark, while training and selecting the behavior predictor using the remaining samples from the same benchmark. Although the individual evaluation examples are disjoint from predictor training, the training and evaluation splits retain the same constituent source datasets and broadly similar data distributions. Thus, the current protocol primarily establishes within-benchmark generalization rather than robustness to previously unseen attack distributions.
- The efficiency claims do not fully account for the cost of preparing and deploying BAVO.
The paper reports substantial speedups by measuring the image optimization stage, excluding response generation, judging, and disk I/O. However, BAVO additionally requires model-specific predictor training, which involves generating responses for 3,303 fitting examples, obtaining ELITE supervision, and selecting a representation layer using a validation split. These offline costs are not reflected in the reported attack-time speedups.
Questions for authors
-
Predictor reliability under adversarial optimization: Can the authors evaluate the correspondence between predicted behavior scores and actual ELITE scores along the adversarial optimization trajectory? In particular, do the predictor's errors increase as perturbations become stronger, and can the authors demonstrate that the selected iterates have consistently higher actual response scores rather than merely higher predicted scores?
-
Comparison with existing methods: How does BAVO compare with recent representation-based or gradient-based visual jailbreak methods, particularly JailBound, ActMan, and FORCE, under comparable white-box settings? If direct comparisons are not feasible, can the authors clarify the differences in threat models and explain why affirmative-target PGD is the most appropriate baseline for evaluating the proposed contribution?
-
Generalization and data splitting: Can the authors evaluate BAVO using a source-disjoint split or an independent multimodal jailbreak benchmark? Such experiments would help establish whether the behavior predictor and its calibrated threshold generalize beyond the data distribution represented in ELITE.
-
End-to-end efficiency: What are the total costs of generating predictor-training responses, computing ELITE supervision, training the probe, and calibrating its threshold? At approximately how many attack instances does the proposed method become computationally advantageous over conventional PGD when these preparation costs are included?
7 comments
Response to Reviewer 2Vqs [2/5]
The retained iterate has the higher actual score. We compare the actual ELITE score at the retained iterate with the scores at fixed iteration budgets, evaluated on the same inputs.
| Target | BAVO (Ours) | Median | Threshold reached (%) | k = 10 | k = 30 | k = 100 | k = 200 | k = 300 |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 17.10 | 29 | 98.0 | +1.41* | +0.76 | −0.44 | −0.41 | −0.07 |
| LLaVA-1.5-7B | 16.69 | 6 | 100.0 | +0.83* | +2.53* | +3.93* | +4.77* | +5.59* |
| InternVL3-8B | 17.60 | 10 | 100.0 | +2.64* | +1.31* | +0.16 | +2.16* | +3.02* |
| Llama-3.2-V-11B | 16.19 | 8 | 99.8 | +2.20* | +1.71* | +1.37* | +1.64* | +2.25* |
Each fixed-budget column reports the mean paired difference in actual ELITE scores across 405 inputs. Positive values indicate higher scores for the retained iterate than for the corresponding fixed iteration. Entries marked with * have a 95% confidence interval that excludes zero.
Across all five fixed iteration budgets (k = 10, 30, 100, 200, 300), the retained iterate achieves higher mean actual ELITE scores on LLaVA-1.5-7B, InternVL3-8B, and Llama-3.2-V-11B, with statistically significant improvements in 14 of the 15 comparisons. On Qwen2.5-VL no significant difference is found against the fixed budgets of 100 to 300 iterations, while the median retained iteration is only , corresponding to approximately 3× to 10× fewer optimization steps.
We further evaluate the retained iterates using HarmBench, an independent evaluator not used for predictor training or iterate selection. Compared with fixed , the retained iterates improve HarmBench ASR by 8.15–33.58 percentage points across all four VLMs, with paired 95% bootstrap confidence intervals excluding zero. For Qwen2.5-VL-7B, the differences are also positive at (+1.98 points) and (+3.21 points).
The above comparisons evaluate the effect of iterate selection within behavior-guided optimization. Notably, the ablation study in Table 3 demonstrates that adding to affirmative-target PGD under the same fixed 300-step budget improves E-ASR by 11.23–26.50 percentage points across all four VLMs. These results highlight the substantial contribution of behavioral guidance itself, alongside the additional benefits of instance-adaptive optimization.
Additionally, the behavior predictor provides a differentiable behavioral signal for image optimization and guides instance-adaptive iterate selection, rather than requiring accurate ELITE-score predictions at every optimization step. Its effectiveness is assessed through improvements in attack success.
Source-disjoint generalization.
| Method | Qwen2.5-VL-7B | LLaVA-1.5-7B | InternVL3-8B | Llama-3.2-V-11B |
|---|---|---|---|---|
| Baseline (no attack) | 62.72 / 60.49 | 78.02 / 52.59 | 41.98 / 38.27 | 27.41 / 21.23 |
| Affirmative-target PGD | 65.19 / 47.41 | 35.80 / 14.81 | 58.77 / 36.79 | 59.75 / 40.74 |
| BAVO (Ours) | 77.78 / 79.26 | 82.22 / 55.80 | 81.23 / 76.30 | 81.98 / 63.70 |
| Delta vs PGD | +12.59 / +31.85 | +46.42 / +40.99 | +22.47 / +39.51 | +22.22 / +22.96 |
E-ASR / HarmBench ASR in percent on the 405 held-out source inputs.
BAVO attains the highest E-ASR and the highest HarmBench ASR on all four targets, improving on affirmative-target PGD by 12.59 to 46.42 points of E-ASR and 22.96 to 40.99 points of HarmBench ASR. PGD does not consistently improve attack success over the clean inputs, with the failure most pronounced on LLaVA-1.5, where E-ASR decreases from 78.02 to 35.80. The objective–behavior mismatch therefore persists under a source-disjoint protocol, and behavioral guidance mitigates it while also outperforming affirmative-target PGD under HarmBench. These results indicate that the behavior predictor and its calibrated threshold generalize beyond the data distribution on which they were fitted.
We will add the source-disjoint protocol and the above analyses to the appendix of the revised manuscript.
Response to Reviewer 2Vqs [3/5]
[W2 & Q2] — Comparison with existing attacks
We thank the reviewer for the suggestion to compare BAVO with existing methods.
Behavior-aligned visual optimization (BAVO) augments affirmative-target PGD with a learned behavior-guidance loss and instance-adaptive optimization. Affirmative-target PGD corresponds to the condition with a full iteration budget, and is therefore the ablation of our own method.
The two conditions share the image-text pair, target model, perturbation budget, maximum iteration budget, affirmative prefix, generation settings, and evaluators. The component-wise ablations in Table 3 of the manuscript further distinguish the contributions of behavioral guidance and instance-adaptive optimization.
The other methods differ from ours in two or more dimensions at once, which makes them unsuitable for a direct performance comparison. We detail these differences and report all four below, each adapted from its authors' released code and run under the protocol of its own paper. JailBound (Song et al., 2025) is excluded, as the authors have not released a code implementation and a faithful reproduction of their results was not feasible.
FORCE (Lin et al., CVPR 2026)
FORCE is a transferability method, which optimizes a perturbation on a source model (LLaVA-1.5-7B) and evaluates it on separate target models. We ran it in two configurations, 1) its own transfer setting with LLaVA-1.5-7B as the source and 2) a white-box setting matching our protocol, where a perturbation is optimized on each target model. We could not complete the white-box evaluation within a practical computational budget because the released optimization loop relies on success-based termination without a fixed iteration limit. Samples converge on LLaVA-1.5 within 109 to 534 steps and the loop ends normally, whereas on the other targets most do not. For example, one Qwen2.5-VL sample ran 3,700 steps without converging, while LLaVA finished the same sample in 406. We therefore report FORCE in its own setting.
ELITE subset, n = 391, E-ASR (%) / HarmBench ASR (%).
| Target | No attack | FORCE | BAVO (Ours) |
|---|---|---|---|
| Qwen2.5-VL-7B | 42.71 / 20.46 | 30.18 / 15.09 | 64.19 / 22.25 |
| LLaVA-1.5-7B (white-box, matched threat model) | 65.47 / 29.92 | 36.83 / 19.18 | 73.91 / 36.57 |
| InternVL3-8B | 43.48 / 23.79 | 36.06 / 18.93 | 71.87 / 35.04 |
| Llama-3.2-V-11B | 33.76 / 18.16 | 29.67 / 18.67 | 62.66 / 32.23 |
On LLaVA-1.5-7B, where both methods optimize the perturbation on the evaluation model, BAVO achieves higher E-ASR and HarmBench ASR than FORCE by 37.1 and 17.4 percentage points, respectively. BAVO also achieves higher E-ASR and HarmBench ASR than FORCE on other three targets, although these comparisons involve different attack settings (white-box versus transfer).
Also, the FORCE comparison is restricted to the 391 samples from SPA-VL (276), VLGuard (105), MLLMGuard (7), and SIUO (3), the four datasets in which the harmful request is carried by the image content rather than by text rendered into the image. The remaining four datasets are excluded for two distinct reasons. FigStep (130), MM-SafetyBench (86), and New (197) use typographic images in which the harmful request is rendered as text, and the perturbation together with the resolution change degrades that text, removing the basis of the attack. In JailbreakV-28k (113), the prompts are long role-play templates that constrain the output format and conflict with FORCE's target prefix "Sure, here is", and no sample we measured converged. Our own results are re-aggregated on the identical indices.
Response to Reviewer 2Vqs [1/5]
Dear Reviewer 2Vqs,
We sincerely thank you for your thoughtful and constructive review. Your comments were extremely helpful in improving the clarity and empirical completeness of the work.
[W1 & Q1, W3 & Q3] — Predictor Reliability and Source-disjoint Generalization
We thank the reviewer for highlighting concerns regarding predictor reliability under adversarial distribution shift and generalization beyond the predictor's training sources.
Source-disjoint evaluation protocol. To jointly investigate these concerns, we conducted an additional source-disjoint evaluation on 405 MM-SafetyBench inputs. MM-SafetyBench is held out in full for testing, and the behavior predictor is trained and calibrated on the seven remaining ELITE source datasets (SPA-VL, New, FigStep, JailbreakV-28K, VLGuard, MLLMGuard, and SIUO), so no test sample shares a source with the training data. The predictor is retrained, the representation layer ℓ is re-selected by validation Pearson correlation, and τ is recalibrated as the 99th percentile of the predictor outputs on the held-in clean data, so neither the predictor nor its threshold has seen the test distribution.
Predictor–ELITE correspondence along the trajectory. At each fixed iteration index k, we computed the Pearson correlation across the 405 inputs between the predicted behavior score of Eq. (6) and the actual ELITE score.
Baseline uses the unperturbed image of the unseen source. Positive predictor–ELITE correlations are observed on these clean inputs across all four VLMs, while the correspondence generally weakens as adversarial optimization proceeds. However, BAVO's instance-adaptive optimization substantially reduces the number of inputs that continue to these later optimization steps, as shown below.
Instance-adaptive optimization limits progression to later iterations. Rather than directly maximizing the predicted behavior score, BAVO employs the saturating hinge loss in Eq. (7), max(0,τ−s^(δ))/τ, which becomes zero once the predicted score reaches τ and provides no further incentive to increase it. Eq. (11) then selects the first threshold-reaching iterate when available. On the source-disjoint evaluation, the median selected iteration ranges from 6 to 29 out of 300. Across the four VLMs, only 0 to 17.0% of inputs remain active at k = 100, and at most 2.0% remain active at k = 300.
The table reports the percentage of the 405 evaluation inputs still being optimized at each iteration.
To verify that adaptive iterate selection improves actual response quality rather than merely increasing predicted scores, we compare the actual ELITE scores of the selected iterates with those at fixed iteration budgets.