- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SCOPE, a training-free method to address jailbreak failures that persist after activation steering in VLMs. It identifies unsafe-related coordinates from base-model responses and zeros them at the final prompt position in one decoder layer during prefill. The selected coordinates remain fixed across steering methods and benchmarks.
Critical strengths
- The intervention is simple and easy to add to existing steering methods. It changes only 139 or 112 coordinates at one layer and prompt position, once during prefill.
- The ablations are well chosen and help justify both the coordinate selection and the zeroing operation.
Critical weaknesses
- The paper's central claim is that coordinates identified from base-model responses provide a general complement to different steering methods. However, the main comparisons with random, activation-magnitude, and negative-difference coordinates are conducted under standalone suppression. CDS in Table 3 restricts the dimensions affected by steering updates, which differs from zeroing hidden-state coordinates at the final prompt position. Although Table 6 controls the edit norm, it uses only SCOPE-selected coordinates and mainly tests the effect of update direction. A direct comparison is therefore still missing.
- The three attack benchmarks provide useful evaluations under fixed attacks and cross-benchmark settings. However, the paper does not report adaptive attacks reoptimized against the full steering-plus-SCOPE defense.
Questions for authors
- How do random or activation-magnitude-based coordinates perform when combined with steering under matched intervention settings?
- Have the authors tested adaptive attacks against the full combined defense? If not, the claims should be limited to the evaluated fixed and transfer attacks.
4 comments
Response to Reviewer 6eFB [2/2]
[W2 & Q2] Adaptive Attacks
Fixed and transfer attacks do not establish robustness to adaptive attackers. We have now evaluated one adaptive setting and will state the scope of our claims explicitly.
Setup. On Qwen2-VL-7B with CAA, we ran the unconstrained white-box universal visual attack of Qi et al. (2024). The adversarial image is prepended as an additional image to each input. We optimized separate images against CAA alone and the full CAA+SCOPE defense for 5,000 PGD steps, differentiating through the exact CAA and SCOPE hooks. Evaluation questions were not used during optimization.
Results. When each defense faces the image optimized against it, CAA+SCOPE has lower ASR than CAA:
| Defense (attack optimized against it) | MMSB ↓ | HADES ↓ | JailbreakV ↓ |
|---|---|---|---|
| CAA | 44.07 | 50.93 | 20.71 |
| CAA + SCOPE | 42.22 | 47.60 | 15.36 |
The table compares separately optimized attacks. In a separate paired comparison, we applied the same image optimized against CAA+SCOPE to both defenses. CAA+SCOPE has lower ASR on all three benchmarks, with paired 95% CIs excluding zero: MMSB, −5.19 pp [−10.00, −0.74]; HADES, −7.20 pp [−10.00, −4.53]; and JailbreakV-mini, −4.64 pp [−8.33, −1.09]. The attack increases ASR for both defenses, but SCOPE retains its additional benefit when the attack is optimized through the full defense, including the fixed support.
We also evaluated a stronger attacker that selects the most effective image for each question from all evaluated candidates (the original and every adaptive image). Under this per-question worst-case selection, CAA+SCOPE still has lower ASR on all three benchmarks: MMSB, 52.96% vs. 57.04% (−4.07 pp [−7.78, −1.11]); HADES, 60.53% vs. 66.13% (−5.60 pp [−8.00, −3.20]); and JailbreakV-mini, 20.00% vs. 26.79% (−6.79 pp [−10.40, −3.44]).
Scope of claims. This evaluation covers one universal attack family and one optimization seed; the attack is not optimized separately for each input. It cannot establish robustness to all adaptive attacks. In the revision, we state that the main evaluation covers fixed and transfer attacks (the abstract now reads "…across two VLMs, three steering methods, and three jailbreak benchmarks with fixed and transfer attacks…"). We will report the adaptive evaluation as a limited, attack-specific test and discuss stronger adaptive attacks as future work.
Please let us know if any questions remain. If these results resolve your concerns, we would be grateful if you could consider raising the score.
Thank you for the additional experiments. Some limitations remain: the adaptive evaluation covers only one universal attack family and one seed on a single model–steering pair. The safety–utility benefit also varies with the configuration: stronger baselines incur utility losses, and Figure 9 includes a high-strength L2S setting where adding SCOPE worsens both ASR and utility.
Follow-up to Reviewer 6eFB
Thank you for the follow-up. We have extended the adaptive evaluation to two VLMs, three steering methods, and five optimization seeds.
Adaptive evaluation. On Qwen2-VL-7B and InternVL2-8B, we ran the same white-box universal attack against CAA, L2S, and ASTRA, each with and without SCOPE. For each model–steering pair, we optimized five images (one per seed) against each defense, yielding 60 adaptive images, and evaluated all 10 images of a pair against both defenses (156,000 responses). Each attack was optimized through the full defended model, including L2S's input-dependent context-to-shift computation and ASTRA's adaptive steering operation. When each defense faces the image optimized against it, ASR (%, steering only → + SCOPE, averaged over five seeds) is:
| Model | Steering | MMSB ↓ | HADES ↓ | JailbreakV ↓ | Macro Δ (pp) [95% CI] |
|---|---|---|---|---|---|
| Qwen2-VL-7B | CAA | 42.74 → 37.11 | 47.17 → 36.29 | 17.93 → 12.36 | −7.36 [−8.68, −6.08] |
| L2S | 40.96 → 22.96 | 68.05 → 51.55 | 20.57 → 16.64 | −12.81 [−14.64, −10.99] | |
| ASTRA | 41.63 → 40.59 | 54.11 → 49.44 | 17.36 → 13.64 | −3.14 [−4.35, −2.00] | |
| InternVL2-8B | CAA | 40.22 → 27.70 | 42.45 → 32.27 | 11.64 → 7.79 | −8.85 [−10.55, −7.21] |
| L2S | 19.93 → 7.33 | 20.56 → 12.24 | 8.14 → 4.64 | −8.14 [−9.78, −6.64] | |
| ASTRA | 42.00 → 35.04 | 43.84 → 39.04 | 12.64 → 10.21 | −4.73 [−6.30, −3.23] |
Averaged over the five seeds, SCOPE lowers ASR in all 18 model–steering–benchmark settings. It also lowers macro ASR in 29 of the 30 model–steering–seed comparisons. We also compared both defenses on the same images (optimized against either defense) and evaluated per-question worst-case ASR separately for each defense, using the same candidate pool consisting of the original input and all 10 adaptive images. SCOPE lowers macro ASR for all six pairs in both analyses (worst case: by 2.01–8.56 pp). All six macro-ASR CIs exclude zero in each analysis; CIs are paired cluster-bootstrap intervals over questions, with the five optimization seeds held fixed. We will present this as an evaluation of one universal attack family and include the per-seed and per-benchmark results in the appendix.
Safety–utility trade-off. The additional baseline experiments show that SCOPE further lowers macro ASR even on top of strong defenses: from 25.04% to 18.37% with Antidote and from 18.39% to 12.04% with refusal-direction steering, at utility costs of 1.75 and 2.46 pp, respectively. This supports its value as a reusable complement to steering, while the balance depends on the configuration. In the Qwen2-VL-7B/L2S sweep (Figure 9), adding SCOPE at the strongest setting (α = 3) worsens both ASR and utility, as reported in Appendix B.3. At the other four strengths, it lowers macro ASR by 3.4–10.7 pp with utility changes between −2.32 and +0.01 pp. We will make both the additional safety gains and this configuration dependence clearer in the main text.
Thank you for the constructive discussion.
Response to Reviewer 6eFB [1/2]
Thank you for the positive assessment and focused questions. We have added:
All differences are calculated from per-item counts before rounding.
[W1 & Q1] Coordinate Controls Combined with Steering
As the reviewer notes, the existing controls do not isolate coordinate selection in the combined setting: the coordinate controls in Table 1 test standalone suppression, CDS restricts steering updates rather than zeroing hidden-state coordinates, and Table 6 compares update directions on SCOPE's own support. To test coordinate selection directly in the combined setting, we evaluated Random (five seeds), Magnitude, and Negative Gap suppression with CAA, L2S, and ASTRA on both VLMs. All controls use SCOPE's layer, final prompt position, first-prefill schedule, support size, and zeroing operator. Steering configurations are fixed, so only the selected coordinates differ. The table reports macro-averaged ASR / utility (%) over the three jailbreak and three benign benchmarks.
SCOPE has the lowest macro ASR in all six configurations and the lowest ASR in 15 of 18 benchmark-level comparisons. The paired cluster-bootstrap 95% CIs for its macro-ASR advantage over each control exclude zero in all six configurations (18 comparisons). The closest control is Magnitude on InternVL2-8B: SCOPE's advantage is 1.49 pp [0.27, 2.74] with CAA, 2.53 pp [1.03, 4.05] with L2S, and 2.95 pp [1.41, 4.49] with ASTRA. SCOPE also has higher macro utility than Magnitude in five of six configurations.
Edit size alone does not explain this pattern. Magnitude zeroing produces edits with a comparable average L2 norm on InternVL2-8B (≈14) and roughly twice SCOPE's norm on Qwen2-VL-7B, but yields smaller ASR reductions on both models. These matched comparisons support an additional safety benefit from response-conditioned coordinate selection over the tested alternatives.
A forward-pass analysis on Qwen2-VL-7B provides further evidence: among these supports, only SCOPE zeroing increases the projection onto the model's refusal direction in later layers on jailbreak prompts (see our response to Reviewer hWTa, W4). We will include the complete benchmark-level results in the appendix.