acceptodds
Under review as a conference paper at ICLR 2027

SCOPE: Sparse Coordinate Suppression Complements Activation Steering in Vision-Language Models

Abstract

Activation steering improves the safety of Vision-Language Models (VLMs), but addressing residual jailbreak failures through stronger steering can degrade benign utility. We find that pre-generation activation differences between prompts yielding safe and unsafe responses are concentrated in a sparse set of hidden-state coordinates. We introduce SCOPE, a training-free complement that zeroes these coordinates at one language decoder layer, only at the final prompt position during prefill. The support and layer are selected once from each unmodified base VLM and reused across steering methods and attack benchmarks. Coordinate and operation controls show that the gains depend on both support selection and zeroing. Across two VLMs, three steering methods, and three jailbreak benchmarks, SCOPE reduces attack success rates by 1.78 to 14.80 percentage points in all 18 evaluated combinations while largely preserving benign utility. At the example level, SCOPE recovers residual steering failures and exhibits combination-specific gains on prompts that neither intervention handles safely on its own.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Multimodal Safety, Activation Steering, Jailbreak Defense, Sparse Representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6eFBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCOPE, a training-free method to address jailbreak failures that persist after activation steering in VLMs. It identifies unsafe-related coordinates from base-model responses and zeros them at the final prompt position in one decoder layer during prefill. The selected coordinates remain fixed across steering methods and benchmarks.

Critical strengths
  1. The intervention is simple and easy to add to existing steering methods. It changes only 139 or 112 coordinates at one layer and prompt position, once during prefill.
  2. The ablations are well chosen and help justify both the coordinate selection and the zeroing operation.
Critical weaknesses
  1. The paper's central claim is that coordinates identified from base-model responses provide a general complement to different steering methods. However, the main comparisons with random, activation-magnitude, and negative-difference coordinates are conducted under standalone suppression. CDS in Table 3 restricts the dimensions affected by steering updates, which differs from zeroing hidden-state coordinates at the final prompt position. Although Table 6 controls the edit norm, it uses only SCOPE-selected coordinates and mainly tests the effect of update direction. A direct comparison is therefore still missing.
  2. The three attack benchmarks provide useful evaluations under fixed attacks and cross-benchmark settings. However, the paper does not report adaptive attacks reoptimized against the full steering-plus-SCOPE defense.
Questions for authors
  1. How do random or activation-magnitude-based coordinates perform when combined with steering under matched intervention settings?
  2. Have the authors tested adaptive attacks against the full combined defense? If not, the claims should be limited to the evaluated fixed and transfer attacks.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6eFB [1/2]

Thank you for the positive assessment and focused questions. We have added:

  • A direct comparison of Random, Magnitude, and Negative Gap suppression with SCOPE, each combined with steering under matched settings. SCOPE has the lowest macro ASR in all six model–steering settings, with a significant advantage over each control (W1/Q1).
  • A white-box adaptive attack optimized through the full steering+SCOPE defense, with an explicit statement of the scope of our robustness claims (W2/Q2).
  • Comparisons with Jailbreak Antidote and refusal-direction steering on Qwen2-VL-7B. SCOPE further reduces ASR even on top of the lowest-ASR baseline on this model (18.39% → 12.04%).
  • A forward-pass analysis showing increased activation along the model's refusal direction after zeroing the SCOPE support, a third VLM (Gemma-3-12B-IT), and paired CIs for all main comparisons. These additions are detailed in our responses to Reviewers xseY and hWTa.

All differences are calculated from per-item counts before rounding.

[W1 & Q1] Coordinate Controls Combined with Steering

As the reviewer notes, the existing controls do not isolate coordinate selection in the combined setting: the coordinate controls in Table 1 test standalone suppression, CDS restricts steering updates rather than zeroing hidden-state coordinates, and Table 6 compares update directions on SCOPE's own support. To test coordinate selection directly in the combined setting, we evaluated Random (five seeds), Magnitude, and Negative Gap suppression with CAA, L2S, and ASTRA on both VLMs. All controls use SCOPE's layer, final prompt position, first-prefill schedule, support size, and zeroing operator. Steering configurations are fixed, so only the selected coordinates differ. The table reports macro-averaged ASR / utility (%) over the three jailbreak and three benign benchmarks.

ModelSteeringSteering only+Random+Magnitude+Negative Gap+SCOPE
QwenCAA33.74 / 72.0233.89 / 71.5733.03 / 68.4337.10 / 70.2227.72 / 73.19
QwenL2S34.84 / 62.5035.04 / 62.4931.72 / 60.1535.47 / 64.1424.14 / 62.50
QwenASTRA38.49 / 65.7738.88 / 65.6737.87 / 64.1640.47 / 64.7333.33 / 65.57
InternVLCAA19.46 / 66.4920.12 / 65.6915.70 / 67.6716.72 / 65.5114.20 / 68.09
InternVLL2S13.73 / 63.1113.43 / 62.538.71 / 60.6711.33 / 61.116.18 / 63.51
InternVLASTRA25.59 / 64.9226.50 / 65.2122.16 / 67.0922.34 / 64.9019.21 / 67.04

SCOPE has the lowest macro ASR in all six configurations and the lowest ASR in 15 of 18 benchmark-level comparisons. The paired cluster-bootstrap 95% CIs for its macro-ASR advantage over each control exclude zero in all six configurations (18 comparisons). The closest control is Magnitude on InternVL2-8B: SCOPE's advantage is 1.49 pp [0.27, 2.74] with CAA, 2.53 pp [1.03, 4.05] with L2S, and 2.95 pp [1.41, 4.49] with ASTRA. SCOPE also has higher macro utility than Magnitude in five of six configurations.

Edit size alone does not explain this pattern. Magnitude zeroing produces edits with a comparable average L2 norm on InternVL2-8B (≈14) and roughly twice SCOPE's norm on Qwen2-VL-7B, but yields smaller ASR reductions on both models. These matched comparisons support an additional safety benefit from response-conditioned coordinate selection over the tested alternatives.

A forward-pass analysis on Qwen2-VL-7B provides further evidence: among these supports, only SCOPE zeroing increases the projection onto the model's refusal direction in later layers on jailbreak prompts (see our response to Reviewer hWTa, W4). We will include the complete benchmark-level results in the appendix.

AuthorsOpenReview ↗

Response to Reviewer 6eFB [2/2]

[W2 & Q2] Adaptive Attacks

Fixed and transfer attacks do not establish robustness to adaptive attackers. We have now evaluated one adaptive setting and will state the scope of our claims explicitly.

Setup. On Qwen2-VL-7B with CAA, we ran the unconstrained white-box universal visual attack of Qi et al. (2024). The adversarial image is prepended as an additional image to each input. We optimized separate images against CAA alone and the full CAA+SCOPE defense for 5,000 PGD steps, differentiating through the exact CAA and SCOPE hooks. Evaluation questions were not used during optimization.

Results. When each defense faces the image optimized against it, CAA+SCOPE has lower ASR than CAA:

Defense (attack optimized against it)MMSB ↓HADES ↓JailbreakV ↓
CAA44.0750.9320.71
CAA + SCOPE42.2247.6015.36

The table compares separately optimized attacks. In a separate paired comparison, we applied the same image optimized against CAA+SCOPE to both defenses. CAA+SCOPE has lower ASR on all three benchmarks, with paired 95% CIs excluding zero: MMSB, −5.19 pp [−10.00, −0.74]; HADES, −7.20 pp [−10.00, −4.53]; and JailbreakV-mini, −4.64 pp [−8.33, −1.09]. The attack increases ASR for both defenses, but SCOPE retains its additional benefit when the attack is optimized through the full defense, including the fixed support.

We also evaluated a stronger attacker that selects the most effective image for each question from all evaluated candidates (the original and every adaptive image). Under this per-question worst-case selection, CAA+SCOPE still has lower ASR on all three benchmarks: MMSB, 52.96% vs. 57.04% (−4.07 pp [−7.78, −1.11]); HADES, 60.53% vs. 66.13% (−5.60 pp [−8.00, −3.20]); and JailbreakV-mini, 20.00% vs. 26.79% (−6.79 pp [−10.40, −3.44]).

Scope of claims. This evaluation covers one universal attack family and one optimization seed; the attack is not optimized separately for each input. It cannot establish robustness to all adaptive attacks. In the revision, we state that the main evaluation covers fixed and transfer attacks (the abstract now reads "…across two VLMs, three steering methods, and three jailbreak benchmarks with fixed and transfer attacks…"). We will report the adaptive evaluation as a limited, attack-specific test and discuss stronger adaptive attacks as future work.

Please let us know if any questions remain. If these results resolve your concerns, we would be grateful if you could consider raising the score.

Thank you for the additional experiments. Some limitations remain: the adaptive evaluation covers only one universal attack family and one seed on a single model–steering pair. The safety–utility benefit also varies with the configuration: stronger baselines incur utility losses, and Figure 9 includes a high-strength L2S setting where adding SCOPE worsens both ASR and utility.

AuthorsOpenReview ↗

Follow-up to Reviewer 6eFB

Thank you for the follow-up. We have extended the adaptive evaluation to two VLMs, three steering methods, and five optimization seeds.

Adaptive evaluation. On Qwen2-VL-7B and InternVL2-8B, we ran the same white-box universal attack against CAA, L2S, and ASTRA, each with and without SCOPE. For each model–steering pair, we optimized five images (one per seed) against each defense, yielding 60 adaptive images, and evaluated all 10 images of a pair against both defenses (156,000 responses). Each attack was optimized through the full defended model, including L2S's input-dependent context-to-shift computation and ASTRA's adaptive steering operation. When each defense faces the image optimized against it, ASR (%, steering only → + SCOPE, averaged over five seeds) is:

ModelSteeringMMSB ↓HADES ↓JailbreakV ↓Macro Δ (pp) [95% CI]
Qwen2-VL-7BCAA42.74 → 37.1147.17 → 36.2917.93 → 12.36−7.36 [−8.68, −6.08]
L2S40.96 → 22.9668.05 → 51.5520.57 → 16.64−12.81 [−14.64, −10.99]
ASTRA41.63 → 40.5954.11 → 49.4417.36 → 13.64−3.14 [−4.35, −2.00]
InternVL2-8BCAA40.22 → 27.7042.45 → 32.2711.64 → 7.79−8.85 [−10.55, −7.21]
L2S19.93 → 7.3320.56 → 12.248.14 → 4.64−8.14 [−9.78, −6.64]
ASTRA42.00 → 35.0443.84 → 39.0412.64 → 10.21−4.73 [−6.30, −3.23]

Averaged over the five seeds, SCOPE lowers ASR in all 18 model–steering–benchmark settings. It also lowers macro ASR in 29 of the 30 model–steering–seed comparisons. We also compared both defenses on the same images (optimized against either defense) and evaluated per-question worst-case ASR separately for each defense, using the same candidate pool consisting of the original input and all 10 adaptive images. SCOPE lowers macro ASR for all six pairs in both analyses (worst case: by 2.01–8.56 pp). All six macro-ASR CIs exclude zero in each analysis; CIs are paired cluster-bootstrap intervals over questions, with the five optimization seeds held fixed. We will present this as an evaluation of one universal attack family and include the per-seed and per-benchmark results in the appendix.

Safety–utility trade-off. The additional baseline experiments show that SCOPE further lowers macro ASR even on top of strong defenses: from 25.04% to 18.37% with Antidote and from 18.39% to 12.04% with refusal-direction steering, at utility costs of 1.75 and 2.46 pp, respectively. This supports its value as a reusable complement to steering, while the balance depends on the configuration. In the Qwen2-VL-7B/L2S sweep (Figure 9), adding SCOPE at the strongest setting (α = 3) worsens both ASR and utility, as reported in Appendix B.3. At the other four strengths, it lowers macro ASR by 3.4–10.7 pp with utility changes between −2.32 and +0.01 pp. We will make both the additional safety gains and this configuration dependence clearer in the main text.

Thank you for the constructive discussion.

Reviewer_hWTaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper tackles the utility degradation caused by activation steering methods for VLM safety. The authors propose SCOPE, a training-free intervention that zeros out a sparse set of hidden-state coordinates (~3–4% of dimensions) at a single decoder layer. Two key designs are introduced: (1) response-conditioned coordinate discovery, which groups calibration prompts by whether the unsteered VLM's own output was safe or unsafe (rather than by input harmfulness), and (2) modality-balanced pooling, which equally weights text and visual tokens. Experiments on two 7–8B VLMs, three jailbreak and three benign benchmarks show that SCOPE, both standalone and composed with steering (CAA, ASTRA, L2S), consistently lowers ASR by 1.78–14.80 pp while preserving or improving utility in 14 of 18 benchmark-level comparisons.

Critical strengths

S1. Novel and well-motivated coordinate discovery strategy. The response-conditioned grouping is the paper's strongest contribution. By conditioning on the VLM's own behavior rather than external input labels, the method focuses on the "hard cases" where the model fails to self-censor. Table 2 demonstrates consistent improvements over input-conditioned discovery (7.73 pp ASR reduction on Qwen, 5.59 pp on InternVL2), validating this insight.

S2. Consistent utility preservation. Unlike CDS, which degrades utility in several settings, SCOPE consistently improves or maintains utility. The 14/18 improvement rate with a maximum decline of only 2.96 pp is a meaningful practical advantage.

S3. The method is training-free and lightweight. SCOPE requires no gradient computation, no model weight updates, and operates at a single layer/position with ~100–140 coordinates (2.7–3.9% of hidden dimensions). This makes it practical for deployment alongside existing steering methods.

Critical weaknesses

W1. The mechanism is not supported by the paper’s own controls. If the support encoded an “unsafe shift,” moving it toward safe statistics should help. Yet safe-mean replacement, mean-gap subtraction, and matched-gap edits are all ineffective; only zeroing works. On InternVL, zeroing Negative Gap or Magnitude coordinates also cuts ASR substantially. The concentration in Fig. 2a is only modestly above a Gaussian null, and the gap is unstandardized. A simpler explanation is that zeroing large-magnitude coordinates at the last token is an out-of-distribution perturbation, which is also consistent with the notable safe→unsafe churn in Fig. 3. The paper does not rule this out.

W2. Incremental novelty and insufficient rigor for the “reusable complement” claim. The method is close to prior sparse or neuron-level interventions). There are no significance tests, and the smallest gain is only 5 prompts. Coordinate controls are never tested in combination with steering. Several baselines look weak: ASTRA is ineffective on InternVL, where ASTRA+SCOPE ≈ SCOPE alone, and L2S+CDS is worse than no defense.

W3. Limited scope and practical cost. Only two older 7–8B VLMs are tested, and only against non-adaptive, stylistically similar attacks. A fixed mask is an easy target for white-box attackers. The method needs about 1,000 human labels per model with no agreement statistics reported. ASR relies solely on text-only Llama Guard, without checking whether “safe” outputs are actually degenerate.

W4. No interpretability. ~100–140 coordinates are identified as safety-critical, but no analysis of what they encode, whether they overlap across models, or why zeroing (vs. other operators) is specifically effective beyond the empirical result in Table 4.

W5: Repeated and suspiciously regular values across independent runs. Several numbers that should come from distinct experiments are identical to two decimals, which is hard to explain if each cell is a separate run. In Table 3, CARE and CAA+CDS report the same HADES change (−5.60) despite different baselines, and the same JailbreakV ASR (17.86); SCOPE and DCD share MMSB 45.19; and MM-Vet 63.30 recurs across SCOPE and SCOPE+CAA. JailbreakV 7.86 appears for both InternVL SCOPE and Qwen CAA+SCOPE, i.e., across different models. These coincidences, together with how often the deltas land on unusually round values, raise concern about whether the cells are independent measurements or partly copied/derived.

Questions for authors

Please refer to weakness.

8 comments
AuthorsOpenReview ↗

Response to Reviewer hWTa [1/5]

Thank you for the careful review. We address W5 first because it concerns the reliability of the reported results. Our main additions are:

  • An audit of the repeated values: all 36 steering vs. steering+SCOPE comparisons in Table 3 are reproduced exactly from the stored per-example judgments (W5).
  • Coordinate controls combined with steering: SCOPE has the lowest macro ASR in all six model–steering settings, with a significant advantage over each control (W1, W2).
  • Paired cluster-bootstrap CIs for the main comparisons, plus Jailbreak Antidote and refusal-direction baselines. SCOPE further reduces ASR when added to either method (W2).
  • Experiments on Gemma-3-12B-IT, a white-box adaptive attack optimized through the full defense, an image-grounded re-evaluation of all jailbreak responses, and annotation agreement statistics (W3).
  • A forward-pass analysis: unlike the control supports, zeroing the SCOPE support increases activation along the model's refusal direction and shifts first tokens toward refusal starts mainly on jailbreak prompts (W4).
  • A correction to the notation in Eq. (3). The implementation and all reported results are unchanged (W1).

All differences are calculated from per-item counts before rounding.

[W5] Verification of Repeated Evaluation Scores

We rechecked the stored per-example evaluation records and recomputed the scores. All 36 steering vs. steering+SCOPE comparisons in Table 3 (18 ASR and 18 utility) are reproduced exactly. The repeated scores have equal outcome counts on the same evaluation sets, but the per-example outcomes differ:

ComparisonReported ScoreCount in Each ConditionDifferent Outcomes
Qwen CARE vs. CAA+CDS (JailbreakV)17.86%50/280 unsafe36
Qwen SCOPE vs. DCD (MMSB)45.19%122/270 unsafe18
Qwen SCOPE vs. CAA+SCOPE (MM-Vet)63.30%138/218 correct10
InternVL SCOPE vs. Qwen CAA+SCOPE (JailbreakV)7.86%22/280 unsafe26

The two HADES changes of −5.60 pp involve different baselines. CARE reduces unsafe responses from 445 to 403 relative to the shared unsteered baseline; CAA+CDS reduces them from 341 to 299 relative to CAA. Both are reductions of 42 out of 750 responses. The scores in these examples are counts divided by fixed benchmark sizes (270, 750, 280, or 218), so they change in fixed steps (e.g., 1/280 = 0.357 pp on JailbreakV-mini). We will add numerator/denominator counts and label the reference baselines in the revised manuscript so that readers can check the reported scores and changes directly.

AuthorsOpenReview ↗

Response to Reviewer hWTa [2/5]

[W1] Mechanism and the Generic-Perturbation Explanation

We agree that the support should not be read as an "unsafe direction." We will use response-conditioned coordinates in the revision to make this distinction clear. Eq. (3) also contains a notation error: discovery uses the magnitudes of modality-wise mean activations, 12∣hˉℓtext∣+12∣hˉℓvis∣\frac{1}{2}|\bar{\mathbf{h}}^{\mathrm{text}}_{\ell}|+\frac{1}{2}|\bar{\mathbf{h}}^{\mathrm{vis}}_{\ell}|, rather than signed activations. The implementation and all reported results are unchanged.

This selection rule does not assign an unsafe direction based on activation sign. SCOPE suppresses the selected coordinates; it is not intended to move them toward safe-group statistics. Neither safe-mean replacement nor mean-gap subtraction yields a statistically supported macro-ASR reduction with CAA on either model (Qwen2-VL-7B: 33.74 → 33.72 / 33.36; InternVL2-8B: 19.46 → 20.81 / 20.83). SCOPE reduces macro ASR to 27.72 and 14.20, respectively, and outperforms both corrections in all 12 model–benchmark comparisons, with paired 95% CIs excluding zero.

We evaluated the proposed out-of-distribution perturbation explanation with three comparisons:

  • Matched norm on the same support (Table 6). Directional edits with exactly SCOPE's per-input edit norm do not reproduce its gains.
  • Other supports with steering (W2). Random, Magnitude, and Negative Gap zeroing at the same layer, position, and support size all give significantly higher macro ASR than SCOPE. On InternVL2-8B, Magnitude zeroing has a comparable average edit norm (≈14) but higher ASR with every steering method.
  • Asymmetry of prompt-level changes. Across the 18 model–steering–benchmark comparisons, SCOPE changes a total of 804 unsafe responses to safe and 224 safe responses to unsafe (3.6:1). The corresponding counts are 608 vs. 426 (1.4:1) for Magnitude and 410 vs. 367 (1.1:1) for Negative Gap. These controls produce more mixed changes, while SCOPE's changes are predominantly corrective. Fig. 3 includes safe→unsafe transitions, but they are outnumbered by unsafe→safe transitions. We will report both counts.

As stated in Sec. 4.2, the gap concentration in Fig. 2a is a descriptive observation that motivates the functional intervention tests. We will make this point more prominent: concentration alone does not establish safety relevance.

We also tested selection by a standardized gap (Cohen's d), keeping the draws, recurrence rule, layer, and support size (139) fixed. The resulting support shares 43 coordinates with SCOPE and produces smaller zeroing edits (mean L2 ≈ 5 vs. ≈ 15 for SCOPE). With CAA on Qwen2-VL-7B, it reduces macro ASR from 33.74% to 32.48%, compared with 27.72% for SCOPE (SCOPE lower by 4.76 pp [2.85, 6.70]). It offers no utility advantage (72.14% vs. 73.19%). Magnitude zeroing produces roughly twice SCOPE's edit norm but also gives higher ASR (33.03%). Neither smaller nor larger edits on these other supports reproduce SCOPE's effect. These comparisons, along with the matched-norm controls above, show that edit size alone does not explain the gain. Our claims are based on these functional tests.

AuthorsOpenReview ↗

Response to Reviewer hWTa [3/5]

[W2] Novelty, Significance, Coordinate Controls, and Baseline Strength

Novelty. Our contribution is the discovery and reuse of the support, rather than sparse suppression itself. We (i) discover coordinates from the base VLM's own response outcomes using modality-balanced pooling, and (ii) show that this support and layer, fixed once per model, complement independently constructed steering methods across attack benchmarks. The input- vs. response-conditioned comparison (Table 2) and the controls below show that the selection rule matters beyond the suppression operator alone.

Significance. We computed paired cluster-bootstrap 95% CIs using 20,000 replicates. MMSB is resampled by its 90 base identities, keeping the three variants together; JailbreakV-mini is resampled by query, and HADES by item. All 18 model–steering–benchmark comparisons in Table 3 show an ASR reduction, and 15 intervals exclude zero. The smallest gain noted by the reviewer (InternVL2-8B/ASTRA/JailbreakV-mini, 23 → 18 unsafe responses) is one of the three exceptions. The others are Qwen2-VL-7B/ASTRA/MMSB and InternVL2-8B/L2S/JailbreakV-mini. We will report all intervals.

Coordinate controls with steering. We combined Random (five seeds), Magnitude, and Negative Gap suppression with CAA, L2S, and ASTRA on both VLMs. All controls use SCOPE's intervention layer, final prompt position, support size, and zeroing operator, with steering configurations fixed. The table reports macro-averaged ASR / utility (%) over the three jailbreak and three benign benchmarks.

ModelSteeringSteering only+Random+Magnitude+Negative Gap+SCOPE
QwenCAA33.74 / 72.0233.89 / 71.5733.03 / 68.4337.10 / 70.2227.72 / 73.19
QwenL2S34.84 / 62.5035.04 / 62.4931.72 / 60.1535.47 / 64.1424.14 / 62.50
QwenASTRA38.49 / 65.7738.88 / 65.6737.87 / 64.1640.47 / 64.7333.33 / 65.57
InternVLCAA19.46 / 66.4920.12 / 65.6915.70 / 67.6716.72 / 65.5114.20 / 68.09
InternVLL2S13.73 / 63.1113.43 / 62.538.71 / 60.6711.33 / 61.116.18 / 63.51
InternVLASTRA25.59 / 64.9226.50 / 65.2122.16 / 67.0922.34 / 64.9019.21 / 67.04

SCOPE has the lowest macro ASR in all six configurations and the lowest ASR in 15 of 18 benchmark-level comparisons. Its macro-ASR advantage over each control has a paired 95% CI excluding zero in all six configurations (18 comparisons). SCOPE also has higher macro utility than Magnitude, the strongest non-SCOPE control, in five of six configurations. We will provide the full benchmark-level results, Random seed variability, and paired CIs in the appendix.

Baseline strength. We followed the official ASTRA procedure, reproducing it for InternVL2-8B with model-targeted PGD images. ASTRA has little effect on that model: ASTRA+SCOPE (19.21%) is close to SCOPE alone (19.35%), so most of the gain comes from SCOPE. CDS serves as a sparse-steering reference with separately tuned strengths (Appendix A.5). Its degradation with L2S shows that restricting steering updates to magnitude-selected dimensions does not preserve their effect. SCOPE does not restrict the steering update to a coordinate subset.

We also tested complementarity with two strong defenses on Qwen2-VL-7B: Jailbreak Antidote and refusal-direction steering (Arditi et al., 2024). Adding SCOPE reduces macro ASR from 25.04% to 18.37% with Antidote and from 18.39% to 12.04% with the refusal direction (−6.35 pp [−8.22, −4.56]). The refusal direction has the lowest baseline ASR in this comparison. The corresponding utility costs are 1.75 and 2.46 pp. Details are in our response to Reviewer xseY (W5).

Reviewer_xseYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is mainly addressing some gaps from activation steering approach (the canonical contrastive activation addition) which might have tendencies to either not fully suppress unwanted behavior or might cause excessive correction behavior on normal inputs.

The paper approaches the problem differently but using different sets of positive prompts and negative prompts based on their human labels as the positive and negative sets. In contrast, CAA style steering approaches use same prompts with positive answers and negative answers as the positive and negative sets. The core idea is that they find under 4% of hidden state coordinates that have the largest difference between the two activation groups and zero them out at one layer and the final prompt token.

The activations groups are selected based on human labeling of a model's response to harmful prompts.

They show in the evaluation that the approach lowers attack success rates in all 18 model, method and benchmark combinations while largely preserving benign utility. It is also proposed as a complement to steering approaches.

Critical strengths

The approach is very simple and if effective is a practical approach to steering that stacks onto other steering techniques. The paper show consistent gains across 18 combinations of model, methods and benchmarks. The limitations are also clearly stated (e.g., in Figure 9 and in showing the -2.96 decline).

The ablations are quite broad. Especially, it tests several alternatives such as coordinate selection, types of operators including safe-mean replacement, mean-gap subtraction, partial suppression across \beta, different types of pooling, edit magnitudes.

Critical weaknesses

Zeroing a coordinate pushes it toward 0, not toward the safe side. If a coordinate has a positive gap but a negative mean activation, zeroing moves it toward unsafe-like values. The method ranks by signed gap without considering the sign of h_j.

The paper provides no statistical uncertainty on the main results. Tables 1, 3, 4, and 6 are all reporting point estimates with no CI or paired tests.

The denominators are small. MM-SafetyBench has 270 items but only 90 independent identities, because the SD, TYPO, and SD+TYPO variants of a question are correlated.

The paper calls the support “unsafe-associated coordinates,” but operators that move activations toward the safe side (safe-mean replacement, mean-gap subtraction, norm-matched gap control) do nothing, while zeroing works.

The closest prior work, Jailbreak Antidote (sparse, last-token, hidden-state edit), is cited but not compared. Also missing are dense refusal-direction projection and safety-neuron methods.

For adaptive methods (ASTRA’s gate, L2S’s learned module), an upstream edit changes the inputs that determine the steering update, so “steering unchanged” is true only for the hyperparameters. The super-additive Qwen/L2S result (MMSB 34.81 to 20.37) is a sign of this. Steering update norms and gate values are not reported.

On InternVL, Magnitude and Negative Gap controls also reduce ASR, which suggests part of the effect is generic perturbation

Questions for authors

How were the steering strengths selected? Appendix A.5 says each strength is the lowest-ASR setting that does not cause response collapse. For CAA on Qwen, α = 0.2 was chosen, yet α = 0.25 and 0.35 appear in Fig. 9 with no visible collapse. Were these collapsed on the selection pool, and how was collapse judged?

What are the sign and scale of the selected coordinates at the final prompt token? Specifically: the fraction of the support where h_j is positive, the ratio ‖Mh‖/‖Mg‖, and the overlap with known massive-activation dimensions.

Which means were used in safe-mean replacement and mean-gap subtraction (Table 4)? Were they computed from pooled prompt tokens or from the final token?

Where are the baselines applied? At which layers and positions are DCD and CARE applied, and did the coordinate controls (Random, Magnitude, Negative Gap) get any layer selection of their own?

Does SCOPE change the steering update for ASTRA and L2S? Please report the update norms and ASTRA’s gate values with and without SCOPE.

Can you evaluate on more recent as well as some larger VLMs with different backbones?

Also curious, how would CAA restricted to the final prompt token would work? Similarly, how would SCOPE applied to all token positions work?

6 comments
AuthorsOpenReview ↗

Response to Reviewer xseY [1/5]

Thank you for the detailed review. We have added:

  • Paired cluster-bootstrap CIs. Adding SCOPE reduces ASR in all 18 model–steering–benchmark settings, with 15 CIs excluding zero. In the new coordinate-control experiment, SCOPE also lowers macro ASR relative to each of three controls in all six model–steering settings (W2, W7).
  • Jailbreak Antidote and refusal-direction baselines. Adding SCOPE further lowers ASR for both, including the lowest-ASR baseline evaluated on Qwen2-VL-7B (W5).
  • Experiments on a larger, more recent VLM, Gemma-3-12B-IT, and analyses of activation sign and scale, massive activations, steering updates, and token positions (Q2–Q7).
  • A correction to the notation in Eq. (3). The implementation and all reported results are unchanged (W1).

All differences are calculated from per-item counts before rounding.

[W1 & W4] Coordinate Selection and the Role of Zeroing

Zeroing does not necessarily move activations toward the safe-group mean. SCOPE suppresses selected coordinates; it does not apply a directional correction. Here, "unsafe-associated" refers to a statistical association with unsafe response outcomes, not a specific unsafe direction. We will use response-conditioned coordinates throughout the revision to make this distinction clear.

Eq. (3) contains a notation error. The implementation takes element-wise absolute values of the modality-wise mean activations before combining them:

hˉℓ(xi)=12∣hˉℓtext(xi)∣+12∣hˉℓvis(xi)∣.\bar{\mathbf{h}}_{\ell}(x_i)=\frac{1}{2}\left|\bar{\mathbf{h}}_{\ell}^{\mathrm{text}}(x_i)\right|+\frac{1}{2}\left|\bar{\mathbf{h}}_{\ell}^{\mathrm{vis}}(x_i)\right|.

The implementation and all reported numerical results are unchanged. We have corrected Eq. (3) and clarified the accompanying descriptions (Lines 172, 189, and 911 of the revised manuscript): discovery compares the magnitudes of modality-wise mean activations. It does not assign an unsafe direction from the activation sign, and zeroing suppresses selected activations of either sign. The operator and coordinate-selection controls provide empirical evidence for the safety gains. Zeroing the support also increases activation along the model's refusal direction (W7).

With CAA on Qwen2-VL-7B, macro ASR decreases as suppression becomes stronger (Table 4): 32.14 → 30.30 → 29.65 → 27.72 for β = 0.25, 0.5, 0.75, 1.0. This trend alone could reflect generic perturbation. The matched controls test this possibility: zeroing SCOPE's coordinates outperforms directional edits on the same support at the same per-input edit norm (Table 6). It also outperforms Magnitude zeroing with comparable average edit norms on InternVL2-8B (W7).

Mean-based directional corrections do not reproduce these gains. The table reports macro ASR over the three jailbreak benchmarks with CAA:

InterventionQwen2-VL-7B ASR ↓InternVL2-8B ASR ↓
CAA only33.7419.46
+ Safe-mean replacement33.7220.81
+ Mean-gap subtraction33.3620.83
+ SCOPE27.7214.20

Neither mean-based correction gives a statistically supported ASR reduction over CAA. SCOPE lowers macro ASR by 6.02 and 5.25 pp on Qwen2-VL-7B and InternVL2-8B, respectively.

[W2 & W3] Statistical Reliability and Sample Dependence

We computed paired cluster-bootstrap 95% CIs with 20,000 replicates for every steering vs. steering+SCOPE comparison in Table 3. For MM-SafetyBench, we resample the 90 base identities, keeping each identity's SD, TYPO, and SD+TYPO variants together. For JailbreakV-mini, we group items by query (248 clusters). HADES items are resampled individually.

All 18 model–steering–benchmark comparisons show lower ASR, and 15 CIs exclude zero. The exceptions are Qwen2-VL-7B/ASTRA on MMSB and InternVL2-8B/ASTRA and L2S on JailbreakV-mini. The CAA results are shown below; negative differences indicate improvement.

ModelMMSB ΔASR [95% CI]HADES ΔASR [95% CI]JailbreakV ΔASR [95% CI]
Qwen2-VL-7B-5.19 [-9.26, -1.48]-5.73 [-8.53, -2.93]-7.14 [-10.58, -3.90]
InternVL2-8B-4.81 [-9.26, -0.74]-7.73 [-10.40, -5.20]-3.21 [-5.82, -0.72]

For the operator comparison in Table 4 and its InternVL2-8B extension in W1, SCOPE improves over CAA with CIs excluding zero in all six model–benchmark pairs. It also outperforms both mean-based corrections with CIs excluding zero in all 12 direct comparisons. The MMSB intervals account for its 90 independent identities by resampling variants jointly. HADES and JailbreakV-mini provide another 1,030 prompts from independently constructed benchmarks. We will add uncertainty estimates to Tables 1, 3, 4, and 6 in the revision.

AuthorsOpenReview ↗

Response to Reviewer xseY [2/5]

[W5] Comparison with Jailbreak Antidote and Related Methods

We added comparisons with Jailbreak Antidote and dense refusal-direction steering. Antidote and SCOPE both make sparse hidden-state edits at the final prompt token, but they use different operations: Antidote adds a steering vector, while SCOPE suppresses selected coordinates. We therefore tested them both separately and together.

Jailbreak Antidote. We estimated steering directions by centered PCA on the final-token activations of 128 benign and 118 harmful text prompts. Following the original method, we retain the top 5% (180) coordinates and apply the intervention at three evenly spaced decoder layers (8, 16, and 24; 0-indexed). We apply it at the last token position of every forward pass, that is, the final prompt token and each cached decoding step. Antidote had little effect at α = 1 in this VLM setting, so we searched a wider strength range on the validation split described below.

Refusal direction. We use the refusal-inducing activation addition of Arditi et al. (2024). Their directional ablation removes this direction and reduces refusal, so it does not serve as a defense. Following their construction, we take the difference in mean final-prompt-token activations between harmful and harmless instructions, using the same 118/128 prompts. We add the direction at the CAA layer (L12), at all prefill and decoding positions.

For both methods, we selected the strength on a validation split of JailbreakV and VLGuard-Safe, disjoint from the fitting prompts and all evaluation sets, from α ∈ {1, 2, 5, 10, 20, 40} per layer for Antidote and α ∈ {0.125, 0.25, 0.5, 1} for the refusal direction. We chose the lowest-ASR setting that kept utility within 2 pp of the unmodified model and introduced no additional collapsed outputs. This selected α = 20 for Antidote and the direction's natural scale, α = 1, for the refusal direction.

In both comparisons, SCOPE uses the support from the paper and acts only at the final prompt token during prefill. We ran these comparisons on Qwen2-VL-7B due to time constraints.

MethodMMSB ASR ↓HADES ASR ↓JailbreakV ASR ↓Utility (macro) ↑
Baseline50.0059.3320.7171.07
SCOPE only45.1949.8714.6472.31
Jailbreak Antidote32.9630.0012.1468.94
+ SCOPE22.5923.608.9367.19
Δ [paired 95% CI]-10.37 [-15.19, -6.30]-6.40 [-8.67, -4.13]-3.21 [-6.74, +0.35]-1.75
Refusal direction18.1524.5312.5072.48
+ SCOPE13.7017.075.3670.02
Δ [paired 95% CI]-4.44 [-8.15, -1.11]-7.47 [-9.87, -5.20]-7.14 [-10.80, -3.64]-2.46

SCOPE lowers ASR on all three benchmarks with both methods; paired 95% CIs exclude zero in five of the six comparisons. This includes the refusal direction, the lowest-ASR baseline evaluated on Qwen2-VL-7B. Adding SCOPE lowers its macro ASR from 18.39% to 12.04% (−6.35 pp [−8.22, −4.56]). The utility costs are 1.75 and 2.46 pp for Antidote and the refusal direction, respectively, larger than those for the steering methods in Table 3. We will report this trade-off as a limitation.

The refusal direction and our CAA vector differ in both construction and scale. CAA uses refusal-versus-compliance response suffixes and α = 0.2, giving an added L2 norm of about 3.4 at L12. The refusal direction contrasts harmful and harmless instructions and uses its natural scale, with norm ≈ 10.6. CAA also becomes stronger at larger α, and SCOPE adds a gain at every tested strength (Q1). We will add both new baselines to Table 3.

Table 3 already includes CARE, which projects out identified unsafe subspaces, and DCD, a coordinate-level zeroing baseline. We will discuss neuron-level detoxification methods in the related work.

AuthorsOpenReview ↗

Response to Reviewer xseY [3/5]

[W6 & Q5] Interaction with Adaptive Steering

As specified in Sec. 4.1, "unchanged" means that adding SCOPE keeps the steering layers, token schedule, strength hyperparameters, and learned parameters fixed; we will state this explicitly in the revision. The realized updates are reported below. In the evaluated implementations, SCOPE cannot change the update directions:

  • CAA adds the fixed vector αv, giving the same update with and without SCOPE.
  • ASTRA applies α·g(h)·‖h‖·v̂ along the fixed vector v̂. Only the scalar gate g(h) and ‖h‖ can change.
  • L2S predicts its shift from its own unsteered context pass, as in the original method. SCOPE does not modify that pass, so the predicted shift is identical.

We measured the remaining interaction by replaying the token sequences generated by steering alone, with and without SCOPE. Both conditions therefore process identical tokens (Qwen2-VL-7B, MMSB):

MethodQuantitySteering only+ SCOPERelative change [95% CI]
ASTRAMean gate value0.04120.0411-0.34% [-0.42, -0.25]
ASTRAUpdate L2 norm20.4820.42-0.28% [-0.36, -0.20]
L2SUpdate L2 norm23.0623.060.00% (max per input < 0.01%)

Under this controlled comparison, SCOPE does not increase the ASTRA update and leaves the L2S update unchanged. In free generation, the mean ASTRA gate is also lower with SCOPE in all six model–jailbreak-benchmark settings, by 2.5–20.1%. The larger changes are consistent with the different generated token trajectories.

The larger-than-additive gain with L2S on Qwen2-VL-7B (MMSB: 34.81% → 20.37%) therefore does not result from a modified L2S update. It reflects complementary effects within the same forward pass: Figure 3 shows that the combination resolves prompts that neither intervention handles alone.

[W7] Effect of Generic Perturbation

Generic coordinate suppression can reduce ASR, as the Magnitude and Negative Gap controls on InternVL2-8B show in Table 1. We tested whether coordinate selection still matters when suppression is combined with steering. We combined these controls and Random suppression (mean over five seeds) with CAA, L2S, and ASTRA, keeping the intervention layer, final prompt position, support size, zeroing operator, and steering configurations fixed. The table reports macro ASR over MMSB, HADES, and JailbreakV-mini.

ModelSteeringSteering only+ Random+ Magnitude+ Negative Gap+ SCOPE
Qwen2-VL-7BCAA33.7433.8933.0337.1027.72
L2S34.8435.0431.7235.4724.14
ASTRA38.4938.8837.8740.4733.33
InternVL2-8BCAA19.4620.1215.7016.7214.20
L2S13.7313.438.7111.336.18
ASTRA25.5926.5022.1622.3419.21

SCOPE has the lowest macro ASR in all six model–steering combinations. Its advantage over each of the three controls has a 95% CI excluding zero in every setting (18 comparisons). We use paired cluster bootstrap, resampling independently within each benchmark and taking the unweighted mean of the three benchmark ASRs in each replicate. The Random CIs are conditional on the five sampled masks. Even against the closest control, Magnitude on InternVL2-8B, the advantages are 1.49 pp [0.27, 2.74] with CAA, 2.53 pp [1.03, 4.05] with L2S, and 2.95 pp [1.41, 4.49] with ASTRA.

Average edit size alone does not explain this pattern. On InternVL2-8B, Magnitude and SCOPE zeroing produce comparable average L2 norms (≈14), yet Magnitude gives higher ASR with every steering method. On Qwen2-VL-7B, Magnitude edits are roughly twice as large but yield markedly smaller gains. Response-conditioned selection therefore adds safety gains beyond those of the tested generic suppression controls.

A forward-pass analysis on Qwen2-VL-7B shows another difference between these supports. On jailbreak prompts, zeroing the SCOPE support at the final prompt token increases the later-layer projection onto the difference-in-means refusal direction (Arditi et al., 2024) by 0.038 of the harmful–harmless separation. This is comparable to CAA's increase (0.044). Magnitude and Negative Gap instead decrease the projection (−0.040 and −0.052).

SCOPE also raises the rate of refusal-start first tokens ("I" or "Sorry") mainly on jailbreak prompts (15.9% → 27.2%, vs. 2.5% → 5.6% on benign prompts). Magnitude instead raises it mainly on benign prompts (2.5% → 11.1%, vs. 15.9% → 17.5% on jailbreak prompts). Only SCOPE shows this combination of effects among the tested supports; our response to Reviewer hWTa, W4, gives the details.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and Summary of Revisions

We thank all reviewers for their constructive feedback. Reviewers noted that SCOPE is simple and practical to add to existing steering methods (xseY, hWTa, 6eFB), that response-conditioned discovery is well motivated (hWTa), and that the ablations are broad and well chosen (xseY, 6eFB). Below we summarize the main additions; details are in the individual responses.

New experiments and analyses

  • Coordinate controls combined with steering (xseY W7, hWTa W2, 6eFB W1). Random, Magnitude, and Negative Gap suppression with CAA, L2S, and ASTRA under matched intervention settings. SCOPE has the lowest macro ASR in all six model–steering settings, with paired 95% CIs excluding zero against every control (18 comparisons).
  • Statistical uncertainty (xseY W2 & W3, hWTa W2). Paired cluster-bootstrap CIs: adding SCOPE lowers ASR in all 18 model–steering–benchmark comparisons, with 15 CIs excluding zero.
  • Stronger baselines (xseY W5, hWTa W2). Jailbreak Antidote and refusal-direction steering, with strengths selected on a held-out validation split. SCOPE further lowers ASR on top of both, including the lowest-ASR baseline on Qwen2-VL-7B (18.39% → 12.04%).
  • Adaptive attack (hWTa W3, 6eFB W2). A white-box universal visual attack optimized through each full defense, on Qwen2-VL-7B and InternVL2-8B with CAA, L2S, and ASTRA (five seeds each). SCOPE lowers macro ASR for all six model–steering pairs (by 3.14–12.81 pp; all 95% CIs exclude zero), and the reduction also holds under per-question worst-case selection over all candidate images (see our follow-up to Reviewer 6eFB).
  • Additional model (xseY Q6, hWTa W3). On Gemma-3-12B-IT, adding SCOPE to CAA lowers macro ASR from 21.22% to 17.01% (−4.21 pp [−6.04, −2.44]) with similar utility.
  • Evaluation and analysis (xseY W6, hWTa W3 & W4). An image-grounded re-evaluation of 15,600 jailbreak responses, annotation agreement statistics (Fleiss' κ 0.86–0.87), steering-update and gate measurements for ASTRA and L2S, and a forward-pass analysis showing that zeroing the SCOPE support increases activation along the model's refusal direction, mainly on jailbreak prompts.
  • Verification of repeated values (hWTa W5). All 36 steering vs. steering+SCOPE comparisons in Table 3 are reproduced exactly from the stored per-example judgments.

Changes in the revised PDF (marked in red)

  • Eq. (3) and the related descriptions (Lines 172, 189, and 911): we corrected a notation error. Discovery uses the magnitudes of modality-wise mean activations; the implementation and all reported results are unchanged.
  • Abstract: we refer to the support as response-conditioned coordinates and state that the main evaluation covers fixed and transfer attacks.

Planned revisions

  • Add the new experiments to the main text and appendix, including the two new baselines in Table 3.
  • Add uncertainty estimates to Tables 1, 3, 4, and 6, and report numerator/denominator counts with the reference baselines.
  • Use response-conditioned coordinates throughout, report the adaptive evaluation as an evaluation of one universal attack family, and report the utility cost with stronger defenses as a limitation.

We appreciate the reviewers' time and are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.